[jira] [Comment Edited] (ATLAS-3755) Allow system attributes to be updated when policy allows
[ https://issues.apache.org/jira/browse/ATLAS-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098286#comment-17098286 ] Bolke de Bruin edited comment on ATLAS-3755 at 5/3/20, 8:20 AM: [~madhan] I have made a couple of updates: 1. I removed the defaults in the AtlasEntity to ensure that at json deserialisation no defaults are set. This was a change very limited in scope as the deserialiser uses the default constructor and the default constructor isn't used anywhere else. All other constructors use "init()" that do set the default values. 2. The update of the system attributes is now behind a feature flag "atlas.store.system_attribute.enable". This defaults to false and disables updating/creation of system attributes if not in "import" mode. 3. Access requests are done in one go having virtually no impact on processing time 4. Unit tests have been updated to verify the feature flags 5. Removed the side effect in preCreateOrUpdate that was setting system attributes, all this is now contained in createOrUpdate *Use case* We intend to receive about 0.5 million entity updates per day from other meta data systems aside from the ~2 million events per day we receive. We use Kafka to handle the backpressure hence the need to have it managed by policy. Also Glossary (which relies on the entity system) needs to expose homeId and other system attributes as well as Glossary items can be created other meta data systems. This needs to be enabled in a follow PR. was (Author: bolke): [~madhan] I have made a couple of updates: 1. I removed the defaults in the AtlasEntity to ensure that at json deserialisation no defaults are set. This was a change very limited in scope as the deserialiser uses the default constructor and the default constructor isn't used anywhere else. All other constructors use "init()" that do set the default values. 2. The update of the system attributes is now behind a feature flag "atlas.store.system_attribute.enable". This defaults to false and disables updating/creation of system attributes if not in "import" mode. 3. Access requests are done in one go having virtually no impact on processing time 4. Unit tests have been updated to verify the feature flags 5. Removed the side effect in preCreateOrUpdate that was setting system attributes, all this is now contained in createOrUpdate *Use case* We intend to receive about 0.5 million entity updates per day from other meta data systems. We use Kafka to handle the backpressure hence the need to have it managed by policy. Also Glossary (which relies on the entity system) needs to expose homeId and other system attributes as well as Glossary items can be created other meta data systems. This needs to be enabled in a follow PR. > Allow system attributes to be updated when policy allows > > > Key: ATLAS-3755 > URL: https://issues.apache.org/jira/browse/ATLAS-3755 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.0.0, 2.1.0 >Reporter: Bolke de Bruin >Assignee: Bolke de Bruin >Priority: Critical > Attachments: > 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch, > 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch, > 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch, > feature.patch, feature.patch > > > Atlas does not operate in a isolated environment, this is one of the reasons > the "homeId" system attribute was introduced. Unfortunately system attributes > can only be updated when importing. This means any integration with other > services is significantly limited (Kafka, Rest API will not work). (See also > ATLAS-3754) > To resolve this I propose to make it possible to update the system attributes > when policy allows it. This introduces new > AtlasPrivilege.ENTITY_UPDATE_SYSTEM_ATTRIBUTE and > AtlasPrivilege.ENTITY_CREATE_SYSTEM_ATTRIBUTE next to > AtlasPrivilege.ENTITY_UPDATE_ATTRIBUTE and > AtlasPrivilege.ENTITY_CREATE_ATTRIBUTE rather than just checking on the > entity level. In certain places we will then drop the requirement for an > import to be active as this can now happen through other channels as well. > This allows operators to specify policies that allow granular controls over > attributes and system attributes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-3755) Allow system attributes to be updated when policy allows
[ https://issues.apache.org/jira/browse/ATLAS-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098286#comment-17098286 ] Bolke de Bruin commented on ATLAS-3755: --- [~madhan] I have made a couple of updates: 1. I removed the defaults in the AtlasEntity to ensure that at json deserialisation no defaults are set. This was a change very limited in scope as the deserialiser uses the default constructor and the default constructor isn't used anywhere else. All other constructors use "init()" that do set the default values. 2. The update of the system attributes is now behind a feature flag "atlas.store.system_attribute.enable". This defaults to false and disables updating/creation of system attributes if not in "import" mode. 3. Access requests are done in one go having virtually no impact on processing time 4. Unit tests have been updated to verify the feature flags 5. Removed the side effect in preCreateOrUpdate that was setting system attributes, all this is now contained in createOrUpdate *Use case* We intend to receive about 0.5 million entity updates per day from other meta data systems. We use Kafka to handle the backpressure hence the need to have it managed by policy. Also Glossary (which relies on the entity system) needs to expose homeId and other system attributes as well as Glossary items can be created other meta data systems. This needs to be enabled in a follow PR. > Allow system attributes to be updated when policy allows > > > Key: ATLAS-3755 > URL: https://issues.apache.org/jira/browse/ATLAS-3755 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.0.0, 2.1.0 >Reporter: Bolke de Bruin >Assignee: Bolke de Bruin >Priority: Critical > Attachments: > 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch, > 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch, > 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch, > feature.patch, feature.patch > > > Atlas does not operate in a isolated environment, this is one of the reasons > the "homeId" system attribute was introduced. Unfortunately system attributes > can only be updated when importing. This means any integration with other > services is significantly limited (Kafka, Rest API will not work). (See also > ATLAS-3754) > To resolve this I propose to make it possible to update the system attributes > when policy allows it. This introduces new > AtlasPrivilege.ENTITY_UPDATE_SYSTEM_ATTRIBUTE and > AtlasPrivilege.ENTITY_CREATE_SYSTEM_ATTRIBUTE next to > AtlasPrivilege.ENTITY_UPDATE_ATTRIBUTE and > AtlasPrivilege.ENTITY_CREATE_ATTRIBUTE rather than just checking on the > entity level. In certain places we will then drop the requirement for an > import to be active as this can now happen through other channels as well. > This allows operators to specify policies that allow granular controls over > attributes and system attributes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3755) Allow system attributes to be updated when policy allows
[ https://issues.apache.org/jira/browse/ATLAS-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin updated ATLAS-3755: -- Attachment: feature.patch > Allow system attributes to be updated when policy allows > > > Key: ATLAS-3755 > URL: https://issues.apache.org/jira/browse/ATLAS-3755 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.0.0, 2.1.0 >Reporter: Bolke de Bruin >Assignee: Bolke de Bruin >Priority: Critical > Attachments: > 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch, > 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch, > 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch, > feature.patch, feature.patch > > > Atlas does not operate in a isolated environment, this is one of the reasons > the "homeId" system attribute was introduced. Unfortunately system attributes > can only be updated when importing. This means any integration with other > services is significantly limited (Kafka, Rest API will not work). (See also > ATLAS-3754) > To resolve this I propose to make it possible to update the system attributes > when policy allows it. This introduces new > AtlasPrivilege.ENTITY_UPDATE_SYSTEM_ATTRIBUTE and > AtlasPrivilege.ENTITY_CREATE_SYSTEM_ATTRIBUTE next to > AtlasPrivilege.ENTITY_UPDATE_ATTRIBUTE and > AtlasPrivilege.ENTITY_CREATE_ATTRIBUTE rather than just checking on the > entity level. In certain places we will then drop the requirement for an > import to be active as this can now happen through other channels as well. > This allows operators to specify policies that allow granular controls over > attributes and system attributes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-3755) Allow system attributes to be updated when policy allows
[ https://issues.apache.org/jira/browse/ATLAS-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17096435#comment-17096435 ] Bolke de Bruin commented on ATLAS-3755: --- [~madhan] I have updated the patch to only use `entity-update` and `entity-create` and do the access request in one go. This makes the access request backwards incompatible and it would require an update to the Ranger plugin. I suggest putting the check against the attributes behind a feature flag so it remains backwards compatible and it would not affect anyone who isn't using access controls on attributes or system attributes. let me know what you think. > Allow system attributes to be updated when policy allows > > > Key: ATLAS-3755 > URL: https://issues.apache.org/jira/browse/ATLAS-3755 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.0.0, 2.1.0 >Reporter: Bolke de Bruin >Assignee: Bolke de Bruin >Priority: Critical > Attachments: > 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch, > 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch, > 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch, feature.patch > > > Atlas does not operate in a isolated environment, this is one of the reasons > the "homeId" system attribute was introduced. Unfortunately system attributes > can only be updated when importing. This means any integration with other > services is significantly limited (Kafka, Rest API will not work). (See also > ATLAS-3754) > To resolve this I propose to make it possible to update the system attributes > when policy allows it. This introduces new > AtlasPrivilege.ENTITY_UPDATE_SYSTEM_ATTRIBUTE and > AtlasPrivilege.ENTITY_CREATE_SYSTEM_ATTRIBUTE next to > AtlasPrivilege.ENTITY_UPDATE_ATTRIBUTE and > AtlasPrivilege.ENTITY_CREATE_ATTRIBUTE rather than just checking on the > entity level. In certain places we will then drop the requirement for an > import to be active as this can now happen through other channels as well. > This allows operators to specify policies that allow granular controls over > attributes and system attributes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3755) Allow system attributes to be updated when policy allows
[ https://issues.apache.org/jira/browse/ATLAS-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin updated ATLAS-3755: -- Attachment: feature.patch > Allow system attributes to be updated when policy allows > > > Key: ATLAS-3755 > URL: https://issues.apache.org/jira/browse/ATLAS-3755 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.0.0, 2.1.0 >Reporter: Bolke de Bruin >Assignee: Bolke de Bruin >Priority: Critical > Attachments: > 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch, > 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch, > 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch, feature.patch > > > Atlas does not operate in a isolated environment, this is one of the reasons > the "homeId" system attribute was introduced. Unfortunately system attributes > can only be updated when importing. This means any integration with other > services is significantly limited (Kafka, Rest API will not work). (See also > ATLAS-3754) > To resolve this I propose to make it possible to update the system attributes > when policy allows it. This introduces new > AtlasPrivilege.ENTITY_UPDATE_SYSTEM_ATTRIBUTE and > AtlasPrivilege.ENTITY_CREATE_SYSTEM_ATTRIBUTE next to > AtlasPrivilege.ENTITY_UPDATE_ATTRIBUTE and > AtlasPrivilege.ENTITY_CREATE_ATTRIBUTE rather than just checking on the > entity level. In certain places we will then drop the requirement for an > import to be active as this can now happen through other channels as well. > This allows operators to specify policies that allow granular controls over > attributes and system attributes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (ATLAS-3755) Allow system attributes to be updated when policy allows
[ https://issues.apache.org/jira/browse/ATLAS-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17095795#comment-17095795 ] Bolke de Bruin edited comment on ATLAS-3755 at 4/29/20, 7:09 PM: - [~madhan] I don't think that would work for the KafkaConsumer, or does it? Please note that also Glossary needs to be updated to contain all system attributes as it is missing at least "homeId". In addition I don't think the risk is that high that system attributes get updated inadvertent. The default authorization model denies access to these attributes. Next to that it would require the incoming message to include those system properties. In this case you could argue that it should not be consumed at all if not allowed as it is a bad behaving client. We would like to integrate Atlas with another metadata system, which would actually make the occurrence much more frequent in around 50% of the updates. Given the fact that system attributes are part of the vertex there is not much additional cost as far as I can see in doing this during entity updates. On your second point. I'm not strongly bound to them so I can merge them. I do think there might be cases that you would like to allow an update but disallow a create. Authorization per attribute allows end-users to edit a particular attribute (say description) without allowing editing of all properties of the entities. This is actually a very common use case as you would like users to be able to enrich the metadata without adjusting some core attributes or system generated attributes. I understand your point about performance. What I could do is to submit an ArrayList and create a RangerCollectionResourceMatcher that requires all items in the submitted array to be matches (as opposed to RangerDefaultResourceMatcher) that should resolve the issue of CPU cycles and audit logs. What do you think? was (Author: bolke): [~madhan] I don't think that would work for the KafkaConsumer, or does it? Please note that also Glossary needs to be updated to contain all system attributes as it is missing at least "homeId". In addition I don't think the risk is that high that system attributes get updated inadvertent. The default authorization model denies access to these attributes. Next to that it would require the incoming message to include those system properties. In this case you could argue that it should not be consumed at all if not allowed as it is a bad behaving client. We would like to integrate Atlas with another metadata system, which would actually make the occurrence much more frequent in around 50% of the updates. Given the fact that system attributes are part of the vertex there is not much additional cost as far as I can see in doing this during entity updates. On your second point. I'm not strongly bound to them so I can merge them. I do think there might be cases that you would like to allow an update but disallow a create. Authorization per attribute allows end-users to edit a particular attribute (say description) without allowing editing of all properties of the entities. This is actually a very common use case as you would like users to be able to enrich the metadata without adjusting some core attributes or system generated attributes. I understand your point about performance. What I could do is to submit an Array in string format (e.g. "attribute1;attribute2;attribute4") and create a RangerArrayResourceMatcher that allows matching on 1 item which should resolve the issue of CPU cycles and audit logs. What do you think? > Allow system attributes to be updated when policy allows > > > Key: ATLAS-3755 > URL: https://issues.apache.org/jira/browse/ATLAS-3755 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.0.0, 2.1.0 >Reporter: Bolke de Bruin >Assignee: Bolke de Bruin >Priority: Critical > Attachments: > 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch, > 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch, > 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch > > > Atlas does not operate in a isolated environment, this is one of the reasons > the "homeId" system attribute was introduced. Unfortunately system attributes > can only be updated when importing. This means any integration with other > services is significantly limited (Kafka, Rest API will not work). (See also > ATLAS-3754) > To resolve this I propose to make it possible to update the system attributes > when policy allows it. This introduces new > AtlasPrivilege.ENTITY_UPDATE_SYSTEM_ATTRIBUTE and > AtlasPrivilege.ENTITY_CREATE_SYSTEM_ATTRIBUTE next to > AtlasPrivilege.ENTITY_UPDATE_ATTRIBUTE and >
[jira] [Commented] (ATLAS-3755) Allow system attributes to be updated when policy allows
[ https://issues.apache.org/jira/browse/ATLAS-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17095795#comment-17095795 ] Bolke de Bruin commented on ATLAS-3755: --- [~madhan] I don't think that would work for the KafkaConsumer, or does it? Please note that also Glossary needs to be updated to contain all system attributes as it is missing at least "homeId". In addition I don't think the risk is that high that system attributes get updated inadvertent. The default authorization model denies access to these attributes. Next to that it would require the incoming message to include those system properties. In this case you could argue that it should not be consumed at all if not allowed as it is a bad behaving client. We would like to integrate Atlas with another metadata system, which would actually make the occurrence much more frequent in around 50% of the updates. Given the fact that system attributes are part of the vertex there is not much additional cost as far as I can see in doing this during entity updates. On your second point. I'm not strongly bound so I can merge them. I do think there might be cases that you would like to allow an update but disallow a create. Authorization per attribute allows end-users to edit a particular attribute (say description) without allowing editing of all properties of the entities. This is actually a very common use case as you would like users to be able to enrich the metadata without adjusting some core attributes or system generated attributes. I understand your point about performance. What I could do is to submit an Array in string format (e.g. "attribute1;attribute2;attribute4") and create a RangerArrayResourceMatcher that allows matching on 1 item which should resolve the issue of CPU cycles and audit logs. What do you think? > Allow system attributes to be updated when policy allows > > > Key: ATLAS-3755 > URL: https://issues.apache.org/jira/browse/ATLAS-3755 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.0.0, 2.1.0 >Reporter: Bolke de Bruin >Assignee: Bolke de Bruin >Priority: Critical > Attachments: > 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch, > 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch, > 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch > > > Atlas does not operate in a isolated environment, this is one of the reasons > the "homeId" system attribute was introduced. Unfortunately system attributes > can only be updated when importing. This means any integration with other > services is significantly limited (Kafka, Rest API will not work). (See also > ATLAS-3754) > To resolve this I propose to make it possible to update the system attributes > when policy allows it. This introduces new > AtlasPrivilege.ENTITY_UPDATE_SYSTEM_ATTRIBUTE and > AtlasPrivilege.ENTITY_CREATE_SYSTEM_ATTRIBUTE next to > AtlasPrivilege.ENTITY_UPDATE_ATTRIBUTE and > AtlasPrivilege.ENTITY_CREATE_ATTRIBUTE rather than just checking on the > entity level. In certain places we will then drop the requirement for an > import to be active as this can now happen through other channels as well. > This allows operators to specify policies that allow granular controls over > attributes and system attributes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (ATLAS-3755) Allow system attributes to be updated when policy allows
[ https://issues.apache.org/jira/browse/ATLAS-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17095795#comment-17095795 ] Bolke de Bruin edited comment on ATLAS-3755 at 4/29/20, 6:51 PM: - [~madhan] I don't think that would work for the KafkaConsumer, or does it? Please note that also Glossary needs to be updated to contain all system attributes as it is missing at least "homeId". In addition I don't think the risk is that high that system attributes get updated inadvertent. The default authorization model denies access to these attributes. Next to that it would require the incoming message to include those system properties. In this case you could argue that it should not be consumed at all if not allowed as it is a bad behaving client. We would like to integrate Atlas with another metadata system, which would actually make the occurrence much more frequent in around 50% of the updates. Given the fact that system attributes are part of the vertex there is not much additional cost as far as I can see in doing this during entity updates. On your second point. I'm not strongly bound to them so I can merge them. I do think there might be cases that you would like to allow an update but disallow a create. Authorization per attribute allows end-users to edit a particular attribute (say description) without allowing editing of all properties of the entities. This is actually a very common use case as you would like users to be able to enrich the metadata without adjusting some core attributes or system generated attributes. I understand your point about performance. What I could do is to submit an Array in string format (e.g. "attribute1;attribute2;attribute4") and create a RangerArrayResourceMatcher that allows matching on 1 item which should resolve the issue of CPU cycles and audit logs. What do you think? was (Author: bolke): [~madhan] I don't think that would work for the KafkaConsumer, or does it? Please note that also Glossary needs to be updated to contain all system attributes as it is missing at least "homeId". In addition I don't think the risk is that high that system attributes get updated inadvertent. The default authorization model denies access to these attributes. Next to that it would require the incoming message to include those system properties. In this case you could argue that it should not be consumed at all if not allowed as it is a bad behaving client. We would like to integrate Atlas with another metadata system, which would actually make the occurrence much more frequent in around 50% of the updates. Given the fact that system attributes are part of the vertex there is not much additional cost as far as I can see in doing this during entity updates. On your second point. I'm not strongly bound so I can merge them. I do think there might be cases that you would like to allow an update but disallow a create. Authorization per attribute allows end-users to edit a particular attribute (say description) without allowing editing of all properties of the entities. This is actually a very common use case as you would like users to be able to enrich the metadata without adjusting some core attributes or system generated attributes. I understand your point about performance. What I could do is to submit an Array in string format (e.g. "attribute1;attribute2;attribute4") and create a RangerArrayResourceMatcher that allows matching on 1 item which should resolve the issue of CPU cycles and audit logs. What do you think? > Allow system attributes to be updated when policy allows > > > Key: ATLAS-3755 > URL: https://issues.apache.org/jira/browse/ATLAS-3755 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.0.0, 2.1.0 >Reporter: Bolke de Bruin >Assignee: Bolke de Bruin >Priority: Critical > Attachments: > 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch, > 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch, > 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch > > > Atlas does not operate in a isolated environment, this is one of the reasons > the "homeId" system attribute was introduced. Unfortunately system attributes > can only be updated when importing. This means any integration with other > services is significantly limited (Kafka, Rest API will not work). (See also > ATLAS-3754) > To resolve this I propose to make it possible to update the system attributes > when policy allows it. This introduces new > AtlasPrivilege.ENTITY_UPDATE_SYSTEM_ATTRIBUTE and > AtlasPrivilege.ENTITY_CREATE_SYSTEM_ATTRIBUTE next to > AtlasPrivilege.ENTITY_UPDATE_ATTRIBUTE and >
[jira] [Updated] (ATLAS-3755) Allow system attributes to be updated when policy allows
[ https://issues.apache.org/jira/browse/ATLAS-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin updated ATLAS-3755: -- Attachment: 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch > Allow system attributes to be updated when policy allows > > > Key: ATLAS-3755 > URL: https://issues.apache.org/jira/browse/ATLAS-3755 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.0.0, 2.1.0 >Reporter: Bolke de Bruin >Assignee: Bolke de Bruin >Priority: Critical > Attachments: > 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch, > 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch, > 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch > > > Atlas does not operate in a isolated environment, this is one of the reasons > the "homeId" system attribute was introduced. Unfortunately system attributes > can only be updated when importing. This means any integration with other > services is significantly limited (Kafka, Rest API will not work). (See also > ATLAS-3754) > To resolve this I propose to make it possible to update the system attributes > when policy allows it. This introduces new > AtlasPrivilege.ENTITY_UPDATE_SYSTEM_ATTRIBUTE and > AtlasPrivilege.ENTITY_CREATE_SYSTEM_ATTRIBUTE next to > AtlasPrivilege.ENTITY_UPDATE_ATTRIBUTE and > AtlasPrivilege.ENTITY_CREATE_ATTRIBUTE rather than just checking on the > entity level. In certain places we will then drop the requirement for an > import to be active as this can now happen through other channels as well. > This allows operators to specify policies that allow granular controls over > attributes and system attributes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3755) Allow system attributes to be updated when policy allows
[ https://issues.apache.org/jira/browse/ATLAS-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin updated ATLAS-3755: -- Attachment: 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch > Allow system attributes to be updated when policy allows > > > Key: ATLAS-3755 > URL: https://issues.apache.org/jira/browse/ATLAS-3755 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.0.0, 2.1.0 >Reporter: Bolke de Bruin >Assignee: Bolke de Bruin >Priority: Critical > Attachments: > 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch, > 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch > > > Atlas does not operate in a isolated environment, this is one of the reasons > the "homeId" system attribute was introduced. Unfortunately system attributes > can only be updated when importing. This means any integration with other > services is significantly limited (Kafka, Rest API will not work). (See also > ATLAS-3754) > To resolve this I propose to make it possible to update the system attributes > when policy allows it. This introduces new > AtlasPrivilege.ENTITY_UPDATE_SYSTEM_ATTRIBUTE and > AtlasPrivilege.ENTITY_CREATE_SYSTEM_ATTRIBUTE next to > AtlasPrivilege.ENTITY_UPDATE_ATTRIBUTE and > AtlasPrivilege.ENTITY_CREATE_ATTRIBUTE rather than just checking on the > entity level. In certain places we will then drop the requirement for an > import to be active as this can now happen through other channels as well. > This allows operators to specify policies that allow granular controls over > attributes and system attributes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3755) Allow system attributes to be updated when policy allows
[ https://issues.apache.org/jira/browse/ATLAS-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin updated ATLAS-3755: -- External issue URL: https://reviews.apache.org/r/72438/ > Allow system attributes to be updated when policy allows > > > Key: ATLAS-3755 > URL: https://issues.apache.org/jira/browse/ATLAS-3755 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.0.0, 2.1.0 >Reporter: Bolke de Bruin >Assignee: Bolke de Bruin >Priority: Critical > Attachments: > 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch > > > Atlas does not operate in a isolated environment, this is one of the reasons > the "homeId" system attribute was introduced. Unfortunately system attributes > can only be updated when importing. This means any integration with other > services is significantly limited (Kafka, Rest API will not work). (See also > ATLAS-3754) > To resolve this I propose to make it possible to update the system attributes > when policy allows it. This introduces new > AtlasPrivilege.ENTITY_UPDATE_SYSTEM_ATTRIBUTE and > AtlasPrivilege.ENTITY_CREATE_SYSTEM_ATTRIBUTE next to > AtlasPrivilege.ENTITY_UPDATE_ATTRIBUTE and > AtlasPrivilege.ENTITY_CREATE_ATTRIBUTE rather than just checking on the > entity level. In certain places we will then drop the requirement for an > import to be active as this can now happen through other channels as well. > This allows operators to specify policies that allow granular controls over > attributes and system attributes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3755) Allow system attributes to be updated when policy allows
[ https://issues.apache.org/jira/browse/ATLAS-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin updated ATLAS-3755: -- Attachment: 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch > Allow system attributes to be updated when policy allows > > > Key: ATLAS-3755 > URL: https://issues.apache.org/jira/browse/ATLAS-3755 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.0.0, 2.1.0 >Reporter: Bolke de Bruin >Assignee: Bolke de Bruin >Priority: Critical > Attachments: > 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch > > > Atlas does not operate in a isolated environment, this is one of the reasons > the "homeId" system attribute was introduced. Unfortunately system attributes > can only be updated when importing. This means any integration with other > services is significantly limited (Kafka, Rest API will not work). (See also > ATLAS-3754) > To resolve this I propose to make it possible to update the system attributes > when policy allows it. This introduces new > AtlasPrivilege.ENTITY_UPDATE_SYSTEM_ATTRIBUTE and > AtlasPrivilege.ENTITY_CREATE_SYSTEM_ATTRIBUTE next to > AtlasPrivilege.ENTITY_UPDATE_ATTRIBUTE and > AtlasPrivilege.ENTITY_CREATE_ATTRIBUTE rather than just checking on the > entity level. In certain places we will then drop the requirement for an > import to be active as this can now happen through other channels as well. > This allows operators to specify policies that allow granular controls over > attributes and system attributes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-3755) Allow system attributes to be updated when policy allows
[ https://issues.apache.org/jira/browse/ATLAS-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17093423#comment-17093423 ] Bolke de Bruin commented on ATLAS-3755: --- cc [~madhan] > Allow system attributes to be updated when policy allows > > > Key: ATLAS-3755 > URL: https://issues.apache.org/jira/browse/ATLAS-3755 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.0.0, 2.1.0 >Reporter: Bolke de Bruin >Assignee: Bolke de Bruin >Priority: Critical > > Atlas does not operate in a isolated environment, this is one of the reasons > the "homeId" system attribute was introduced. Unfortunately system attributes > can only be updated when importing. This means any integration with other > services is significantly limited (Kafka, Rest API will not work). (See also > ATLAS-3754) > To resolve this I propose to make it possible to update the system attributes > when policy allows it. This introduces new > AtlasPrivilege.ENTITY_UPDATE_SYSTEM_ATTRIBUTE and > AtlasPrivilege.ENTITY_CREATE_SYSTEM_ATTRIBUTE next to > AtlasPrivilege.ENTITY_UPDATE_ATTRIBUTE and > AtlasPrivilege.ENTITY_CREATE_ATTRIBUTE rather than just checking on the > entity level. In certain places we will then drop the requirement for an > import to be active as this can now happen through other channels as well. > This allows operators to specify policies that allow granular controls over > attributes and system attributes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-3754) System/Business Attributes (like homeId) not being saved
[
https://issues.apache.org/jira/browse/ATLAS-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17093419#comment-17093419
]
Bolke de Bruin commented on ATLAS-3754:
---
[~madhan] See the linked issue please. I am working on a patch to refactors
preCreateOrUpdate and createOrUpdate to allow for this per policy. Changes are
being detected and only set if the policy allows it. Of course if a client
still specifies an (system)attribute but is not allowed to set it, it would
fail now instead of silently dropping such items. Imho, that should be fine as
the client would then be misbehaving.
> System/Business Attributes (like homeId) not being saved
>
>
> Key: ATLAS-3754
> URL: https://issues.apache.org/jira/browse/ATLAS-3754
> Project: Atlas
> Issue Type: Bug
>Affects Versions: trunk
>Reporter: Bolke de Bruin
>Priority: Blocker
>
> System attributes and business attributes are not being saved (homeId at
> least isn't)
> {code}
> skipping unchanged entity: AtlasEntity{AtlasStruct{typeName='hdfs_path',
> attributes=[name:electionresults,
> qualifiedName:electionresults.electionresults@Sandbox,
> path:/user/dmaster/electionresults/ls2014.tsv,
> clusterName:Sandbox]}guid='8e6ede0c-22c5-4b41-bee4-757568b4b99a',
> homeId='my_home_id_x', isProxy='false', isIncomplete=false, provenanceType=0,
> status=null, createdBy='null', updatedBy='null', createTime=null,
> updateTime=null, version=0, relationshipAttributes=[], classifications=[],
> meanings=[], customAttributes=[], businessAttributes=[], labels=[]}
> (AtlasEntityStoreV2:1204)
> {code}
> {code}
> skipping unchanged entity: AtlasEntity{AtlasStruct{typeName='hdfs_path',
> attributes=[name:electionresults,
> qualifiedName:electionresults.electionresults@Sandbox,
> path:/user/dmaster/electionresults/ls2014.tsv,
> clusterName:Sandbox]}guid='8e6ede0c-22c5-4b41-bee4-757568b4b99a',
> homeId='my_home_id_2', isProxy='false', isIncomplete=false, provenanceType=0,
> status=null, createdBy='null', updatedBy='null', createTime=null,
> updateTime=null, version=0, relationshipAttributes=[], classifications=[],
> meanings=[], customAttributes=[], businessAttributes=[], labels=[]}
> (AtlasEntityStoreV2:1204)
> {code}
> Note the change in homeId and the entity not being updated
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ATLAS-3755) Allow system attributes to be updated when policy allows
[ https://issues.apache.org/jira/browse/ATLAS-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin updated ATLAS-3755: -- Description: Atlas does not operate in a isolated environment, this is one of the reasons the "homeId" system attribute was introduced. Unfortunately system attributes can only be updated when importing. This means any integration with other services is significantly limited (Kafka, Rest API will not work). (See also ATLAS-3754) To resolve this I propose to make it possible to update the system attributes when policy allows it. This introduces new AtlasPrivilege.ENTITY_UPDATE_SYSTEM_ATTRIBUTE and AtlasPrivilege.ENTITY_CREATE_SYSTEM_ATTRIBUTE next to AtlasPrivilege.ENTITY_UPDATE_ATTRIBUTE and AtlasPrivilege.ENTITY_CREATE_ATTRIBUTE rather than just checking on the entity level. In certain places we will then drop the requirement for an import to be active as this can now happen through other channels as well. This allows operators to specify policies that allow granular controls over attributes and system attributes. was: Atlas does not operate in a isolated environment, this is one of the reasons the "homeId" system attribute was introduced. Unfortunately system attributes can only be updated when importing. This means any integration with other services is significantly limited (Kafka, Rest API will not work). (See also ATLAS-3754) To resolve this I propose to make it possible to update the system attributes when `homeId` is present in the entity update. An access check should then verify whether such an update can happen if a `homeId` is present. > Allow system attributes to be updated when policy allows > > > Key: ATLAS-3755 > URL: https://issues.apache.org/jira/browse/ATLAS-3755 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.0.0, 2.1.0 >Reporter: Bolke de Bruin >Assignee: Bolke de Bruin >Priority: Critical > > Atlas does not operate in a isolated environment, this is one of the reasons > the "homeId" system attribute was introduced. Unfortunately system attributes > can only be updated when importing. This means any integration with other > services is significantly limited (Kafka, Rest API will not work). (See also > ATLAS-3754) > To resolve this I propose to make it possible to update the system attributes > when policy allows it. This introduces new > AtlasPrivilege.ENTITY_UPDATE_SYSTEM_ATTRIBUTE and > AtlasPrivilege.ENTITY_CREATE_SYSTEM_ATTRIBUTE next to > AtlasPrivilege.ENTITY_UPDATE_ATTRIBUTE and > AtlasPrivilege.ENTITY_CREATE_ATTRIBUTE rather than just checking on the > entity level. In certain places we will then drop the requirement for an > import to be active as this can now happen through other channels as well. > This allows operators to specify policies that allow granular controls over > attributes and system attributes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3755) Allow system attributes to be updated when policy allows
[ https://issues.apache.org/jira/browse/ATLAS-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin updated ATLAS-3755: -- Summary: Allow system attributes to be updated when policy allows (was: Allow system attributes to be updated when homeId is specified) > Allow system attributes to be updated when policy allows > > > Key: ATLAS-3755 > URL: https://issues.apache.org/jira/browse/ATLAS-3755 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.0.0, 2.1.0 >Reporter: Bolke de Bruin >Assignee: Bolke de Bruin >Priority: Critical > > Atlas does not operate in a isolated environment, this is one of the reasons > the "homeId" system attribute was introduced. Unfortunately system attributes > can only be updated when importing. This means any integration with other > services is significantly limited (Kafka, Rest API will not work). (See also > ATLAS-3754) > To resolve this I propose to make it possible to update the system attributes > when `homeId` is present in the entity update. An access check should then > verify whether such an update can happen if a `homeId` is present. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ATLAS-3755) Allow system attributes to be updated when homeId is specified
Bolke de Bruin created ATLAS-3755: - Summary: Allow system attributes to be updated when homeId is specified Key: ATLAS-3755 URL: https://issues.apache.org/jira/browse/ATLAS-3755 Project: Atlas Issue Type: Improvement Components: atlas-core Affects Versions: 2.0.0, 2.1.0 Reporter: Bolke de Bruin Assignee: Bolke de Bruin Atlas does not operate in a isolated environment, this is one of the reasons the "homeId" system attribute was introduced. Unfortunately system attributes can only be updated when importing. This means any integration with other services is significantly limited (Kafka, Rest API will not work). (See also ATLAS-3754) To resolve this I propose to make it possible to update the system attributes when `homeId` is present in the entity update. An access check should then verify whether such an update can happen if a `homeId` is present. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3754) System/Business Attributes (like homeId) not being saved
[
https://issues.apache.org/jira/browse/ATLAS-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bolke de Bruin updated ATLAS-3754:
--
Description:
System attributes and business attributes are not being saved (homeId at least
isn't)
{code}
skipping unchanged entity: AtlasEntity{AtlasStruct{typeName='hdfs_path',
attributes=[name:electionresults,
qualifiedName:electionresults.electionresults@Sandbox,
path:/user/dmaster/electionresults/ls2014.tsv,
clusterName:Sandbox]}guid='8e6ede0c-22c5-4b41-bee4-757568b4b99a',
homeId='my_home_id_x', isProxy='false', isIncomplete=false, provenanceType=0,
status=null, createdBy='null', updatedBy='null', createTime=null,
updateTime=null, version=0, relationshipAttributes=[], classifications=[],
meanings=[], customAttributes=[], businessAttributes=[], labels=[]}
(AtlasEntityStoreV2:1204)
{code}
{code}
skipping unchanged entity: AtlasEntity{AtlasStruct{typeName='hdfs_path',
attributes=[name:electionresults,
qualifiedName:electionresults.electionresults@Sandbox,
path:/user/dmaster/electionresults/ls2014.tsv,
clusterName:Sandbox]}guid='8e6ede0c-22c5-4b41-bee4-757568b4b99a',
homeId='my_home_id_2', isProxy='false', isIncomplete=false, provenanceType=0,
status=null, createdBy='null', updatedBy='null', createTime=null,
updateTime=null, version=0, relationshipAttributes=[], classifications=[],
meanings=[], customAttributes=[], businessAttributes=[], labels=[]}
(AtlasEntityStoreV2:1204)
{code}
Note the change in homeId and the entity not being updated
was:
Since introducing the name spacing attributes that fall outside of the typedef
i.e. system attributes and business attributes are not being saved anymore due
to a Noop in AtlasStructType
{code}
public AtlasAttribute getAttribute(String attributeName) {
AtlasAttribute ret = allAttributes.get(attributeName);
if (ret == null) {
ret = getSystemAttribute(attributeName);
}
if (ret == null) {
ret = getBusinesAAttribute(attributeName);
}
return ret;
}
public AtlasAttribute getSystemAttribute(String attributeName) {
return null;
}
public AtlasAttribute getBusinesAAttribute(String attributeName) {
return null;
}
{code}
{code}
skipping unchanged entity: AtlasEntity{AtlasStruct{typeName='hdfs_path',
attributes=[name:electionresults,
qualifiedName:electionresults.electionresults@Sandbox,
path:/user/dmaster/electionresults/ls2014.tsv,
clusterName:Sandbox]}guid='8e6ede0c-22c5-4b41-bee4-757568b4b99a',
homeId='my_home_id_x', isProxy='false', isIncomplete=false, provenanceType=0,
status=null, createdBy='null', updatedBy='null', createTime=null,
updateTime=null, version=0, relationshipAttributes=[], classifications=[],
meanings=[], customAttributes=[], businessAttributes=[], labels=[]}
(AtlasEntityStoreV2:1204)
{code}
{code}
skipping unchanged entity: AtlasEntity{AtlasStruct{typeName='hdfs_path',
attributes=[name:electionresults,
qualifiedName:electionresults.electionresults@Sandbox,
path:/user/dmaster/electionresults/ls2014.tsv,
clusterName:Sandbox]}guid='8e6ede0c-22c5-4b41-bee4-757568b4b99a',
homeId='my_home_id_2', isProxy='false', isIncomplete=false, provenanceType=0,
status=null, createdBy='null', updatedBy='null', createTime=null,
updateTime=null, version=0, relationshipAttributes=[], classifications=[],
meanings=[], customAttributes=[], businessAttributes=[], labels=[]}
(AtlasEntityStoreV2:1204)
{code}
Note the change in homeId and the entity not being updated
> System/Business Attributes (like homeId) not being saved
>
>
> Key: ATLAS-3754
> URL: https://issues.apache.org/jira/browse/ATLAS-3754
> Project: Atlas
> Issue Type: Bug
>Affects Versions: trunk
>Reporter: Bolke de Bruin
>Priority: Blocker
>
> System attributes and business attributes are not being saved (homeId at
> least isn't)
> {code}
> skipping unchanged entity: AtlasEntity{AtlasStruct{typeName='hdfs_path',
> attributes=[name:electionresults,
> qualifiedName:electionresults.electionresults@Sandbox,
> path:/user/dmaster/electionresults/ls2014.tsv,
> clusterName:Sandbox]}guid='8e6ede0c-22c5-4b41-bee4-757568b4b99a',
> homeId='my_home_id_x', isProxy='false', isIncomplete=false, provenanceType=0,
> status=null, createdBy='null', updatedBy='null', createTime=null,
> updateTime=null, version=0, relationshipAttributes=[], classifications=[],
> meanings=[], customAttributes=[], businessAttributes=[], labels=[]}
> (AtlasEntityStoreV2:1204)
> {code}
> {code}
> skipping unchanged entity: AtlasEntity{AtlasStruct{typeName='hdfs_path',
> attributes=[name:electionresults,
> qualifiedName:electionresults.electionresults@Sandbox,
> path:/user/dmaster/electionresults/ls2014.tsv,
>
[jira] [Commented] (ATLAS-3754) System/Business Attributes (like homeId) not being saved
[
https://issues.apache.org/jira/browse/ATLAS-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091455#comment-17091455
]
Bolke de Bruin commented on ATLAS-3754:
---
cc [~madhan]
> System/Business Attributes (like homeId) not being saved
>
>
> Key: ATLAS-3754
> URL: https://issues.apache.org/jira/browse/ATLAS-3754
> Project: Atlas
> Issue Type: Bug
>Affects Versions: trunk
>Reporter: Bolke de Bruin
>Priority: Blocker
>
> Since introducing the name spacing attributes that fall outside of the
> typedef i.e. system attributes and business attributes are not being saved
> anymore due to a Noop in AtlasStructType
> {code}
> public AtlasAttribute getAttribute(String attributeName) {
> AtlasAttribute ret = allAttributes.get(attributeName);
> if (ret == null) {
> ret = getSystemAttribute(attributeName);
> }
> if (ret == null) {
> ret = getBusinesAAttribute(attributeName);
> }
> return ret;
> }
> public AtlasAttribute getSystemAttribute(String attributeName) {
> return null;
> }
> public AtlasAttribute getBusinesAAttribute(String attributeName) {
> return null;
> }
> {code}
> {code}
> skipping unchanged entity: AtlasEntity{AtlasStruct{typeName='hdfs_path',
> attributes=[name:electionresults,
> qualifiedName:electionresults.electionresults@Sandbox,
> path:/user/dmaster/electionresults/ls2014.tsv,
> clusterName:Sandbox]}guid='8e6ede0c-22c5-4b41-bee4-757568b4b99a',
> homeId='my_home_id_x', isProxy='false', isIncomplete=false, provenanceType=0,
> status=null, createdBy='null', updatedBy='null', createTime=null,
> updateTime=null, version=0, relationshipAttributes=[], classifications=[],
> meanings=[], customAttributes=[], businessAttributes=[], labels=[]}
> (AtlasEntityStoreV2:1204)
> {code}
> {code}
> skipping unchanged entity: AtlasEntity{AtlasStruct{typeName='hdfs_path',
> attributes=[name:electionresults,
> qualifiedName:electionresults.electionresults@Sandbox,
> path:/user/dmaster/electionresults/ls2014.tsv,
> clusterName:Sandbox]}guid='8e6ede0c-22c5-4b41-bee4-757568b4b99a',
> homeId='my_home_id_2', isProxy='false', isIncomplete=false, provenanceType=0,
> status=null, createdBy='null', updatedBy='null', createTime=null,
> updateTime=null, version=0, relationshipAttributes=[], classifications=[],
> meanings=[], customAttributes=[], businessAttributes=[], labels=[]}
> (AtlasEntityStoreV2:1204)
> {code}
> Note the change in homeId and the entity not being updated
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (ATLAS-3754) Attributes (homeId) not being saved
Bolke de Bruin created ATLAS-3754:
-
Summary: Attributes (homeId) not being saved
Key: ATLAS-3754
URL: https://issues.apache.org/jira/browse/ATLAS-3754
Project: Atlas
Issue Type: Bug
Affects Versions: trunk
Reporter: Bolke de Bruin
Since introducing the name spacing attributes that fall outside of the typedef
i.e. system attributes and business attributes are not being saved anymore due
to a Noop in AtlasStructType
{code}
public AtlasAttribute getAttribute(String attributeName) {
AtlasAttribute ret = allAttributes.get(attributeName);
if (ret == null) {
ret = getSystemAttribute(attributeName);
}
if (ret == null) {
ret = getBusinesAAttribute(attributeName);
}
return ret;
}
public AtlasAttribute getSystemAttribute(String attributeName) {
return null;
}
public AtlasAttribute getBusinesAAttribute(String attributeName) {
return null;
}
{code}
{code}
skipping unchanged entity: AtlasEntity{AtlasStruct{typeName='hdfs_path',
attributes=[name:electionresults,
qualifiedName:electionresults.electionresults@Sandbox,
path:/user/dmaster/electionresults/ls2014.tsv,
clusterName:Sandbox]}guid='8e6ede0c-22c5-4b41-bee4-757568b4b99a',
homeId='my_home_id_x', isProxy='false', isIncomplete=false, provenanceType=0,
status=null, createdBy='null', updatedBy='null', createTime=null,
updateTime=null, version=0, relationshipAttributes=[], classifications=[],
meanings=[], customAttributes=[], businessAttributes=[], labels=[]}
(AtlasEntityStoreV2:1204)
{code}
{code}
skipping unchanged entity: AtlasEntity{AtlasStruct{typeName='hdfs_path',
attributes=[name:electionresults,
qualifiedName:electionresults.electionresults@Sandbox,
path:/user/dmaster/electionresults/ls2014.tsv,
clusterName:Sandbox]}guid='8e6ede0c-22c5-4b41-bee4-757568b4b99a',
homeId='my_home_id_2', isProxy='false', isIncomplete=false, provenanceType=0,
status=null, createdBy='null', updatedBy='null', createTime=null,
updateTime=null, version=0, relationshipAttributes=[], classifications=[],
meanings=[], customAttributes=[], businessAttributes=[], labels=[]}
(AtlasEntityStoreV2:1204)
{code}
Note the change in homeId and the entity not being updated
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ATLAS-3754) System/Business Attributes (like homeId) not being saved
[
https://issues.apache.org/jira/browse/ATLAS-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bolke de Bruin updated ATLAS-3754:
--
Summary: System/Business Attributes (like homeId) not being saved (was:
Attributes (homeId) not being saved)
> System/Business Attributes (like homeId) not being saved
>
>
> Key: ATLAS-3754
> URL: https://issues.apache.org/jira/browse/ATLAS-3754
> Project: Atlas
> Issue Type: Bug
>Affects Versions: trunk
>Reporter: Bolke de Bruin
>Priority: Blocker
>
> Since introducing the name spacing attributes that fall outside of the
> typedef i.e. system attributes and business attributes are not being saved
> anymore due to a Noop in AtlasStructType
> {code}
> public AtlasAttribute getAttribute(String attributeName) {
> AtlasAttribute ret = allAttributes.get(attributeName);
> if (ret == null) {
> ret = getSystemAttribute(attributeName);
> }
> if (ret == null) {
> ret = getBusinesAAttribute(attributeName);
> }
> return ret;
> }
> public AtlasAttribute getSystemAttribute(String attributeName) {
> return null;
> }
> public AtlasAttribute getBusinesAAttribute(String attributeName) {
> return null;
> }
> {code}
> {code}
> skipping unchanged entity: AtlasEntity{AtlasStruct{typeName='hdfs_path',
> attributes=[name:electionresults,
> qualifiedName:electionresults.electionresults@Sandbox,
> path:/user/dmaster/electionresults/ls2014.tsv,
> clusterName:Sandbox]}guid='8e6ede0c-22c5-4b41-bee4-757568b4b99a',
> homeId='my_home_id_x', isProxy='false', isIncomplete=false, provenanceType=0,
> status=null, createdBy='null', updatedBy='null', createTime=null,
> updateTime=null, version=0, relationshipAttributes=[], classifications=[],
> meanings=[], customAttributes=[], businessAttributes=[], labels=[]}
> (AtlasEntityStoreV2:1204)
> {code}
> {code}
> skipping unchanged entity: AtlasEntity{AtlasStruct{typeName='hdfs_path',
> attributes=[name:electionresults,
> qualifiedName:electionresults.electionresults@Sandbox,
> path:/user/dmaster/electionresults/ls2014.tsv,
> clusterName:Sandbox]}guid='8e6ede0c-22c5-4b41-bee4-757568b4b99a',
> homeId='my_home_id_2', isProxy='false', isIncomplete=false, provenanceType=0,
> status=null, createdBy='null', updatedBy='null', createTime=null,
> updateTime=null, version=0, relationshipAttributes=[], classifications=[],
> meanings=[], customAttributes=[], businessAttributes=[], labels=[]}
> (AtlasEntityStoreV2:1204)
> {code}
> Note the change in homeId and the entity not being updated
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ATLAS-3546) isOptional for composition relationship category?
[ https://issues.apache.org/jira/browse/ATLAS-3546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16998540#comment-16998540 ] Bolke de Bruin commented on ATLAS-3546: --- +1 on that last comment see also ATLAS-3254 > isOptional for composition relationship category? > - > > Key: ATLAS-3546 > URL: https://issues.apache.org/jira/browse/ATLAS-3546 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.0.0 >Reporter: charles shen >Priority: Major > Attachments: json.jpg, xml.jpg, xml.jpg > > > I noticed since > [ATLAS-3051|https://issues.apache.org/jira/browse/ATLAS-3051], the > relationship attribute must be specified in the end def which is not > container and relationship category is composition. > I understand it's to prohibit orphan children but is it too strong? Reason > below: > # I have to provide all the entities along the relationship path in the > payload when creating a child, eg, for RDBMS, I have to provide > rdbms_instance, rdbms_db, rdbms_table, rdbms_column where I just want to > create a single rdbms_column, it brings performance overhead to check > existence of rdbms_instance, rdbms_db, etc..., > # I have defined a composition relationship type where each end is the same > entity type, it couldn't be created successfully anymore since it always > requires the mandatory attribute where it's the same type and then falls into > infinite loop. > Three possible fixes: > # Remove the isOptional constraint, since ownedRef/inverseRef doesn't have > such constraint. > # Add isOptional to relationship type end def. > # Add option in Rest to ignore the isOptional constraint for relationship > type. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (ATLAS-3254) Atlas entity with large array of refs causes performance issues for lineage
[ https://issues.apache.org/jira/browse/ATLAS-3254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16998538#comment-16998538 ] Bolke de Bruin edited comment on ATLAS-3254 at 12/17/19 8:08 PM: - What [~mayank_nj] do you consider to load “properly”? What is the time taken to show the properties? What is the size of the json sent over the network (ours is > 27mb)? What is the load time? What is the render time? Are you saying the loading of 200K objects in a pseudodir is taking over 1h? That is not “proper” I think? was (Author: bolke): What [~mayank_nj] do you consider to load “properly”? What is the time taken to show the properties? What is the size of the json sent over the network (ours is > 27mb)? What is the load time? What is the render time? > Atlas entity with large array of refs causes performance issues for lineage > --- > > Key: ATLAS-3254 > URL: https://issues.apache.org/jira/browse/ATLAS-3254 > Project: Atlas > Issue Type: Bug > Components: atlas-core, atlas-webui >Affects Versions: 1.0.0, 2.0.0 >Reporter: Adam Rempter >Assignee: Mayank Jain >Priority: Major > Labels: performance > Attachments: Screenshot 2019-11-28 at 21.18.44.png, > entity_auto_create.sh, example_create_entities.json, > rest_entity_get_pseudodir.json > > > We use “aws_s3_pseudo_dir” type from 3020-aws_s3_typedefs.json model. > It has following property: > "name": "s3Objects", > "typeName": "array" > > Now in AWS buckets you can have thousands of objects. This causes that > s3Objects array grows quite quickly, causing aws_s3_pseudo_dir entity Json to > rich easly few MBs. > > Then we start seeing problems like: > * UI is dying on displaying entity properties or lineage > * Error in logs: audit record too long: entityType=aws_s3_pseudo_dir, > guid=24398271-6ba0-4db5-adfa-38e432dc55ce, size=1053931; maxSize=1048576. > entity attribute values not stored in audit (EntityAuditListenerV2:234) > * Some errors with write to HBase (java.lang.IllegalArgumentException: > KeyValue size too large, as workaround we set hbase.client.keyvalue.maxsize > param to 0) > * kafka consumer errors (we can of course set some parameters on consumer, > but I think it is just workaround) > … > Exception in NotificationHookConsumer (NotificationHookConsumer:332) > org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be > completed since the group has already rebalanced and assigned the partitions > to another member. This means that the time between subsequen > t calls to poll() was longer than the configured max.poll.interval.ms, which > typically implies that the poll loop is spending too much time message > processing. You can address this either by increasing the sessio > n timeout or by reducing the maximum size of batches returned in poll() with > max.poll.records. > … > Specifying pseudo_dir is required for s3objects: > name": "pseudoDirectory", > "typeName": "aws_s3_pseudo_dir", > "cardinality": "SINGLE", > "isIndexable": false, > *"isOptional": false,* > "isUnique": false, > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-3254) Atlas entity with large array of refs causes performance issues for lineage
[ https://issues.apache.org/jira/browse/ATLAS-3254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16998538#comment-16998538 ] Bolke de Bruin commented on ATLAS-3254: --- What [~mayank_nj] do you consider to load “properly”? What is the time taken to show the properties? What is the size of the json sent over the network (ours is > 27mb)? What is the load time? What is the render time? > Atlas entity with large array of refs causes performance issues for lineage > --- > > Key: ATLAS-3254 > URL: https://issues.apache.org/jira/browse/ATLAS-3254 > Project: Atlas > Issue Type: Bug > Components: atlas-core, atlas-webui >Affects Versions: 1.0.0, 2.0.0 >Reporter: Adam Rempter >Assignee: Mayank Jain >Priority: Major > Labels: performance > Attachments: Screenshot 2019-11-28 at 21.18.44.png, > entity_auto_create.sh, example_create_entities.json, > rest_entity_get_pseudodir.json > > > We use “aws_s3_pseudo_dir” type from 3020-aws_s3_typedefs.json model. > It has following property: > "name": "s3Objects", > "typeName": "array" > > Now in AWS buckets you can have thousands of objects. This causes that > s3Objects array grows quite quickly, causing aws_s3_pseudo_dir entity Json to > rich easly few MBs. > > Then we start seeing problems like: > * UI is dying on displaying entity properties or lineage > * Error in logs: audit record too long: entityType=aws_s3_pseudo_dir, > guid=24398271-6ba0-4db5-adfa-38e432dc55ce, size=1053931; maxSize=1048576. > entity attribute values not stored in audit (EntityAuditListenerV2:234) > * Some errors with write to HBase (java.lang.IllegalArgumentException: > KeyValue size too large, as workaround we set hbase.client.keyvalue.maxsize > param to 0) > * kafka consumer errors (we can of course set some parameters on consumer, > but I think it is just workaround) > … > Exception in NotificationHookConsumer (NotificationHookConsumer:332) > org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be > completed since the group has already rebalanced and assigned the partitions > to another member. This means that the time between subsequen > t calls to poll() was longer than the configured max.poll.interval.ms, which > typically implies that the poll loop is spending too much time message > processing. You can address this either by increasing the sessio > n timeout or by reducing the maximum size of batches returned in poll() with > max.poll.records. > … > Specifying pseudo_dir is required for s3objects: > name": "pseudoDirectory", > "typeName": "aws_s3_pseudo_dir", > "cardinality": "SINGLE", > "isIndexable": false, > *"isOptional": false,* > "isUnique": false, > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3553) Integrity issue: Recreated deleted (propagated) classification has same entities
[ https://issues.apache.org/jira/browse/ATLAS-3553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin updated ATLAS-3553: -- Affects Version/s: trunk 2.0.0 > Integrity issue: Recreated deleted (propagated) classification has same > entities > > > Key: ATLAS-3553 > URL: https://issues.apache.org/jira/browse/ATLAS-3553 > Project: Atlas > Issue Type: Bug >Affects Versions: 2.0.0, trunk >Reporter: Bolke de Bruin >Priority: Blocker > > ATLAS-3545 solved the issue to the null pointer exception. The classification > still lists entities in the classification part of the UI. Now when trying to > remove the classification this works, but when recreating the classification > with the same name the same entities shows up again (same list). > Another observation is that looking up the properties of the entity does not > show the classification so this is contrary what the classification UI is > saying. This reduces confidence in the security (i.e. can we rely on the > classifications) of Atlas. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ATLAS-3553) Integrity issue: Recreated deleted (propagated) classification has same entities
Bolke de Bruin created ATLAS-3553: - Summary: Integrity issue: Recreated deleted (propagated) classification has same entities Key: ATLAS-3553 URL: https://issues.apache.org/jira/browse/ATLAS-3553 Project: Atlas Issue Type: Bug Reporter: Bolke de Bruin ATLAS-3545 solved the issue to the null pointer exception. The classification still lists entities in the classification part of the UI. Now when trying to remove the classification this works, but when recreating the classification with the same name the same entities shows up again (same list). Another observation is that looking up the properties of the entity does not show the classification so this is contrary what the classification UI is saying. This reduces confidence in the security (i.e. can we rely on the classifications) of Atlas. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3552) Race condition while adding a classification
[ https://issues.apache.org/jira/browse/ATLAS-3552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin updated ATLAS-3552: -- Description: Following up on ATLAS-3545 we set a classification on a aws_s3_bucket with several thousands (200K+) of objects. As the UI wasn’t clear if the classification was applied so we tried it a couple of times. This resulted in the UI displaying it 3 times was: Following up on ATLAS-3454 we set a classification on a aws_s3_bucket with several thousands (200K+) of objects. As the UI wasn’t clear if the classification was applied so we tried it a couple of times. This resulted in the UI displaying it 3 times > Race condition while adding a classification > > > Key: ATLAS-3552 > URL: https://issues.apache.org/jira/browse/ATLAS-3552 > Project: Atlas > Issue Type: Bug >Affects Versions: 2.0.0, trunk >Reporter: Bolke de Bruin >Priority: Critical > > Following up on ATLAS-3545 > we set a classification on a aws_s3_bucket with several thousands (200K+) of > objects. As the UI wasn’t clear if the classification was applied so we tried > it a couple of times. This resulted in the UI displaying it 3 times -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ATLAS-3552) Race condition while adding a classification
Bolke de Bruin created ATLAS-3552: - Summary: Race condition while adding a classification Key: ATLAS-3552 URL: https://issues.apache.org/jira/browse/ATLAS-3552 Project: Atlas Issue Type: Bug Affects Versions: 2.0.0, trunk Reporter: Bolke de Bruin Following up on ATLAS-3454 we set a classification on a aws_s3_bucket with several thousands (200K+) of objects. As the UI wasn’t clear if the classification was applied so we tried it a couple of times. This resulted in the UI displaying it 3 times -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-3254) Atlas entity with large array of refs causes performance issues for lineage
[ https://issues.apache.org/jira/browse/ATLAS-3254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16989021#comment-16989021 ] Bolke de Bruin commented on ATLAS-3254: --- 1. Create aws_s3_bucket 2. Add 200K+ objects in the same pseudodir to this bucket 3. Try to load the properties from the pseudodirs > Atlas entity with large array of refs causes performance issues for lineage > --- > > Key: ATLAS-3254 > URL: https://issues.apache.org/jira/browse/ATLAS-3254 > Project: Atlas > Issue Type: Bug > Components: atlas-core, atlas-webui >Affects Versions: 1.0.0, 2.0.0 >Reporter: Adam Rempter >Assignee: Mayank Jain >Priority: Major > Labels: performance > Attachments: Screenshot 2019-11-28 at 21.18.44.png, > example_create_entities.json, rest_entity_get_pseudodir.json > > > We use “aws_s3_pseudo_dir” type from 3020-aws_s3_typedefs.json model. > It has following property: > "name": "s3Objects", > "typeName": "array" > > Now in AWS buckets you can have thousands of objects. This causes that > s3Objects array grows quite quickly, causing aws_s3_pseudo_dir entity Json to > rich easly few MBs. > > Then we start seeing problems like: > * UI is dying on displaying entity properties or lineage > * Error in logs: audit record too long: entityType=aws_s3_pseudo_dir, > guid=24398271-6ba0-4db5-adfa-38e432dc55ce, size=1053931; maxSize=1048576. > entity attribute values not stored in audit (EntityAuditListenerV2:234) > * Some errors with write to HBase (java.lang.IllegalArgumentException: > KeyValue size too large, as workaround we set hbase.client.keyvalue.maxsize > param to 0) > * kafka consumer errors (we can of course set some parameters on consumer, > but I think it is just workaround) > … > Exception in NotificationHookConsumer (NotificationHookConsumer:332) > org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be > completed since the group has already rebalanced and assigned the partitions > to another member. This means that the time between subsequen > t calls to poll() was longer than the configured max.poll.interval.ms, which > typically implies that the poll loop is spending too much time message > processing. You can address this either by increasing the sessio > n timeout or by reducing the maximum size of batches returned in poll() with > max.poll.records. > … > Specifying pseudo_dir is required for s3objects: > name": "pseudoDirectory", > "typeName": "aws_s3_pseudo_dir", > "cardinality": "SINGLE", > "isIndexable": false, > *"isOptional": false,* > "isUnique": false, > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-3545) NullPointerException while trying to delete classification
[ https://issues.apache.org/jira/browse/ATLAS-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987628#comment-16987628 ] Bolke de Bruin commented on ATLAS-3545: --- Thanks [~sarath]. We will apply the patch and observe the behavior. Please note that this only removes the symptom from search/retrieval. On entity update an invalid propagating classification is still being evaluated. That problem is worse as it creates performance issues. The question thus also becomes, apart from the race condition, how to delete the invalid vertex? > NullPointerException while trying to delete classification > -- > > Key: ATLAS-3545 > URL: https://issues.apache.org/jira/browse/ATLAS-3545 > Project: Atlas > Issue Type: Bug >Affects Versions: trunk >Reporter: Bolke de Bruin >Assignee: Sarath Subramanian >Priority: Critical > Attachments: ATLAS-3545.001.patch, Screenshot 2019-11-30 at > 22.16.44.png, Screenshot 2019-11-30 at 22.28.00.png, Screenshot 2019-12-02 at > 21.43.29.png > > > We see an issue where there is a NullPointerException while trying to delete > a classification that is propagating. > It seems (stack trace is as of yet unavailable) that it is caused due to a > typeName being NULL in AtlasTypeRegistry.getType. The UI looks fishy too see > screenshots. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ATLAS-3547) Don’t evaluate propagating classifications for non relevant update
Bolke de Bruin created ATLAS-3547: - Summary: Don’t evaluate propagating classifications for non relevant update Key: ATLAS-3547 URL: https://issues.apache.org/jira/browse/ATLAS-3547 Project: Atlas Issue Type: Bug Affects Versions: trunk Reporter: Bolke de Bruin In case of an aws_s3_bucket with a propagating classification and with several thousands (200K+) objects we are observing that every addition of an object to this aws_s3_bucket triggers a re-evaluation of all entities with the propagating tag. This exponentially increases the time that it takes a message to be consumed if the bucket is growing in size. We send a create or update message for an aws_s3_object by Kafka which has all relevant information inside the message (ie. Aws_s3_bucket, aws_s3_pseudodir, aws_s3_object) as the producer system is unaware if the relevant metadata for the bucket and pseudodir are already available. This triggers an evaluation of the classifications and all propagating classifications. This seems over eager. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-3545) NullPointerException while trying to delete classification
[ https://issues.apache.org/jira/browse/ATLAS-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987118#comment-16987118 ] Bolke de Bruin commented on ATLAS-3545: --- [~ayubpathan] they were included above, but just to get it clear: 1. Add a propagating classification to an entity with a large container (e.g. aws_s3_bucket). 2. Add the same propagating classification again before the UI shows the classification is applied 3. Observe 2 classifications have been applied with the same name 4. Remove one of the propagating classifications 5. A Try to remove the other one(s), this should fail B Observe that the count in the UI basic search view is 1 or higher (e.g. it reads something like MANAGED (2) ) C Try to search for this classification with basic search, this should fail with an error message D Try to read classification properties in the classification view, this should fail E Observer that the propagating classification still is applied to derived entities (e.g. aws_s3_objects) Obviously we are seeing multiple issues here (race condition, null pointer). Our goal is first to remove the “orphan” propagating classification 4. Observer > NullPointerException while trying to delete classification > -- > > Key: ATLAS-3545 > URL: https://issues.apache.org/jira/browse/ATLAS-3545 > Project: Atlas > Issue Type: Bug >Affects Versions: trunk >Reporter: Bolke de Bruin >Priority: Critical > Attachments: Screenshot 2019-11-30 at 22.16.44.png, Screenshot > 2019-11-30 at 22.28.00.png, Screenshot 2019-12-02 at 21.43.29.png > > > We see an issue where there is a NullPointerException while trying to delete > a classification that is propagating. > It seems (stack trace is as of yet unavailable) that it is caused due to a > typeName being NULL in AtlasTypeRegistry.getType. The UI looks fishy too see > screenshots. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-3532) Test case failure in GremlinQueryComposerTest
[
https://issues.apache.org/jira/browse/ATLAS-3532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16986402#comment-16986402
]
Bolke de Bruin commented on ATLAS-3532:
---
Sorry got confused I guess. Master looks okay.
> Test case failure in GremlinQueryComposerTest
> -
>
> Key: ATLAS-3532
> URL: https://issues.apache.org/jira/browse/ATLAS-3532
> Project: Atlas
> Issue Type: Bug
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
> Attachments:
> ATLAS-3532-DSL-Composer-update-to-address-_-test-cas.patch,
> ATLAS-3532-Test-case-failure-in-GremlinQueryComposer.patch
>
>
> {code:java}
> Tests run: 39, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 15.464 sec
> <<< FAILURE! - in
> org.apache.atlas.query.GremlinQueryComposerTestwhereClauseTextContains(org.apache.atlas.query.GremlinQueryComposerTest)
> Time elapsed: 0.697 sec <<< FAILURE!java.lang.AssertionError: Table where
> owner like "Tab*" expected [g.V().has('__typeName',
> 'Table').has('Table.owner',
> org.janusgraph.core.attribute.Text.textContainsRegex("Tab.*")).dedup().limit(25).toList()]
> but found [g.V().has('__typeName', 'Table').has('Table.owner',
> org.janusgraph.core.attribute.Text.textRegex("Tab.*")).dedup().limit(25).toList()]
> at org.testng.Assert.fail(Assert.java:94)
> at org.testng.Assert.failNotEquals(Assert.java:496)
> at org.testng.Assert.assertEquals(Assert.java:125)
> at org.testng.Assert.assertEquals(Assert.java:178)
> at
> org.apache.atlas.query.GremlinQueryComposerTest.verify(GremlinQueryComposerTest.java:362)
> at
> org.apache.atlas.query.GremlinQueryComposerTest.verify(GremlinQueryComposerTest.java:371)
> at
> org.apache.atlas.query.GremlinQueryComposerTest.whereClauseTextContains(GremlinQueryComposerTest.java:170)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ATLAS-3545) NullPointerException while trying to delete classification
[ https://issues.apache.org/jira/browse/ATLAS-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16986354#comment-16986354 ] Bolke de Bruin commented on ATLAS-3545: --- [~madhan] latest screenshot gives a stack trace > NullPointerException while trying to delete classification > -- > > Key: ATLAS-3545 > URL: https://issues.apache.org/jira/browse/ATLAS-3545 > Project: Atlas > Issue Type: Bug >Affects Versions: trunk >Reporter: Bolke de Bruin >Priority: Critical > Attachments: Screenshot 2019-11-30 at 22.16.44.png, Screenshot > 2019-11-30 at 22.28.00.png, Screenshot 2019-12-02 at 21.43.29.png > > > We see an issue where there is a NullPointerException while trying to delete > a classification that is propagating. > It seems (stack trace is as of yet unavailable) that it is caused due to a > typeName being NULL in AtlasTypeRegistry.getType. The UI looks fishy too see > screenshots. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3545) NullPointerException while trying to delete classification
[ https://issues.apache.org/jira/browse/ATLAS-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin updated ATLAS-3545: -- Attachment: Screenshot 2019-12-02 at 21.43.29.png > NullPointerException while trying to delete classification > -- > > Key: ATLAS-3545 > URL: https://issues.apache.org/jira/browse/ATLAS-3545 > Project: Atlas > Issue Type: Bug >Affects Versions: trunk >Reporter: Bolke de Bruin >Priority: Critical > Attachments: Screenshot 2019-11-30 at 22.16.44.png, Screenshot > 2019-11-30 at 22.28.00.png, Screenshot 2019-12-02 at 21.43.29.png > > > We see an issue where there is a NullPointerException while trying to delete > a classification that is propagating. > It seems (stack trace is as of yet unavailable) that it is caused due to a > typeName being NULL in AtlasTypeRegistry.getType. The UI looks fishy too see > screenshots. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (ATLAS-3305) Unable to scale atlas kafka consumers
[
https://issues.apache.org/jira/browse/ATLAS-3305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16986051#comment-16986051
]
Bolke de Bruin edited comment on ATLAS-3305 at 12/2/19 1:34 PM:
I suggest using a different topic for this (ATLAS_PARTIONED) or having a
different config item (token.partioned) to ensure administrators know what to
do with this (making sure that the messages are atomic and/or partioned by key).
was (Author: bolke):
I suggest using a different topic for this (ATLAS_PARTIONED) to ensure
administrators know what to do with this (making sure that the messages are
atomic and/or partioned by key).
> Unable to scale atlas kafka consumers
> -
>
> Key: ATLAS-3305
> URL: https://issues.apache.org/jira/browse/ATLAS-3305
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core, atlas-intg
>Affects Versions: 1.1.0, 2.0.0
>Reporter: Adam Rempter
>Priority: Major
> Labels: performance
> Attachments: ATLAS-3305_multiple_kafka_consumers.patch,
> multiple_consumers_perf.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> We wanted to scale kafka consumers for atlas, as we are getting many lineage
> messages and processing them just with one consumer is not enough.
>
> There is parameter atlas.notification.hook.numthreads to scale consumers in
> NotificationHookConsumer.
> But the method:
>
> notificationInterface.createConsumers(NotificationType.HOOK, numThreads)
>
> is always returning one element list, which effectively always starts one
> consumer
> List> consumers =
> Collections.singletonList(kafkaConsumer);
>
> Log incorrectly says that nuber of consumers has been created:
> LOG.info("<== KafkaNotification.createConsumers(notificationType={},
> numConsumers={}, autoCommitEnabled={})", notificationType, numConsumers,
> autoCommitEnabled)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ATLAS-3305) Unable to scale atlas kafka consumers
[
https://issues.apache.org/jira/browse/ATLAS-3305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16986051#comment-16986051
]
Bolke de Bruin commented on ATLAS-3305:
---
I suggest using a different topic for this (ATLAS_PARTIONED) to ensure
administrators know what to do with this (making sure that the messages are
atomic and/or partioned by key).
> Unable to scale atlas kafka consumers
> -
>
> Key: ATLAS-3305
> URL: https://issues.apache.org/jira/browse/ATLAS-3305
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core, atlas-intg
>Affects Versions: 1.1.0, 2.0.0
>Reporter: Adam Rempter
>Priority: Major
> Labels: performance
> Attachments: ATLAS-3305_multiple_kafka_consumers.patch,
> multiple_consumers_perf.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> We wanted to scale kafka consumers for atlas, as we are getting many lineage
> messages and processing them just with one consumer is not enough.
>
> There is parameter atlas.notification.hook.numthreads to scale consumers in
> NotificationHookConsumer.
> But the method:
>
> notificationInterface.createConsumers(NotificationType.HOOK, numThreads)
>
> is always returning one element list, which effectively always starts one
> consumer
> List> consumers =
> Collections.singletonList(kafkaConsumer);
>
> Log incorrectly says that nuber of consumers has been created:
> LOG.info("<== KafkaNotification.createConsumers(notificationType={},
> numConsumers={}, autoCommitEnabled={})", notificationType, numConsumers,
> autoCommitEnabled)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ATLAS-3545) NullPointerException while trying to delete classification
[ https://issues.apache.org/jira/browse/ATLAS-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16985659#comment-16985659 ] Bolke de Bruin commented on ATLAS-3545: --- Stack trace is unavailable at the moment as Atlas swallows the exception. We will patch it to make sure the whole trace is available. We are observing 3 separate issues that are related. 1) we set a classification on a aws_s3_bucket with several thousands (200K+) of objects. As the UI wasn’t clear if the classification was applied so we tried it a couple of times. This resulted in the UI displaying it 3 (!) times. We then tried to remove one, which seemed to work but trying to remove the others fails. Moreover, the original classification still seems to be there as we face scaleability issues when updating the aws_s3_bucket in question: it then eagerly fetches all vertices that have a propagating tag which exponentially increases the time spend on this when s3_objects are added. This will be filed separately in a jira. > NullPointerException while trying to delete classification > -- > > Key: ATLAS-3545 > URL: https://issues.apache.org/jira/browse/ATLAS-3545 > Project: Atlas > Issue Type: Bug >Affects Versions: trunk >Reporter: Bolke de Bruin >Priority: Critical > Attachments: Screenshot 2019-11-30 at 22.16.44.png, Screenshot > 2019-11-30 at 22.28.00.png > > > We see an issue where there is a NullPointerException while trying to delete > a classification that is propagating. > It seems (stack trace is as of yet unavailable) that it is caused due to a > typeName being NULL in AtlasTypeRegistry.getType. The UI looks fishy too see > screenshots. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3545) NullPointerException while trying to delete classification
[ https://issues.apache.org/jira/browse/ATLAS-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin updated ATLAS-3545: -- Attachment: Screenshot 2019-11-30 at 22.16.44.png > NullPointerException while trying to delete classification > -- > > Key: ATLAS-3545 > URL: https://issues.apache.org/jira/browse/ATLAS-3545 > Project: Atlas > Issue Type: Bug >Affects Versions: trunk >Reporter: Bolke de Bruin >Priority: Critical > Attachments: Screenshot 2019-11-30 at 22.16.44.png, Screenshot > 2019-11-30 at 22.28.00.png > > > We see an issue where there is a NullPointerException while trying to delete > a classification that is propagating. > It seems (stack trace is as of yet unavailable) that it is caused due to a > typeName being NULL in AtlasTypeRegistry.getType. The UI looks fishy too see > screenshots. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3545) NullPointerException while trying to delete classification
[ https://issues.apache.org/jira/browse/ATLAS-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin updated ATLAS-3545: -- Attachment: Screenshot 2019-11-30 at 22.28.00.png > NullPointerException while trying to delete classification > -- > > Key: ATLAS-3545 > URL: https://issues.apache.org/jira/browse/ATLAS-3545 > Project: Atlas > Issue Type: Bug >Affects Versions: trunk >Reporter: Bolke de Bruin >Priority: Critical > Attachments: Screenshot 2019-11-30 at 22.16.44.png, Screenshot > 2019-11-30 at 22.28.00.png > > > We see an issue where there is a NullPointerException while trying to delete > a classification that is propagating. > It seems (stack trace is as of yet unavailable) that it is caused due to a > typeName being NULL in AtlasTypeRegistry.getType. The UI looks fishy too see > screenshots. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ATLAS-3545) NullPointerException while trying to delete classification
Bolke de Bruin created ATLAS-3545: - Summary: NullPointerException while trying to delete classification Key: ATLAS-3545 URL: https://issues.apache.org/jira/browse/ATLAS-3545 Project: Atlas Issue Type: Bug Affects Versions: trunk Reporter: Bolke de Bruin We see an issue where there is a NullPointerException while trying to delete a classification that is propagating. It seems (stack trace is as of yet unavailable) that it is caused due to a typeName being NULL in AtlasTypeRegistry.getType. The UI looks fishy too see screenshots. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3544) NullPointerException in AtlasGraphUtilsV2.getStateAsString
[ https://issues.apache.org/jira/browse/ATLAS-3544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin updated ATLAS-3544: -- Attachment: Screenshot 2019-11-29 at 19.06.47.png > NullPointerException in AtlasGraphUtilsV2.getStateAsString > -- > > Key: ATLAS-3544 > URL: https://issues.apache.org/jira/browse/ATLAS-3544 > Project: Atlas > Issue Type: Bug >Reporter: Bolke de Bruin >Assignee: Sidharth Kumar Mishra >Priority: Major > Attachments: Screenshot 2019-11-29 at 19.06.47.png > > > ATLAS-3477 creates a NullPointerException in getStateAsString of > AtlasGraphUtilsV2 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ATLAS-3544) NullPointerException in AtlasGraphUtilsV2.getStateAsString
Bolke de Bruin created ATLAS-3544: - Summary: NullPointerException in AtlasGraphUtilsV2.getStateAsString Key: ATLAS-3544 URL: https://issues.apache.org/jira/browse/ATLAS-3544 Project: Atlas Issue Type: Bug Reporter: Bolke de Bruin Assignee: Sidharth Kumar Mishra ATLAS-3477 creates a NullPointerException in getStateAsString of AtlasGraphUtilsV2 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (ATLAS-3254) Atlas entity with large array of refs causes performance issues for lineage
[ https://issues.apache.org/jira/browse/ATLAS-3254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16984631#comment-16984631 ] Bolke de Bruin edited comment on ATLAS-3254 at 11/28/19 8:24 PM: - [~madhan] we see now serious issues with entity deletion. The attached screenshot shows 30min processing time. So this not only has an effect on the UI. This is on master btw. was (Author: bolke): [~madhan] we see now serious issues with entity deletion. The attached screenshot shows 30min processing time. So this not only has an effect on the UI. > Atlas entity with large array of refs causes performance issues for lineage > --- > > Key: ATLAS-3254 > URL: https://issues.apache.org/jira/browse/ATLAS-3254 > Project: Atlas > Issue Type: Bug > Components: atlas-core, atlas-webui >Affects Versions: 1.0.0, 2.0.0 >Reporter: Adam Rempter >Priority: Major > Labels: performance > Attachments: Screenshot 2019-11-28 at 21.18.44.png, > example_create_entities.json, rest_entity_get_pseudodir.json > > > We use “aws_s3_pseudo_dir” type from 3020-aws_s3_typedefs.json model. > It has following property: > "name": "s3Objects", > "typeName": "array" > > Now in AWS buckets you can have thousands of objects. This causes that > s3Objects array grows quite quickly, causing aws_s3_pseudo_dir entity Json to > rich easly few MBs. > > Then we start seeing problems like: > * UI is dying on displaying entity properties or lineage > * Error in logs: audit record too long: entityType=aws_s3_pseudo_dir, > guid=24398271-6ba0-4db5-adfa-38e432dc55ce, size=1053931; maxSize=1048576. > entity attribute values not stored in audit (EntityAuditListenerV2:234) > * Some errors with write to HBase (java.lang.IllegalArgumentException: > KeyValue size too large, as workaround we set hbase.client.keyvalue.maxsize > param to 0) > * kafka consumer errors (we can of course set some parameters on consumer, > but I think it is just workaround) > … > Exception in NotificationHookConsumer (NotificationHookConsumer:332) > org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be > completed since the group has already rebalanced and assigned the partitions > to another member. This means that the time between subsequen > t calls to poll() was longer than the configured max.poll.interval.ms, which > typically implies that the poll loop is spending too much time message > processing. You can address this either by increasing the sessio > n timeout or by reducing the maximum size of batches returned in poll() with > max.poll.records. > … > Specifying pseudo_dir is required for s3objects: > name": "pseudoDirectory", > "typeName": "aws_s3_pseudo_dir", > "cardinality": "SINGLE", > "isIndexable": false, > *"isOptional": false,* > "isUnique": false, > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-3254) Atlas entity with large array of refs causes performance issues for lineage
[ https://issues.apache.org/jira/browse/ATLAS-3254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16984631#comment-16984631 ] Bolke de Bruin commented on ATLAS-3254: --- [~madhan] we see now serious issues with entity deletion. The attached screenshot shows 30min processing time. So this not only has an effect on the UI. > Atlas entity with large array of refs causes performance issues for lineage > --- > > Key: ATLAS-3254 > URL: https://issues.apache.org/jira/browse/ATLAS-3254 > Project: Atlas > Issue Type: Bug > Components: atlas-core, atlas-webui >Affects Versions: 1.0.0, 2.0.0 >Reporter: Adam Rempter >Priority: Major > Labels: performance > Attachments: Screenshot 2019-11-28 at 21.18.44.png, > example_create_entities.json, rest_entity_get_pseudodir.json > > > We use “aws_s3_pseudo_dir” type from 3020-aws_s3_typedefs.json model. > It has following property: > "name": "s3Objects", > "typeName": "array" > > Now in AWS buckets you can have thousands of objects. This causes that > s3Objects array grows quite quickly, causing aws_s3_pseudo_dir entity Json to > rich easly few MBs. > > Then we start seeing problems like: > * UI is dying on displaying entity properties or lineage > * Error in logs: audit record too long: entityType=aws_s3_pseudo_dir, > guid=24398271-6ba0-4db5-adfa-38e432dc55ce, size=1053931; maxSize=1048576. > entity attribute values not stored in audit (EntityAuditListenerV2:234) > * Some errors with write to HBase (java.lang.IllegalArgumentException: > KeyValue size too large, as workaround we set hbase.client.keyvalue.maxsize > param to 0) > * kafka consumer errors (we can of course set some parameters on consumer, > but I think it is just workaround) > … > Exception in NotificationHookConsumer (NotificationHookConsumer:332) > org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be > completed since the group has already rebalanced and assigned the partitions > to another member. This means that the time between subsequen > t calls to poll() was longer than the configured max.poll.interval.ms, which > typically implies that the poll loop is spending too much time message > processing. You can address this either by increasing the sessio > n timeout or by reducing the maximum size of batches returned in poll() with > max.poll.records. > … > Specifying pseudo_dir is required for s3objects: > name": "pseudoDirectory", > "typeName": "aws_s3_pseudo_dir", > "cardinality": "SINGLE", > "isIndexable": false, > *"isOptional": false,* > "isUnique": false, > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (ATLAS-3532) Test case failure in GremlinQueryComposerTest
[
https://issues.apache.org/jira/browse/ATLAS-3532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bolke de Bruin reopened ATLAS-3532:
---
> Test case failure in GremlinQueryComposerTest
> -
>
> Key: ATLAS-3532
> URL: https://issues.apache.org/jira/browse/ATLAS-3532
> Project: Atlas
> Issue Type: Bug
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
> Attachments:
> ATLAS-3532-Test-case-failure-in-GremlinQueryComposer.patch
>
>
> {code:java}
> Tests run: 39, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 15.464 sec
> <<< FAILURE! - in
> org.apache.atlas.query.GremlinQueryComposerTestwhereClauseTextContains(org.apache.atlas.query.GremlinQueryComposerTest)
> Time elapsed: 0.697 sec <<< FAILURE!java.lang.AssertionError: Table where
> owner like "Tab*" expected [g.V().has('__typeName',
> 'Table').has('Table.owner',
> org.janusgraph.core.attribute.Text.textContainsRegex("Tab.*")).dedup().limit(25).toList()]
> but found [g.V().has('__typeName', 'Table').has('Table.owner',
> org.janusgraph.core.attribute.Text.textRegex("Tab.*")).dedup().limit(25).toList()]
> at org.testng.Assert.fail(Assert.java:94)
> at org.testng.Assert.failNotEquals(Assert.java:496)
> at org.testng.Assert.assertEquals(Assert.java:125)
> at org.testng.Assert.assertEquals(Assert.java:178)
> at
> org.apache.atlas.query.GremlinQueryComposerTest.verify(GremlinQueryComposerTest.java:362)
> at
> org.apache.atlas.query.GremlinQueryComposerTest.verify(GremlinQueryComposerTest.java:371)
> at
> org.apache.atlas.query.GremlinQueryComposerTest.whereClauseTextContains(GremlinQueryComposerTest.java:170)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ATLAS-3532) Test case failure in GremlinQueryComposerTest
[
https://issues.apache.org/jira/browse/ATLAS-3532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16981747#comment-16981747
]
Bolke de Bruin commented on ATLAS-3532:
---
This does not seem to be the right chance as there is no test for
textContainsRegex anymore now afaik. Does the query contain the special '_' so
it warrants switching to textRegex which is *slow*?
> Test case failure in GremlinQueryComposerTest
> -
>
> Key: ATLAS-3532
> URL: https://issues.apache.org/jira/browse/ATLAS-3532
> Project: Atlas
> Issue Type: Bug
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
> Attachments:
> ATLAS-3532-Test-case-failure-in-GremlinQueryComposer.patch
>
>
> {code:java}
> Tests run: 39, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 15.464 sec
> <<< FAILURE! - in
> org.apache.atlas.query.GremlinQueryComposerTestwhereClauseTextContains(org.apache.atlas.query.GremlinQueryComposerTest)
> Time elapsed: 0.697 sec <<< FAILURE!java.lang.AssertionError: Table where
> owner like "Tab*" expected [g.V().has('__typeName',
> 'Table').has('Table.owner',
> org.janusgraph.core.attribute.Text.textContainsRegex("Tab.*")).dedup().limit(25).toList()]
> but found [g.V().has('__typeName', 'Table').has('Table.owner',
> org.janusgraph.core.attribute.Text.textRegex("Tab.*")).dedup().limit(25).toList()]
> at org.testng.Assert.fail(Assert.java:94)
> at org.testng.Assert.failNotEquals(Assert.java:496)
> at org.testng.Assert.assertEquals(Assert.java:125)
> at org.testng.Assert.assertEquals(Assert.java:178)
> at
> org.apache.atlas.query.GremlinQueryComposerTest.verify(GremlinQueryComposerTest.java:362)
> at
> org.apache.atlas.query.GremlinQueryComposerTest.verify(GremlinQueryComposerTest.java:371)
> at
> org.apache.atlas.query.GremlinQueryComposerTest.whereClauseTextContains(GremlinQueryComposerTest.java:170)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ATLAS-3420) New logo for Apache Atlas
[ https://issues.apache.org/jira/browse/ATLAS-3420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16936433#comment-16936433 ] Bolke de Bruin commented on ATLAS-3420: --- [~sarath] if you mean by that that i should upload a vector image of the logo, i can ask my team to design one. It will look different of course. I was (wrongfully?) assuming that the basis for these images was a vector image. let me know. > New logo for Apache Atlas > - > > Key: ATLAS-3420 > URL: https://issues.apache.org/jira/browse/ATLAS-3420 > Project: Atlas > Issue Type: Improvement >Reporter: Keval Bhatt >Assignee: Keval Bhatt >Priority: Major > Attachments: Atlas_Favicon.ico, Atlas_Logo.png, > Atlas_Logo_monogram.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-3409) Remove the duplicate code in Solr6Index.
[ https://issues.apache.org/jira/browse/ATLAS-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928633#comment-16928633 ] Bolke de Bruin commented on ATLAS-3409: --- This is a duplicate of atlas-3397 > Remove the duplicate code in Solr6Index. > > > Key: ATLAS-3409 > URL: https://issues.apache.org/jira/browse/ATLAS-3409 > Project: Atlas > Issue Type: Improvement >Reporter: Sridhar >Assignee: Sridhar >Priority: Major > > Historically, we made a copy of SolrIndex from Janus graph and referred to it > as Solr6Index. Over time, SolrIndex in janus graph has mode enough progress > w.r.t Kerberos support etc. So, we should remove the duplicate code > Solr6Index. > This ticket should be done after ATLAS-3378 is completed. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (ATLAS-3378) Update JanusGraph to 0.4.0
[ https://issues.apache.org/jira/browse/ATLAS-3378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926837#comment-16926837 ] Bolke de Bruin commented on ATLAS-3378: --- see also: ATLAS-3399 > Update JanusGraph to 0.4.0 > -- > > Key: ATLAS-3378 > URL: https://issues.apache.org/jira/browse/ATLAS-3378 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.0.0 >Reporter: Bolke de Bruin >Assignee: Sarath Subramanian >Priority: Major > Fix For: 2.1.0 > > > This release supports hbase 2 out of the box -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Assigned] (ATLAS-3399) Add order by (sort) to basic search
[ https://issues.apache.org/jira/browse/ATLAS-3399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin reassigned ATLAS-3399: - Assignee: Bolke de Bruin > Add order by (sort) to basic search > --- > > Key: ATLAS-3399 > URL: https://issues.apache.org/jira/browse/ATLAS-3399 > Project: Atlas > Issue Type: New Feature > Components: atlas-core >Affects Versions: 2.0.0 >Reporter: Bolke de Bruin >Assignee: Bolke de Bruin >Priority: Major > Fix For: trunk > > > JanusGraph 0.4.0 has added support for indexed "Order By" using the search > backend. This would allow sorting on names, popularity scores etc en improve > the capabilities for data discovery -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (ATLAS-3399) Add order by (sort) to basic search
Bolke de Bruin created ATLAS-3399: - Summary: Add order by (sort) to basic search Key: ATLAS-3399 URL: https://issues.apache.org/jira/browse/ATLAS-3399 Project: Atlas Issue Type: New Feature Components: atlas-core Affects Versions: 2.0.0 Reporter: Bolke de Bruin Fix For: trunk JanusGraph 0.4.0 has added support for indexed "Order By" using the search backend. This would allow sorting on names, popularity scores etc en improve the capabilities for data discovery -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Assigned] (ATLAS-3393) Fix inconsistency with attribute createtime in aws_s3_bucket
[ https://issues.apache.org/jira/browse/ATLAS-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin reassigned ATLAS-3393: - Assignee: Bolke de Bruin > Fix inconsistency with attribute createtime in aws_s3_bucket > > > Key: ATLAS-3393 > URL: https://issues.apache.org/jira/browse/ATLAS-3393 > Project: Atlas > Issue Type: Bug >Affects Versions: 1.2.0, 2.0.0 >Reporter: Bolke de Bruin >Assignee: Bolke de Bruin >Priority: Critical > > attributes are in camelcase. in aws_s3_bukcket createtime isnt > > (fix available in linked issue/pr) > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (ATLAS-3393) Fix inconsistency with attribute createtime in aws_s3_bucket
[ https://issues.apache.org/jira/browse/ATLAS-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16922253#comment-16922253 ] Bolke de Bruin commented on ATLAS-3393: --- [~madhan.neethiraj] got your point. Thanks for being elaborate. > Fix inconsistency with attribute createtime in aws_s3_bucket > > > Key: ATLAS-3393 > URL: https://issues.apache.org/jira/browse/ATLAS-3393 > Project: Atlas > Issue Type: Bug >Affects Versions: 1.2.0, 2.0.0 >Reporter: Bolke de Bruin >Priority: Critical > > attributes are in camelcase. in aws_s3_bukcket createtime isnt > > (fix available in linked issue/pr) > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (ATLAS-3398) Duplicates for unique attributes
[ https://issues.apache.org/jira/browse/ATLAS-3398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin updated ATLAS-3398: -- Description: We are seeing issues with entities being added to Atlas with duplicate "qualifiedName". The guids differ and other attributes do also differ. Below a graph that shows the distribution over time for duplicates. We have difficulty determining which one is the right one (as they are different) in order to clean them up. We are also not the only ones encountering this as you can in the linked issue. We have noticed that Atlas does not use the [locking|https://docs.janusgraph.org/master/advanced-topics/eventual-consistency/] mechanism of Janus to prevent this: !zrzut_ekranu_2019-09-03_o_10.28.50.png! was: We are seeing issues with entities being added to Atlas with duplicate "qualifiedName". The guids differ and other attributes do also differ. Below a graph that shows the distribution over time for duplicates. We have difficulty determining which one is the right one (as they are different) in order to clean them up. We are also not the only ones encountering this as you can in the linked issue. !zrzut_ekranu_2019-09-03_o_10.28.50.png! > Duplicates for unique attributes > - > > Key: ATLAS-3398 > URL: https://issues.apache.org/jira/browse/ATLAS-3398 > Project: Atlas > Issue Type: Bug >Affects Versions: 2.0.0, trunk >Reporter: Bolke de Bruin >Priority: Blocker > Attachments: zrzut_ekranu_2019-09-03_o_10.28.50.png > > > We are seeing issues with entities being added to Atlas with duplicate > "qualifiedName". The guids differ and other attributes do also differ. Below > a graph that shows the distribution over time for duplicates. We have > difficulty determining which one is the right one (as they are different) in > order to clean them up. > We are also not the only ones encountering this as you can in the linked > issue. > We have noticed that Atlas does not use the > [locking|https://docs.janusgraph.org/master/advanced-topics/eventual-consistency/] > mechanism of Janus to prevent this: > > !zrzut_ekranu_2019-09-03_o_10.28.50.png! -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (ATLAS-3398) Duplicates for unique attributes
[ https://issues.apache.org/jira/browse/ATLAS-3398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin updated ATLAS-3398: -- Component/s: atlas-core > Duplicates for unique attributes > - > > Key: ATLAS-3398 > URL: https://issues.apache.org/jira/browse/ATLAS-3398 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.0.0, trunk >Reporter: Bolke de Bruin >Priority: Blocker > Attachments: zrzut_ekranu_2019-09-03_o_10.28.50.png > > > We are seeing issues with entities being added to Atlas with duplicate > "qualifiedName". The guids differ and other attributes do also differ. Below > a graph that shows the distribution over time for duplicates. We have > difficulty determining which one is the right one (as they are different) in > order to clean them up. > We are also not the only ones encountering this as you can in the linked > issue. > We have noticed that Atlas does not use the > [locking|https://docs.janusgraph.org/master/advanced-topics/eventual-consistency/] > mechanism of Janus to prevent this: > > !zrzut_ekranu_2019-09-03_o_10.28.50.png! -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (ATLAS-3398) Duplicates for unique attributes
[ https://issues.apache.org/jira/browse/ATLAS-3398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin updated ATLAS-3398: -- Description: We are seeing issues with entities being added to Atlas with duplicate "qualifiedName". The guids differ and other attributes do also differ. Below a graph that shows the distribution over time for duplicates. We have difficulty determining which one is the right one (as they are different) in order to clean them up. We are also not the only ones encountering this as you can in the linked issue. !zrzut_ekranu_2019-09-03_o_10.28.50.png! was: We are seeing issues with entities being added to Atlas with duplicate "qualifiedName". The guids differ and other attributes do also differ. Below a graph that shows the distribution over time for duplicates. We have difficulty determining which one is the right one (as they are different) in order to clean them up. We are also not the only ones encountering this as you can in the linked issue. !https://files.slack.com/files-pri/T9LB9QRT8-FMM67SSQK/zrzut_ekranu_2019-09-03_o_10.28.50.png! > Duplicates for unique attributes > - > > Key: ATLAS-3398 > URL: https://issues.apache.org/jira/browse/ATLAS-3398 > Project: Atlas > Issue Type: Bug >Affects Versions: 2.0.0, trunk >Reporter: Bolke de Bruin >Priority: Blocker > Attachments: zrzut_ekranu_2019-09-03_o_10.28.50.png > > > We are seeing issues with entities being added to Atlas with duplicate > "qualifiedName". The guids differ and other attributes do also differ. Below > a graph that shows the distribution over time for duplicates. We have > difficulty determining which one is the right one (as they are different) in > order to clean them up. > We are also not the only ones encountering this as you can in the linked > issue. > > !zrzut_ekranu_2019-09-03_o_10.28.50.png! -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (ATLAS-3398) Duplicates for unique attributes
[ https://issues.apache.org/jira/browse/ATLAS-3398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin updated ATLAS-3398: -- Attachment: zrzut_ekranu_2019-09-03_o_10.28.50.png > Duplicates for unique attributes > - > > Key: ATLAS-3398 > URL: https://issues.apache.org/jira/browse/ATLAS-3398 > Project: Atlas > Issue Type: Bug >Affects Versions: 2.0.0, trunk >Reporter: Bolke de Bruin >Priority: Blocker > Attachments: zrzut_ekranu_2019-09-03_o_10.28.50.png > > > We are seeing issues with entities being added to Atlas with duplicate > "qualifiedName". The guids differ and other attributes do also differ. Below > a graph that shows the distribution over time for duplicates. We have > difficulty determining which one is the right one (as they are different) in > order to clean them up. > We are also not the only ones encountering this as you can in the linked > issue. > > !https://files.slack.com/files-pri/T9LB9QRT8-FMM67SSQK/zrzut_ekranu_2019-09-03_o_10.28.50.png! -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (ATLAS-3398) Duplicates for unique attributes
Bolke de Bruin created ATLAS-3398: - Summary: Duplicates for unique attributes Key: ATLAS-3398 URL: https://issues.apache.org/jira/browse/ATLAS-3398 Project: Atlas Issue Type: Bug Affects Versions: 2.0.0, trunk Reporter: Bolke de Bruin We are seeing issues with entities being added to Atlas with duplicate "qualifiedName". The guids differ and other attributes do also differ. Below a graph that shows the distribution over time for duplicates. We have difficulty determining which one is the right one (as they are different) in order to clean them up. We are also not the only ones encountering this as you can in the linked issue. !https://files.slack.com/files-pri/T9LB9QRT8-FMM67SSQK/zrzut_ekranu_2019-09-03_o_10.28.50.png! -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (ATLAS-3393) Fix inconsistency with attribute createtime in aws_s3_bucket
[ https://issues.apache.org/jira/browse/ATLAS-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16921622#comment-16921622 ] Bolke de Bruin commented on ATLAS-3393: --- ping [~madhan.neethiraj] > Fix inconsistency with attribute createtime in aws_s3_bucket > > > Key: ATLAS-3393 > URL: https://issues.apache.org/jira/browse/ATLAS-3393 > Project: Atlas > Issue Type: Bug >Affects Versions: 1.2.0, 2.0.0 >Reporter: Bolke de Bruin >Priority: Critical > > attributes are in camelcase. in aws_s3_bukcket createtime isnt > > (fix available in linked issue/pr) > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (ATLAS-3397) Remove Solr6Index and use upstream version
[ https://issues.apache.org/jira/browse/ATLAS-3397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin updated ATLAS-3397: -- Summary: Remove Solr6Index and use upstream version (was: Push Solr6Index improvements upstream and use Janus' version) > Remove Solr6Index and use upstream version > -- > > Key: ATLAS-3397 > URL: https://issues.apache.org/jira/browse/ATLAS-3397 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.0.0 >Reporter: Bolke de Bruin >Priority: Critical > > Solr6Index has changes to support Kerberos and multiple Zookeeper clients. > There are several issues with Atlas' version: > * LICENSE for this file is incorrect as it is not licensed to the ASF. The > file says it is copied from Janus. > * It is outdated and is not benefitting from upstream improvements (like > indexed OrderBy of Janus 0.4) > * Kerberos has been integrated into Janus' version > * multiple Zookeeper clients have been implemented (although still using > ZOOKEEPER_URL, but it supports multiple entries: > [https://github.com/JanusGraph/janusgraph/blob/master/janusgraph-solr/src/main/java/org/janusgraph/diskstorage/solr/SolrIndex.java#L176] > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (ATLAS-3397) Push Solr6Index improvements upstream and use Janus' version
[ https://issues.apache.org/jira/browse/ATLAS-3397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin updated ATLAS-3397: -- Description: Solr6Index has changes to support Kerberos and multiple Zookeeper clients. There are several issues with Atlas' version: * LICENSE for this file is incorrect as it is not licensed to the ASF. The file says it is copied from Janus. * It is outdated and is not benefitting from upstream improvements (like indexed OrderBy of Janus 0.4) * Kerberos has been integrated into Janus' version * multiple Zookeeper clients have been implemented (although still using ZOOKEEPER_URL, but it supports multiple entries: [https://github.com/JanusGraph/janusgraph/blob/master/janusgraph-solr/src/main/java/org/janusgraph/diskstorage/solr/SolrIndex.java#L176] was: Solr6Index has changes to support Kerberos and multiple Zookeeper clients. There are several issues with Atlas' version: * LICENSE for this file is incorrect as it is not licensed to the ASF. * It is outdated and is not benefitting from upstream improvements (like indexed OrderBy of Janus 0.4) * Kerberos has been integrated into Janus' version * multiple Zookeeper clients have been implemented (although still using ZOOKEEPER_URL, but it supports multiple entries: [https://github.com/JanusGraph/janusgraph/blob/master/janusgraph-solr/src/main/java/org/janusgraph/diskstorage/solr/SolrIndex.java#L176] > Push Solr6Index improvements upstream and use Janus' version > > > Key: ATLAS-3397 > URL: https://issues.apache.org/jira/browse/ATLAS-3397 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.0.0 >Reporter: Bolke de Bruin >Priority: Critical > > Solr6Index has changes to support Kerberos and multiple Zookeeper clients. > There are several issues with Atlas' version: > * LICENSE for this file is incorrect as it is not licensed to the ASF. The > file says it is copied from Janus. > * It is outdated and is not benefitting from upstream improvements (like > indexed OrderBy of Janus 0.4) > * Kerberos has been integrated into Janus' version > * multiple Zookeeper clients have been implemented (although still using > ZOOKEEPER_URL, but it supports multiple entries: > [https://github.com/JanusGraph/janusgraph/blob/master/janusgraph-solr/src/main/java/org/janusgraph/diskstorage/solr/SolrIndex.java#L176] > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Comment Edited] (ATLAS-3393) Fix inconsistency with attribute createtime in aws_s3_bucket
[ https://issues.apache.org/jira/browse/ATLAS-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918344#comment-16918344 ] Bolke de Bruin edited comment on ATLAS-3393 at 8/29/19 7:12 AM: I understand that backward compatibility is important. However, we encountered this bug due to integration issues with our tooling expecting "createTime" as it is used everywhere else[1]. I do think it will create more problems down the road if you leave this inconstency in. "createTime" is also used hardcoded in [2]. If it is considered to be an "internal" attribute what happens if you mix the two? Is there a way to alias one to the other? So the trade off is that this inconsistency needs to be tested for in donwstream tooling and its not described in the documentation. So the question becomes how much tooling is already integrating with "createtime" and how much future tooling will assume "createTime". I think the latter will be bigger. [1] [https://github.com/apache/atlas/search?q=createTime_q=createTime] [2] [https://github.com/apache/atlas/blob/7e4788ed00c6a0ef4e8c89f0486b8073a56409e4/intg/src/main/java/org/apache/atlas/model/instance/AtlasRelationship.java#L63] was (Author: bolke): I understand that backward compatibility is important. However, we encountered this bug due to integration issues with our tooling expecting "createTime" as it is used everywhere else[1]. I do think it will create more problem down the road if you leave this inconstency in. "createTime" is also used hardcoded in [2]. If it is considered to be an "internal" attribute what happens if you mix the two? Is there a way to alias one to the other? So the trade off is that this inconsistency needs to be tested for in donwstream tooling and its not described in the documentation. So the question becomes how much tooling is already integrating with "createtime" and how much future tooling will assume "createTime". I think the latter will be bigger. [1] [https://github.com/apache/atlas/search?q=createTime_q=createTime] [2] [https://github.com/apache/atlas/blob/7e4788ed00c6a0ef4e8c89f0486b8073a56409e4/intg/src/main/java/org/apache/atlas/model/instance/AtlasRelationship.java#L63] > Fix inconsistency with attribute createtime in aws_s3_bucket > > > Key: ATLAS-3393 > URL: https://issues.apache.org/jira/browse/ATLAS-3393 > Project: Atlas > Issue Type: Bug >Affects Versions: 1.2.0, 2.0.0 >Reporter: Bolke de Bruin >Priority: Critical > > attributes are in camelcase. in aws_s3_bukcket createtime isnt > > (fix available in linked issue/pr) > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Comment Edited] (ATLAS-3393) Fix inconsistency with attribute createtime in aws_s3_bucket
[ https://issues.apache.org/jira/browse/ATLAS-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918344#comment-16918344 ] Bolke de Bruin edited comment on ATLAS-3393 at 8/29/19 7:12 AM: I understand that backward compatibility is important. However, we encountered this bug due to integration issues with our tooling expecting "createTime" as it is used everywhere else[1]. I do think it will create more problems down the road if you leave this inconstency in. "createTime" is also used hardcoded in [2]. If it is considered to be an "internal" attribute what happens if you mix the two? Is there a way to alias one to the other? So the trade off is that this inconsistency needs to be tested for in downstream tooling and it is not described in the documentation. So the question becomes how much tooling is already integrating with "createtime" and how much future tooling will assume "createTime". I think the latter will be bigger. [1] [https://github.com/apache/atlas/search?q=createTime_q=createTime] [2] [https://github.com/apache/atlas/blob/7e4788ed00c6a0ef4e8c89f0486b8073a56409e4/intg/src/main/java/org/apache/atlas/model/instance/AtlasRelationship.java#L63] was (Author: bolke): I understand that backward compatibility is important. However, we encountered this bug due to integration issues with our tooling expecting "createTime" as it is used everywhere else[1]. I do think it will create more problems down the road if you leave this inconstency in. "createTime" is also used hardcoded in [2]. If it is considered to be an "internal" attribute what happens if you mix the two? Is there a way to alias one to the other? So the trade off is that this inconsistency needs to be tested for in donwstream tooling and it is not described in the documentation. So the question becomes how much tooling is already integrating with "createtime" and how much future tooling will assume "createTime". I think the latter will be bigger. [1] [https://github.com/apache/atlas/search?q=createTime_q=createTime] [2] [https://github.com/apache/atlas/blob/7e4788ed00c6a0ef4e8c89f0486b8073a56409e4/intg/src/main/java/org/apache/atlas/model/instance/AtlasRelationship.java#L63] > Fix inconsistency with attribute createtime in aws_s3_bucket > > > Key: ATLAS-3393 > URL: https://issues.apache.org/jira/browse/ATLAS-3393 > Project: Atlas > Issue Type: Bug >Affects Versions: 1.2.0, 2.0.0 >Reporter: Bolke de Bruin >Priority: Critical > > attributes are in camelcase. in aws_s3_bukcket createtime isnt > > (fix available in linked issue/pr) > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Comment Edited] (ATLAS-3393) Fix inconsistency with attribute createtime in aws_s3_bucket
[ https://issues.apache.org/jira/browse/ATLAS-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918344#comment-16918344 ] Bolke de Bruin edited comment on ATLAS-3393 at 8/29/19 7:12 AM: I understand that backward compatibility is important. However, we encountered this bug due to integration issues with our tooling expecting "createTime" as it is used everywhere else[1]. I do think it will create more problems down the road if you leave this inconstency in. "createTime" is also used hardcoded in [2]. If it is considered to be an "internal" attribute what happens if you mix the two? Is there a way to alias one to the other? So the trade off is that this inconsistency needs to be tested for in donwstream tooling and it is not described in the documentation. So the question becomes how much tooling is already integrating with "createtime" and how much future tooling will assume "createTime". I think the latter will be bigger. [1] [https://github.com/apache/atlas/search?q=createTime_q=createTime] [2] [https://github.com/apache/atlas/blob/7e4788ed00c6a0ef4e8c89f0486b8073a56409e4/intg/src/main/java/org/apache/atlas/model/instance/AtlasRelationship.java#L63] was (Author: bolke): I understand that backward compatibility is important. However, we encountered this bug due to integration issues with our tooling expecting "createTime" as it is used everywhere else[1]. I do think it will create more problems down the road if you leave this inconstency in. "createTime" is also used hardcoded in [2]. If it is considered to be an "internal" attribute what happens if you mix the two? Is there a way to alias one to the other? So the trade off is that this inconsistency needs to be tested for in donwstream tooling and its not described in the documentation. So the question becomes how much tooling is already integrating with "createtime" and how much future tooling will assume "createTime". I think the latter will be bigger. [1] [https://github.com/apache/atlas/search?q=createTime_q=createTime] [2] [https://github.com/apache/atlas/blob/7e4788ed00c6a0ef4e8c89f0486b8073a56409e4/intg/src/main/java/org/apache/atlas/model/instance/AtlasRelationship.java#L63] > Fix inconsistency with attribute createtime in aws_s3_bucket > > > Key: ATLAS-3393 > URL: https://issues.apache.org/jira/browse/ATLAS-3393 > Project: Atlas > Issue Type: Bug >Affects Versions: 1.2.0, 2.0.0 >Reporter: Bolke de Bruin >Priority: Critical > > attributes are in camelcase. in aws_s3_bukcket createtime isnt > > (fix available in linked issue/pr) > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Comment Edited] (ATLAS-3393) Fix inconsistency with attribute createtime in aws_s3_bucket
[ https://issues.apache.org/jira/browse/ATLAS-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918344#comment-16918344 ] Bolke de Bruin edited comment on ATLAS-3393 at 8/29/19 7:11 AM: I understand that backward compatibility is important. However, we encountered this bug due to integration issues with our tooling expecting "createTime" as it is used everywhere else[1]. I do think it will create more problem down the road if you leave this inconstency in. "createTime" is also used hardcoded in [2]. If it is considered to be an "internal" attribute what happens if you mix the two? Is there a way to alias one to the other? So the trade off is that this inconsistency needs to be tested for in donwstream tooling and its not described in the documentation. So the question becomes how much tooling is already integrating with "createtime" and how much future tooling will assume "createTime". I think the latter will be bigger. [1] [https://github.com/apache/atlas/search?q=createTime_q=createTime] [2] [https://github.com/apache/atlas/blob/7e4788ed00c6a0ef4e8c89f0486b8073a56409e4/intg/src/main/java/org/apache/atlas/model/instance/AtlasRelationship.java#L63] was (Author: bolke): I understand that backward compatibility is important. However, we encountered this bug due to integration issues with our tooling expecting "createTime" as it is used everywhere else[1]. I do think it will create more problem down the road if you leave this inconstency in. "createTime" is also used hardcoded in [2]. If it is considered to be an "internal" attribute what happens if you mix the two? Is there a way to alias one to the other? [1] https://github.com/apache/atlas/search?q=createTime_q=createTime [2] https://github.com/apache/atlas/blob/7e4788ed00c6a0ef4e8c89f0486b8073a56409e4/intg/src/main/java/org/apache/atlas/model/instance/AtlasRelationship.java#L63 > Fix inconsistency with attribute createtime in aws_s3_bucket > > > Key: ATLAS-3393 > URL: https://issues.apache.org/jira/browse/ATLAS-3393 > Project: Atlas > Issue Type: Bug >Affects Versions: 1.2.0, 2.0.0 >Reporter: Bolke de Bruin >Priority: Critical > > attributes are in camelcase. in aws_s3_bukcket createtime isnt > > (fix available in linked issue/pr) > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (ATLAS-3393) Fix inconsistency with attribute createtime in aws_s3_bucket
[ https://issues.apache.org/jira/browse/ATLAS-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin updated ATLAS-3393: -- Affects Version/s: 1.2.0 > Fix inconsistency with attribute createtime in aws_s3_bucket > > > Key: ATLAS-3393 > URL: https://issues.apache.org/jira/browse/ATLAS-3393 > Project: Atlas > Issue Type: Bug >Affects Versions: 1.2.0, 2.0.0 >Reporter: Bolke de Bruin >Priority: Critical > > attributes are in camelcase. in aws_s3_bukcket createtime isnt > > (fix available in linked issue/pr) > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (ATLAS-3393) Fix inconsistency with attribute createtime in aws_s3_bucket
[ https://issues.apache.org/jira/browse/ATLAS-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918344#comment-16918344 ] Bolke de Bruin commented on ATLAS-3393: --- I understand that backward compatibility is important. However, we encountered this bug due to integration issues with our tooling expecting "createTime" as it is used everywhere else[1]. I do think it will create more problem down the road if you leave this inconstency in. "createTime" is also used hardcoded in [2]. If it is considered to be an "internal" attribute what happens if you mix the two? Is there a way to alias one to the other? [1] https://github.com/apache/atlas/search?q=createTime_q=createTime [2] https://github.com/apache/atlas/blob/7e4788ed00c6a0ef4e8c89f0486b8073a56409e4/intg/src/main/java/org/apache/atlas/model/instance/AtlasRelationship.java#L63 > Fix inconsistency with attribute createtime in aws_s3_bucket > > > Key: ATLAS-3393 > URL: https://issues.apache.org/jira/browse/ATLAS-3393 > Project: Atlas > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Bolke de Bruin >Priority: Critical > > attributes are in camelcase. in aws_s3_bukcket createtime isnt > > (fix available in linked issue/pr) > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (ATLAS-3393) Fix inconsistency with attribute createtime in aws_s3_bucket
Bolke de Bruin created ATLAS-3393: - Summary: Fix inconsistency with attribute createtime in aws_s3_bucket Key: ATLAS-3393 URL: https://issues.apache.org/jira/browse/ATLAS-3393 Project: Atlas Issue Type: Bug Affects Versions: 2.0.0 Reporter: Bolke de Bruin attributes are in camelcase. in aws_s3_bukcket createtime isnt (fix available in linked issue/pr) -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (ATLAS-3378) Update JanusGraph to 0.4.0
[ https://issues.apache.org/jira/browse/ATLAS-3378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916317#comment-16916317 ] Bolke de Bruin commented on ATLAS-3378: --- JanusGraph 0.4.0 now also supports native OrderBy giving a nice performance boost to sort clauses (when refactored to use OrderBy) > Update JanusGraph to 0.4.0 > -- > > Key: ATLAS-3378 > URL: https://issues.apache.org/jira/browse/ATLAS-3378 > Project: Atlas > Issue Type: Improvement >Reporter: Bolke de Bruin >Priority: Major > > This release supports hbase 2 out of the box -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Comment Edited] (ATLAS-3370) Aggregation Metrics with quick search, Counts don't add up
[
https://issues.apache.org/jira/browse/ATLAS-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16913656#comment-16913656
]
Bolke de Bruin edited comment on ATLAS-3370 at 8/22/19 8:04 PM:
[~Koritala] This is not congruent with our observations and I think there there
was at a minimum an oversight that all fields that are not indexed with
"STRING" are automatically indexed as "TEXT" _("By default, strings are indexed
as text. To make this indexing option explicit, one can define a mapping when
indexing a property key as text." [1])_, this means all text attributes from
the past. This is not tied to the Fulltext index created by Atlas. Also note
that Freetext is not supported on Elastic.
We were debugging DSL search performance issues on Atlas 2.0 configured with
Solr (non embedded) and hbase 2. It uses the Freetext search (confirmed there
is no fulltext index in Solr). The basic search was returning subsecond, but
for the same query the DSL search came back 20 seconds later or more. We traced
it down to the use of incorrect predicates per the documentation of
JanusGraph[1]. When we changed it to use the correct predicates we had the same
response times.
So this change has the consequence that indices are now mixed instead of one
type, however the predicate usage in master is always geared to use STRING and
thus reverts to in memory sorting in many cases which is slow. The change we
were proposing in the linked issue would change the predicate to use the
correct one for the index.
This change complicates matters in that it requires the GremlinQueryComposer to
be aware of the index used for the attribute and then select the right
predicate usage. Or expose this complexity to the user by having something like
"FULLTEXT_LIKE".
1. [https://docs.janusgraph.org/latest/index-parameters.html]
was (Author: bolke):
[~Koritala] This is not congruent with our observations and I think there there
was at a minimum an oversight that all fields that are not indexed with
"STRING" are automatically indexed as "TEXT" _("By default, strings are indexed
as text. To make this indexing option explicit, one can define a mapping when
indexing a property key as text." [1])_, this means all text attributes from
the past. This is not tight to the Fulltext index created by Atlas. Also note
that Freetext is not supported on Elastic.
We were debugging DSL search performance issues on Atlas 2.0 configured with
Solr (non embedded) and hbase 2. It uses the Freetext search (confirmed there
is no fulltext index in Solr). The basic search was returning subsecond, but
for the same query the DSL search came back 20 seconds later or more. We traced
it down to the use of incorrect predicates per the documentation of
JanusGraph[1]. When we changed it to use the correct predicates we had the same
response times.
So this change has the consequence that indices are now mixed instead of one
type, however the predicate usage in master is always geared to use STRING and
thus reverts to in memory sorting in many cases which is slow. The change we
were proposing in the linked issue would change the predicate to use the
correct one for the index.
This change complicates matters in that it requires the GremlinQueryComposer to
be aware of the index used for the attribute and then select the right
predicate usage. Or expose this complexity to the user by having something like
"FULLTEXT_LIKE".
1. https://docs.janusgraph.org/latest/index-parameters.html
> Aggregation Metrics with quick search, Counts don't add up
> --
>
> Key: ATLAS-3370
> URL: https://issues.apache.org/jira/browse/ATLAS-3370
> Project: Atlas
> Issue Type: Bug
>Reporter: Sridhar
>Assignee: Sridhar
>Priority: Major
>
> The issue was happening because of tokenization done for the fields in issue.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (ATLAS-3370) Aggregation Metrics with quick search, Counts don't add up
[ https://issues.apache.org/jira/browse/ATLAS-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16912945#comment-16912945 ] Bolke de Bruin commented on ATLAS-3370: --- why is the indexing changed from fulltext to string based? this changes search behavior. per documentation of janus: When a string mapping is configured, the string value is indexed and can be queried "as-is" - including stop words and non-letter characters. However, in this case the query must match the entire string value. Hence, the string mapping is useful when indexing short character sequences that are considered to be one token. > Aggregation Metrics with quick search, Counts don't add up > -- > > Key: ATLAS-3370 > URL: https://issues.apache.org/jira/browse/ATLAS-3370 > Project: Atlas > Issue Type: Bug >Reporter: Sridhar >Assignee: Sridhar >Priority: Major > > The issue was happening because of tokenization done for the fields in issue. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (ATLAS-3384) Use proper indexed queries for DSL search
Bolke de Bruin created ATLAS-3384:
-
Summary: Use proper indexed queries for DSL search
Key: ATLAS-3384
URL: https://issues.apache.org/jira/browse/ATLAS-3384
Project: Atlas
Issue Type: Improvement
Affects Versions: 2.0.0
Reporter: Bolke de Bruin
Per janusgraph documentation
[https://docs.janusgraph.org/latest/index-parameters.html] strings are indexed
as text by default. To use the indexes one is required to use
* {{textContains}}: is true if (at least) one word inside the text string
matches the query string
* {{textContainsPrefix}}: is true if (at least) one word inside the text
string begins with the query string
* {{textContainsRegex}}: is true if (at least) one word inside the text string
matches the given regular expression
* {{textContainsFuzzy}}: is true if (at least) one word inside the text string
is similar to the query String (based on Levenshtein edit distance)
Atlas uses string search which is suboptimal and leads to significant
performance loss. We have observed subsecond queries vs half minute pkus
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Created] (ATLAS-3378) Update JanusGraph to 0.4.0
Bolke de Bruin created ATLAS-3378: - Summary: Update JanusGraph to 0.4.0 Key: ATLAS-3378 URL: https://issues.apache.org/jira/browse/ATLAS-3378 Project: Atlas Issue Type: Improvement Reporter: Bolke de Bruin This release supports hbase 2 out of the box -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (ATLAS-3367) Expose total search results when using IndexQuery
[ https://issues.apache.org/jira/browse/ATLAS-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin updated ATLAS-3367: -- Attachment: (was: 0001-ATLAS-3367-Expose-total-count-of-index-searches.patch) > Expose total search results when using IndexQuery > - > > Key: ATLAS-3367 > URL: https://issues.apache.org/jira/browse/ATLAS-3367 > Project: Atlas > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Bolke de Bruin >Priority: Major > Attachments: > 0001-ATLAS-3367-Expose-total-count-of-index-searches.patch > > > ElasticSearch and SOLR both can expose the total number of search results > without returning all results. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (ATLAS-3367) Expose total search results when using IndexQuery
[ https://issues.apache.org/jira/browse/ATLAS-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin updated ATLAS-3367: -- Attachment: 0001-ATLAS-3367-Expose-total-count-of-index-searches.patch > Expose total search results when using IndexQuery > - > > Key: ATLAS-3367 > URL: https://issues.apache.org/jira/browse/ATLAS-3367 > Project: Atlas > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Bolke de Bruin >Priority: Major > Attachments: > 0001-ATLAS-3367-Expose-total-count-of-index-searches.patch > > > ElasticSearch and SOLR both can expose the total number of search results > without returning all results. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (ATLAS-3368) Very slow creation of entities with referred entities without relationshipDef
Bolke de Bruin created ATLAS-3368:
-
Summary: Very slow creation of entities with referred entities
without relationshipDef
Key: ATLAS-3368
URL: https://issues.apache.org/jira/browse/ATLAS-3368
Project: Atlas
Issue Type: Bug
Affects Versions: 2.0.0
Reporter: Bolke de Bruin
If an entity is created that has a definition with a reference to a non built
in type, e.g.
{code:java}
,{
"name" : "server",
"typeName" : "server",
"isOptional" : false,
"cardinality" : "SINGLE",
"valuesMinCount" : 1,
"valuesMaxCount" : 1,
"isUnique" : false,
"isIndexable" : true
}{code}
It will trigger a legacy code path in EntityGraphMapper. This path can take
excessive amounts of time:
{code}2019-08-12 12:10:23,323 DEBUG - [NotificationHookConsumer thread-0:] ~ No
RelationshipDef defined between rokku_client and server on attribute: server
(EntityGraphMapper:907)
2019-08-12 12:10:23,323 DEBUG - [NotificationHookConsumer thread-0:] ~ ==>
mapObjectIdValue(org.apache.atlas.repository.store.graph.v2.AttributeMutationContext@3f7c5b66)
(EntityGraphMapper:791)
2019-08-12 12:10:28,825 DEBUG - [NotificationHookConsumer thread-0:] ~ ==>
setProperty(edge[id=v6kwld-biutqg-xel1-x8oqg label=__rokku_client.server from
vertex[id=696799240 type=rokku_client
guid=30db90a8-8236-41c6-9438-4fa2ec14f349] -> to vertex[id=55832632 type=server
guid=9144e224-089f-4ae3-83bd-3472175d92a4]], __state, ACTIVE)
(AtlasGraphUtilsV2:212)
2019-08-12 12:10:23,323 DEBUG - [NotificationHookConsumer thread-0:] ~ ==>
mapObjectIdValue(org.apache.atlas.repository.store.graph.v2.AttributeMutationContext@3f7c5b66)
(EntityGraphMapper:791)
2019-08-12 12:10:28,825 DEBUG - [NotificationHookConsumer thread-0:] ~ ==>
setProperty(edge[id=v6kwld-biutqg-xel1-x8oqg label=__rokku_client.server from
vertex[id=696799240 type=rokku_client
guid=30db90a8-8236-41c6-9438-4fa2ec14f349] -> to vertex[id=55832632 type=server
guid=9144e224-089f-4ae3-83bd-3472175d92a4]], __state, ACTIVE)
(AtlasGraphUtilsV2:212)
{code}
Notice the 5 second(!) jump between the last two log lines. At a minimum the
log about triggering the legacy code path should be at WARN level and extended
with a suggestion to update the definition for better performance.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Created] (ATLAS-3367) Expose total search results when using IndexQuery
Bolke de Bruin created ATLAS-3367: - Summary: Expose total search results when using IndexQuery Key: ATLAS-3367 URL: https://issues.apache.org/jira/browse/ATLAS-3367 Project: Atlas Issue Type: Improvement Affects Versions: 2.0.0 Reporter: Bolke de Bruin ElasticSearch and SOLR both can expose the total number of search results without returning all results. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (ATLAS-3305) Unable to scale atlas kafka consumers
[
https://issues.apache.org/jira/browse/ATLAS-3305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903328#comment-16903328
]
Bolke de Bruin commented on ATLAS-3305:
---
Why not use partitions with hashed ordering by qualified name? Or is this what
you meant [~arempter]?
Can we have consensus here [~sarath.ku...@gmail.com] cause this scalability
issue is stopping deployment at the moment for us
> Unable to scale atlas kafka consumers
> -
>
> Key: ATLAS-3305
> URL: https://issues.apache.org/jira/browse/ATLAS-3305
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core, atlas-intg
>Affects Versions: 1.1.0, 2.0.0
>Reporter: Adam Rempter
>Priority: Major
> Labels: performance
>
> We wanted to scale kafka consumers for atlas, as we are getting many lineage
> messages and processing them just with one consumer is not enough.
>
> There is parameter atlas.notification.hook.numthreads to scale consumers in
> NotificationHookConsumer.
> But the method:
>
> notificationInterface.createConsumers(NotificationType.HOOK, numThreads)
>
> is always returning one element list, which effectively always starts one
> consumer
> List> consumers =
> Collections.singletonList(kafkaConsumer);
>
> Log incorrectly says that nuber of consumers has been created:
> LOG.info("<== KafkaNotification.createConsumers(notificationType={},
> numConsumers={}, autoCommitEnabled={})", notificationType, numConsumers,
> autoCommitEnabled)
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (ATLAS-3309) Support OIDC bearer token in QuickStart and Hive import
[ https://issues.apache.org/jira/browse/ATLAS-3309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16894089#comment-16894089 ] Bolke de Bruin commented on ATLAS-3309: --- On holiday. Will do when iget the chance > Support OIDC bearer token in QuickStart and Hive import > --- > > Key: ATLAS-3309 > URL: https://issues.apache.org/jira/browse/ATLAS-3309 > Project: Atlas > Issue Type: Improvement >Reporter: Bolke de Bruin >Priority: Major > Attachments: > 0002-ATLAS-3309-Add-support-for-OIDC-bearer-token-in-exam.patch > > > In ATLAS-3153 support for keycloak (OIDC) authentication was added. The > QuickStart examples and Hive import need to be updated to support bearer > token authentication. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (ATLAS-3317) java.lang.Exception: Timeout scanning annotations while running Integration Tests
[
https://issues.apache.org/jira/browse/ATLAS-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880664#comment-16880664
]
Bolke de Bruin commented on ATLAS-3317:
---
I'll have a stab at it, but it is bedtime here as well to be honest.
> java.lang.Exception: Timeout scanning annotations while running Integration
> Tests
> -
>
> Key: ATLAS-3317
> URL: https://issues.apache.org/jira/browse/ATLAS-3317
> Project: Atlas
> Issue Type: Bug
>Reporter: Nixon Rodrigues
>Priority: Major
> Attachments: maxWait.diff
>
>
> run :- *mvn clean install -DskipUTs*
>
> {noformat}
> [INFO] --- jetty-maven-plugin:9.3.14.v20161028:deploy-war (start-jetty) @
> atlas-webapp ---
> [INFO] Logging initialized @220653ms
> [INFO] Configuring Jetty for project: Apache Atlas Web Application
> [INFO] Context path = /
> [INFO] Tmp directory =
> /home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/tmp
> [INFO] Web defaults = org/eclipse/jetty/webapp/webdefault.xml
> [INFO] Web overrides = none
> [INFO] jetty-9.3.14.v20161028
> [INFO] Scanning elapsed time=67258ms
> [WARNING] Failed startup of context o.e.j.m.p.JettyWebAppContext@2eb2d10c{/,
> [file:///home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT/,UNAVAILABLE}\{/home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT.war}|file:///home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT/,UNAVAILABLE%7D%7B/home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT.war%7D]
> java.lang.Exception: Timeout scanning annotations
> at
> org.eclipse.jetty.annotations.AnnotationConfiguration.scanForAnnotations
> (AnnotationConfiguration.java:578)
> at org.eclipse.jetty.annotations.AnnotationConfiguration.configure
> (AnnotationConfiguration.java:447)
> at org.eclipse.jetty.webapp.WebAppContext.configure
> (WebAppContext.java:494)
> at org.eclipse.jetty.webapp.WebAppContext.startContext
> (WebAppContext.java:1361)
> at org.eclipse.jetty.server.handler.ContextHandler.doStart
> (ContextHandler.java:778)
> at org.eclipse.jetty.servlet.ServletContextHandler.doStart
> (ServletContextHandler.java:262)
> at org.eclipse.jetty.webapp.WebAppContext.doStart (WebAppContext.java:520)
> at org.eclipse.jetty.maven.plugin.JettyWebAppContext.doStart
> (JettyWebAppContext.java:398)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart
> (ContainerLifeCycle.java:113)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart
> (AbstractHandler.java:61)
> at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart
> (ContextHandlerCollection.java:161)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart
> (ContainerLifeCycle.java:113)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart
> (AbstractHandler.java:61)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.server.Server.start (Server.java:422)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart
> (ContainerLifeCycle.java:105)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart
> (AbstractHandler.java:61)
> at org.eclipse.jetty.server.Server.doStart (Server.java:389)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.maven.plugin.AbstractJettyMojo.startJetty
> (AbstractJettyMojo.java:460)
> at org.eclipse.jetty.maven.plugin.AbstractJettyMojo.execute
> (AbstractJettyMojo.java:328)
> at org.eclipse.jetty.maven.plugin.JettyRunWarMojo.execute
> (JettyRunWarMojo.java:64)
> at org.eclipse.jetty.maven.plugin.JettyDeployWar.execute
> (JettyDeployWar.java:65)
> at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo
> (DefaultBuildPluginManager.java:137)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute
> (MojoExecutor.java:210)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute
> (MojoExecutor.java:156)
> at
[jira] [Comment Edited] (ATLAS-3317) java.lang.Exception: Timeout scanning annotations while running Integration Tests
[
https://issues.apache.org/jira/browse/ATLAS-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880657#comment-16880657
]
Bolke de Bruin edited comment on ATLAS-3317 at 7/8/19 7:43 PM:
---
That indeed fixes the issue. Now I am also getting an application.log, which
wasn't there previously.
Ah okay I was thrown of by the reference to my patch i guess. Obviously in the
patch itself annotations have been added and keycloak probably brings some
itself as well. This increases scanning time. The default of 60s might just not
have been enough anymore now.
Note that the setting is in seconds so it is set to 3000 seconds now, which
might not be what you want.
was (Author: bolke):
That indeed fixes the issue. Now I am also getting an application.log, which
wasn't there previously.
Ah okay I was thrown of by the reference to my patch i guess. Obviously in the
patch itself annotations have been added and keycloak probably brings some
itself as well. This increases scanning time. The default of 60s might just not
have been enough anymore now.
Note that the setting is in seconds so it is set to an hour now.
> java.lang.Exception: Timeout scanning annotations while running Integration
> Tests
> -
>
> Key: ATLAS-3317
> URL: https://issues.apache.org/jira/browse/ATLAS-3317
> Project: Atlas
> Issue Type: Bug
>Reporter: Nixon Rodrigues
>Priority: Major
> Attachments: maxWait.diff
>
>
> run :- *mvn clean install -DskipUTs*
>
> {noformat}
> [INFO] --- jetty-maven-plugin:9.3.14.v20161028:deploy-war (start-jetty) @
> atlas-webapp ---
> [INFO] Logging initialized @220653ms
> [INFO] Configuring Jetty for project: Apache Atlas Web Application
> [INFO] Context path = /
> [INFO] Tmp directory =
> /home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/tmp
> [INFO] Web defaults = org/eclipse/jetty/webapp/webdefault.xml
> [INFO] Web overrides = none
> [INFO] jetty-9.3.14.v20161028
> [INFO] Scanning elapsed time=67258ms
> [WARNING] Failed startup of context o.e.j.m.p.JettyWebAppContext@2eb2d10c{/,
> [file:///home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT/,UNAVAILABLE}\{/home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT.war}|file:///home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT/,UNAVAILABLE%7D%7B/home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT.war%7D]
> java.lang.Exception: Timeout scanning annotations
> at
> org.eclipse.jetty.annotations.AnnotationConfiguration.scanForAnnotations
> (AnnotationConfiguration.java:578)
> at org.eclipse.jetty.annotations.AnnotationConfiguration.configure
> (AnnotationConfiguration.java:447)
> at org.eclipse.jetty.webapp.WebAppContext.configure
> (WebAppContext.java:494)
> at org.eclipse.jetty.webapp.WebAppContext.startContext
> (WebAppContext.java:1361)
> at org.eclipse.jetty.server.handler.ContextHandler.doStart
> (ContextHandler.java:778)
> at org.eclipse.jetty.servlet.ServletContextHandler.doStart
> (ServletContextHandler.java:262)
> at org.eclipse.jetty.webapp.WebAppContext.doStart (WebAppContext.java:520)
> at org.eclipse.jetty.maven.plugin.JettyWebAppContext.doStart
> (JettyWebAppContext.java:398)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart
> (ContainerLifeCycle.java:113)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart
> (AbstractHandler.java:61)
> at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart
> (ContextHandlerCollection.java:161)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart
> (ContainerLifeCycle.java:113)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart
> (AbstractHandler.java:61)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.server.Server.start (Server.java:422)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart
> (ContainerLifeCycle.java:105)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart
>
[jira] [Comment Edited] (ATLAS-3317) java.lang.Exception: Timeout scanning annotations while running Integration Tests
[
https://issues.apache.org/jira/browse/ATLAS-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880657#comment-16880657
]
Bolke de Bruin edited comment on ATLAS-3317 at 7/8/19 7:42 PM:
---
That indeed fixes the issue. Now I am also getting an application.log, which
wasn't there previously.
Ah okay I was thrown of by the reference to my patch i guess. Obviously in the
patch itself annotations have been added and keycloak probably brings some
itself as well. This increases scanning time. The default of 60s might just not
have been enough anymore now.
Note that the setting is in seconds so it is set to an hour now.
was (Author: bolke):
That indeed fixes the issue. Now I am also getting an application.log, which
wasn't there previously. Maybe server didn't have time to startup?
> java.lang.Exception: Timeout scanning annotations while running Integration
> Tests
> -
>
> Key: ATLAS-3317
> URL: https://issues.apache.org/jira/browse/ATLAS-3317
> Project: Atlas
> Issue Type: Bug
>Reporter: Nixon Rodrigues
>Priority: Major
> Attachments: maxWait.diff
>
>
> run :- *mvn clean install -DskipUTs*
>
> {noformat}
> [INFO] --- jetty-maven-plugin:9.3.14.v20161028:deploy-war (start-jetty) @
> atlas-webapp ---
> [INFO] Logging initialized @220653ms
> [INFO] Configuring Jetty for project: Apache Atlas Web Application
> [INFO] Context path = /
> [INFO] Tmp directory =
> /home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/tmp
> [INFO] Web defaults = org/eclipse/jetty/webapp/webdefault.xml
> [INFO] Web overrides = none
> [INFO] jetty-9.3.14.v20161028
> [INFO] Scanning elapsed time=67258ms
> [WARNING] Failed startup of context o.e.j.m.p.JettyWebAppContext@2eb2d10c{/,
> [file:///home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT/,UNAVAILABLE}\{/home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT.war}|file:///home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT/,UNAVAILABLE%7D%7B/home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT.war%7D]
> java.lang.Exception: Timeout scanning annotations
> at
> org.eclipse.jetty.annotations.AnnotationConfiguration.scanForAnnotations
> (AnnotationConfiguration.java:578)
> at org.eclipse.jetty.annotations.AnnotationConfiguration.configure
> (AnnotationConfiguration.java:447)
> at org.eclipse.jetty.webapp.WebAppContext.configure
> (WebAppContext.java:494)
> at org.eclipse.jetty.webapp.WebAppContext.startContext
> (WebAppContext.java:1361)
> at org.eclipse.jetty.server.handler.ContextHandler.doStart
> (ContextHandler.java:778)
> at org.eclipse.jetty.servlet.ServletContextHandler.doStart
> (ServletContextHandler.java:262)
> at org.eclipse.jetty.webapp.WebAppContext.doStart (WebAppContext.java:520)
> at org.eclipse.jetty.maven.plugin.JettyWebAppContext.doStart
> (JettyWebAppContext.java:398)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart
> (ContainerLifeCycle.java:113)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart
> (AbstractHandler.java:61)
> at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart
> (ContextHandlerCollection.java:161)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart
> (ContainerLifeCycle.java:113)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart
> (AbstractHandler.java:61)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.server.Server.start (Server.java:422)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart
> (ContainerLifeCycle.java:105)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart
> (AbstractHandler.java:61)
> at org.eclipse.jetty.server.Server.doStart (Server.java:389)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.maven.plugin.AbstractJettyMojo.startJetty
> (AbstractJettyMojo.java:460)
> at
[jira] [Commented] (ATLAS-3317) java.lang.Exception: Timeout scanning annotations while running Integration Tests
[
https://issues.apache.org/jira/browse/ATLAS-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880657#comment-16880657
]
Bolke de Bruin commented on ATLAS-3317:
---
That indeed fixes the issue. Now I am also getting an application.log, which
wasn't there previously. Maybe server didn't have time to startup?
> java.lang.Exception: Timeout scanning annotations while running Integration
> Tests
> -
>
> Key: ATLAS-3317
> URL: https://issues.apache.org/jira/browse/ATLAS-3317
> Project: Atlas
> Issue Type: Bug
>Reporter: Nixon Rodrigues
>Priority: Major
> Attachments: maxWait.diff
>
>
> run :- *mvn clean install -DskipUTs*
>
> {noformat}
> [INFO] --- jetty-maven-plugin:9.3.14.v20161028:deploy-war (start-jetty) @
> atlas-webapp ---
> [INFO] Logging initialized @220653ms
> [INFO] Configuring Jetty for project: Apache Atlas Web Application
> [INFO] Context path = /
> [INFO] Tmp directory =
> /home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/tmp
> [INFO] Web defaults = org/eclipse/jetty/webapp/webdefault.xml
> [INFO] Web overrides = none
> [INFO] jetty-9.3.14.v20161028
> [INFO] Scanning elapsed time=67258ms
> [WARNING] Failed startup of context o.e.j.m.p.JettyWebAppContext@2eb2d10c{/,
> [file:///home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT/,UNAVAILABLE}\{/home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT.war}|file:///home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT/,UNAVAILABLE%7D%7B/home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT.war%7D]
> java.lang.Exception: Timeout scanning annotations
> at
> org.eclipse.jetty.annotations.AnnotationConfiguration.scanForAnnotations
> (AnnotationConfiguration.java:578)
> at org.eclipse.jetty.annotations.AnnotationConfiguration.configure
> (AnnotationConfiguration.java:447)
> at org.eclipse.jetty.webapp.WebAppContext.configure
> (WebAppContext.java:494)
> at org.eclipse.jetty.webapp.WebAppContext.startContext
> (WebAppContext.java:1361)
> at org.eclipse.jetty.server.handler.ContextHandler.doStart
> (ContextHandler.java:778)
> at org.eclipse.jetty.servlet.ServletContextHandler.doStart
> (ServletContextHandler.java:262)
> at org.eclipse.jetty.webapp.WebAppContext.doStart (WebAppContext.java:520)
> at org.eclipse.jetty.maven.plugin.JettyWebAppContext.doStart
> (JettyWebAppContext.java:398)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart
> (ContainerLifeCycle.java:113)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart
> (AbstractHandler.java:61)
> at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart
> (ContextHandlerCollection.java:161)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart
> (ContainerLifeCycle.java:113)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart
> (AbstractHandler.java:61)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.server.Server.start (Server.java:422)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart
> (ContainerLifeCycle.java:105)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart
> (AbstractHandler.java:61)
> at org.eclipse.jetty.server.Server.doStart (Server.java:389)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.maven.plugin.AbstractJettyMojo.startJetty
> (AbstractJettyMojo.java:460)
> at org.eclipse.jetty.maven.plugin.AbstractJettyMojo.execute
> (AbstractJettyMojo.java:328)
> at org.eclipse.jetty.maven.plugin.JettyRunWarMojo.execute
> (JettyRunWarMojo.java:64)
> at org.eclipse.jetty.maven.plugin.JettyDeployWar.execute
> (JettyDeployWar.java:65)
> at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo
> (DefaultBuildPluginManager.java:137)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute
> (MojoExecutor.java:210)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute
>
[jira] [Commented] (ATLAS-3317) java.lang.Exception: Timeout scanning annotations while running Integration Tests
[
https://issues.apache.org/jira/browse/ATLAS-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880650#comment-16880650
]
Bolke de Bruin commented on ATLAS-3317:
---
Trying now. If that works, I'm suspecting a lookup timeout somehow.
the failsafe reports don't give me a server side trace btw, just the trace of
the failed test. A would assume a 503 error being returned should somehow end
up being logged server side as well.
> java.lang.Exception: Timeout scanning annotations while running Integration
> Tests
> -
>
> Key: ATLAS-3317
> URL: https://issues.apache.org/jira/browse/ATLAS-3317
> Project: Atlas
> Issue Type: Bug
>Reporter: Nixon Rodrigues
>Priority: Major
> Attachments: maxWait.diff
>
>
> run :- *mvn clean install -DskipUTs*
>
> {noformat}
> [INFO] --- jetty-maven-plugin:9.3.14.v20161028:deploy-war (start-jetty) @
> atlas-webapp ---
> [INFO] Logging initialized @220653ms
> [INFO] Configuring Jetty for project: Apache Atlas Web Application
> [INFO] Context path = /
> [INFO] Tmp directory =
> /home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/tmp
> [INFO] Web defaults = org/eclipse/jetty/webapp/webdefault.xml
> [INFO] Web overrides = none
> [INFO] jetty-9.3.14.v20161028
> [INFO] Scanning elapsed time=67258ms
> [WARNING] Failed startup of context o.e.j.m.p.JettyWebAppContext@2eb2d10c{/,
> [file:///home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT/,UNAVAILABLE}\{/home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT.war}|file:///home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT/,UNAVAILABLE%7D%7B/home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT.war%7D]
> java.lang.Exception: Timeout scanning annotations
> at
> org.eclipse.jetty.annotations.AnnotationConfiguration.scanForAnnotations
> (AnnotationConfiguration.java:578)
> at org.eclipse.jetty.annotations.AnnotationConfiguration.configure
> (AnnotationConfiguration.java:447)
> at org.eclipse.jetty.webapp.WebAppContext.configure
> (WebAppContext.java:494)
> at org.eclipse.jetty.webapp.WebAppContext.startContext
> (WebAppContext.java:1361)
> at org.eclipse.jetty.server.handler.ContextHandler.doStart
> (ContextHandler.java:778)
> at org.eclipse.jetty.servlet.ServletContextHandler.doStart
> (ServletContextHandler.java:262)
> at org.eclipse.jetty.webapp.WebAppContext.doStart (WebAppContext.java:520)
> at org.eclipse.jetty.maven.plugin.JettyWebAppContext.doStart
> (JettyWebAppContext.java:398)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart
> (ContainerLifeCycle.java:113)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart
> (AbstractHandler.java:61)
> at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart
> (ContextHandlerCollection.java:161)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart
> (ContainerLifeCycle.java:113)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart
> (AbstractHandler.java:61)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.server.Server.start (Server.java:422)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart
> (ContainerLifeCycle.java:105)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart
> (AbstractHandler.java:61)
> at org.eclipse.jetty.server.Server.doStart (Server.java:389)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.maven.plugin.AbstractJettyMojo.startJetty
> (AbstractJettyMojo.java:460)
> at org.eclipse.jetty.maven.plugin.AbstractJettyMojo.execute
> (AbstractJettyMojo.java:328)
> at org.eclipse.jetty.maven.plugin.JettyRunWarMojo.execute
> (JettyRunWarMojo.java:64)
> at org.eclipse.jetty.maven.plugin.JettyDeployWar.execute
> (JettyDeployWar.java:65)
> at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo
> (DefaultBuildPluginManager.java:137)
> at
[jira] [Comment Edited] (ATLAS-3317) java.lang.Exception: Timeout scanning annotations while running Integration Tests
[
https://issues.apache.org/jira/browse/ATLAS-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880638#comment-16880638
]
Bolke de Bruin edited comment on ATLAS-3317 at 7/8/19 7:05 PM:
---
Ok. Do you know if there is a way to get to the application logs when running
the IT? I see only client side at the moment. By default Keycloak is inactive
so this looks a bit weird. Where can I find the config of the IT (ie. server
config)?
was (Author: bolke):
Ok. Do you know if there is a way to get to the application logs when running
the IT? I see only client side at the moment. By default Keycloak is inactive
so this looks a bit weird.
> java.lang.Exception: Timeout scanning annotations while running Integration
> Tests
> -
>
> Key: ATLAS-3317
> URL: https://issues.apache.org/jira/browse/ATLAS-3317
> Project: Atlas
> Issue Type: Bug
>Reporter: Nixon Rodrigues
>Priority: Major
>
> run :- *mvn clean install -DskipUTs*
>
> {noformat}
> [INFO] --- jetty-maven-plugin:9.3.14.v20161028:deploy-war (start-jetty) @
> atlas-webapp ---
> [INFO] Logging initialized @220653ms
> [INFO] Configuring Jetty for project: Apache Atlas Web Application
> [INFO] Context path = /
> [INFO] Tmp directory =
> /home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/tmp
> [INFO] Web defaults = org/eclipse/jetty/webapp/webdefault.xml
> [INFO] Web overrides = none
> [INFO] jetty-9.3.14.v20161028
> [INFO] Scanning elapsed time=67258ms
> [WARNING] Failed startup of context o.e.j.m.p.JettyWebAppContext@2eb2d10c{/,
> [file:///home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT/,UNAVAILABLE}\{/home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT.war}|file:///home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT/,UNAVAILABLE%7D%7B/home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT.war%7D]
> java.lang.Exception: Timeout scanning annotations
> at
> org.eclipse.jetty.annotations.AnnotationConfiguration.scanForAnnotations
> (AnnotationConfiguration.java:578)
> at org.eclipse.jetty.annotations.AnnotationConfiguration.configure
> (AnnotationConfiguration.java:447)
> at org.eclipse.jetty.webapp.WebAppContext.configure
> (WebAppContext.java:494)
> at org.eclipse.jetty.webapp.WebAppContext.startContext
> (WebAppContext.java:1361)
> at org.eclipse.jetty.server.handler.ContextHandler.doStart
> (ContextHandler.java:778)
> at org.eclipse.jetty.servlet.ServletContextHandler.doStart
> (ServletContextHandler.java:262)
> at org.eclipse.jetty.webapp.WebAppContext.doStart (WebAppContext.java:520)
> at org.eclipse.jetty.maven.plugin.JettyWebAppContext.doStart
> (JettyWebAppContext.java:398)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart
> (ContainerLifeCycle.java:113)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart
> (AbstractHandler.java:61)
> at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart
> (ContextHandlerCollection.java:161)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart
> (ContainerLifeCycle.java:113)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart
> (AbstractHandler.java:61)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.server.Server.start (Server.java:422)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart
> (ContainerLifeCycle.java:105)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart
> (AbstractHandler.java:61)
> at org.eclipse.jetty.server.Server.doStart (Server.java:389)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.maven.plugin.AbstractJettyMojo.startJetty
> (AbstractJettyMojo.java:460)
> at org.eclipse.jetty.maven.plugin.AbstractJettyMojo.execute
> (AbstractJettyMojo.java:328)
> at org.eclipse.jetty.maven.plugin.JettyRunWarMojo.execute
> (JettyRunWarMojo.java:64)
> at
[jira] [Commented] (ATLAS-3317) java.lang.Exception: Timeout scanning annotations while running Integration Tests
[
https://issues.apache.org/jira/browse/ATLAS-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880638#comment-16880638
]
Bolke de Bruin commented on ATLAS-3317:
---
Ok. Do you know if there is a way to get to the application logs when running
the IT? I see only client side at the moment. By default Keycloak is inactive
so this looks a bit weird.
> java.lang.Exception: Timeout scanning annotations while running Integration
> Tests
> -
>
> Key: ATLAS-3317
> URL: https://issues.apache.org/jira/browse/ATLAS-3317
> Project: Atlas
> Issue Type: Bug
>Reporter: Nixon Rodrigues
>Priority: Major
>
> run :- *mvn clean install -DskipUTs*
>
> {noformat}
> [INFO] --- jetty-maven-plugin:9.3.14.v20161028:deploy-war (start-jetty) @
> atlas-webapp ---
> [INFO] Logging initialized @220653ms
> [INFO] Configuring Jetty for project: Apache Atlas Web Application
> [INFO] Context path = /
> [INFO] Tmp directory =
> /home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/tmp
> [INFO] Web defaults = org/eclipse/jetty/webapp/webdefault.xml
> [INFO] Web overrides = none
> [INFO] jetty-9.3.14.v20161028
> [INFO] Scanning elapsed time=67258ms
> [WARNING] Failed startup of context o.e.j.m.p.JettyWebAppContext@2eb2d10c{/,
> [file:///home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT/,UNAVAILABLE}\{/home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT.war}|file:///home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT/,UNAVAILABLE%7D%7B/home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT.war%7D]
> java.lang.Exception: Timeout scanning annotations
> at
> org.eclipse.jetty.annotations.AnnotationConfiguration.scanForAnnotations
> (AnnotationConfiguration.java:578)
> at org.eclipse.jetty.annotations.AnnotationConfiguration.configure
> (AnnotationConfiguration.java:447)
> at org.eclipse.jetty.webapp.WebAppContext.configure
> (WebAppContext.java:494)
> at org.eclipse.jetty.webapp.WebAppContext.startContext
> (WebAppContext.java:1361)
> at org.eclipse.jetty.server.handler.ContextHandler.doStart
> (ContextHandler.java:778)
> at org.eclipse.jetty.servlet.ServletContextHandler.doStart
> (ServletContextHandler.java:262)
> at org.eclipse.jetty.webapp.WebAppContext.doStart (WebAppContext.java:520)
> at org.eclipse.jetty.maven.plugin.JettyWebAppContext.doStart
> (JettyWebAppContext.java:398)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart
> (ContainerLifeCycle.java:113)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart
> (AbstractHandler.java:61)
> at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart
> (ContextHandlerCollection.java:161)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart
> (ContainerLifeCycle.java:113)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart
> (AbstractHandler.java:61)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.server.Server.start (Server.java:422)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart
> (ContainerLifeCycle.java:105)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart
> (AbstractHandler.java:61)
> at org.eclipse.jetty.server.Server.doStart (Server.java:389)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.maven.plugin.AbstractJettyMojo.startJetty
> (AbstractJettyMojo.java:460)
> at org.eclipse.jetty.maven.plugin.AbstractJettyMojo.execute
> (AbstractJettyMojo.java:328)
> at org.eclipse.jetty.maven.plugin.JettyRunWarMojo.execute
> (JettyRunWarMojo.java:64)
> at org.eclipse.jetty.maven.plugin.JettyDeployWar.execute
> (JettyDeployWar.java:65)
> at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo
> (DefaultBuildPluginManager.java:137)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute
> (MojoExecutor.java:210)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute
>
[jira] [Commented] (ATLAS-3317) java.lang.Exception: Timeout scanning annotations while running Integration Tests
[ https://issues.apache.org/jira/browse/ATLAS-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880624#comment-16880624 ] Bolke de Bruin commented on ATLAS-3317: --- I'm also seeing errors with commit f75d7f409f3323e799bb33e6ae1da89526fc475c although a bit different (this is the commit before my patch) ``` 127.0.0.1 - - [08/Jul/2019:18:40:22 +] "GET //localhost:31000/api/atlas/lineage/d8cc5cd7-625b-4c0b-874d-703feba86fd9/inputs/graph HTTP/1.1" 200 2423 "-" "Java/1.8.0_212" Jul 08, 2019 6:40:23 PM com.sun.jersey.spi.container.ContainerResponse logException SEVERE: Mapped exception to response: 500 (Internal Server Error) javax.ws.rs.WebApplicationException at org.apache.atlas.web.resources.DataSetLineageResource.schema(DataSetLineageResource.java:203) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185) at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:845) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1712) at org.apache.atlas.web.filters.AuditFilter.doFilter(AuditFilter.java:100) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.apache.atlas.web.filters.AtlasCSRFPreventionFilter$ServletFilterHttpInteraction.proceed(AtlasCSRFPreventionFilter.java:235) at org.apache.atlas.web.filters.AtlasCSRFPreventionFilter.handleHttpInteraction(AtlasCSRFPreventionFilter.java:177) at org.apache.atlas.web.filters.AtlasCSRFPreventionFilter.doFilter(AtlasCSRFPreventionFilter.java:190) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.apache.atlas.web.filters.AtlasAuthenticationFilter.doFilter(AtlasAuthenticationFilter.java:358) at
[jira] [Comment Edited] (ATLAS-3317) java.lang.Exception: Timeout scanning annotations while running Integration Tests
[
https://issues.apache.org/jira/browse/ATLAS-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880624#comment-16880624
]
Bolke de Bruin edited comment on ATLAS-3317 at 7/8/19 6:46 PM:
---
I'm also seeing errors with commit f75d7f409f3323e799bb33e6ae1da89526fc475c
although a bit different (this is the commit before my patch)
{code:java}
127.0.0.1 - - [08/Jul/2019:18:40:22 +] "GET
//localhost:31000/api/atlas/lineage/d8cc5cd7-625b-4c0b-874d-703feba86fd9/inputs/graph
HTTP/1.1" 200 2423 "-" "Java/1.8.0_212"
Jul 08, 2019 6:40:23 PM com.sun.jersey.spi.container.ContainerResponse
logException
SEVERE: Mapped exception to response: 500 (Internal Server Error)
javax.ws.rs.WebApplicationException
at
org.apache.atlas.web.resources.DataSetLineageResource.schema(DataSetLineageResource.java:203)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
at
com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at
com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at
com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at
com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
at
com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:845)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1712)
at org.apache.atlas.web.filters.AuditFilter.doFilter(AuditFilter.java:100)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
at
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
at
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at
org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at
org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at
org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at
org.apache.atlas.web.filters.AtlasCSRFPreventionFilter$ServletFilterHttpInteraction.proceed(AtlasCSRFPreventionFilter.java:235)
at
org.apache.atlas.web.filters.AtlasCSRFPreventionFilter.handleHttpInteraction(AtlasCSRFPreventionFilter.java:177)
at
org.apache.atlas.web.filters.AtlasCSRFPreventionFilter.doFilter(AtlasCSRFPreventionFilter.java:190)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at
[jira] [Commented] (ATLAS-3317) java.lang.Exception: Timeout scanning annotations while running Integration Tests
[
https://issues.apache.org/jira/browse/ATLAS-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880588#comment-16880588
]
Bolke de Bruin commented on ATLAS-3317:
---
[~nixonrodrigues] I'll have a look. Please note that the keycloak patch was
tested (mvn clean install) with 2.0 rcX. I'll check that build and will check
out master.
> java.lang.Exception: Timeout scanning annotations while running Integration
> Tests
> -
>
> Key: ATLAS-3317
> URL: https://issues.apache.org/jira/browse/ATLAS-3317
> Project: Atlas
> Issue Type: Bug
>Reporter: Nixon Rodrigues
>Priority: Major
>
> run :- *mvn clean install -DskipUTs*
>
> {noformat}
> [INFO] --- jetty-maven-plugin:9.3.14.v20161028:deploy-war (start-jetty) @
> atlas-webapp ---
> [INFO] Logging initialized @220653ms
> [INFO] Configuring Jetty for project: Apache Atlas Web Application
> [INFO] Context path = /
> [INFO] Tmp directory =
> /home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/tmp
> [INFO] Web defaults = org/eclipse/jetty/webapp/webdefault.xml
> [INFO] Web overrides = none
> [INFO] jetty-9.3.14.v20161028
> [INFO] Scanning elapsed time=67258ms
> [WARNING] Failed startup of context o.e.j.m.p.JettyWebAppContext@2eb2d10c{/,
> [file:///home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT/,UNAVAILABLE}\{/home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT.war}|file:///home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT/,UNAVAILABLE%7D%7B/home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT.war%7D]
> java.lang.Exception: Timeout scanning annotations
> at
> org.eclipse.jetty.annotations.AnnotationConfiguration.scanForAnnotations
> (AnnotationConfiguration.java:578)
> at org.eclipse.jetty.annotations.AnnotationConfiguration.configure
> (AnnotationConfiguration.java:447)
> at org.eclipse.jetty.webapp.WebAppContext.configure
> (WebAppContext.java:494)
> at org.eclipse.jetty.webapp.WebAppContext.startContext
> (WebAppContext.java:1361)
> at org.eclipse.jetty.server.handler.ContextHandler.doStart
> (ContextHandler.java:778)
> at org.eclipse.jetty.servlet.ServletContextHandler.doStart
> (ServletContextHandler.java:262)
> at org.eclipse.jetty.webapp.WebAppContext.doStart (WebAppContext.java:520)
> at org.eclipse.jetty.maven.plugin.JettyWebAppContext.doStart
> (JettyWebAppContext.java:398)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart
> (ContainerLifeCycle.java:113)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart
> (AbstractHandler.java:61)
> at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart
> (ContextHandlerCollection.java:161)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart
> (ContainerLifeCycle.java:113)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart
> (AbstractHandler.java:61)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.server.Server.start (Server.java:422)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart
> (ContainerLifeCycle.java:105)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart
> (AbstractHandler.java:61)
> at org.eclipse.jetty.server.Server.doStart (Server.java:389)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.maven.plugin.AbstractJettyMojo.startJetty
> (AbstractJettyMojo.java:460)
> at org.eclipse.jetty.maven.plugin.AbstractJettyMojo.execute
> (AbstractJettyMojo.java:328)
> at org.eclipse.jetty.maven.plugin.JettyRunWarMojo.execute
> (JettyRunWarMojo.java:64)
> at org.eclipse.jetty.maven.plugin.JettyDeployWar.execute
> (JettyDeployWar.java:65)
> at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo
> (DefaultBuildPluginManager.java:137)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute
> (MojoExecutor.java:210)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute
>
[jira] [Comment Edited] (ATLAS-3153) Support OpenID Connect directly rather than through Knox
[ https://issues.apache.org/jira/browse/ATLAS-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880583#comment-16880583 ] Bolke de Bruin edited comment on ATLAS-3153 at 7/8/19 5:53 PM: --- [~nixonrodrigues] can you share a bit more what is failing exactly? I assume IT is integration test? Can I reproduce the test somehow? Is it possible to get access to the jenkins logs? was (Author: bolke): [~nixonrodrigues]can you share a bit more what is failing exactly? I assume IT is integration test? Can I reproduce the test somehow? > Support OpenID Connect directly rather than through Knox > > > Key: ATLAS-3153 > URL: https://issues.apache.org/jira/browse/ATLAS-3153 > Project: Atlas > Issue Type: Improvement > Components: atlas-core, atlas-webui >Affects Versions: 2.0.0 >Reporter: Bolke de Bruin >Priority: Major > Labels: authentication, authorization > Fix For: 3.0.0 > > Attachments: 0001-ATLAS-3153-Add-keycloak-authentication.patch, > 0001-ATLAS-3153-Testcase-fix-due-to-Keycloak-authenticati.patch, > application.log, keycloak.json, openid_connect_atlas.md > > Time Spent: 40m > Remaining Estimate: 0h > > The current SSO implementation with Apache Knox is limiting SSO > interoperability to Apache Knox. Knox uses JWT verification which could > easily be extended to allow for direct OpenID Connect support and doesn't > require organizations to deploy Knox. > Required changes: > * Pickup bearer token from headers > * Improve and standardize redirecting > * Optionally: obtain certificates from well_known uri > * Optionally: obtain user groups from userinfo endpoint rather than UGI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ATLAS-3153) Support OpenID Connect directly rather than through Knox
[ https://issues.apache.org/jira/browse/ATLAS-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880583#comment-16880583 ] Bolke de Bruin commented on ATLAS-3153: --- [~nixonrodrigues]can you share a bit more what is failing exactly? I assume IT is integration test? Can I reproduce the test somehow? > Support OpenID Connect directly rather than through Knox > > > Key: ATLAS-3153 > URL: https://issues.apache.org/jira/browse/ATLAS-3153 > Project: Atlas > Issue Type: Improvement > Components: atlas-core, atlas-webui >Affects Versions: 2.0.0 >Reporter: Bolke de Bruin >Priority: Major > Labels: authentication, authorization > Fix For: 3.0.0 > > Attachments: 0001-ATLAS-3153-Add-keycloak-authentication.patch, > 0001-ATLAS-3153-Testcase-fix-due-to-Keycloak-authenticati.patch, > application.log, keycloak.json, openid_connect_atlas.md > > Time Spent: 40m > Remaining Estimate: 0h > > The current SSO implementation with Apache Knox is limiting SSO > interoperability to Apache Knox. Knox uses JWT verification which could > easily be extended to allow for direct OpenID Connect support and doesn't > require organizations to deploy Knox. > Required changes: > * Pickup bearer token from headers > * Improve and standardize redirecting > * Optionally: obtain certificates from well_known uri > * Optionally: obtain user groups from userinfo endpoint rather than UGI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ATLAS-3309) Support OIDC bearer token in QuickStart and Hive import
[ https://issues.apache.org/jira/browse/ATLAS-3309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16879035#comment-16879035 ] Bolke de Bruin commented on ATLAS-3309: --- Cc [~nixonrodrigues] > Support OIDC bearer token in QuickStart and Hive import > --- > > Key: ATLAS-3309 > URL: https://issues.apache.org/jira/browse/ATLAS-3309 > Project: Atlas > Issue Type: Improvement >Reporter: Bolke de Bruin >Priority: Major > Attachments: > 0002-ATLAS-3309-Add-support-for-OIDC-bearer-token-in-exam.patch > > > In ATLAS-3153 support for keycloak (OIDC) authentication was added. The > QuickStart examples and Hive import need to be updated to support bearer > token authentication. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ATLAS-3153) Support OpenID Connect directly rather than through Knox
[ https://issues.apache.org/jira/browse/ATLAS-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878538#comment-16878538 ] Bolke de Bruin commented on ATLAS-3153: --- [~saqeeb.shaikh136] can we have this merged please? > Support OpenID Connect directly rather than through Knox > > > Key: ATLAS-3153 > URL: https://issues.apache.org/jira/browse/ATLAS-3153 > Project: Atlas > Issue Type: Improvement > Components: atlas-core, atlas-webui >Affects Versions: 2.0.0 >Reporter: Bolke de Bruin >Priority: Major > Labels: authentication, authorization > Attachments: 0001-ATLAS-3153-Add-keycloak-authentication.patch, > application.log, keycloak.json, openid_connect_atlas.md > > Time Spent: 20m > Remaining Estimate: 0h > > The current SSO implementation with Apache Knox is limiting SSO > interoperability to Apache Knox. Knox uses JWT verification which could > easily be extended to allow for direct OpenID Connect support and doesn't > require organizations to deploy Knox. > Required changes: > * Pickup bearer token from headers > * Improve and standardize redirecting > * Optionally: obtain certificates from well_known uri > * Optionally: obtain user groups from userinfo endpoint rather than UGI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ATLAS-3203) Incorrect qualifiedName for QuickStartV2 examples
[ https://issues.apache.org/jira/browse/ATLAS-3203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16875579#comment-16875579 ] Bolke de Bruin commented on ATLAS-3203: --- [~saqeeb.shaikh136] can this be merged please? its a simple fix > Incorrect qualifiedName for QuickStartV2 examples > - > > Key: ATLAS-3203 > URL: https://issues.apache.org/jira/browse/ATLAS-3203 > Project: Atlas > Issue Type: Bug >Reporter: Nanne Wielinga >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > The qualifiedName for columns should be: 'database.table.column@cluster'. > The qualifiedName for columns currently is: 'column@cluster' > > The qualifiedName for tables should be: 'database.table@cluster'. > The qualifiedName for tables currently is: 'table@cluster' -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ATLAS-3153) Support OpenID Connect directly rather than through Knox
[ https://issues.apache.org/jira/browse/ATLAS-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16875578#comment-16875578 ] Bolke de Bruin commented on ATLAS-3153: --- ping [~saqeeb.shaikh136] . I have created the first version for bridges, quickstart and exampkes, but it is dependent on this one. Also note that Knox does not have support in Atlas client side. > Support OpenID Connect directly rather than through Knox > > > Key: ATLAS-3153 > URL: https://issues.apache.org/jira/browse/ATLAS-3153 > Project: Atlas > Issue Type: Improvement > Components: atlas-core, atlas-webui >Affects Versions: 2.0.0 >Reporter: Bolke de Bruin >Priority: Major > Labels: authentication, authorization > Attachments: 0001-ATLAS-3153-Add-keycloak-authentication.patch, > application.log, keycloak.json, openid_connect_atlas.md > > Time Spent: 20m > Remaining Estimate: 0h > > The current SSO implementation with Apache Knox is limiting SSO > interoperability to Apache Knox. Knox uses JWT verification which could > easily be extended to allow for direct OpenID Connect support and doesn't > require organizations to deploy Knox. > Required changes: > * Pickup bearer token from headers > * Improve and standardize redirecting > * Optionally: obtain certificates from well_known uri > * Optionally: obtain user groups from userinfo endpoint rather than UGI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ATLAS-3153) Support OpenID Connect directly rather than through Knox
[ https://issues.apache.org/jira/browse/ATLAS-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873590#comment-16873590 ] Bolke de Bruin commented on ATLAS-3153: --- [~saqeeb.shaikh136] I have created ATLAS-3309 to track this. Can this now be merged? > Support OpenID Connect directly rather than through Knox > > > Key: ATLAS-3153 > URL: https://issues.apache.org/jira/browse/ATLAS-3153 > Project: Atlas > Issue Type: Improvement > Components: atlas-core, atlas-webui >Affects Versions: 2.0.0 >Reporter: Bolke de Bruin >Priority: Major > Labels: authentication, authorization > Attachments: 0001-ATLAS-3153-Add-keycloak-authentication.patch, > application.log, keycloak.json, openid_connect_atlas.md > > Time Spent: 20m > Remaining Estimate: 0h > > The current SSO implementation with Apache Knox is limiting SSO > interoperability to Apache Knox. Knox uses JWT verification which could > easily be extended to allow for direct OpenID Connect support and doesn't > require organizations to deploy Knox. > Required changes: > * Pickup bearer token from headers > * Improve and standardize redirecting > * Optionally: obtain certificates from well_known uri > * Optionally: obtain user groups from userinfo endpoint rather than UGI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ATLAS-3309) Support OIDC bearer token in QuickStart and Hive import
[ https://issues.apache.org/jira/browse/ATLAS-3309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873589#comment-16873589 ] Bolke de Bruin commented on ATLAS-3309: --- Cc [~saqeeb.shaikh136] > Support OIDC bearer token in QuickStart and Hive import > --- > > Key: ATLAS-3309 > URL: https://issues.apache.org/jira/browse/ATLAS-3309 > Project: Atlas > Issue Type: Improvement >Reporter: Bolke de Bruin >Priority: Major > > In ATLAS-3153 support for keycloak (OIDC) authentication was added. The > QuickStart examples and Hive import need to be updated to support bearer > token authentication. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ATLAS-3309) Support OIDC bearer token in QuickStart and Hive import
[ https://issues.apache.org/jira/browse/ATLAS-3309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin updated ATLAS-3309: -- Description: In ATLAS-3153 support for keycloak (OIDC) authentication was added. The QuickStart examples and Hive import need to be updated to support bearer token authentication. Summary: Support OIDC bearer token in QuickStart and Hive import (was: Support OIDC bearer to) > Support OIDC bearer token in QuickStart and Hive import > --- > > Key: ATLAS-3309 > URL: https://issues.apache.org/jira/browse/ATLAS-3309 > Project: Atlas > Issue Type: Improvement >Reporter: Bolke de Bruin >Priority: Major > > In ATLAS-3153 support for keycloak (OIDC) authentication was added. The > QuickStart examples and Hive import need to be updated to support bearer > token authentication. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ATLAS-3309) Support OIDC bearer to
Bolke de Bruin created ATLAS-3309: - Summary: Support OIDC bearer to Key: ATLAS-3309 URL: https://issues.apache.org/jira/browse/ATLAS-3309 Project: Atlas Issue Type: Improvement Reporter: Bolke de Bruin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
