[jira] [Updated] (ATLAS-2932) Update DSL to use Java Traversal API
[ https://issues.apache.org/jira/browse/ATLAS-2932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry updated ATLAS-2932: --- Attachment: (was: dsl-traversal-2.1.patch) > Update DSL to use Java Traversal API > > > Key: ATLAS-2932 > URL: https://issues.apache.org/jira/browse/ATLAS-2932 > Project: Atlas > Issue Type: Bug >Affects Versions: 1.0.0, trunk >Reporter: Apoorv Naik >Assignee: Ashutosh Mestry >Priority: Major > Attachments: > 0002-ATLAS-2932-Update-DSL-to-use-Tinkerpop-Java-APIs-ins.patch, > dsl-traversal-v2.3.patch > > > Change DSL code to use Java Tinkerpop Traversals instead of > GremlinScriptEngine -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-2932) Update DSL to use Java Traversal API
[ https://issues.apache.org/jira/browse/ATLAS-2932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry updated ATLAS-2932: --- Attachment: dsl-traversal-2.1.patch > Update DSL to use Java Traversal API > > > Key: ATLAS-2932 > URL: https://issues.apache.org/jira/browse/ATLAS-2932 > Project: Atlas > Issue Type: Bug >Affects Versions: 1.0.0, trunk >Reporter: Apoorv Naik >Assignee: Ashutosh Mestry >Priority: Major > Attachments: > 0002-ATLAS-2932-Update-DSL-to-use-Tinkerpop-Java-APIs-ins.patch, > dsl-traversal-2.1.patch > > > Change DSL code to use Java Tinkerpop Traversals instead of > GremlinScriptEngine -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4068) Export/Import Service: Conditionally Support Simultaneous Operations
[ https://issues.apache.org/jira/browse/ATLAS-4068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry updated ATLAS-4068: --- Attachment: ATLAS-4068-Export-Import-Conditionally-Support-Simpu.patch > Export/Import Service: Conditionally Support Simultaneous Operations > > > Key: ATLAS-4068 > URL: https://issues.apache.org/jira/browse/ATLAS-4068 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Ashutosh Mestry >Assignee: Ashutosh Mestry >Priority: Major > Fix For: trunk > > Attachments: > ATLAS-4068-Export-Import-Conditionally-Support-Simpu.patch > > > *Background* > Existing implementation for Export & Import REST APIs disallows more than one > operation at a time. > *Solution* > Loosen the criteria based on request: > * Export: _AtlasExportRequest_: _skipLineage_ and _replicatedTo_ is selected. > * Import: _AtlasImportRequest_: _replicatedFrom_ is selected. > Operations where these options are not specified, the existing behavior will > continue. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-4068) Export/Import Service: Conditionally Support Simultaneous Operations
[ https://issues.apache.org/jira/browse/ATLAS-4068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17246872#comment-17246872 ] Ashutosh Mestry commented on ATLAS-4068: Addressed review comments. > Export/Import Service: Conditionally Support Simultaneous Operations > > > Key: ATLAS-4068 > URL: https://issues.apache.org/jira/browse/ATLAS-4068 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Ashutosh Mestry >Assignee: Ashutosh Mestry >Priority: Major > Fix For: trunk > > Attachments: > ATLAS-4068-Export-Import-Conditionally-Support-Simpu.patch > > > *Background* > Existing implementation for Export & Import REST APIs disallows more than one > operation at a time. > *Solution* > Loosen the criteria based on request: > * Export: _AtlasExportRequest_: _skipLineage_ and _replicatedTo_ is selected. > * Import: _AtlasImportRequest_: _replicatedFrom_ is selected. > Operations where these options are not specified, the existing behavior will > continue. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4068) Export/Import Service: Conditionally Support Simultaneous Operations
[ https://issues.apache.org/jira/browse/ATLAS-4068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry updated ATLAS-4068: --- Attachment: (was: ATLAS-4068-Support-simultaneous-export-and-import-op.patch) > Export/Import Service: Conditionally Support Simultaneous Operations > > > Key: ATLAS-4068 > URL: https://issues.apache.org/jira/browse/ATLAS-4068 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Ashutosh Mestry >Assignee: Ashutosh Mestry >Priority: Major > Fix For: trunk > > > *Background* > Existing implementation for Export & Import REST APIs disallows more than one > operation at a time. > *Solution* > Loosen the criteria based on request: > * Export: _AtlasExportRequest_: _skipLineage_ and _replicatedTo_ is selected. > * Import: _AtlasImportRequest_: _replicatedFrom_ is selected. > Operations where these options are not specified, the existing behavior will > continue. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4068) Export/Import Service: Conditionally Support Simultaneous Operations
[ https://issues.apache.org/jira/browse/ATLAS-4068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry updated ATLAS-4068: --- Summary: Export/Import Service: Conditionally Support Simultaneous Operations (was: Export/Import Service: Conditionally Support Multiple Operations) > Export/Import Service: Conditionally Support Simultaneous Operations > > > Key: ATLAS-4068 > URL: https://issues.apache.org/jira/browse/ATLAS-4068 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Ashutosh Mestry >Assignee: Ashutosh Mestry >Priority: Major > Fix For: trunk > > Attachments: > ATLAS-4068-Support-simultaneous-export-and-import-op.patch > > > *Background* > Existing implementation for Export & Import REST APIs disallows more than one > operation at a time. > *Solution* > Loosen the criteria based on request: > * Export: _AtlasExportRequest_: _skipLineage_ and _replicatedTo_ is selected. > * Import: _AtlasImportRequest_: _replicatedFrom_ is selected. > Operations where these options are not specified, the existing behavior will > continue. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-4068) Export/Import Service: Conditionally Support Multiple Operations
[ https://issues.apache.org/jira/browse/ATLAS-4068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17246838#comment-17246838 ] Ashutosh Mestry commented on ATLAS-4068: PC build: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/257/ > Export/Import Service: Conditionally Support Multiple Operations > > > Key: ATLAS-4068 > URL: https://issues.apache.org/jira/browse/ATLAS-4068 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Ashutosh Mestry >Assignee: Ashutosh Mestry >Priority: Major > Fix For: trunk > > Attachments: > ATLAS-4068-Support-simultaneous-export-and-import-op.patch > > > *Background* > Existing implementation for Export & Import REST APIs disallows more than one > operation at a time. > *Solution* > Loosen the criteria based on request: > * Export: _AtlasExportRequest_: _skipLineage_ and _replicatedTo_ is selected. > * Import: _AtlasImportRequest_: _replicatedFrom_ is selected. > Operations where these options are not specified, the existing behavior will > continue. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (ATLAS-4069) Export/Importing a recreated deleted entity has the relationshipStatus set to ACTIVE instead of DELETED for deleted entity
[
https://issues.apache.org/jira/browse/ATLAS-4069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Mestry reassigned ATLAS-4069:
--
Assignee: Ashutosh Mestry
> Export/Importing a recreated deleted entity has the relationshipStatus set to
> ACTIVE instead of DELETED for deleted entity
> --
>
> Key: ATLAS-4069
> URL: https://issues.apache.org/jira/browse/ATLAS-4069
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
>Affects Versions: 2.1.0
>Reporter: Umesh Padashetty
>Assignee: Ashutosh Mestry
>Priority: Major
> Attachments: Screenshot 2020-12-09 at 5.51.10 PM.png
>
>
> Run the below queries:
> # create database IF NOT EXISTS database34_${var:rand_str};
> # create table IF NOT EXISTS
> database34_${var:rand_str}.table_${var:rand_str}_34(id int,name string,dob
> date);
> # drop table database34_${var:rand_str}.table_${var:rand_str}_34;
> # drop database database34_${var:rand_str};
> # create database IF NOT EXISTS database34_${var:rand_str};
> # create table IF NOT EXISTS
> database34_${var:rand_str}.table_${var:rand_str}_34(id int,name string,dob
> date);
> As you can see, create a database/table, followed by drop statements, and
> then recreated them.
> Hence, there are 2 entities with same name, but the state is different (1 is
> ACTIVE, 1 is DELETED).
> Text compare: !Screenshot 2020-12-09 at 5.51.10 PM.png!
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ATLAS-4068) Export/Import Service: Conditionally Support Multiple Operations
[ https://issues.apache.org/jira/browse/ATLAS-4068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry updated ATLAS-4068: --- Attachment: ATLAS-4068-Support-simultaneous-export-and-import-op.patch > Export/Import Service: Conditionally Support Multiple Operations > > > Key: ATLAS-4068 > URL: https://issues.apache.org/jira/browse/ATLAS-4068 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Ashutosh Mestry >Assignee: Ashutosh Mestry >Priority: Major > Fix For: trunk > > Attachments: > ATLAS-4068-Support-simultaneous-export-and-import-op.patch > > > *Background* > Existing implementation for Export & Import REST APIs disallows more than one > operation at a time. > *Solution* > Loosen the criteria based on request: > * Export: _AtlasExportRequest_: _skipLineage_ and _replicatedTo_ is selected. > * Import: _AtlasImportRequest_: _replicatedFrom_ is selected. > Operations where these options are not specified, the existing behavior will > continue. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ATLAS-4068) Export/Import Service: Conditionally Support Multiple Operations
Ashutosh Mestry created ATLAS-4068: -- Summary: Export/Import Service: Conditionally Support Multiple Operations Key: ATLAS-4068 URL: https://issues.apache.org/jira/browse/ATLAS-4068 Project: Atlas Issue Type: Improvement Components: atlas-core Affects Versions: 2.1.0 Reporter: Ashutosh Mestry Assignee: Ashutosh Mestry Fix For: trunk *Background* Existing implementation for Export & Import REST APIs disallows more than one operation at a time. *Solution* Loosen the criteria based on request: * Export: _AtlasExportRequest_: _skipLineage_ and _replicatedTo_ is selected. * Import: _AtlasImportRequest_: _replicatedFrom_ is selected. Operations where these options are not specified, the existing behavior will continue. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (ATLAS-4047) Audits Framework: Handling Long List
[
https://issues.apache.org/jira/browse/ATLAS-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Mestry reopened ATLAS-4047:
> Audits Framework: Handling Long List
>
>
> Key: ATLAS-4047
> URL: https://issues.apache.org/jira/browse/ATLAS-4047
> Project: Atlas
> Issue Type: Bug
>Reporter: Ashutosh Mestry
>Assignee: Ashutosh Mestry
>Priority: Major
> Attachments: ATLAS-4047-AtlasAuditService-Handling-large-strings.patch
>
>
> *Background*
> Existing _AtlasAuditService_ implementation, accepts list of GUIDs (object
> ids) and attempts to store it in a vertex.
> *Steps to Duplicate*
> Setup:
> * Create an export payload with 50 tables each with about 50 columns.
> Steps to duplicate:
> * Attempt to import. Import succeeds.
> Expected result: Import completes successfully.
> Actual result: Following exception is seen within the log, import returns and
> error code:
> {code:java}
> at
> org.apache.atlas.GraphTransactionInterceptor.doCommitOrRollback(GraphTransactionInterceptor.java:177)
> at
> org.apache.atlas.GraphTransactionInterceptor.doCommitOrRollback(GraphTransactionInterceptor.java:177)
> at
> org.apache.atlas.GraphTransactionInterceptor.invoke(GraphTransactionInterceptor.java:116)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
> at
> org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:672)
> at
> org.apache.atlas.repository.store.graph.v2.AtlasEntityStoreV2$$EnhancerBySpringCGLIB$$f28200fc.createOrUpdate()
> at org.apache.atlas.repository.ogm.DataAccess.saveNoLoad(DataAccess.java:73)
> at
> org.apache.atlas.repository.audit.AtlasAuditService.save(AtlasAuditService.java:62)
> at
> org.apache.atlas.repository.audit.AtlasAuditService.add(AtlasAuditService.java:82)
> at
> org.apache.atlas.repository.audit.AtlasAuditService$$FastClassBySpringCGLIB$$2eddda47.invoke()
> at
> org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204){code}
> CC: [~mandar.ambaw...@freestoneinfotech.com]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (ATLAS-3032) Importing an entity with tag attribute of type float data type fails with an BigDecimal Exception
[
https://issues.apache.org/jira/browse/ATLAS-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Mestry resolved ATLAS-3032.
Resolution: Fixed
Fixed by: https://issues.apache.org/jira/browse/ATLAS-1876
> Importing an entity with tag attribute of type float data type fails with an
> BigDecimal Exception
> -
>
> Key: ATLAS-3032
> URL: https://issues.apache.org/jira/browse/ATLAS-3032
> Project: Atlas
> Issue Type: Bug
>Reporter: Kapildeo Nayak
>Assignee: Ashutosh Mestry
>Priority: Critical
> Attachments: ATLAS-3032.patch
>
>
> # In source cluster , created a tag with float attribute.
> 2. Created an hive_db and associated the tag to the hive_db with value
> "-3.4028235E+38"
> 3. On importing the exported hive_db entity, exception is seen and import
> fails.
> Following stack trace is generated:
> {code:java}
> org.apache.atlas.repository.graphdb.AtlasSchemaViolationException:
> com.thinkaurelius.titan.core.SchemaViolationException: Value [-3.4028235E+38]
> is not an instance of the expected data type for property key
> [tag_float.float] and cannot be converted. Expected: class java.lang.Float,
> found: class java.math.BigDecimal{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ATLAS-4048) Import Service: Entity Creation: Updates to Business Metadata
[ https://issues.apache.org/jira/browse/ATLAS-4048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17238443#comment-17238443 ] Ashutosh Mestry commented on ATLAS-4048: [~madhan] I have modified the steps. My earlier steps were inaccurate. > Import Service: Entity Creation: Updates to Business Metadata > - > > Key: ATLAS-4048 > URL: https://issues.apache.org/jira/browse/ATLAS-4048 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Umesh Padashetty >Assignee: Ashutosh Mestry >Priority: Major > Attachments: ATLAS-4048-Entity-Update-Businessmetada-changes.patch > > > *Background* > Business Metadata attributes provide a simple way to augment attributes to > existing entity types. > Updating Business Metadata for existing entities via REST APIs did not happen > as the changes were not tracked when considering entity for update. > *Steps to Duplicate* > Setup: > * Create a source cluster, say cl1. > * Create a destination cluster, say cl2. > Steps to duplicate: > # On cl1: Create Business Metadata say bm1. Add 3 attributes to an entity, > say t1. Perform export. > # On cl2: Use Import REST API, perform import on a different cluster, say > cl2. > # On cl1: Edit the entity t1. Remove 2 of the 3 BM attributes. Perform > export. > # On cl2: Perform import. > Expected results: On cl2: Entity t1 will have 1 business metadata attribute. > Actual result: On cl2, Entity t1 contains all 3 attributes from step 1. > *Root Cause* > During entity creation flow, the Business Metadata attributes are not > considered when detecting if entity has changes. > Solution: Add a step to detect changes to Business Metadata attributes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4048) Import Service: Entity Creation: Updates to Business Metadata
[ https://issues.apache.org/jira/browse/ATLAS-4048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry updated ATLAS-4048: --- Description: *Background* Business Metadata attributes provide a simple way to augment attributes to existing entity types. Updating Business Metadata for existing entities via REST APIs did not happen as the changes were not tracked when considering entity for update. *Steps to Duplicate* Setup: * Create a source cluster, say cl1. * Create a destination cluster, say cl2. Steps to duplicate: # On cl1: Create Business Metadata say bm1. Add 3 attributes to an entity, say t1. Perform export. # On cl2: Use Import REST API, perform import on a different cluster, say cl2. # On cl1: Edit the entity t1. Remove 2 of the 3 BM attributes. Perform export. # On cl2: Perform import. Expected results: On cl2: Entity t1 will have 1 business metadata attribute. Actual result: On cl2, Entity t1 contains all 3 attributes from step 1. *Root Cause* During entity creation flow, the Business Metadata attributes are not considered when detecting if entity has changes. Solution: Add a step to detect changes to Business Metadata attributes. was: *Background* Business Metadata attributes provide a simple way to augment attributes to existing entity types. Updating Business Metadata for existing entities via REST APIs did not happen as the changes were not tracked when considering entity for update. *Steps to Duplicate* Setup: * Create Business Metadata and add 3 attributes. * Create an exported ZIP file. Steps to duplicate: # Via REST API, create an entity with 3 Business Metadata attributes. # From the web UI, load the entity where attributes have been added. # Delete 1 attribute. # Use entity update REST API to update the entity from step 1. Expected results: Entity will have all 3 attributes. Actual result: Entity will contain only 2 attributes that were present in step 2. *Root Cause* During entity creation flow, the Business Metadata attributes are not considered when detecting if entity has changes. Solution: Add a step to detect changes to Business Metadata attributes. > Import Service: Entity Creation: Updates to Business Metadata > - > > Key: ATLAS-4048 > URL: https://issues.apache.org/jira/browse/ATLAS-4048 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Umesh Padashetty >Assignee: Ashutosh Mestry >Priority: Major > Attachments: ATLAS-4048-Entity-Update-Businessmetada-changes.patch > > > *Background* > Business Metadata attributes provide a simple way to augment attributes to > existing entity types. > Updating Business Metadata for existing entities via REST APIs did not happen > as the changes were not tracked when considering entity for update. > *Steps to Duplicate* > Setup: > * Create a source cluster, say cl1. > * Create a destination cluster, say cl2. > Steps to duplicate: > # On cl1: Create Business Metadata say bm1. Add 3 attributes to an entity, > say t1. Perform export. > # On cl2: Use Import REST API, perform import on a different cluster, say > cl2. > # On cl1: Edit the entity t1. Remove 2 of the 3 BM attributes. Perform > export. > # On cl2: Perform import. > Expected results: On cl2: Entity t1 will have 1 business metadata attribute. > Actual result: On cl2, Entity t1 contains all 3 attributes from step 1. > *Root Cause* > During entity creation flow, the Business Metadata attributes are not > considered when detecting if entity has changes. > Solution: Add a step to detect changes to Business Metadata attributes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4048) Import Service: Entity Creation: Updates to Business Metadata
[ https://issues.apache.org/jira/browse/ATLAS-4048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry updated ATLAS-4048: --- Summary: Import Service: Entity Creation: Updates to Business Metadata (was: Entity Creation: Updates to Business Metadata) > Import Service: Entity Creation: Updates to Business Metadata > - > > Key: ATLAS-4048 > URL: https://issues.apache.org/jira/browse/ATLAS-4048 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Umesh Padashetty >Assignee: Ashutosh Mestry >Priority: Major > Attachments: ATLAS-4048-Entity-Update-Businessmetada-changes.patch > > > *Background* > Business Metadata attributes provide a simple way to augment attributes to > existing entity types. > Updating Business Metadata for existing entities via REST APIs did not happen > as the changes were not tracked when considering entity for update. > *Steps to Duplicate* > Setup: > * Create Business Metadata and add 3 attributes. > * Create an exported ZIP file. > Steps to duplicate: > # Via REST API, create an entity with 3 Business Metadata attributes. > # From the web UI, load the entity where attributes have been added. > # Delete 1 attribute. > # Use entity update REST API to update the entity from step 1. > Expected results: Entity will have all 3 attributes. > Actual result: Entity will contain only 2 attributes that were present in > step 2. > *Root Cause* > During entity creation flow, the Business Metadata attributes are not > considered when detecting if entity has changes. > Solution: Add a step to detect changes to Business Metadata attributes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4048) Entity Creation: Updates to Business Metadata
[ https://issues.apache.org/jira/browse/ATLAS-4048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry updated ATLAS-4048: --- Description: *Background* Business Metadata attributes provide a simple way to augment attributes to existing entity types. Updating Business Metadata for existing entities via REST APIs did not happen as the changes were not tracked when considering entity for update. *Steps to Duplicate* Setup: * Create Business Metadata and add 3 attributes. * Create an exported ZIP file. Steps to duplicate: # Via REST API, create an entity with 3 Business Metadata attributes. # From the web UI, load the entity where attributes have been added. # Delete 1 attribute. # Use entity update REST API to update the entity from step 1. Expected results: Entity will have all 3 attributes. Actual result: Entity will contain only 2 attributes that were present in step 2. *Root Cause* During entity creation flow, the Business Metadata attributes are not considered when detecting if entity has changes. Solution: Add a step to detect changes to Business Metadata attributes. was: *Background* Business Metadata attributes provide a simple way to augment attributes to existing entity types. Updating Business Metadata for existing entities via REST APIs did not happen as the changes were not tracked when considering entity for update. *Steps to Duplicate* Setup: * Create Business Metadata and add 3 attributes. Steps to duplicate: # Via REST API, create an entity with 3 Business Metadata attributes. # From the web UI, load the entity where attributes have been added. # Delete 1 attribute. # Use entity update REST API to update the entity from step 1. Expected results: Entity will have all 3 attributes. Actual result: Entity will contain only 2 attributes that were present in step 2. *Root Cause* During entity creation flow, the Business Metadata attributes are not considered when detecting if entity has changes. Solution: Add a step to detect changes to Business Metadata attributes. > Entity Creation: Updates to Business Metadata > - > > Key: ATLAS-4048 > URL: https://issues.apache.org/jira/browse/ATLAS-4048 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Umesh Padashetty >Assignee: Ashutosh Mestry >Priority: Major > Attachments: ATLAS-4048-Entity-Update-Businessmetada-changes.patch > > > *Background* > Business Metadata attributes provide a simple way to augment attributes to > existing entity types. > Updating Business Metadata for existing entities via REST APIs did not happen > as the changes were not tracked when considering entity for update. > *Steps to Duplicate* > Setup: > * Create Business Metadata and add 3 attributes. > * Create an exported ZIP file. > Steps to duplicate: > # Via REST API, create an entity with 3 Business Metadata attributes. > # From the web UI, load the entity where attributes have been added. > # Delete 1 attribute. > # Use entity update REST API to update the entity from step 1. > Expected results: Entity will have all 3 attributes. > Actual result: Entity will contain only 2 attributes that were present in > step 2. > *Root Cause* > During entity creation flow, the Business Metadata attributes are not > considered when detecting if entity has changes. > Solution: Add a step to detect changes to Business Metadata attributes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4048) Entity Creation: Updates to Business Metadata
[ https://issues.apache.org/jira/browse/ATLAS-4048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry updated ATLAS-4048: --- Description: *Background* Business Metadata attributes provide a simple way to augment attributes to existing entity types. Updating Business Metadata for existing entities via REST APIs did not happen as the changes were not tracked when considering entity for update. *Steps to Duplicate* Setup: * Create Business Metadata and add 3 attributes. Steps to duplicate: # Via REST API, create an entity with 3 Business Metadata attributes. # From the web UI, load the entity where attributes have been added. # Delete 1 attribute. # Use entity update REST API to update the entity from step 1. Expected results: Entity will have all 3 attributes. Actual result: Entity will contain only 2 attributes that were present in step 2. *Root Cause* During entity creation flow, the Business Metadata attributes are not considered when detecting if entity has changes. Solution: Add a step to detect changes to Business Metadata attributes. was: *Background* Business Metadata attributes provide a simple way to augment attributes to existing entity types. Updating Business Metadata for existing entities via REST APIs did not happen as the changes were not tracked when considering entity for update. *Steps to Duplicate* Setup: * Create Business Metadata and add 3 attributes. Steps to duplicate: # Via REST API, create an entity with 3 Business Metadata attributes. # From the web UI, load the entity where attributes have been added. # Delete 1 attribute. # Use entity update REST API to update the entity from step 1. Expected results: Entity will have all 3 attributes. Actual result: Entity will contain only 2 attributes that were present in step 2. > Entity Creation: Updates to Business Metadata > - > > Key: ATLAS-4048 > URL: https://issues.apache.org/jira/browse/ATLAS-4048 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Umesh Padashetty >Assignee: Ashutosh Mestry >Priority: Major > Attachments: ATLAS-4048-Entity-Update-Businessmetada-changes.patch > > > *Background* > Business Metadata attributes provide a simple way to augment attributes to > existing entity types. > Updating Business Metadata for existing entities via REST APIs did not happen > as the changes were not tracked when considering entity for update. > *Steps to Duplicate* > Setup: > * Create Business Metadata and add 3 attributes. > Steps to duplicate: > # Via REST API, create an entity with 3 Business Metadata attributes. > # From the web UI, load the entity where attributes have been added. > # Delete 1 attribute. > # Use entity update REST API to update the entity from step 1. > Expected results: Entity will have all 3 attributes. > Actual result: Entity will contain only 2 attributes that were present in > step 2. > *Root Cause* > During entity creation flow, the Business Metadata attributes are not > considered when detecting if entity has changes. > Solution: Add a step to detect changes to Business Metadata attributes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4048) Entity Creation: Updates to Business Metadata
[ https://issues.apache.org/jira/browse/ATLAS-4048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry updated ATLAS-4048: --- Description: *Background* Business Metadata attributes provide a simple way to augment attributes to existing entity types. Updating Business Metadata for existing entities via REST APIs did not happen as the changes were not tracked when considering entity for update. *Steps to Duplicate* Setup: * Create Business Metadata and add 3 attributes. Steps to duplicate: # Via REST API, create an entity with 3 Business Metadata attributes. # From the web UI, load the entity where attributes have been added. # Delete 1 attribute. # Use entity update REST API to update the entity from step 1. Expected results: Entity will have all 3 attributes. Actual result: Entity will contain only 2 attributes that were present in step 2. > Entity Creation: Updates to Business Metadata > - > > Key: ATLAS-4048 > URL: https://issues.apache.org/jira/browse/ATLAS-4048 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Umesh Padashetty >Assignee: Ashutosh Mestry >Priority: Major > Attachments: ATLAS-4048-Entity-Update-Businessmetada-changes.patch > > > *Background* > Business Metadata attributes provide a simple way to augment attributes to > existing entity types. > Updating Business Metadata for existing entities via REST APIs did not happen > as the changes were not tracked when considering entity for update. > *Steps to Duplicate* > Setup: > * Create Business Metadata and add 3 attributes. > Steps to duplicate: > # Via REST API, create an entity with 3 Business Metadata attributes. > # From the web UI, load the entity where attributes have been added. > # Delete 1 attribute. > # Use entity update REST API to update the entity from step 1. > Expected results: Entity will have all 3 attributes. > Actual result: Entity will contain only 2 attributes that were present in > step 2. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4048) Entity Creation: Updates to Business Metadata
[ https://issues.apache.org/jira/browse/ATLAS-4048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry updated ATLAS-4048: --- Priority: Major (was: Blocker) > Entity Creation: Updates to Business Metadata > - > > Key: ATLAS-4048 > URL: https://issues.apache.org/jira/browse/ATLAS-4048 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Umesh Padashetty >Assignee: Ashutosh Mestry >Priority: Major > Attachments: ATLAS-4048-Entity-Update-Businessmetada-changes.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4048) Entity Creation: Updates to Business Metadata
[ https://issues.apache.org/jira/browse/ATLAS-4048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry updated ATLAS-4048: --- Summary: Entity Creation: Updates to Business Metadata (was: Issue with Business Metadata in Export/Import APIs) > Entity Creation: Updates to Business Metadata > - > > Key: ATLAS-4048 > URL: https://issues.apache.org/jira/browse/ATLAS-4048 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Umesh Padashetty >Assignee: Ashutosh Mestry >Priority: Blocker > Attachments: ATLAS-4048-Entity-Update-Businessmetada-changes.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4048) Issue with Business Metadata in Export/Import APIs
[ https://issues.apache.org/jira/browse/ATLAS-4048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry updated ATLAS-4048: --- Attachment: ATLAS-4048-Entity-Update-Businessmetada-changes.patch > Issue with Business Metadata in Export/Import APIs > -- > > Key: ATLAS-4048 > URL: https://issues.apache.org/jira/browse/ATLAS-4048 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Umesh Padashetty >Assignee: Ashutosh Mestry >Priority: Blocker > Attachments: ATLAS-4048-Entity-Update-Businessmetada-changes.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (ATLAS-4048) Issue with Business Metadata in Export/Import APIs
[ https://issues.apache.org/jira/browse/ATLAS-4048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry reassigned ATLAS-4048: -- Assignee: Ashutosh Mestry > Issue with Business Metadata in Export/Import APIs > -- > > Key: ATLAS-4048 > URL: https://issues.apache.org/jira/browse/ATLAS-4048 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Umesh Padashetty >Assignee: Ashutosh Mestry >Priority: Blocker > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4047) Audits Framework: Handling Large Object List
[
https://issues.apache.org/jira/browse/ATLAS-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Mestry updated ATLAS-4047:
---
Attachment: ATLAS-4047-AtlasAuditService-Handling-large-strings.patch
> Audits Framework: Handling Large Object List
>
>
> Key: ATLAS-4047
> URL: https://issues.apache.org/jira/browse/ATLAS-4047
> Project: Atlas
> Issue Type: Bug
>Reporter: Ashutosh Mestry
>Assignee: Ashutosh Mestry
>Priority: Major
> Attachments: ATLAS-4047-AtlasAuditService-Handling-large-strings.patch
>
>
> *Background*
> Existing _AtlasAuditService_ implementation, accepts list of GUIDs (object
> ids) and attempts to store it in a vertex.
> *Steps to Duplicate*
> Setup:
> * Create an export payload with 50 tables each with about 50 columns.
> Steps to duplicate:
> * Attempt to import. Import succeeds.
> Expected result: Import completes successfully.
> Actual result: Following exception is seen within the log, import returns and
> error code:
> {code:java}
> at
> org.apache.atlas.GraphTransactionInterceptor.doCommitOrRollback(GraphTransactionInterceptor.java:177)
> at
> org.apache.atlas.GraphTransactionInterceptor.doCommitOrRollback(GraphTransactionInterceptor.java:177)
> at
> org.apache.atlas.GraphTransactionInterceptor.invoke(GraphTransactionInterceptor.java:116)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
> at
> org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:672)
> at
> org.apache.atlas.repository.store.graph.v2.AtlasEntityStoreV2$$EnhancerBySpringCGLIB$$f28200fc.createOrUpdate()
> at org.apache.atlas.repository.ogm.DataAccess.saveNoLoad(DataAccess.java:73)
> at
> org.apache.atlas.repository.audit.AtlasAuditService.save(AtlasAuditService.java:62)
> at
> org.apache.atlas.repository.audit.AtlasAuditService.add(AtlasAuditService.java:82)
> at
> org.apache.atlas.repository.audit.AtlasAuditService$$FastClassBySpringCGLIB$$2eddda47.invoke()
> at
> org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204){code}
> CC: [~mandar.ambaw...@freestoneinfotech.com]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ATLAS-4047) Audits Framework: Handling Long List
[
https://issues.apache.org/jira/browse/ATLAS-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Mestry updated ATLAS-4047:
---
Summary: Audits Framework: Handling Long List (was: Audits Framework:
Handling Large Object List)
> Audits Framework: Handling Long List
>
>
> Key: ATLAS-4047
> URL: https://issues.apache.org/jira/browse/ATLAS-4047
> Project: Atlas
> Issue Type: Bug
>Reporter: Ashutosh Mestry
>Assignee: Ashutosh Mestry
>Priority: Major
> Attachments: ATLAS-4047-AtlasAuditService-Handling-large-strings.patch
>
>
> *Background*
> Existing _AtlasAuditService_ implementation, accepts list of GUIDs (object
> ids) and attempts to store it in a vertex.
> *Steps to Duplicate*
> Setup:
> * Create an export payload with 50 tables each with about 50 columns.
> Steps to duplicate:
> * Attempt to import. Import succeeds.
> Expected result: Import completes successfully.
> Actual result: Following exception is seen within the log, import returns and
> error code:
> {code:java}
> at
> org.apache.atlas.GraphTransactionInterceptor.doCommitOrRollback(GraphTransactionInterceptor.java:177)
> at
> org.apache.atlas.GraphTransactionInterceptor.doCommitOrRollback(GraphTransactionInterceptor.java:177)
> at
> org.apache.atlas.GraphTransactionInterceptor.invoke(GraphTransactionInterceptor.java:116)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
> at
> org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:672)
> at
> org.apache.atlas.repository.store.graph.v2.AtlasEntityStoreV2$$EnhancerBySpringCGLIB$$f28200fc.createOrUpdate()
> at org.apache.atlas.repository.ogm.DataAccess.saveNoLoad(DataAccess.java:73)
> at
> org.apache.atlas.repository.audit.AtlasAuditService.save(AtlasAuditService.java:62)
> at
> org.apache.atlas.repository.audit.AtlasAuditService.add(AtlasAuditService.java:82)
> at
> org.apache.atlas.repository.audit.AtlasAuditService$$FastClassBySpringCGLIB$$2eddda47.invoke()
> at
> org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204){code}
> CC: [~mandar.ambaw...@freestoneinfotech.com]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ATLAS-4047) Audits Framework: Handling Large Object List
[
https://issues.apache.org/jira/browse/ATLAS-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Mestry updated ATLAS-4047:
---
Description:
*Background*
Existing _AtlasAuditService_ implementation, accepts list of GUIDs (object ids)
and attempts to store it in a vertex.
*Steps to Duplicate*
Setup:
* Create an export payload with 50 tables each with about 50 columns.
Steps to duplicate:
* Attempt to import. Import succeeds.
Expected result: Import completes successfully.
Actual result: Following exception is seen within the log, import returns and
error code:
{code:java}
at
org.apache.atlas.GraphTransactionInterceptor.doCommitOrRollback(GraphTransactionInterceptor.java:177)
at
org.apache.atlas.GraphTransactionInterceptor.doCommitOrRollback(GraphTransactionInterceptor.java:177)
at
org.apache.atlas.GraphTransactionInterceptor.invoke(GraphTransactionInterceptor.java:116)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at
org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:672)
at
org.apache.atlas.repository.store.graph.v2.AtlasEntityStoreV2$$EnhancerBySpringCGLIB$$f28200fc.createOrUpdate()
at org.apache.atlas.repository.ogm.DataAccess.saveNoLoad(DataAccess.java:73)
at
org.apache.atlas.repository.audit.AtlasAuditService.save(AtlasAuditService.java:62)
at
org.apache.atlas.repository.audit.AtlasAuditService.add(AtlasAuditService.java:82)
at
org.apache.atlas.repository.audit.AtlasAuditService$$FastClassBySpringCGLIB$$2eddda47.invoke()
at
org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204){code}
CC: [~mandar.ambaw...@freestoneinfotech.com]
was:
*Background*
Existing _AtlasAuditService_ implementation, accepts list of GUIDs (object ids)
and attempts to store it in a vertex.
*Steps to Duplicate*
Setup:
* Create an export payload with 50 tables each with about 50 columns.
Steps to duplicate:
* Attempt to import. Import succeeds.
Expected result: Import completes successfully.
Actual result: Following exception is seen within the log, import returns and
error code:
{code:java}
at
org.apache.atlas.GraphTransactionInterceptor.doCommitOrRollback(GraphTransactionInterceptor.java:177)
at
org.apache.atlas.GraphTransactionInterceptor.doCommitOrRollback(GraphTransactionInterceptor.java:177)
at
org.apache.atlas.GraphTransactionInterceptor.invoke(GraphTransactionInterceptor.java:116)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at
org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:672)
at
org.apache.atlas.repository.store.graph.v2.AtlasEntityStoreV2$$EnhancerBySpringCGLIB$$f28200fc.createOrUpdate()
at org.apache.atlas.repository.ogm.DataAccess.saveNoLoad(DataAccess.java:73)
at
org.apache.atlas.repository.audit.AtlasAuditService.save(AtlasAuditService.java:62)
at
org.apache.atlas.repository.audit.AtlasAuditService.add(AtlasAuditService.java:82)
at
org.apache.atlas.repository.audit.AtlasAuditService$$FastClassBySpringCGLIB$$2eddda47.invoke()
at
org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204){code}
> Audits Framework: Handling Large Object List
>
>
> Key: ATLAS-4047
> URL: https://issues.apache.org/jira/browse/ATLAS-4047
> Project: Atlas
> Issue Type: Bug
>Reporter: Ashutosh Mestry
>Assignee: Ashutosh Mestry
>Priority: Major
>
> *Background*
> Existing _AtlasAuditService_ implementation, accepts list of GUIDs (object
> ids) and attempts to store it in a vertex.
> *Steps to Duplicate*
> Setup:
> * Create an export payload with 50 tables each with about 50 columns.
> Steps to duplicate:
> * Attempt to import. Import succeeds.
> Expected result: Import completes successfully.
> Actual result: Following exception is seen within the log, import returns and
> error code:
> {code:java}
> at
> org.apache.atlas.GraphTransactionInterceptor.doCommitOrRollback(GraphTransactionInterceptor.java:177)
> at
> org.apache.atlas.GraphTransactionInterceptor.doCommitOrRollback(GraphTransactionInterceptor.java:177)
> at
> org.apache.atlas.GraphTransactionInterceptor.invoke(GraphTransactionInterceptor.java:116)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
> at
> org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:672)
> at
> org.apache.atlas.repository.store.graph.v2.AtlasEntityStoreV2$$EnhancerBySpringCGLIB$$f28200fc.createOrUpdate()
> at org.apache.atlas.repository.ogm.DataAccess.saveNoLoad(DataAccess.java:73)
> at
>
[jira] [Created] (ATLAS-4047) Audits Framework: Handling Large Object List
Ashutosh Mestry created ATLAS-4047:
--
Summary: Audits Framework: Handling Large Object List
Key: ATLAS-4047
URL: https://issues.apache.org/jira/browse/ATLAS-4047
Project: Atlas
Issue Type: Bug
Reporter: Ashutosh Mestry
Assignee: Ashutosh Mestry
*Background*
Existing _AtlasAuditService_ implementation, accepts list of GUIDs (object ids)
and attempts to store it in a vertex.
*Steps to Duplicate*
Setup:
* Create an export payload with 50 tables each with about 50 columns.
Steps to duplicate:
* Attempt to import. Import succeeds.
Expected result: Import completes successfully.
Actual result: Following exception is seen within the log, import returns and
error code:
{code:java}
at
org.apache.atlas.GraphTransactionInterceptor.doCommitOrRollback(GraphTransactionInterceptor.java:177)
at
org.apache.atlas.GraphTransactionInterceptor.doCommitOrRollback(GraphTransactionInterceptor.java:177)
at
org.apache.atlas.GraphTransactionInterceptor.invoke(GraphTransactionInterceptor.java:116)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at
org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:672)
at
org.apache.atlas.repository.store.graph.v2.AtlasEntityStoreV2$$EnhancerBySpringCGLIB$$f28200fc.createOrUpdate()
at org.apache.atlas.repository.ogm.DataAccess.saveNoLoad(DataAccess.java:73)
at
org.apache.atlas.repository.audit.AtlasAuditService.save(AtlasAuditService.java:62)
at
org.apache.atlas.repository.audit.AtlasAuditService.add(AtlasAuditService.java:82)
at
org.apache.atlas.repository.audit.AtlasAuditService$$FastClassBySpringCGLIB$$2eddda47.invoke()
at
org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ATLAS-4024) Atlas terms are not imported to target cluster using Atlas export/import API
[
https://issues.apache.org/jira/browse/ATLAS-4024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Mestry updated ATLAS-4024:
---
Attachment: ATLAS-4024-Terms-not-getting-exported-in-incremental.patch
> Atlas terms are not imported to target cluster using Atlas export/import API
>
>
> Key: ATLAS-4024
> URL: https://issues.apache.org/jira/browse/ATLAS-4024
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
>Affects Versions: 2.1.0
>Reporter: Umesh Padashetty
>Assignee: Ashutosh Mestry
>Priority: Major
> Attachments:
> ATLAS-4024-Terms-not-getting-exported-in-incremental.patch, destination.png,
> source.png
>
>
> Created a table with name "src_test5_t1" and assigned "test term" to
> it.Created a table with name "src_test5_t1" and assigned "test term" to it.
> Atlas export API was then run with below payload:
> {code:java}
> { "itemsToExport": [{ "typeName": "hive_db", "uniqueAttributes": {
> "qualifiedName": "src_test5@cm" } }], "options": { "replicatedTo": "cm",
> "fetchType": "incremental", "skipLineage": true, "changeMarker": 0 },
> "fetchTypeOptionValue": "incremental", "skipLineageOptionValue": true,
> "changeTokenFromOptions": 0} {code}
> Atlas Import was run with below payload:
> {code:java}
> { { options = { transformers: [{ "conditions": { "__entity": "topLevel: " },
> "action": { "__entity": "ADD_CLASSIFICATION: cm_replicated" } }, { "action":
> { "__entity.replicatedTo": "CLEAR:", "__entity.replicatedFrom": "CLEAR:" } },
> { "conditions": { "hive_db.clusterName": "EQUALS: cm" }, "action": {
> "hive_db.clusterName": "SET: cm" } }, { "conditions": { "hive_db.name":
> "EQUALS: src_test5" }, "action": { "hive_db.name": "SET: tgt_test5" } }, {
> "conditions": { "hive_db.location": "STARTS_WITH_IGNORE_CASE:
> hdfs://host:8020" }, "action": { "hive_db.location": "REPLACE_PREFIX: =
> :hdfs://host:8020=hdfs://host:8020" } }, { "conditions": {
> "hive_storagedesc.location": "STARTS_WITH_IGNORE_CASE: hdfs://host:8020" },
> "action": { "hive_storagedesc.location": "REPLACE_PREFIX: =
> :hdfs://host:8020=hdfs://host:8020" } }], replicatedFrom: cm, size: 1 } }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (ATLAS-4024) Atlas terms are not imported to target cluster using Atlas export/import API
[
https://issues.apache.org/jira/browse/ATLAS-4024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Mestry reassigned ATLAS-4024:
--
Assignee: Ashutosh Mestry
> Atlas terms are not imported to target cluster using Atlas export/import API
>
>
> Key: ATLAS-4024
> URL: https://issues.apache.org/jira/browse/ATLAS-4024
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
>Affects Versions: 2.1.0
>Reporter: Umesh Padashetty
>Assignee: Ashutosh Mestry
>Priority: Major
> Attachments:
> ATLAS-4024-Terms-not-getting-exported-in-incremental.patch, destination.png,
> source.png
>
>
> Created a table with name "src_test5_t1" and assigned "test term" to
> it.Created a table with name "src_test5_t1" and assigned "test term" to it.
> Atlas export API was then run with below payload:
> {code:java}
> { "itemsToExport": [{ "typeName": "hive_db", "uniqueAttributes": {
> "qualifiedName": "src_test5@cm" } }], "options": { "replicatedTo": "cm",
> "fetchType": "incremental", "skipLineage": true, "changeMarker": 0 },
> "fetchTypeOptionValue": "incremental", "skipLineageOptionValue": true,
> "changeTokenFromOptions": 0} {code}
> Atlas Import was run with below payload:
> {code:java}
> { { options = { transformers: [{ "conditions": { "__entity": "topLevel: " },
> "action": { "__entity": "ADD_CLASSIFICATION: cm_replicated" } }, { "action":
> { "__entity.replicatedTo": "CLEAR:", "__entity.replicatedFrom": "CLEAR:" } },
> { "conditions": { "hive_db.clusterName": "EQUALS: cm" }, "action": {
> "hive_db.clusterName": "SET: cm" } }, { "conditions": { "hive_db.name":
> "EQUALS: src_test5" }, "action": { "hive_db.name": "SET: tgt_test5" } }, {
> "conditions": { "hive_db.location": "STARTS_WITH_IGNORE_CASE:
> hdfs://host:8020" }, "action": { "hive_db.location": "REPLACE_PREFIX: =
> :hdfs://host:8020=hdfs://host:8020" } }, { "conditions": {
> "hive_storagedesc.location": "STARTS_WITH_IGNORE_CASE: hdfs://host:8020" },
> "action": { "hive_storagedesc.location": "REPLACE_PREFIX: =
> :hdfs://host:8020=hdfs://host:8020" } }], replicatedFrom: cm, size: 1 } }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (ATLAS-3941) NotificationHookConsumer: Reduce Retry Pause Interval
[ https://issues.apache.org/jira/browse/ATLAS-3941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry resolved ATLAS-3941. Resolution: Fixed > NotificationHookConsumer: Reduce Retry Pause Interval > - > > Key: ATLAS-3941 > URL: https://issues.apache.org/jira/browse/ATLAS-3941 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: trunk, 2.1.0 >Reporter: Ashutosh Mestry >Assignee: Ashutosh Mestry >Priority: Major > Fix For: trunk > > Attachments: > ATLAS-3941-Notification-Consumer-Reduce-pause-time-o.patch > > > *Background* > The retry logic introduced earlier, had a long wait time in case a concurrent > entity create was detected. This adversely affect ingest speed in the case > where there are a lot of errors in the data being ingested. > *Solution* > Reduce the wait time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ATLAS-4015) Indexes: Add Conditional Re-Indexing Patch
[ https://issues.apache.org/jira/browse/ATLAS-4015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry resolved ATLAS-4015. Resolution: Fixed > Indexes: Add Conditional Re-Indexing Patch > -- > > Key: ATLAS-4015 > URL: https://issues.apache.org/jira/browse/ATLAS-4015 > Project: Atlas > Issue Type: Improvement >Reporter: Ashutosh Mestry >Assignee: Ashutosh Mestry >Priority: Major > Attachments: ATLAS-4015-Add-Re-indexing-as-JAVA_PATCH-master.patch > > > *Background* > Every so often, we encounter cases where the Solr indexes are out of sync > with the data stored in the database. This leads to incorrect basic search > output. Additionally the facet numbers displayed are also incorrect. > *Solution* > It may help to make the ReIndex functionality to be made part of one of > JAVA_PATCH handlers that can be applied only when end user notices a problem > like described above. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4025) [Export/Import] Removing a classification from the entity in source cluster does not remove the classification in target cluster
[ https://issues.apache.org/jira/browse/ATLAS-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry updated ATLAS-4025: --- Attachment: ATLAS-4023-Import-Service-Label-and-Classification-e.patch > [Export/Import] Removing a classification from the entity in source cluster > does not remove the classification in target cluster > - > > Key: ATLAS-4025 > URL: https://issues.apache.org/jira/browse/ATLAS-4025 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Umesh Padashetty >Assignee: Ashutosh Mestry >Priority: Major > Attachments: > ATLAS-4023-Import-Service-Label-and-Classification-e.patch > > > Steps to reproduce: > # Created a table src_test8_t1 in the source cluster > # Assigned the classification "test_classification" to the src_test8_t1 > hive_table entity in source cluster > # Ran Atlas export in source, followed by import in target cluster > # Entity src_test8_t1 was created in target cluster with classification > "test_classification" assigned to it correctly > # Now, I removed the classification "test_classification" from the entity > src_test8_t1 in source cluster > # Ran Atlas export in source, followed by import in target cluster > # Now, the expectation is the classification "test_classification" is > removed in the target cluster from the entity src_test8_t1. But it is not, > classification is still assigned to the entity -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (ATLAS-4025) [Export/Import] Removing a classification from the entity in source cluster does not remove the classification in target cluster
[ https://issues.apache.org/jira/browse/ATLAS-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry reassigned ATLAS-4025: -- Assignee: Ashutosh Mestry > [Export/Import] Removing a classification from the entity in source cluster > does not remove the classification in target cluster > - > > Key: ATLAS-4025 > URL: https://issues.apache.org/jira/browse/ATLAS-4025 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Umesh Padashetty >Assignee: Ashutosh Mestry >Priority: Major > > Steps to reproduce: > # Created a table src_test8_t1 in the source cluster > # Assigned the classification "test_classification" to the src_test8_t1 > hive_table entity in source cluster > # Ran Atlas export in source, followed by import in target cluster > # Entity src_test8_t1 was created in target cluster with classification > "test_classification" assigned to it correctly > # Now, I removed the classification "test_classification" from the entity > src_test8_t1 in source cluster > # Ran Atlas export in source, followed by import in target cluster > # Now, the expectation is the classification "test_classification" is > removed in the target cluster from the entity src_test8_t1. But it is not, > classification is still assigned to the entity -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4023) Import Service: Labels Not Importing Correctly
[ https://issues.apache.org/jira/browse/ATLAS-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry updated ATLAS-4023: --- Summary: Import Service: Labels Not Importing Correctly (was: Import Service: Labels and Classifications Not Importing Correctly) > Import Service: Labels Not Importing Correctly > -- > > Key: ATLAS-4023 > URL: https://issues.apache.org/jira/browse/ATLAS-4023 > Project: Atlas > Issue Type: Bug >Reporter: Ashutosh Mestry >Assignee: Ashutosh Mestry >Priority: Major > Attachments: > ATLAS-4023-Import-Service-Label-and-Classification-e.patch > > > *Steps to Duplicate* > # Create dataset with _hive_db, hive_table,_ say _t1_. > # Add label to _hive_table,_ say _label1, label2_. > # Perform _Import_. Import succeeds with _hive_table_ getting assigned > labels _label1_ and _label2_. > # Modify _t1_ such that it has only _label1_. > # Perform Import. > Expected result: _t1_ should have _label1._ > Actual result: _t1_ has _label1_ and _label2._ > *Additional Information* > Similar exercise can be performed with classifications. > > Original reporter: [~umesh.padashetty] Thanks you! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4023) Import Service: Labels Not Importing Correctly
[ https://issues.apache.org/jira/browse/ATLAS-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry updated ATLAS-4023: --- Description: *Steps to Duplicate* # Create dataset with _hive_db, hive_table,_ say _t1_. # Add label to _hive_table,_ say _label1, label2_. # Perform _Import_. Import succeeds with _hive_table_ getting assigned labels _label1_ and _label2_. # Modify _t1_ such that it has only _label1_. # Perform Import. Expected result: _t1_ should have _label1._ Actual result: _t1_ has _label1_ and _label2._ Original reporter: [~umesh.padashetty] Thanks you! was: *Steps to Duplicate* # Create dataset with _hive_db, hive_table,_ say _t1_. # Add label to _hive_table,_ say _label1, label2_. # Perform _Import_. Import succeeds with _hive_table_ getting assigned labels _label1_ and _label2_. # Modify _t1_ such that it has only _label1_. # Perform Import. Expected result: _t1_ should have _label1._ Actual result: _t1_ has _label1_ and _label2._ *Additional Information* Similar exercise can be performed with classifications. Original reporter: [~umesh.padashetty] Thanks you! > Import Service: Labels Not Importing Correctly > -- > > Key: ATLAS-4023 > URL: https://issues.apache.org/jira/browse/ATLAS-4023 > Project: Atlas > Issue Type: Bug >Reporter: Ashutosh Mestry >Assignee: Ashutosh Mestry >Priority: Major > Attachments: > ATLAS-4023-Import-Service-Label-and-Classification-e.patch > > > *Steps to Duplicate* > # Create dataset with _hive_db, hive_table,_ say _t1_. > # Add label to _hive_table,_ say _label1, label2_. > # Perform _Import_. Import succeeds with _hive_table_ getting assigned > labels _label1_ and _label2_. > # Modify _t1_ such that it has only _label1_. > # Perform Import. > Expected result: _t1_ should have _label1._ > Actual result: _t1_ has _label1_ and _label2._ > Original reporter: [~umesh.padashetty] Thanks you! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4023) Import Service: Labels and Classifications Not Importing Correctly
[ https://issues.apache.org/jira/browse/ATLAS-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry updated ATLAS-4023: --- Description: *Steps to Duplicate* # Create dataset with _hive_db, hive_table,_ say _t1_. # Add label to _hive_table,_ say _label1, label2_. # Perform _Import_. Import succeeds with _hive_table_ getting assigned labels _label1_ and _label2_. # Modify _t1_ such that it has only _label1_. # Perform Import. Expected result: _t1_ should have _label1._ Actual result: _t1_ has _label1_ and _label2._ *Additional Information* Similar exercise can be performed with classifications. Original reporter: [~umesh.padashetty] Thanks you! was: *Steps to Duplicate* # Create dataset with _hive_db, hive_table,_ say _t1_. # Add label to _hive_table,_ say _label1, label2_. # Perform _Import_. Import succeeds with _hive_table_ getting assigned labels _label1_ and _label2_. # Modify _t1_ such that it has only _label1_. # Perform Import. Expected result: _t1_ should have _label1._ Actual result: _t1_ has _label1_ and _label2._ *Additional Information* Similar exercise can be performed with classifications. > Import Service: Labels and Classifications Not Importing Correctly > -- > > Key: ATLAS-4023 > URL: https://issues.apache.org/jira/browse/ATLAS-4023 > Project: Atlas > Issue Type: Bug >Reporter: Ashutosh Mestry >Assignee: Ashutosh Mestry >Priority: Major > Attachments: > ATLAS-4023-Import-Service-Label-and-Classification-e.patch > > > *Steps to Duplicate* > # Create dataset with _hive_db, hive_table,_ say _t1_. > # Add label to _hive_table,_ say _label1, label2_. > # Perform _Import_. Import succeeds with _hive_table_ getting assigned > labels _label1_ and _label2_. > # Modify _t1_ such that it has only _label1_. > # Perform Import. > Expected result: _t1_ should have _label1._ > Actual result: _t1_ has _label1_ and _label2._ > *Additional Information* > Similar exercise can be performed with classifications. > > Original reporter: [~umesh.padashetty] Thanks you! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-4023) Import Service: Labels and Classifications Not Importing Correctly
[ https://issues.apache.org/jira/browse/ATLAS-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17231167#comment-17231167 ] Ashutosh Mestry commented on ATLAS-4023: Pre-commit build: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/188/ > Import Service: Labels and Classifications Not Importing Correctly > -- > > Key: ATLAS-4023 > URL: https://issues.apache.org/jira/browse/ATLAS-4023 > Project: Atlas > Issue Type: Bug >Reporter: Ashutosh Mestry >Assignee: Ashutosh Mestry >Priority: Major > Attachments: > ATLAS-4023-Import-Service-Label-and-Classification-e.patch > > > *Steps to Duplicate* > # Create dataset with _hive_db, hive_table,_ say _t1_. > # Add label to _hive_table,_ say _label1, label2_. > # Perform _Import_. Import succeeds with _hive_table_ getting assigned > labels _label1_ and _label2_. > # Modify _t1_ such that it has only _label1_. > # Perform Import. > Expected result: _t1_ should have _label1._ > Actual result: _t1_ has _label1_ and _label2._ > *Additional Information* > Similar exercise can be performed with classifications. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3941) NotificationHookConsumer: Reduce Retry Pause Interval
[ https://issues.apache.org/jira/browse/ATLAS-3941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry updated ATLAS-3941: --- Attachment: (was: ATLAS-3941-NotificationHookConsumer-Reduce-wait-time.patch) > NotificationHookConsumer: Reduce Retry Pause Interval > - > > Key: ATLAS-3941 > URL: https://issues.apache.org/jira/browse/ATLAS-3941 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: trunk, 2.1.0 >Reporter: Ashutosh Mestry >Assignee: Ashutosh Mestry >Priority: Major > Fix For: trunk > > Attachments: > ATLAS-3941-Notification-Consumer-Reduce-pause-time-o.patch > > > *Background* > The retry logic introduced earlier, had a long wait time in case a concurrent > entity create was detected. This adversely affect ingest speed in the case > where there are a lot of errors in the data being ingested. > *Solution* > Reduce the wait time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3941) NotificationHookConsumer: Reduce Retry Pause Interval
[ https://issues.apache.org/jira/browse/ATLAS-3941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry updated ATLAS-3941: --- Attachment: ATLAS-3941-Notification-Consumer-Reduce-pause-time-o.patch > NotificationHookConsumer: Reduce Retry Pause Interval > - > > Key: ATLAS-3941 > URL: https://issues.apache.org/jira/browse/ATLAS-3941 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: trunk, 2.1.0 >Reporter: Ashutosh Mestry >Assignee: Ashutosh Mestry >Priority: Major > Fix For: trunk > > Attachments: > ATLAS-3941-Notification-Consumer-Reduce-pause-time-o.patch > > > *Background* > The retry logic introduced earlier, had a long wait time in case a concurrent > entity create was detected. This adversely affect ingest speed in the case > where there are a lot of errors in the data being ingested. > *Solution* > Reduce the wait time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4023) Import Service: Labels and Classifications Not Importing Correctly
[ https://issues.apache.org/jira/browse/ATLAS-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry updated ATLAS-4023: --- Attachment: ATLAS-4023-Import-Service-Label-and-Classification-e.patch > Import Service: Labels and Classifications Not Importing Correctly > -- > > Key: ATLAS-4023 > URL: https://issues.apache.org/jira/browse/ATLAS-4023 > Project: Atlas > Issue Type: Bug >Reporter: Ashutosh Mestry >Assignee: Ashutosh Mestry >Priority: Major > Attachments: > ATLAS-4023-Import-Service-Label-and-Classification-e.patch > > > *Steps to Duplicate* > # Create dataset with _hive_db, hive_table,_ say _t1_. > # Add label to _hive_table,_ say _label1, label2_. > # Perform _Import_. Import succeeds with _hive_table_ getting assigned > labels _label1_ and _label2_. > # Modify _t1_ such that it has only _label1_. > # Perform Import. > Expected result: _t1_ should have _label1._ > Actual result: _t1_ has _label1_ and _label2._ > *Additional Information* > Similar exercise can be performed with classifications. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ATLAS-4023) Import Service: Labels and Classifications Not Importing Correctly
Ashutosh Mestry created ATLAS-4023: -- Summary: Import Service: Labels and Classifications Not Importing Correctly Key: ATLAS-4023 URL: https://issues.apache.org/jira/browse/ATLAS-4023 Project: Atlas Issue Type: Bug Reporter: Ashutosh Mestry Assignee: Ashutosh Mestry *Steps to Duplicate* # Create dataset with _hive_db, hive_table,_ say _t1_. # Add label to _hive_table,_ say _label1, label2_. # Perform _Import_. Import succeeds with _hive_table_ getting assigned labels _label1_ and _label2_. # Modify _t1_ such that it has only _label1_. # Perform Import. Expected result: _t1_ should have _label1._ Actual result: _t1_ has _label1_ and _label2._ *Additional Information* Similar exercise can be performed with classifications. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4015) Indexes: Add Conditional Re-Indexing Patch
[ https://issues.apache.org/jira/browse/ATLAS-4015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry updated ATLAS-4015: --- Description: *Background* Every so often, we encounter cases where the Solr indexes are out of sync with the data stored in the database. This leads to incorrect basic search output. Additionally the facet numbers displayed are also incorrect. *Solution* It may help to make the ReIndex functionality to be made part of one of JAVA_PATCH handlers that can be applied only when end user notices a problem like described above. was: *Background* Every so often, we encounter cases where the Solr indexes are out of sync with the data stored in the database. The is leads basic search output. Additionally the facet numbers displayed are also incorrect. *Solution* It may help to make the ReIndex functionality to be made part of one of JAVA_PATCH handlers that can be applied only when end user notices a problem like described above. > Indexes: Add Conditional Re-Indexing Patch > -- > > Key: ATLAS-4015 > URL: https://issues.apache.org/jira/browse/ATLAS-4015 > Project: Atlas > Issue Type: Improvement >Reporter: Ashutosh Mestry >Assignee: Ashutosh Mestry >Priority: Major > Attachments: ATLAS-4015-Add-Re-indexing-as-JAVA_PATCH-master.patch > > > *Background* > Every so often, we encounter cases where the Solr indexes are out of sync > with the data stored in the database. This leads to incorrect basic search > output. Additionally the facet numbers displayed are also incorrect. > *Solution* > It may help to make the ReIndex functionality to be made part of one of > JAVA_PATCH handlers that can be applied only when end user notices a problem > like described above. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4015) Indexes: Add Conditional Re-Indexing Patch
[ https://issues.apache.org/jira/browse/ATLAS-4015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry updated ATLAS-4015: --- Attachment: (was: ATLAS-4015-Add-Re-indexing-as-JAVA_PATCH.patch) > Indexes: Add Conditional Re-Indexing Patch > -- > > Key: ATLAS-4015 > URL: https://issues.apache.org/jira/browse/ATLAS-4015 > Project: Atlas > Issue Type: Improvement >Reporter: Ashutosh Mestry >Assignee: Ashutosh Mestry >Priority: Major > Attachments: ATLAS-4015-Add-Re-indexing-as-JAVA_PATCH-master.patch > > > *Background* > Every so often, we encounter cases where the Solr indexes are out of sync > with the data stored in the database. The is leads basic search output. > Additionally the facet numbers displayed are also incorrect. > *Solution* > It may help to make the ReIndex functionality to be made part of one of > JAVA_PATCH handlers that can be applied only when end user notices a problem > like described above. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4015) Indexes: Add Conditional Re-Indexing Patch
[ https://issues.apache.org/jira/browse/ATLAS-4015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry updated ATLAS-4015: --- Attachment: ATLAS-4015-Add-Re-indexing-as-JAVA_PATCH-master.patch > Indexes: Add Conditional Re-Indexing Patch > -- > > Key: ATLAS-4015 > URL: https://issues.apache.org/jira/browse/ATLAS-4015 > Project: Atlas > Issue Type: Improvement >Reporter: Ashutosh Mestry >Assignee: Ashutosh Mestry >Priority: Major > Attachments: ATLAS-4015-Add-Re-indexing-as-JAVA_PATCH-master.patch > > > *Background* > Every so often, we encounter cases where the Solr indexes are out of sync > with the data stored in the database. The is leads basic search output. > Additionally the facet numbers displayed are also incorrect. > *Solution* > It may help to make the ReIndex functionality to be made part of one of > JAVA_PATCH handlers that can be applied only when end user notices a problem > like described above. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-4015) Indexes: Add Conditional Re-Indexing Patch
[ https://issues.apache.org/jira/browse/ATLAS-4015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17228711#comment-17228711 ] Ashutosh Mestry commented on ATLAS-4015: With the new implementation I am able to process 16M vertices in ~5 hrs. I will open RB item for regular review. > Indexes: Add Conditional Re-Indexing Patch > -- > > Key: ATLAS-4015 > URL: https://issues.apache.org/jira/browse/ATLAS-4015 > Project: Atlas > Issue Type: Improvement >Reporter: Ashutosh Mestry >Assignee: Ashutosh Mestry >Priority: Major > Attachments: ATLAS-4015-Add-Re-indexing-as-JAVA_PATCH.patch > > > *Background* > Every so often, we encounter cases where the Solr indexes are out of sync > with the data stored in the database. The is leads basic search output. > Additionally the facet numbers displayed are also incorrect. > *Solution* > It may help to make the ReIndex functionality to be made part of one of > JAVA_PATCH handlers that can be applied only when end user notices a problem > like described above. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4015) Indexes: Add Conditional Re-Indexing Patch
[ https://issues.apache.org/jira/browse/ATLAS-4015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry updated ATLAS-4015: --- Attachment: ATLAS-4015-Add-Re-indexing-as-JAVA_PATCH.patch > Indexes: Add Conditional Re-Indexing Patch > -- > > Key: ATLAS-4015 > URL: https://issues.apache.org/jira/browse/ATLAS-4015 > Project: Atlas > Issue Type: Improvement >Reporter: Ashutosh Mestry >Assignee: Ashutosh Mestry >Priority: Major > Attachments: ATLAS-4015-Add-Re-indexing-as-JAVA_PATCH.patch > > > *Background* > Every so often, we encounter cases where the Solr indexes are out of sync > with the data stored in the database. The is leads basic search output. > Additionally the facet numbers displayed are also incorrect. > *Solution* > It may help to make the ReIndex functionality to be made part of one of > JAVA_PATCH handlers that can be applied only when end user notices a problem > like described above. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4015) Indexes: Add Conditional Re-Indexing Patch
[ https://issues.apache.org/jira/browse/ATLAS-4015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry updated ATLAS-4015: --- Attachment: (was: ATLAS-4015-Add-Re-indexing-as-JAVA_PATCH.patch) > Indexes: Add Conditional Re-Indexing Patch > -- > > Key: ATLAS-4015 > URL: https://issues.apache.org/jira/browse/ATLAS-4015 > Project: Atlas > Issue Type: Improvement >Reporter: Ashutosh Mestry >Assignee: Ashutosh Mestry >Priority: Major > Attachments: ATLAS-4015-Add-Re-indexing-as-JAVA_PATCH.patch > > > *Background* > Every so often, we encounter cases where the Solr indexes are out of sync > with the data stored in the database. The is leads basic search output. > Additionally the facet numbers displayed are also incorrect. > *Solution* > It may help to make the ReIndex functionality to be made part of one of > JAVA_PATCH handlers that can be applied only when end user notices a problem > like described above. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4015) Indexes: Add Conditional Re-Indexing Patch
[ https://issues.apache.org/jira/browse/ATLAS-4015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry updated ATLAS-4015: --- Attachment: ATLAS-4015-Add-Re-indexing-as-JAVA_PATCH.patch > Indexes: Add Conditional Re-Indexing Patch > -- > > Key: ATLAS-4015 > URL: https://issues.apache.org/jira/browse/ATLAS-4015 > Project: Atlas > Issue Type: Improvement >Reporter: Ashutosh Mestry >Assignee: Ashutosh Mestry >Priority: Major > Attachments: ATLAS-4015-Add-Re-indexing-as-JAVA_PATCH.patch > > > *Background* > Every so often, we encounter cases where the Solr indexes are out of sync > with the data stored in the database. The is leads basic search output. > Additionally the facet numbers displayed are also incorrect. > *Solution* > It may help to make the ReIndex functionality to be made part of one of > JAVA_PATCH handlers that can be applied only when end user notices a problem > like described above. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ATLAS-4015) Indexes: Add Conditional Re-Indexing Patch
Ashutosh Mestry created ATLAS-4015: -- Summary: Indexes: Add Conditional Re-Indexing Patch Key: ATLAS-4015 URL: https://issues.apache.org/jira/browse/ATLAS-4015 Project: Atlas Issue Type: Improvement Reporter: Ashutosh Mestry Assignee: Ashutosh Mestry *Background* Every so often, we encounter cases where the Solr indexes are out of sync with the data stored in the database. The is leads basic search output. Additionally the facet numbers displayed are also incorrect. *Solution* It may help to make the ReIndex functionality to be made part of one of JAVA_PATCH handlers that can be applied only when end user notices a problem like described above. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-4010) Sort properties lexicographically in pom.xml for readability
[ https://issues.apache.org/jira/browse/ATLAS-4010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17223403#comment-17223403 ] Ashutosh Mestry commented on ATLAS-4010: PC build: [https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/168/] > Sort properties lexicographically in pom.xml for readability > > > Key: ATLAS-4010 > URL: https://issues.apache.org/jira/browse/ATLAS-4010 > Project: Atlas > Issue Type: Task >Reporter: Radhika Kundam >Assignee: Radhika Kundam >Priority: Trivial > > Properties are in random order under section in pom.xml > For better readability, need to sort properties in lexicographic order. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4007) Support for Business Metadata in Atlas Import API
[ https://issues.apache.org/jira/browse/ATLAS-4007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry updated ATLAS-4007: --- Attachment: ATLAS-4006-ATLAS-4007-Export-Import-Added-support-for-Business-.patch > Support for Business Metadata in Atlas Import API > - > > Key: ATLAS-4007 > URL: https://issues.apache.org/jira/browse/ATLAS-4007 > Project: Atlas > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Umesh Padashetty >Assignee: Ashutosh Mestry >Priority: Critical > Attachments: > ATLAS-4006-ATLAS-4007-Export-Import-Added-support-for-Business-.patch > > > Business metadata definition and Entity's Business metadata details are not > imported currently via Atlas Import API, we need to build support for this. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4006) Support for Business Metadata in Atlas Export API
[ https://issues.apache.org/jira/browse/ATLAS-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry updated ATLAS-4006: --- Attachment: ATLAS-4006-ATLAS-4007-Export-Import-Added-support-for-Business-.patch > Support for Business Metadata in Atlas Export API > - > > Key: ATLAS-4006 > URL: https://issues.apache.org/jira/browse/ATLAS-4006 > Project: Atlas > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Umesh Padashetty >Assignee: Ashutosh Mestry >Priority: Critical > Attachments: > ATLAS-4006-ATLAS-4007-Export-Import-Added-support-for-Business-.patch > > > Business metadata definition and Entity's Business metadata details are not > exported currently via Atlas Export API, we need to build support for this. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (ATLAS-4007) Support for Business Metadata in Atlas Import API
[ https://issues.apache.org/jira/browse/ATLAS-4007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry reassigned ATLAS-4007: -- Assignee: Ashutosh Mestry > Support for Business Metadata in Atlas Import API > - > > Key: ATLAS-4007 > URL: https://issues.apache.org/jira/browse/ATLAS-4007 > Project: Atlas > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Umesh Padashetty >Assignee: Ashutosh Mestry >Priority: Critical > > Business metadata definition and Entity's Business metadata details are not > imported currently via Atlas Import API, we need to build support for this. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (ATLAS-4006) Support for Business Metadata in Atlas Export API
[ https://issues.apache.org/jira/browse/ATLAS-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry reassigned ATLAS-4006: -- Assignee: Ashutosh Mestry > Support for Business Metadata in Atlas Export API > - > > Key: ATLAS-4006 > URL: https://issues.apache.org/jira/browse/ATLAS-4006 > Project: Atlas > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Umesh Padashetty >Assignee: Ashutosh Mestry >Priority: Critical > > Business metadata definition and Entity's Business metadata details are not > exported currently via Atlas Export API, we need to build support for this. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3977) Migration Script: Deleted entities in Atlas1.0 miss few attributes after migrated to Atlas2.0
[ https://issues.apache.org/jira/browse/ATLAS-3977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry updated ATLAS-3977: --- Attachment: ATLAS-3977-Deleted-entity-behavior-fix-during-migrat-Part-2.patch > Migration Script: Deleted entities in Atlas1.0 miss few attributes after > migrated to Atlas2.0 > - > > Key: ATLAS-3977 > URL: https://issues.apache.org/jira/browse/ATLAS-3977 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Reporter: Sharmadha S >Assignee: Ashutosh Mestry >Priority: Major > Attachments: ATLAS-3977-Addressed-patch-processing-for.patch, > ATLAS-3977-Deleted-entity-behavior-fix-during-migrat-Part-2.patch, > am2cm_deleted_active_entity.png > > > table6 is DELETED in Atlas1.0 and it is migrated to Atals2.0 > Attributes like name , db , owner are missing in DELETED entities whereas > they are populated correctly in ACTIVE entities. > Attaching the screenshot. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3989) Behavior change in Atlas API to get Atlas Server object during metadata replication
[
https://issues.apache.org/jira/browse/ATLAS-3989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Mestry updated ATLAS-3989:
---
Attachment: ATLAS-3989-Updated-Export-Import-Audits-Writer-to-us.patch
> Behavior change in Atlas API to get Atlas Server object during metadata
> replication
> ---
>
> Key: ATLAS-3989
> URL: https://issues.apache.org/jira/browse/ATLAS-3989
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
>Reporter: Ashutosh Mestry
>Assignee: Ashutosh Mestry
>Priority: Major
> Attachments:
> ATLAS-3989-Updated-Export-Import-Audits-Writer-to-us.patch
>
>
> There is a change being noticed in the Atlas server API used during atlas
> metadata replication in hive.
> Earlier the API: '/api/atlas/admin/server/cl1' where 'cl1' is the cluster
> name, used to work.
> Now, it doesn't work any more. This Api is used during replication to get
> the changeMarker info.
> Now the API works with some thing called server name which is 'default' by
> default.
> Also, now the prerequisite is to at least have one export or import API call
> in order to get this 'default' initialized.
> The change-marker is obtained prior to invoking export and hence the all
> policy is bound to fail unless we mandate calling Export or Import API
> outside hive replication once to get this object initialized, even if hive
> uses 'default' in the call.
>
> *Root cause*
> We switched from using
> {code:java}
> atlas.cluster.name{code}
> to
> {code:java}
> atlas.metadata.namespace{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ATLAS-3989) Behavior change in Atlas API to get Atlas Server object during metadata replication
[
https://issues.apache.org/jira/browse/ATLAS-3989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Mestry updated ATLAS-3989:
---
Attachment: (was:
ATLAS-3989-Updated-Export-Import-Audits-Writer-to-us.patch)
> Behavior change in Atlas API to get Atlas Server object during metadata
> replication
> ---
>
> Key: ATLAS-3989
> URL: https://issues.apache.org/jira/browse/ATLAS-3989
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
>Reporter: Ashutosh Mestry
>Assignee: Ashutosh Mestry
>Priority: Major
>
> There is a change being noticed in the Atlas server API used during atlas
> metadata replication in hive.
> Earlier the API: '/api/atlas/admin/server/cl1' where 'cl1' is the cluster
> name, used to work.
> Now, it doesn't work any more. This Api is used during replication to get
> the changeMarker info.
> Now the API works with some thing called server name which is 'default' by
> default.
> Also, now the prerequisite is to at least have one export or import API call
> in order to get this 'default' initialized.
> The change-marker is obtained prior to invoking export and hence the all
> policy is bound to fail unless we mandate calling Export or Import API
> outside hive replication once to get this object initialized, even if hive
> uses 'default' in the call.
>
> *Root cause*
> We switched from using
> {code:java}
> atlas.cluster.name{code}
> to
> {code:java}
> atlas.metadata.namespace{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ATLAS-3989) Behavior change in Atlas API to get Atlas Server object during metadata replication
[
https://issues.apache.org/jira/browse/ATLAS-3989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Mestry updated ATLAS-3989:
---
Description:
There is a change being noticed in the Atlas server API used during atlas
metadata replication in hive.
Earlier the API: '/api/atlas/admin/server/cl1' where 'cl1' is the cluster
name, used to work.
Now, it doesn't work any more. This Api is used during replication to get the
changeMarker info.
Now the API works with some thing called server name which is 'default' by
default.
Also, now the prerequisite is to at least have one export or import API call
in order to get this 'default' initialized.
The change-marker is obtained prior to invoking export and hence the all
policy is bound to fail unless we mandate calling Export or Import API outside
hive replication once to get this object initialized, even if hive uses
'default' in the call.
*Root cause*
We switched from using
{code:java}
atlas.cluster.name{code}
to
{code:java}
atlas.metadata.namespace{code}
was:
There is a change being noticed in the Atlas server API used during atlas
metadata replication in hive.
Earlier the API: '/api/atlas/admin/server/cl1' where 'cl1' is the cluster name,
used to work.
Now, it doesn't work any more. This Api is used during replication to get the
changeMarker info.
Now the API works with some thing called server name which is 'default' by
default.
Also, now the prerequisite is to at least have one export or import API call in
order to get this 'default' initialized.
The change-marker is obtained prior to invoking export and hence the all policy
is bound to fail unless we mandate calling Export or Import API outside hive
replication once to get this object initialized, even if hive uses 'default' in
the call.
> Behavior change in Atlas API to get Atlas Server object during metadata
> replication
> ---
>
> Key: ATLAS-3989
> URL: https://issues.apache.org/jira/browse/ATLAS-3989
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
>Reporter: Ashutosh Mestry
>Assignee: Ashutosh Mestry
>Priority: Major
> Attachments:
> ATLAS-3989-Updated-Export-Import-Audits-Writer-to-us.patch
>
>
> There is a change being noticed in the Atlas server API used during atlas
> metadata replication in hive.
> Earlier the API: '/api/atlas/admin/server/cl1' where 'cl1' is the cluster
> name, used to work.
> Now, it doesn't work any more. This Api is used during replication to get
> the changeMarker info.
> Now the API works with some thing called server name which is 'default' by
> default.
> Also, now the prerequisite is to at least have one export or import API call
> in order to get this 'default' initialized.
> The change-marker is obtained prior to invoking export and hence the all
> policy is bound to fail unless we mandate calling Export or Import API
> outside hive replication once to get this object initialized, even if hive
> uses 'default' in the call.
>
> *Root cause*
> We switched from using
> {code:java}
> atlas.cluster.name{code}
> to
> {code:java}
> atlas.metadata.namespace{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ATLAS-3989) Behavior change in Atlas API to get Atlas Server object during metadata replication
[ https://issues.apache.org/jira/browse/ATLAS-3989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry updated ATLAS-3989: --- Attachment: ATLAS-3989-Updated-Export-Import-Audits-Writer-to-us.patch > Behavior change in Atlas API to get Atlas Server object during metadata > replication > --- > > Key: ATLAS-3989 > URL: https://issues.apache.org/jira/browse/ATLAS-3989 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Reporter: Ashutosh Mestry >Assignee: Ashutosh Mestry >Priority: Major > Attachments: > ATLAS-3989-Updated-Export-Import-Audits-Writer-to-us.patch > > > There is a change being noticed in the Atlas server API used during atlas > metadata replication in hive. > Earlier the API: '/api/atlas/admin/server/cl1' where 'cl1' is the cluster > name, used to work. > Now, it doesn't work any more. This Api is used during replication to get the > changeMarker info. > Now the API works with some thing called server name which is 'default' by > default. > Also, now the prerequisite is to at least have one export or import API call > in order to get this 'default' initialized. > The change-marker is obtained prior to invoking export and hence the all > policy is bound to fail unless we mandate calling Export or Import API > outside hive replication once to get this object initialized, even if hive > uses 'default' in the call. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ATLAS-3989) Behavior change in Atlas API to get Atlas Server object during metadata replication
Ashutosh Mestry created ATLAS-3989: -- Summary: Behavior change in Atlas API to get Atlas Server object during metadata replication Key: ATLAS-3989 URL: https://issues.apache.org/jira/browse/ATLAS-3989 Project: Atlas Issue Type: Bug Components: atlas-core Reporter: Ashutosh Mestry Assignee: Ashutosh Mestry There is a change being noticed in the Atlas server API used during atlas metadata replication in hive. Earlier the API: '/api/atlas/admin/server/cl1' where 'cl1' is the cluster name, used to work. Now, it doesn't work any more. This Api is used during replication to get the changeMarker info. Now the API works with some thing called server name which is 'default' by default. Also, now the prerequisite is to at least have one export or import API call in order to get this 'default' initialized. The change-marker is obtained prior to invoking export and hence the all policy is bound to fail unless we mandate calling Export or Import API outside hive replication once to get this object initialized, even if hive uses 'default' in the call. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (ATLAS-3977) Migration Script: Deleted entities in Atlas1.0 miss few attributes after migrated to Atlas2.0
[ https://issues.apache.org/jira/browse/ATLAS-3977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17208482#comment-17208482 ] Ashutosh Mestry edited comment on ATLAS-3977 at 10/6/20, 9:41 PM: -- [~sarath] / [~nixon] / [~nikhilbonte] Can you please review. Pre-commit build: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/63/ was (Author: ashutoshm): [~sarath] / [~nixon] / [~nikhilbonte] Can you please review. > Migration Script: Deleted entities in Atlas1.0 miss few attributes after > migrated to Atlas2.0 > - > > Key: ATLAS-3977 > URL: https://issues.apache.org/jira/browse/ATLAS-3977 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Reporter: Sharmadha S >Assignee: Ashutosh Mestry >Priority: Major > Attachments: ATLAS-3977-Addressed-patch-processing-for.patch, > am2cm_deleted_active_entity.png > > > table6 is DELETED in Atlas1.0 and it is migrated to Atals2.0 > Attributes like name , db , owner are missing in DELETED entities whereas > they are populated correctly in ACTIVE entities. > Attaching the screenshot. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-3977) Migration Script: Deleted entities in Atlas1.0 miss few attributes after migrated to Atlas2.0
[ https://issues.apache.org/jira/browse/ATLAS-3977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17208482#comment-17208482 ] Ashutosh Mestry commented on ATLAS-3977: [~sarath] / [~nixon] / [~nikhilbonte] Can you please review. > Migration Script: Deleted entities in Atlas1.0 miss few attributes after > migrated to Atlas2.0 > - > > Key: ATLAS-3977 > URL: https://issues.apache.org/jira/browse/ATLAS-3977 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Reporter: Sharmadha S >Assignee: Ashutosh Mestry >Priority: Major > Attachments: ATLAS-3977-Addressed-patch-processing-for.patch, > am2cm_deleted_active_entity.png > > > table6 is DELETED in Atlas1.0 and it is migrated to Atals2.0 > Attributes like name , db , owner are missing in DELETED entities whereas > they are populated correctly in ACTIVE entities. > Attaching the screenshot. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (ATLAS-3977) Migration Script: Deleted entities in Atlas1.0 miss few attributes after migrated to Atlas2.0
[ https://issues.apache.org/jira/browse/ATLAS-3977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry reassigned ATLAS-3977: -- Attachment: ATLAS-3977-Addressed-patch-processing-for.patch Assignee: Ashutosh Mestry > Migration Script: Deleted entities in Atlas1.0 miss few attributes after > migrated to Atlas2.0 > - > > Key: ATLAS-3977 > URL: https://issues.apache.org/jira/browse/ATLAS-3977 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Reporter: Sharmadha S >Assignee: Ashutosh Mestry >Priority: Major > Attachments: ATLAS-3977-Addressed-patch-processing-for.patch, > am2cm_deleted_active_entity.png > > > table6 is DELETED in Atlas1.0 and it is migrated to Atals2.0 > Attributes like name , db , owner are missing in DELETED entities whereas > they are populated correctly in ACTIVE entities. > Attaching the screenshot. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ATLAS-3953) JSON Files from Export API with "?" char for string with special chars
[ https://issues.apache.org/jira/browse/ATLAS-3953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry resolved ATLAS-3953. Resolution: Fixed > JSON Files from Export API with "?" char for string with special chars > --- > > Key: ATLAS-3953 > URL: https://issues.apache.org/jira/browse/ATLAS-3953 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 > Environment: Apache Atlas 2.1.0 embedded HBASE and SOLR >Reporter: Carlos Alberto Rocha Cardoso >Assignee: Ashutosh Mestry >Priority: Major > Attachments: 9fdc3ad0-46c2-430a-89c4-4a751d31c064.json, > ATLAS-3953-Export-ZipSink-Specify-character-endcodin.patch, > Asset_Imported.PNG, AtlasServer.PNG, > a5c148bf-5ab6-4c49-853e-855842102128.json, atlas_export.zip, path.zip > > > The Export API returns a ZIP file with some JSON files describing Atlas > Entities and TypeDefs. > I am having an issue where some special chars in JSON are being replaced by > "?" chars. > An Entity name like "Distribuição" was exported in JSON file like > "Distribui??o". The special chars "çã" was replaced for the "??" chars. > I tried to change the exported JSON file encoding and the request header for > Export API but without success. > After analyzing the Atlas source code, especially the *splitAndWriteBytes* > method of the > *[ZipSink|https://github.com/apache/atlas/blob/cc601d7371fae1dbc16b55d1ca84f06b745700dc/repository/src/main/java/org/apache/atlas/repository/impexp/ZipSink.java] > class*, I thought if maybe the problem is because the *s.getBytes()* is > returning the JSON string to be written to ZIP with another encoding than > *UTF-8*, and maybe set the encode like *s.getBytes(StandardCharsets.UTF_8)* > could be a solution. > It's my first contact with the Atlas source code, and I'm not a JAVA > programmer, so it's only a guess. > I saw that it's possible to set the default to encode to the platform or JVM, > but how they said in this below discussion, perhaps this doesn't work > properly in all situations. > [https://stackoverflow.com/questions/361975/setting-the-default-java-character-encoding] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-3953) JSON Files from Export API with "?" char for string with special chars
[ https://issues.apache.org/jira/browse/ATLAS-3953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17204390#comment-17204390 ] Ashutosh Mestry commented on ATLAS-3953: Pre-commit build: [https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/33/] [~nikhilbonte] / [~nixon] / [~sarath]: If you could add +1 to this I can proceed with commit. > JSON Files from Export API with "?" char for string with special chars > --- > > Key: ATLAS-3953 > URL: https://issues.apache.org/jira/browse/ATLAS-3953 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 > Environment: Apache Atlas 2.1.0 embedded HBASE and SOLR >Reporter: Carlos Alberto Rocha Cardoso >Assignee: Ashutosh Mestry >Priority: Major > Attachments: 9fdc3ad0-46c2-430a-89c4-4a751d31c064.json, > ATLAS-3953-Export-ZipSink-Specify-character-endcodin.patch, > Asset_Imported.PNG, AtlasServer.PNG, > a5c148bf-5ab6-4c49-853e-855842102128.json, atlas_export.zip, path.zip > > > The Export API returns a ZIP file with some JSON files describing Atlas > Entities and TypeDefs. > I am having an issue where some special chars in JSON are being replaced by > "?" chars. > An Entity name like "Distribuição" was exported in JSON file like > "Distribui??o". The special chars "çã" was replaced for the "??" chars. > I tried to change the exported JSON file encoding and the request header for > Export API but without success. > After analyzing the Atlas source code, especially the *splitAndWriteBytes* > method of the > *[ZipSink|https://github.com/apache/atlas/blob/cc601d7371fae1dbc16b55d1ca84f06b745700dc/repository/src/main/java/org/apache/atlas/repository/impexp/ZipSink.java] > class*, I thought if maybe the problem is because the *s.getBytes()* is > returning the JSON string to be written to ZIP with another encoding than > *UTF-8*, and maybe set the encode like *s.getBytes(StandardCharsets.UTF_8)* > could be a solution. > It's my first contact with the Atlas source code, and I'm not a JAVA > programmer, so it's only a guess. > I saw that it's possible to set the default to encode to the platform or JVM, > but how they said in this below discussion, perhaps this doesn't work > properly in all situations. > [https://stackoverflow.com/questions/361975/setting-the-default-java-character-encoding] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-3953) JSON Files from Export API with "?" char for string with special chars
[ https://issues.apache.org/jira/browse/ATLAS-3953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17204389#comment-17204389 ] Ashutosh Mestry commented on ATLAS-3953: [~carlosrochacardoso] That is great news! I will proceed with next steps. Thanks for verifying this for me! I appreciate it. > JSON Files from Export API with "?" char for string with special chars > --- > > Key: ATLAS-3953 > URL: https://issues.apache.org/jira/browse/ATLAS-3953 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 > Environment: Apache Atlas 2.1.0 embedded HBASE and SOLR >Reporter: Carlos Alberto Rocha Cardoso >Assignee: Ashutosh Mestry >Priority: Major > Attachments: 9fdc3ad0-46c2-430a-89c4-4a751d31c064.json, > ATLAS-3953-Export-ZipSink-Specify-character-endcodin.patch, > Asset_Imported.PNG, AtlasServer.PNG, > a5c148bf-5ab6-4c49-853e-855842102128.json, atlas_export.zip, path.zip > > > The Export API returns a ZIP file with some JSON files describing Atlas > Entities and TypeDefs. > I am having an issue where some special chars in JSON are being replaced by > "?" chars. > An Entity name like "Distribuição" was exported in JSON file like > "Distribui??o". The special chars "çã" was replaced for the "??" chars. > I tried to change the exported JSON file encoding and the request header for > Export API but without success. > After analyzing the Atlas source code, especially the *splitAndWriteBytes* > method of the > *[ZipSink|https://github.com/apache/atlas/blob/cc601d7371fae1dbc16b55d1ca84f06b745700dc/repository/src/main/java/org/apache/atlas/repository/impexp/ZipSink.java] > class*, I thought if maybe the problem is because the *s.getBytes()* is > returning the JSON string to be written to ZIP with another encoding than > *UTF-8*, and maybe set the encode like *s.getBytes(StandardCharsets.UTF_8)* > could be a solution. > It's my first contact with the Atlas source code, and I'm not a JAVA > programmer, so it's only a guess. > I saw that it's possible to set the default to encode to the platform or JVM, > but how they said in this below discussion, perhaps this doesn't work > properly in all situations. > [https://stackoverflow.com/questions/361975/setting-the-default-java-character-encoding] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-3953) JSON Files from Export API with "?" char for string with special chars
[ https://issues.apache.org/jira/browse/ATLAS-3953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17201698#comment-17201698 ] Ashutosh Mestry commented on ATLAS-3953: Thanks for your interest. Please take a look at: [http://atlas.apache.org/#/BuildInstallation] This patch will apply to master: [https://github.com/apache/atlas.git] I will try at my end to duplicate the issue and then use the solution. > JSON Files from Export API with "?" char for string with special chars > --- > > Key: ATLAS-3953 > URL: https://issues.apache.org/jira/browse/ATLAS-3953 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 > Environment: Apache Atlas 2.1.0 embedded HBASE and SOLR >Reporter: Carlos Alberto Rocha Cardoso >Assignee: Ashutosh Mestry >Priority: Major > Attachments: 9fdc3ad0-46c2-430a-89c4-4a751d31c064.json, > ATLAS-3953-Export-ZipSink-Specify-character-endcodin.patch, AtlasServer.PNG, > a5c148bf-5ab6-4c49-853e-855842102128.json, path.zip > > > The Export API returns a ZIP file with some JSON files describing Atlas > Entities and TypeDefs. > I am having an issue where some special chars in JSON are being replaced by > "?" chars. > An Entity name like "Distribuição" was exported in JSON file like > "Distribui??o". The special chars "çã" was replaced for the "??" chars. > I tried to change the exported JSON file encoding and the request header for > Export API but without success. > After analyzing the Atlas source code, especially the *splitAndWriteBytes* > method of the > *[ZipSink|https://github.com/apache/atlas/blob/cc601d7371fae1dbc16b55d1ca84f06b745700dc/repository/src/main/java/org/apache/atlas/repository/impexp/ZipSink.java] > class*, I thought if maybe the problem is because the *s.getBytes()* is > returning the JSON string to be written to ZIP with another encoding than > *UTF-8*, and maybe set the encode like *s.getBytes(StandardCharsets.UTF_8)* > could be a solution. > It's my first contact with the Atlas source code, and I'm not a JAVA > programmer, so it's only a guess. > I saw that it's possible to set the default to encode to the platform or JVM, > but how they said in this below discussion, perhaps this doesn't work > properly in all situations. > [https://stackoverflow.com/questions/361975/setting-the-default-java-character-encoding] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3953) JSON Files from Export API with "?" char for string with special chars
[ https://issues.apache.org/jira/browse/ATLAS-3953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry updated ATLAS-3953: --- Attachment: ATLAS-3953-Export-ZipSink-Specify-character-endcodin.patch > JSON Files from Export API with "?" char for string with special chars > --- > > Key: ATLAS-3953 > URL: https://issues.apache.org/jira/browse/ATLAS-3953 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 > Environment: Apache Atlas 2.1.0 embedded HBASE and SOLR >Reporter: Carlos Alberto Rocha Cardoso >Assignee: Ashutosh Mestry >Priority: Major > Attachments: 9fdc3ad0-46c2-430a-89c4-4a751d31c064.json, > ATLAS-3953-Export-ZipSink-Specify-character-endcodin.patch, > a5c148bf-5ab6-4c49-853e-855842102128.json, path.zip > > > The Export API return a ZIP file with some JSON files describing Atlas > Entities and TypeDefs. > I am having a issue where some special chars in JSON are being replaced by > "?" chars. > An Entity name like "Distribuição" was exported in JSON file like > "Distribui??o". The special chars "çã" was replaced for the "??" chars. > I tried to change the exported JSON file encoding and the request header for > Export API, but without success. > After analyzing the Atlas source code, specialy the *splitAndWriteBytes* > method of the > *[ZipSink|https://github.com/apache/atlas/blob/cc601d7371fae1dbc16b55d1ca84f06b745700dc/repository/src/main/java/org/apache/atlas/repository/impexp/ZipSink.java] > class*, I tought if maybe the problem is because the *s.getBytes()* is > returning the JSON string to be written to ZIP with other encode than > *UTF-8*, and maybe set the encode like *s.getBytes(StandardCharsets.UTF_8)* > could be a solution. > Its my first contact with the Atlas source code, and I'm not a JAVA > programmer, so it's only a guess. > I saw that it's possible set the default encode to the plataform, or JVM, but > how they said in this below discussion, perhaps this doesn't work properly in > all situations. > [https://stackoverflow.com/questions/361975/setting-the-default-java-character-encoding] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-3953) JSON Files from Export API with "?" char for string with special chars
[ https://issues.apache.org/jira/browse/ATLAS-3953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17201000#comment-17201000 ] Ashutosh Mestry commented on ATLAS-3953: I have attached couple of things: * _path.zip_: Exported zip file that contains entity with special characters in it. * _a5c...json_: Entity represented within Atlas. I was not able to duplicate the problem. I also have a patch that specifies _UTF-8_ encoding when writing string to the ZIP stream. Would you be able to try out the patch? > JSON Files from Export API with "?" char for string with special chars > --- > > Key: ATLAS-3953 > URL: https://issues.apache.org/jira/browse/ATLAS-3953 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 > Environment: Apache Atlas 2.1.0 embedded HBASE and SOLR >Reporter: Carlos Alberto Rocha Cardoso >Assignee: Ashutosh Mestry >Priority: Major > Attachments: 9fdc3ad0-46c2-430a-89c4-4a751d31c064.json, > a5c148bf-5ab6-4c49-853e-855842102128.json, path.zip > > > The Export API return a ZIP file with some JSON files describing Atlas > Entities and TypeDefs. > I am having a issue where some special chars in JSON are being replaced by > "?" chars. > An Entity name like "Distribuição" was exported in JSON file like > "Distribui??o". The special chars "çã" was replaced for the "??" chars. > I tried to change the exported JSON file encoding and the request header for > Export API, but without success. > After analyzing the Atlas source code, specialy the *splitAndWriteBytes* > method of the > *[ZipSink|https://github.com/apache/atlas/blob/cc601d7371fae1dbc16b55d1ca84f06b745700dc/repository/src/main/java/org/apache/atlas/repository/impexp/ZipSink.java] > class*, I tought if maybe the problem is because the *s.getBytes()* is > returning the JSON string to be written to ZIP with other encode than > *UTF-8*, and maybe set the encode like *s.getBytes(StandardCharsets.UTF_8)* > could be a solution. > Its my first contact with the Atlas source code, and I'm not a JAVA > programmer, so it's only a guess. > I saw that it's possible set the default encode to the plataform, or JVM, but > how they said in this below discussion, perhaps this doesn't work properly in > all situations. > [https://stackoverflow.com/questions/361975/setting-the-default-java-character-encoding] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3953) JSON Files from Export API with "?" char for string with special chars
[ https://issues.apache.org/jira/browse/ATLAS-3953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry updated ATLAS-3953: --- Attachment: path.zip > JSON Files from Export API with "?" char for string with special chars > --- > > Key: ATLAS-3953 > URL: https://issues.apache.org/jira/browse/ATLAS-3953 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 > Environment: Apache Atlas 2.1.0 embedded HBASE and SOLR >Reporter: Carlos Alberto Rocha Cardoso >Assignee: Ashutosh Mestry >Priority: Major > Attachments: 9fdc3ad0-46c2-430a-89c4-4a751d31c064.json, > a5c148bf-5ab6-4c49-853e-855842102128.json, path.zip > > > The Export API return a ZIP file with some JSON files describing Atlas > Entities and TypeDefs. > I am having a issue where some special chars in JSON are being replaced by > "?" chars. > An Entity name like "Distribuição" was exported in JSON file like > "Distribui??o". The special chars "çã" was replaced for the "??" chars. > I tried to change the exported JSON file encoding and the request header for > Export API, but without success. > After analyzing the Atlas source code, specialy the *splitAndWriteBytes* > method of the > *[ZipSink|https://github.com/apache/atlas/blob/cc601d7371fae1dbc16b55d1ca84f06b745700dc/repository/src/main/java/org/apache/atlas/repository/impexp/ZipSink.java] > class*, I tought if maybe the problem is because the *s.getBytes()* is > returning the JSON string to be written to ZIP with other encode than > *UTF-8*, and maybe set the encode like *s.getBytes(StandardCharsets.UTF_8)* > could be a solution. > Its my first contact with the Atlas source code, and I'm not a JAVA > programmer, so it's only a guess. > I saw that it's possible set the default encode to the plataform, or JVM, but > how they said in this below discussion, perhaps this doesn't work properly in > all situations. > [https://stackoverflow.com/questions/361975/setting-the-default-java-character-encoding] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3953) JSON Files from Export API with "?" char for string with special chars
[ https://issues.apache.org/jira/browse/ATLAS-3953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry updated ATLAS-3953: --- Attachment: a5c148bf-5ab6-4c49-853e-855842102128.json > JSON Files from Export API with "?" char for string with special chars > --- > > Key: ATLAS-3953 > URL: https://issues.apache.org/jira/browse/ATLAS-3953 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 > Environment: Apache Atlas 2.1.0 embedded HBASE and SOLR >Reporter: Carlos Alberto Rocha Cardoso >Assignee: Ashutosh Mestry >Priority: Major > Attachments: 9fdc3ad0-46c2-430a-89c4-4a751d31c064.json, > a5c148bf-5ab6-4c49-853e-855842102128.json, path.zip > > > The Export API return a ZIP file with some JSON files describing Atlas > Entities and TypeDefs. > I am having a issue where some special chars in JSON are being replaced by > "?" chars. > An Entity name like "Distribuição" was exported in JSON file like > "Distribui??o". The special chars "çã" was replaced for the "??" chars. > I tried to change the exported JSON file encoding and the request header for > Export API, but without success. > After analyzing the Atlas source code, specialy the *splitAndWriteBytes* > method of the > *[ZipSink|https://github.com/apache/atlas/blob/cc601d7371fae1dbc16b55d1ca84f06b745700dc/repository/src/main/java/org/apache/atlas/repository/impexp/ZipSink.java] > class*, I tought if maybe the problem is because the *s.getBytes()* is > returning the JSON string to be written to ZIP with other encode than > *UTF-8*, and maybe set the encode like *s.getBytes(StandardCharsets.UTF_8)* > could be a solution. > Its my first contact with the Atlas source code, and I'm not a JAVA > programmer, so it's only a guess. > I saw that it's possible set the default encode to the plataform, or JVM, but > how they said in this below discussion, perhaps this doesn't work properly in > all situations. > [https://stackoverflow.com/questions/361975/setting-the-default-java-character-encoding] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-3953) JSON Files from Export API with "?" char for string with special chars
[ https://issues.apache.org/jira/browse/ATLAS-3953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200984#comment-17200984 ] Ashutosh Mestry commented on ATLAS-3953: [~carlosrochacardoso] Thanks for filing this JIRA. If you attach a sample, I can verify it with my fix. > JSON Files from Export API with "?" char for string with special chars > --- > > Key: ATLAS-3953 > URL: https://issues.apache.org/jira/browse/ATLAS-3953 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 > Environment: Apache Atlas 2.1.0 embedded HBASE and SOLR >Reporter: Carlos Alberto Rocha Cardoso >Assignee: Ashutosh Mestry >Priority: Major > > The Export API return a ZIP file with some JSON files describing Atlas > Entities and TypeDefs. > I am having a issue where some special chars in JSON are being replaced by > "?" chars. > An Entity name like "Distribuição" was exported in JSON file like > "Distribui??o". The special chars "çã" was replaced for the "??" chars. > I tried to change the exported JSON file encoding and the request header for > Export API, but without success. > After analyzing the Atlas source code, specialy the *splitAndWriteBytes* > method of the > *[ZipSink|https://github.com/apache/atlas/blob/cc601d7371fae1dbc16b55d1ca84f06b745700dc/repository/src/main/java/org/apache/atlas/repository/impexp/ZipSink.java] > class*, I tought if maybe the problem is because the *s.getBytes()* is > returning the JSON string to be written to ZIP with other encode than > *UTF-8*, and maybe set the encode like *s.getBytes(StandardCharsets.UTF_8)* > could be a solution. > Its my first contact with the Atlas source code, and I'm not a JAVA > programmer, so it's only a guess. > I saw that it's possible set the default encode to the plataform, or JVM, but > how they said in this below discussion, perhaps this doesn't work properly in > all situations. > [https://stackoverflow.com/questions/361975/setting-the-default-java-character-encoding] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3953) JSON Files from Export API with "?" char for string with special chars
[ https://issues.apache.org/jira/browse/ATLAS-3953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry updated ATLAS-3953: --- Priority: Major (was: Minor) > JSON Files from Export API with "?" char for string with special chars > --- > > Key: ATLAS-3953 > URL: https://issues.apache.org/jira/browse/ATLAS-3953 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 > Environment: Apache Atlas 2.1.0 embedded HBASE and SOLR >Reporter: Carlos Alberto Rocha Cardoso >Assignee: Ashutosh Mestry >Priority: Major > > The Export API return a ZIP file with some JSON files describing Atlas > Entities and TypeDefs. > I am having a issue where some special chars in JSON are being replaced by > "?" chars. > An Entity name like "Distribuição" was exported in JSON file like > "Distribui??o". The special chars "çã" was replaced for the "??" chars. > I tried to change the exported JSON file encoding and the request header for > Export API, but without success. > After analyzing the Atlas source code, specialy the *splitAndWriteBytes* > method of the > *[ZipSink|https://github.com/apache/atlas/blob/cc601d7371fae1dbc16b55d1ca84f06b745700dc/repository/src/main/java/org/apache/atlas/repository/impexp/ZipSink.java] > class*, I tought if maybe the problem is because the *s.getBytes()* is > returning the JSON string to be written to ZIP with other encode than > *UTF-8*, and maybe set the encode like *s.getBytes(StandardCharsets.UTF_8)* > could be a solution. > Its my first contact with the Atlas source code, and I'm not a JAVA > programmer, so it's only a guess. > I saw that it's possible set the default encode to the plataform, or JVM, but > how they said in this below discussion, perhaps this doesn't work properly in > all situations. > [https://stackoverflow.com/questions/361975/setting-the-default-java-character-encoding] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (ATLAS-3953) JSON Files from Export API with "?" char for string with special chars
[ https://issues.apache.org/jira/browse/ATLAS-3953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry reassigned ATLAS-3953: -- Assignee: Ashutosh Mestry > JSON Files from Export API with "?" char for string with special chars > --- > > Key: ATLAS-3953 > URL: https://issues.apache.org/jira/browse/ATLAS-3953 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 > Environment: Apache Atlas 2.1.0 embedded HBASE and SOLR >Reporter: Carlos Alberto Rocha Cardoso >Assignee: Ashutosh Mestry >Priority: Minor > > The Export API return a ZIP file with some JSON files describing Atlas > Entities and TypeDefs. > I am having a issue where some special chars in JSON are being replaced by > "?" chars. > An Entity name like "Distribuição" was exported in JSON file like > "Distribui??o". The special chars "çã" was replaced for the "??" chars. > I tried to change the exported JSON file encoding and the request header for > Export API, but without success. > After analyzing the Atlas source code, specialy the *splitAndWriteBytes* > method of the > *[ZipSink|https://github.com/apache/atlas/blob/cc601d7371fae1dbc16b55d1ca84f06b745700dc/repository/src/main/java/org/apache/atlas/repository/impexp/ZipSink.java] > class*, I tought if maybe the problem is because the *s.getBytes()* is > returning the JSON string to be written to ZIP with other encode than > *UTF-8*, and maybe set the encode like *s.getBytes(StandardCharsets.UTF_8)* > could be a solution. > Its my first contact with the Atlas source code, and I'm not a JAVA > programmer, so it's only a guess. > I saw that it's possible set the default encode to the plataform, or JVM, but > how they said in this below discussion, perhaps this doesn't work properly in > all situations. > [https://stackoverflow.com/questions/361975/setting-the-default-java-character-encoding] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3427) Atlas hooks enhancements for better fault-tolerance i.e. Kafka unavailability
[ https://issues.apache.org/jira/browse/ATLAS-3427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry updated ATLAS-3427: --- Description: *Background* Existing implementation for sending messages from the hooks relies on Kafka. Which means that if Kafka is not available for any reasons, the messages do not make it to Atlas. They are simply added to the _failed_._log_. *Solution* Introduce a mechanism which preserves messages if destination for the messages is not reachable. This new mechanism will also have logic to retry these messages. *Requirements* The mechanism should be: * Transparent: Existing hooks should not have to rework their implementation. * Configurable: This should be introduced optionally. If hook chooses not to use this, it should fallback to existing implementation. The behavior should be configurable via properties. * High-performance: Using this mechanism should not introduce additional overhead. Ideally, there should not be additional serialization introduced. * Reliable: The new mechanism should relay these messages reliably. Existing message formats should be supported viz. plain, compressed, compress & multi-part. was: *Background* Existing implementation for sending messages from the hooks relies on Kafka. Which means that if Kafka is not available for any reasons, the messages do not make it to Atlas. They are simply added to the _failed_._log_. *Solution* Introduce a mechanism which preserves messages if destination for the messages is not reachable. This new mechanism will also have logic to retry these messages. > Atlas hooks enhancements for better fault-tolerance i.e. Kafka unavailability > - > > Key: ATLAS-3427 > URL: https://issues.apache.org/jira/browse/ATLAS-3427 > Project: Atlas > Issue Type: Improvement >Reporter: Nixon Rodrigues >Assignee: Ashutosh Mestry >Priority: Major > > *Background* > Existing implementation for sending messages from the hooks relies on Kafka. > Which means that if Kafka is not available for any reasons, the messages do > not make it to Atlas. They are simply added to the _failed_._log_. > *Solution* > Introduce a mechanism which preserves messages if destination for the > messages is not reachable. This new mechanism will also have logic to retry > these messages. > *Requirements* > The mechanism should be: > * Transparent: Existing hooks should not have to rework their > implementation. > * Configurable: This should be introduced optionally. If hook chooses not to > use this, it should fallback to existing implementation. The behavior should > be configurable via properties. > * High-performance: Using this mechanism should not introduce additional > overhead. Ideally, there should not be additional serialization introduced. > * Reliable: The new mechanism should relay these messages reliably. Existing > message formats should be supported viz. plain, compressed, compress & > multi-part. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3427) Atlas hooks enhancements for better fault-tolerance i.e. Kafka unavailability
[ https://issues.apache.org/jira/browse/ATLAS-3427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry updated ATLAS-3427: --- Description: *Background* Existing implementation for sending messages from the hooks relies on Kafka. Which means that if Kafka is not available for any reasons, the messages do not make it to Atlas. They are simply added to the _failed_._log_. *Solution* Introduce a mechanism which preserves messages if destination for the messages is not reachable. This new mechanism will also have logic to retry these messages. was: Currently if hooks are not able to reach Kafka, metadata updates are not posted and received by Atlas service. Add a mechanism to make Hooks fault tolerant. > Atlas hooks enhancements for better fault-tolerance i.e. Kafka unavailability > - > > Key: ATLAS-3427 > URL: https://issues.apache.org/jira/browse/ATLAS-3427 > Project: Atlas > Issue Type: Improvement >Reporter: Nixon Rodrigues >Assignee: Ashutosh Mestry >Priority: Major > > *Background* > Existing implementation for sending messages from the hooks relies on Kafka. > Which means that if Kafka is not available for any reasons, the messages do > not make it to Atlas. They are simply added to the _failed_._log_. > > *Solution* > Introduce a mechanism which preserves messages if destination for the > messages is not reachable. This new mechanism will also have logic to retry > these messages. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (ATLAS-3427) Atlas hooks enhancements for better fault-tolerance i.e. Kafka unavailability
[ https://issues.apache.org/jira/browse/ATLAS-3427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry reassigned ATLAS-3427: -- Assignee: Ashutosh Mestry (was: Nixon Rodrigues) > Atlas hooks enhancements for better fault-tolerance i.e. Kafka unavailability > - > > Key: ATLAS-3427 > URL: https://issues.apache.org/jira/browse/ATLAS-3427 > Project: Atlas > Issue Type: Improvement >Reporter: Nixon Rodrigues >Assignee: Ashutosh Mestry >Priority: Major > > Currently if hooks are not able to reach Kafka, metadata updates are not > posted and received by Atlas service. > Add a mechanism to make Hooks fault tolerant. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3948) Entity Creation: Index Consistency: Java Patch Handler: Provide Option to Disable
[ https://issues.apache.org/jira/browse/ATLAS-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry updated ATLAS-3948: --- Attachment: ATLAS-3948-index-consistency-conditional.patch > Entity Creation: Index Consistency: Java Patch Handler: Provide Option to > Disable > - > > Key: ATLAS-3948 > URL: https://issues.apache.org/jira/browse/ATLAS-3948 > Project: Atlas > Issue Type: Bug >Reporter: Ashutosh Mestry >Assignee: Ashutosh Mestry >Priority: Major > Attachments: ATLAS-3948-index-consistency-conditional.patch > > > *Background* > The ATLAS-3907 provided ability to apply index consistency to existing > indexes via Java patch handler. > However, when _atlas.graph.storage.consistency-lock.enabled=false_ this patch > still gets applied. > *Solution* > Check for the property in Java patch handler and avoid applying it if the > property is set to false. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ATLAS-3948) Entity Creation: Index Consistency: Java Patch Handler: Provide Option to Disable
Ashutosh Mestry created ATLAS-3948: -- Summary: Entity Creation: Index Consistency: Java Patch Handler: Provide Option to Disable Key: ATLAS-3948 URL: https://issues.apache.org/jira/browse/ATLAS-3948 Project: Atlas Issue Type: Bug Reporter: Ashutosh Mestry Assignee: Ashutosh Mestry *Background* The ATLAS-3907 provided ability to apply index consistency to existing indexes via Java patch handler. However, when _atlas.graph.storage.consistency-lock.enabled=false_ this patch still gets applied. *Solution* Check for the property in Java patch handler and avoid applying it if the property is set to false. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3941) NotificationHookConsumer: Reduce Retry Pause Interval
[ https://issues.apache.org/jira/browse/ATLAS-3941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry updated ATLAS-3941: --- Attachment: ATLAS-3941-NotificationHookConsumer-Reduce-wait-time.patch > NotificationHookConsumer: Reduce Retry Pause Interval > - > > Key: ATLAS-3941 > URL: https://issues.apache.org/jira/browse/ATLAS-3941 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: trunk, 2.1.0 >Reporter: Ashutosh Mestry >Assignee: Ashutosh Mestry >Priority: Major > Fix For: trunk > > Attachments: > ATLAS-3941-NotificationHookConsumer-Reduce-wait-time.patch > > > *Background* > The retry logic introduced earlier, had a long wait time in case a concurrent > entity create was detected. This adversely affect ingest speed in the case > where there are a lot of errors in the data being ingested. > *Solution* > Reduce the wait time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3941) NotificationHookConsumer: Reduce Retry Pause Interval
[ https://issues.apache.org/jira/browse/ATLAS-3941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry updated ATLAS-3941: --- Attachment: (was: ATLAS-3941-NotificationHookConsumer-Reduce-wait-time.patch) > NotificationHookConsumer: Reduce Retry Pause Interval > - > > Key: ATLAS-3941 > URL: https://issues.apache.org/jira/browse/ATLAS-3941 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: trunk, 2.1.0 >Reporter: Ashutosh Mestry >Assignee: Ashutosh Mestry >Priority: Major > Fix For: trunk > > Attachments: > ATLAS-3941-NotificationHookConsumer-Reduce-wait-time.patch > > > *Background* > The retry logic introduced earlier, had a long wait time in case a concurrent > entity create was detected. This adversely affect ingest speed in the case > where there are a lot of errors in the data being ingested. > *Solution* > Reduce the wait time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3941) NotificationHookConsumer: Reduce Retry Pause Interval
[ https://issues.apache.org/jira/browse/ATLAS-3941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry updated ATLAS-3941: --- Attachment: ATLAS-3941-NotificationHookConsumer-Reduce-wait-time.patch > NotificationHookConsumer: Reduce Retry Pause Interval > - > > Key: ATLAS-3941 > URL: https://issues.apache.org/jira/browse/ATLAS-3941 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: trunk, 2.1.0 >Reporter: Ashutosh Mestry >Assignee: Ashutosh Mestry >Priority: Major > Fix For: trunk > > Attachments: > ATLAS-3941-NotificationHookConsumer-Reduce-wait-time.patch > > > *Background* > The retry logic introduced earlier, had a long wait time in case a concurrent > entity create was detected. This adversely affect ingest speed in the case > where there are a lot of errors in the data being ingested. > *Solution* > Reduce the wait time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ATLAS-3941) NotificationHookConsumer: Reduce Retry Pause Interval
Ashutosh Mestry created ATLAS-3941: -- Summary: NotificationHookConsumer: Reduce Retry Pause Interval Key: ATLAS-3941 URL: https://issues.apache.org/jira/browse/ATLAS-3941 Project: Atlas Issue Type: Bug Components: atlas-core Affects Versions: 2.1.0, trunk Reporter: Ashutosh Mestry Assignee: Ashutosh Mestry Fix For: trunk *Background* The retry logic introduced earlier, had a long wait time in case a concurrent entity create was detected. This adversely affect ingest speed in the case where there are a lot of errors in the data being ingested. *Solution* Reduce the wait time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (ATLAS-2932) Update DSL to use Java Traversal API
[ https://issues.apache.org/jira/browse/ATLAS-2932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry reassigned ATLAS-2932: -- Assignee: Ashutosh Mestry (was: Apoorv Naik) > Update DSL to use Java Traversal API > > > Key: ATLAS-2932 > URL: https://issues.apache.org/jira/browse/ATLAS-2932 > Project: Atlas > Issue Type: Bug >Affects Versions: 1.0.0, trunk >Reporter: Apoorv Naik >Assignee: Ashutosh Mestry >Priority: Major > Attachments: > 0002-ATLAS-2932-Update-DSL-to-use-Tinkerpop-Java-APIs-ins.patch > > > Change DSL code to use Java Tinkerpop Traversals instead of > GremlinScriptEngine -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3082) /var/log/atlas/application.log has unclear message
[
https://issues.apache.org/jira/browse/ATLAS-3082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Mestry updated ATLAS-3082:
---
Priority: Major (was: Minor)
> /var/log/atlas/application.log has unclear message
> ---
>
> Key: ATLAS-3082
> URL: https://issues.apache.org/jira/browse/ATLAS-3082
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
>Affects Versions: 2.0.0
>Reporter: Xinran Tinney
>Assignee: Ashutosh Mestry
>Priority: Major
>
> In /var/log/atlas/application.log every time the home page loads, there is
> this error in the logs:
> "graph rollback due to exception AtlasBaseException:Instance
> __AtlasUserProfile with unique attribute \{name=admin} does not exist" This
> should be removed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (ATLAS-3082) /var/log/atlas/application.log has unclear message
[
https://issues.apache.org/jira/browse/ATLAS-3082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Mestry reassigned ATLAS-3082:
--
Assignee: Ashutosh Mestry (was: Xinran Tinney)
> /var/log/atlas/application.log has unclear message
> ---
>
> Key: ATLAS-3082
> URL: https://issues.apache.org/jira/browse/ATLAS-3082
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
>Affects Versions: 2.0.0
>Reporter: Xinran Tinney
>Assignee: Ashutosh Mestry
>Priority: Minor
>
> In /var/log/atlas/application.log every time the home page loads, there is
> this error in the logs:
> "graph rollback due to exception AtlasBaseException:Instance
> __AtlasUserProfile with unique attribute \{name=admin} does not exist" This
> should be removed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ATLAS-3874) NotificationHookConsumer: Shell Entity Resolution When Multiple Consumers Process Same Entity
[ https://issues.apache.org/jira/browse/ATLAS-3874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry updated ATLAS-3874: --- Attachment: (was: ATLAS-3874-Notification-hook-processing-concurrent-e.patch) > NotificationHookConsumer: Shell Entity Resolution When Multiple Consumers > Process Same Entity > - > > Key: ATLAS-3874 > URL: https://issues.apache.org/jira/browse/ATLAS-3874 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.0.0, trunk >Reporter: Ashutosh Mestry >Assignee: Ashutosh Mestry >Priority: Major > Fix For: trunk > > > *Background* > Atlas has ability to consume messages from multiple hooks. When doing so, if > there are messages in multiple queue requesting creation of the same entity, > then this case leads to a race condition. > *Steps to Duplicate* > _Pre-requisites_ > Atlas should be configured to consume from 2 Kafka queues viz. _queue1_ and > _queue2_. > _Steps:_ > # Add a message in _queue1_ that references a non-existent entity. > # Add a message in _queue2_ that creates the entity in step 1. > _Expected behavior_: > Shell entity should get created in step 1 that is resolved into first class > entity in step 2. > _Actual behavior:_ > Shell entity is not resolved. A duplicate entity with same _qualifiedName_ > gets created. > *Root Cause* > The _NotificationHookConsumer_ is not aware of messages processed by other > consumers there by leading to concurrent entity creation. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ATLAS-3878) Notifications: Improve Memory Usage in Scale Enviroment
[ https://issues.apache.org/jira/browse/ATLAS-3878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry resolved ATLAS-3878. Resolution: Fixed > Notifications: Improve Memory Usage in Scale Enviroment > --- > > Key: ATLAS-3878 > URL: https://issues.apache.org/jira/browse/ATLAS-3878 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.0.0, trunk >Reporter: Ashutosh Mestry >Assignee: Ashutosh Mestry >Priority: Major > Fix For: trunk > > Attachments: > ATLAS-3878-Notifications-Improve-Memory-Usage-in-Sca-v2.patch > > > *Background* > As part of entity creation, Atlas sends notifications of different types. > Current implementation, to listeners. Listeners in turn perform specific > tasks. > At a more concrete level, the _EntityAuditListenerV2_ will write audits and > the _NotificationEntityChangeListener_ will send Kafka notifications. > Each of the listeners create notification objects. The notification objects > are large in number and are short lived. > The transient nature of the notification objects causes memory pressure in > scale environment. > *Solution* > Create object pool for notification objects. This way objects can be > reused.and existing design can be kept in tact. This will also offer benefit > of using existing test setup for verification. > *Tests Used* > _Setup_ > Create a test rig that will spawn multiple works that will invoke Atlas' bulk > APIs for entity creation. > Node: 40 workers, 8 GB allocated memory and 40 cores. > _Observation_ > About 40 mins into the exercise, memory pressure builds up causing GC > collects to take longer. This causes ZK timeout and finally Atlas process > crashes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (ATLAS-3398) Duplicates for unique attributes
[ https://issues.apache.org/jira/browse/ATLAS-3398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry reassigned ATLAS-3398: -- Assignee: Ashutosh Mestry > Duplicates for unique attributes > - > > Key: ATLAS-3398 > URL: https://issues.apache.org/jira/browse/ATLAS-3398 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.0.0, trunk >Reporter: Bolke de Bruin >Assignee: Ashutosh Mestry >Priority: Blocker > Labels: integrity > Attachments: zrzut_ekranu_2019-09-03_o_10.28.50.png > > > We are seeing issues with entities being added to Atlas with duplicate > "qualifiedName". The guids differ and other attributes do also differ. Below > a graph that shows the distribution over time for duplicates. We have > difficulty determining which one is the right one (as they are different) in > order to clean them up. > We are also not the only ones encountering this as you can in the linked > issue. > We have noticed that Atlas does not use the > [locking|https://docs.janusgraph.org/master/advanced-topics/eventual-consistency/] > mechanism of Janus to prevent this: > > !zrzut_ekranu_2019-09-03_o_10.28.50.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-3398) Duplicates for unique attributes
[ https://issues.apache.org/jira/browse/ATLAS-3398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17168967#comment-17168967 ] Ashutosh Mestry commented on ATLAS-3398: [~dwarszawski] I am not able to assign this issue to you. Your name does not show up in the list for some reason. I have assigned it to me for tracking. Hope that is OK. > Duplicates for unique attributes > - > > Key: ATLAS-3398 > URL: https://issues.apache.org/jira/browse/ATLAS-3398 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.0.0, trunk >Reporter: Bolke de Bruin >Assignee: Ashutosh Mestry >Priority: Blocker > Labels: integrity > Attachments: zrzut_ekranu_2019-09-03_o_10.28.50.png > > > We are seeing issues with entities being added to Atlas with duplicate > "qualifiedName". The guids differ and other attributes do also differ. Below > a graph that shows the distribution over time for duplicates. We have > difficulty determining which one is the right one (as they are different) in > order to clean them up. > We are also not the only ones encountering this as you can in the linked > issue. > We have noticed that Atlas does not use the > [locking|https://docs.janusgraph.org/master/advanced-topics/eventual-consistency/] > mechanism of Janus to prevent this: > > !zrzut_ekranu_2019-09-03_o_10.28.50.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ATLAS-3398) Duplicates for unique attributes
[ https://issues.apache.org/jira/browse/ATLAS-3398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry resolved ATLAS-3398. Resolution: Fixed > Duplicates for unique attributes > - > > Key: ATLAS-3398 > URL: https://issues.apache.org/jira/browse/ATLAS-3398 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.0.0, trunk >Reporter: Bolke de Bruin >Assignee: Ashutosh Mestry >Priority: Blocker > Labels: integrity > Attachments: zrzut_ekranu_2019-09-03_o_10.28.50.png > > > We are seeing issues with entities being added to Atlas with duplicate > "qualifiedName". The guids differ and other attributes do also differ. Below > a graph that shows the distribution over time for duplicates. We have > difficulty determining which one is the right one (as they are different) in > order to clean them up. > We are also not the only ones encountering this as you can in the linked > issue. > We have noticed that Atlas does not use the > [locking|https://docs.janusgraph.org/master/advanced-topics/eventual-consistency/] > mechanism of Janus to prevent this: > > !zrzut_ekranu_2019-09-03_o_10.28.50.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ATLAS-3907) Entity Creation: Index Consistency: Java Patch Handler
Ashutosh Mestry created ATLAS-3907: -- Summary: Entity Creation: Index Consistency: Java Patch Handler Key: ATLAS-3907 URL: https://issues.apache.org/jira/browse/ATLAS-3907 Project: Atlas Issue Type: Bug Components: atlas-core Affects Versions: trunk Reporter: Ashutosh Mestry Assignee: Ashutosh Mestry Fix For: trunk *Background* [ATLAS-3398|https://issues.apache.org/jira/browse/ATLAS-3398] adds _setConsistency_ during index creation. This works well for net new Atlas deployments. This Jira attempts to address the case where Atlas has been deployed and in-use without this change to the indexes. *Proposed Approach* * Use Atlas' java patch framework. * Create a java patch that enumerates existing indexes. This should be possible using JanusGraph's native APIs. * Add the _setConsistency_ to existing indexes. [~dwarszawski] : Thanks for your earlier patch. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3902) Import Service: Importing Data With Differing GUIDs for Same Unique Attributes Causes Errors in Certain Cases
[ https://issues.apache.org/jira/browse/ATLAS-3902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry updated ATLAS-3902: --- Summary: Import Service: Importing Data With Differing GUIDs for Same Unique Attributes Causes Errors in Certain Cases (was: Import Service: Importing Data With Differing GUIDs for Same Unique Attributes Causes Errors) > Import Service: Importing Data With Differing GUIDs for Same Unique > Attributes Causes Errors in Certain Cases > - > > Key: ATLAS-3902 > URL: https://issues.apache.org/jira/browse/ATLAS-3902 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.0.0, trunk, 2.1.0 >Reporter: Ashutosh Mestry >Assignee: Ashutosh Mestry >Priority: Major > Fix For: trunk, 2.1.0 > > > *Background* > Consider the scenario where 2 clusters containing Atlas are setup to be > synchronized Atlas' export and import APIs. If the source Atlas has changes > where table is dropped and re-created with same name. The table's entity > within Atlas will get a new GUID but will continue to have the same > _qualifiedName_. > This case is handled within the Import API. > However, the case that is not handled is to perform similar update on to the > table's storage descriptor. > *Steps to Duplicate* > # Create a schema within Hive containing database, tables, columns and > views. Atlas will reflect the changes. Perform export. Generate _s1.zip_. > # Drop schema. > # Re-create the same schema within Hive. Perform export. Generate _s2.zip_. > # Clear Atlas database. > # Import _s1.zip_. Observe _application.log_. > # Import s2.zip. Observe _application.log_. During import log will generate > messages like '_GUID Updated: Entity..._' > _Expected result:_ Import should succeed with messages indicating changes > entity's GUID. > _Actual result_: Import fails with errors indicating schema violation > (_AtlasSchemaViolation_) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3902) Import Service: Importing Data With Differing GUIDs for Same Unique Attributes Causes Errors
[ https://issues.apache.org/jira/browse/ATLAS-3902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry updated ATLAS-3902: --- Description: *Background* Consider the scenario where 2 clusters containing Atlas are setup to be synchronized Atlas' export and import APIs. If the source Atlas has changes where table is dropped and re-created with same name. The table's entity within Atlas will get a new GUID but will continue to have the same _qualifiedName_. This case is handled within the Import API. However, the case that is not handled is to perform similar update on to the table's storage descriptor. *Steps to Duplicate* # Create a schema within Hive containing database, tables, columns and views. Atlas will reflect the changes. Perform export. Generate _s1.zip_. # Drop schema. # Re-create the same schema within Hive. Perform export. Generate _s2.zip_. # Clear Atlas database. # Import _s1.zip_. Observe _application.log_. # Import s2.zip. Observe _application.log_. During import log will generate messages like '_GUID Updated: Entity..._' _Expected result:_ Import should succeed with messages indicating changes entity's GUID. _Actual result_: Import fails with errors indicating schema violation (_AtlasSchemaViolation_) was: *Background* *Steps to Duplicate* # Create a schema within Hive containing database, tables, columns and views. Atlas will reflect the changes. Perform export. Generate _s1.zip_. # Drop schema. # Re-create the same schema within Hive. Perform export. Generate _s2.zip_. # Clear Atlas database. # Import _s1.zip_. Observe _application.log_. # Import s2.zip. Observe _application.log_. During import log will generate messages like '_GUID Updated: Entity..._' _Expected result:_ Import should succeed with messages indicating changes entity's GUID. _Actual result_: Import fails with errors indicating schema violation (_AtlasSchemaViolation_) > Import Service: Importing Data With Differing GUIDs for Same Unique > Attributes Causes Errors > > > Key: ATLAS-3902 > URL: https://issues.apache.org/jira/browse/ATLAS-3902 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.0.0, trunk, 2.1.0 >Reporter: Ashutosh Mestry >Assignee: Ashutosh Mestry >Priority: Major > Fix For: trunk, 2.1.0 > > > *Background* > Consider the scenario where 2 clusters containing Atlas are setup to be > synchronized Atlas' export and import APIs. If the source Atlas has changes > where table is dropped and re-created with same name. The table's entity > within Atlas will get a new GUID but will continue to have the same > _qualifiedName_. > This case is handled within the Import API. > However, the case that is not handled is to perform similar update on to the > table's storage descriptor. > *Steps to Duplicate* > # Create a schema within Hive containing database, tables, columns and > views. Atlas will reflect the changes. Perform export. Generate _s1.zip_. > # Drop schema. > # Re-create the same schema within Hive. Perform export. Generate _s2.zip_. > # Clear Atlas database. > # Import _s1.zip_. Observe _application.log_. > # Import s2.zip. Observe _application.log_. During import log will generate > messages like '_GUID Updated: Entity..._' > _Expected result:_ Import should succeed with messages indicating changes > entity's GUID. > _Actual result_: Import fails with errors indicating schema violation > (_AtlasSchemaViolation_) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ATLAS-3902) Import Service: Importing Data With Differing GUIDs for Same Unique Attributes Causes Errors
Ashutosh Mestry created ATLAS-3902: -- Summary: Import Service: Importing Data With Differing GUIDs for Same Unique Attributes Causes Errors Key: ATLAS-3902 URL: https://issues.apache.org/jira/browse/ATLAS-3902 Project: Atlas Issue Type: Bug Components: atlas-core Affects Versions: 2.1.0, 2.0.0, trunk Reporter: Ashutosh Mestry Assignee: Ashutosh Mestry Fix For: trunk, 2.1.0 *Background* *Steps to Duplicate* # Create a schema within Hive containing database, tables, columns and views. Atlas will reflect the changes. Perform export. Generate _s1.zip_. # Drop schema. # Re-create the same schema within Hive. Perform export. Generate _s2.zip_. # Clear Atlas database. # Import _s1.zip_. Observe _application.log_. # Import s2.zip. Observe _application.log_. During import log will generate messages like '_GUID Updated: Entity..._' _Expected result:_ Import should succeed with messages indicating changes entity's GUID. _Actual result_: Import fails with errors indicating schema violation (_AtlasSchemaViolation_) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-3398) Duplicates for unique attributes
[ https://issues.apache.org/jira/browse/ATLAS-3398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17161588#comment-17161588 ] Ashutosh Mestry commented on ATLAS-3398: I suppose the way to duplicate would be to attempt to create entities via REST APIs? Test case would be to have entity payload with same _qualifiedName_ but different GUIDs and execute them at the same moment. > Duplicates for unique attributes > - > > Key: ATLAS-3398 > URL: https://issues.apache.org/jira/browse/ATLAS-3398 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.0.0, trunk >Reporter: Bolke de Bruin >Priority: Blocker > Labels: integrity > Attachments: zrzut_ekranu_2019-09-03_o_10.28.50.png > > > We are seeing issues with entities being added to Atlas with duplicate > "qualifiedName". The guids differ and other attributes do also differ. Below > a graph that shows the distribution over time for duplicates. We have > difficulty determining which one is the right one (as they are different) in > order to clean them up. > We are also not the only ones encountering this as you can in the linked > issue. > We have noticed that Atlas does not use the > [locking|https://docs.janusgraph.org/master/advanced-topics/eventual-consistency/] > mechanism of Janus to prevent this: > > !zrzut_ekranu_2019-09-03_o_10.28.50.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3878) Notifications: Improve Memory Usage in Scale Enviroment
[ https://issues.apache.org/jira/browse/ATLAS-3878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry updated ATLAS-3878: --- Attachment: (was: notifications-memory-fix-v3.patch) > Notifications: Improve Memory Usage in Scale Enviroment > --- > > Key: ATLAS-3878 > URL: https://issues.apache.org/jira/browse/ATLAS-3878 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.0.0, trunk >Reporter: Ashutosh Mestry >Assignee: Ashutosh Mestry >Priority: Major > Fix For: trunk > > Attachments: > ATLAS-3878-Notifications-Improve-Memory-Usage-in-Sca-v2.patch > > > *Background* > As part of entity creation, Atlas sends notifications of different types. > Current implementation, to listeners. Listeners in turn perform specific > tasks. > At a more concrete level, the _EntityAuditListenerV2_ will write audits and > the _NotificationEntityChangeListener_ will send Kafka notifications. > Each of the listeners create notification objects. The notification objects > are large in number and are short lived. > The transient nature of the notification objects causes memory pressure in > scale environment. > *Solution* > Create object pool for notification objects. This way objects can be > reused.and existing design can be kept in tact. This will also offer benefit > of using existing test setup for verification. > *Tests Used* > _Setup_ > Create a test rig that will spawn multiple works that will invoke Atlas' bulk > APIs for entity creation. > Node: 40 workers, 8 GB allocated memory and 40 cores. > _Observation_ > About 40 mins into the exercise, memory pressure builds up causing GC > collects to take longer. This causes ZK timeout and finally Atlas process > crashes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3878) Notifications: Improve Memory Usage in Scale Enviroment
[ https://issues.apache.org/jira/browse/ATLAS-3878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry updated ATLAS-3878: --- Attachment: ATLAS-3878-Notifications-Improve-Memory-Usage-in-Sca-v2.patch > Notifications: Improve Memory Usage in Scale Enviroment > --- > > Key: ATLAS-3878 > URL: https://issues.apache.org/jira/browse/ATLAS-3878 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.0.0, trunk >Reporter: Ashutosh Mestry >Assignee: Ashutosh Mestry >Priority: Major > Fix For: trunk > > Attachments: > ATLAS-3878-Notifications-Improve-Memory-Usage-in-Sca-v2.patch > > > *Background* > As part of entity creation, Atlas sends notifications of different types. > Current implementation, to listeners. Listeners in turn perform specific > tasks. > At a more concrete level, the _EntityAuditListenerV2_ will write audits and > the _NotificationEntityChangeListener_ will send Kafka notifications. > Each of the listeners create notification objects. The notification objects > are large in number and are short lived. > The transient nature of the notification objects causes memory pressure in > scale environment. > *Solution* > Create object pool for notification objects. This way objects can be > reused.and existing design can be kept in tact. This will also offer benefit > of using existing test setup for verification. > *Tests Used* > _Setup_ > Create a test rig that will spawn multiple works that will invoke Atlas' bulk > APIs for entity creation. > Node: 40 workers, 8 GB allocated memory and 40 cores. > _Observation_ > About 40 mins into the exercise, memory pressure builds up causing GC > collects to take longer. This causes ZK timeout and finally Atlas process > crashes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3878) Notifications: Improve Memory Usage in Scale Enviroment
[ https://issues.apache.org/jira/browse/ATLAS-3878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry updated ATLAS-3878: --- Attachment: notifications-memory-fix-v3.patch > Notifications: Improve Memory Usage in Scale Enviroment > --- > > Key: ATLAS-3878 > URL: https://issues.apache.org/jira/browse/ATLAS-3878 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.0.0, trunk >Reporter: Ashutosh Mestry >Assignee: Ashutosh Mestry >Priority: Major > Fix For: trunk > > Attachments: notifications-memory-fix-v3.patch > > > *Background* > As part of entity creation, Atlas sends notifications of different types. > Current implementation, to listeners. Listeners in turn perform specific > tasks. > At a more concrete level, the _EntityAuditListenerV2_ will write audits and > the _NotificationEntityChangeListener_ will send Kafka notifications. > Each of the listeners create notification objects. The notification objects > are large in number and are short lived. > The transient nature of the notification objects causes memory pressure in > scale environment. > *Solution* > Create object pool for notification objects. This way objects can be > reused.and existing design can be kept in tact. This will also offer benefit > of using existing test setup for verification. > *Tests Used* > _Setup_ > Create a test rig that will spawn multiple works that will invoke Atlas' bulk > APIs for entity creation. > Node: 40 workers, 8 GB allocated memory and 40 cores. > _Observation_ > About 40 mins into the exercise, memory pressure builds up causing GC > collects to take longer. This causes ZK timeout and finally Atlas process > crashes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3878) Notifications: Improve Memory Usage in Scale Enviroment
[ https://issues.apache.org/jira/browse/ATLAS-3878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry updated ATLAS-3878: --- Description: *Background* As part of entity creation, Atlas sends notifications of different types. Current implementation, to listeners. Listeners in turn perform specific tasks. At a more concrete level, the _EntityAuditListenerV2_ will write audits and the _NotificationEntityChangeListener_ will send Kafka notifications. Each of the listeners create notification objects. The notification objects are large in number and are short lived. The transient nature of the notification objects causes memory pressure in scale environment. *Solution* Create object pool for notification objects. This way objects can be reused.and existing design can be kept in tact. This will also offer benefit of using existing test setup for verification. *Tests Used* _Setup_ Create a test rig that will spawn multiple works that will invoke Atlas' bulk APIs for entity creation. Node: 40 workers, 8 GB allocated memory and 40 cores. _Observation_ About 40 mins into the exercise, memory pressure builds up causing GC collects to take longer. This causes ZK timeout and finally Atlas process crashes. was: *Background* As part of entity creation, Atlas sends notifications of different types. Current implementation, to listeners. Listeners in turn perform specific tasks. At a more concrete level, the _EntityAuditListenerV2_ will write audits and the _NotificationEntityChangeListener_ will send Kafka notifications. Each of the listeners create notification objects. The notification objects are large in number and are short lived. The transient nature of the notification objects causes memory pressure in scale environment. *Solution* Create object pool for notification objects. This way objects can be reused.and existing design can be kept in tact. This will also offer benefit of using existing test setup for verification. *Tests Used* _Setup_ Create a test rig that will spawn multiple works that will invoke Atlas' bulk APIs for entity creation. Node: 40 workers, 8 GB allocated memory and 40 cores. **_Observation_ About 40 mins into the exercise, memory pressure builds up causing GC collects to take longer. This causes ZK timeout and finally Atlas process crashes. > Notifications: Improve Memory Usage in Scale Enviroment > --- > > Key: ATLAS-3878 > URL: https://issues.apache.org/jira/browse/ATLAS-3878 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.0.0, trunk >Reporter: Ashutosh Mestry >Assignee: Ashutosh Mestry >Priority: Major > Fix For: trunk > > > *Background* > As part of entity creation, Atlas sends notifications of different types. > Current implementation, to listeners. Listeners in turn perform specific > tasks. > At a more concrete level, the _EntityAuditListenerV2_ will write audits and > the _NotificationEntityChangeListener_ will send Kafka notifications. > Each of the listeners create notification objects. The notification objects > are large in number and are short lived. > The transient nature of the notification objects causes memory pressure in > scale environment. > *Solution* > Create object pool for notification objects. This way objects can be > reused.and existing design can be kept in tact. This will also offer benefit > of using existing test setup for verification. > *Tests Used* > _Setup_ > Create a test rig that will spawn multiple works that will invoke Atlas' bulk > APIs for entity creation. > Node: 40 workers, 8 GB allocated memory and 40 cores. > _Observation_ > About 40 mins into the exercise, memory pressure builds up causing GC > collects to take longer. This causes ZK timeout and finally Atlas process > crashes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ATLAS-3878) Notifications: Improve Memory Usage in Scale Enviroment
Ashutosh Mestry created ATLAS-3878: -- Summary: Notifications: Improve Memory Usage in Scale Enviroment Key: ATLAS-3878 URL: https://issues.apache.org/jira/browse/ATLAS-3878 Project: Atlas Issue Type: Improvement Components: atlas-core Affects Versions: 2.0.0, trunk Reporter: Ashutosh Mestry Assignee: Ashutosh Mestry Fix For: trunk *Background* As part of entity creation, Atlas sends notifications of different types. Current implementation, to listeners. Listeners in turn perform specific tasks. At a more concrete level, the _EntityAuditListenerV2_ will write audits and the _NotificationEntityChangeListener_ will send Kafka notifications. Each of the listeners create notification objects. The notification objects are large in number and are short lived. The transient nature of the notification objects causes memory pressure in scale environment. *Solution* Create object pool for notification objects. This way objects can be reused.and existing design can be kept in tact. This will also offer benefit of using existing test setup for verification. *Tests Used* _Setup_ Create a test rig that will spawn multiple works that will invoke Atlas' bulk APIs for entity creation. Node: 40 workers, 8 GB allocated memory and 40 cores. **_Observation_ About 40 mins into the exercise, memory pressure builds up causing GC collects to take longer. This causes ZK timeout and finally Atlas process crashes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
