[jira] [Created] (ATLAS-4859) NPE during Atlas Import
Priyanshi Shah created ATLAS-4859: - Summary: NPE during Atlas Import Key: ATLAS-4859 URL: https://issues.apache.org/jira/browse/ATLAS-4859 Project: Atlas Issue Type: Bug Components: atlas-core Reporter: Priyanshi Shah Assignee: Priyanshi Shah Fix For: 3.0.0, 2.3.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
Review Request 74967: ATLAS-4856 : Parallel Atlas Import is failing with unexpected errors
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74967/ --- Review request for atlas, Jayendra Parab, Mandar Ambawane, Pinal Shah, and Sheetal Shah. Bugs: ATLAS-4856 https://issues.apache.org/jira/browse/ATLAS-4856 Repository: atlas Description --- Problem Statement: This is an intermittent issue the possibilities of reproducing this issue are: 1. When 2 parallel imports running which has some overlapping entities. 2. If more than 10 parallel imports are running which has no overlapping entities. 3. Performing parallel imports on fresh atlas server which has no entities present. Fix: As permanent failures can be caused by complete connection loss, hardware failure or lock contention. So, found one of the solution to fix this exception, is by retrying the transaction. Diffs - repository/src/main/java/org/apache/atlas/repository/impexp/AuditsWriter.java c4de0ed27 repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/RegularImport.java eca6b0e68 Diff: https://reviews.apache.org/r/74967/diff/1/ Testing --- Thanks, Priyanshi Shah
[jira] [Updated] (ATLAS-4856) Parallel Atlas Import is failing with unexpected errors
[ https://issues.apache.org/jira/browse/ATLAS-4856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyanshi Shah updated ATLAS-4856: -- Description: PermanentLockingException was thrown while running parallel imports. > Parallel Atlas Import is failing with unexpected errors > --- > > Key: ATLAS-4856 > URL: https://issues.apache.org/jira/browse/ATLAS-4856 > Project: Atlas > Issue Type: Bug > Components: atlas-core > Reporter: Priyanshi Shah > Assignee: Priyanshi Shah >Priority: Major > Fix For: 3.0.0, 2.3.0 > > > PermanentLockingException was thrown while running parallel imports. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Review Request 74965: ATLAS-4844 : Atlas - Upgrade Common Configuration2 to 2.10.1
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74965/ --- Review request for atlas, Jayendra Parab, Mandar Ambawane, Pinal Shah, and Sheetal Shah. Bugs: ATLAS-4844 https://issues.apache.org/jira/browse/ATLAS-4844 Repository: atlas Description --- Currently Atlas is using common configuration2 version 2.8.0 upgrading it to 2.10.1 Diffs - pom.xml 6e6724275 Diff: https://reviews.apache.org/r/74965/diff/1/ Testing --- Thanks, Priyanshi Shah
[jira] [Assigned] (ATLAS-4856) Parallel Atlas Import is failing with unexpected errors
[ https://issues.apache.org/jira/browse/ATLAS-4856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyanshi Shah reassigned ATLAS-4856: - Assignee: Priyanshi Shah > Parallel Atlas Import is failing with unexpected errors > --- > > Key: ATLAS-4856 > URL: https://issues.apache.org/jira/browse/ATLAS-4856 > Project: Atlas > Issue Type: Bug > Components: atlas-core > Reporter: Priyanshi Shah > Assignee: Priyanshi Shah >Priority: Major > Fix For: 3.0.0, 2.3.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (ATLAS-4856) Parallel Atlas Import is failing with unexpected errors
Priyanshi Shah created ATLAS-4856: - Summary: Parallel Atlas Import is failing with unexpected errors Key: ATLAS-4856 URL: https://issues.apache.org/jira/browse/ATLAS-4856 Project: Atlas Issue Type: Bug Components: atlas-core Reporter: Priyanshi Shah Fix For: 3.0.0, 2.3.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
Review Request 74961: ATLAS-4854 : Atlas - Upgrade Spring Security to 5.8.11
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74961/ --- Review request for atlas, Jayendra Parab, Mandar Ambawane, Pinal Shah, and Sheetal Shah. Bugs: ATLAS-4854 https://issues.apache.org/jira/browse/ATLAS-4854 Repository: atlas Description --- Currently Atlas is using Spring security version 5.8.5 upgrading it to 5.8.11 Diffs - pom.xml 6e6724275 Diff: https://reviews.apache.org/r/74961/diff/1/ Testing --- Thanks, Priyanshi Shah
[jira] [Updated] (ATLAS-4854) Atlas - Upgrade Spring Security to 5.8.11
[ https://issues.apache.org/jira/browse/ATLAS-4854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyanshi Shah updated ATLAS-4854: -- Description: Upgrading spring security from 5.8.5 to 5.8.11 > Atlas - Upgrade Spring Security to 5.8.11 > - > > Key: ATLAS-4854 > URL: https://issues.apache.org/jira/browse/ATLAS-4854 > Project: Atlas > Issue Type: Improvement > Components: atlas-core > Reporter: Priyanshi Shah > Assignee: Priyanshi Shah >Priority: Major > Fix For: 3.0.0 > > > Upgrading spring security from 5.8.5 to 5.8.11 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (ATLAS-4854) Atlas - Upgrade Spring Security to 5.8.11
Priyanshi Shah created ATLAS-4854: - Summary: Atlas - Upgrade Spring Security to 5.8.11 Key: ATLAS-4854 URL: https://issues.apache.org/jira/browse/ATLAS-4854 Project: Atlas Issue Type: Improvement Components: atlas-core Reporter: Priyanshi Shah Fix For: 3.0.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (ATLAS-4854) Atlas - Upgrade Spring Security to 5.8.11
[ https://issues.apache.org/jira/browse/ATLAS-4854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyanshi Shah reassigned ATLAS-4854: - Assignee: Priyanshi Shah > Atlas - Upgrade Spring Security to 5.8.11 > - > > Key: ATLAS-4854 > URL: https://issues.apache.org/jira/browse/ATLAS-4854 > Project: Atlas > Issue Type: Improvement > Components: atlas-core > Reporter: Priyanshi Shah > Assignee: Priyanshi Shah >Priority: Major > Fix For: 3.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (ATLAS-4844) Atlas - Upgrade Common Configuration2 to 2.10.1
[ https://issues.apache.org/jira/browse/ATLAS-4844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyanshi Shah updated ATLAS-4844: -- Due Date: (was: 2/Apr/24) > Atlas - Upgrade Common Configuration2 to 2.10.1 > --- > > Key: ATLAS-4844 > URL: https://issues.apache.org/jira/browse/ATLAS-4844 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 3.0.0 > Reporter: Priyanshi Shah > Assignee: Priyanshi Shah >Priority: Major > > Upgrading common configurations2 from 2.8.0 to 2.10.1 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (ATLAS-4847) Export/Import : Atlas export fails and throws NullPointerException.
[ https://issues.apache.org/jira/browse/ATLAS-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyanshi Shah updated ATLAS-4847: -- Fix Version/s: 3.0.0 2.3.0 > Export/Import : Atlas export fails and throws NullPointerException. > --- > > Key: ATLAS-4847 > URL: https://issues.apache.org/jira/browse/ATLAS-4847 > Project: Atlas > Issue Type: Bug > Components: atlas-core > Reporter: Priyanshi Shah > Assignee: Priyanshi Shah >Priority: Major > Fix For: 3.0.0, 2.3.0 > > Attachments: ATLAS-4847.patch, NullPointerException_stackTrace.txt > > > Steps to reproduce: > 1. Create a hive_table entity and assign a term to it. > 2. Perform 1st incremental export of that entity without using changeMarker. > 3. Without any modification on that entity perform 2nd consecutive > incremental export with changeMarker from previous export. > Problem is it throws NullPointerException and export is failed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (ATLAS-4842) Export/Import: fetchType as "incremental" does full export instead of connected
[ https://issues.apache.org/jira/browse/ATLAS-4842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyanshi Shah updated ATLAS-4842: -- Fix Version/s: 3.0.0 2.3.0 > Export/Import: fetchType as "incremental" does full export instead of > connected > --- > > Key: ATLAS-4842 > URL: https://issues.apache.org/jira/browse/ATLAS-4842 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Reporter: Sheetal Shah >Assignee: Priyanshi Shah >Priority: Major > Fix For: 3.0.0, 2.3.0 > > Attachments: ATLAS-4842.patch > > > Create two hive_tables in same database: > # create table hive1(id int); > # create table hive1_ctas as select * from hive1; > # create table hive2(name string); > Perform incremental export of hive1 with changeMarker as 0 and skipLineage > value as false. > Import the zip. > Expectation is only the requested hive_table hive1 and its related entities > should be exported, but it does full export and hive2 is also imported. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (ATLAS-4845) Atlas Import is failing with fetchType: Incremental if there are no changes between two consecutive runs
[ https://issues.apache.org/jira/browse/ATLAS-4845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyanshi Shah updated ATLAS-4845: -- Fix Version/s: 3.0.0 2.3.0 > Atlas Import is failing with fetchType: Incremental if there are no changes > between two consecutive runs > > > Key: ATLAS-4845 > URL: https://issues.apache.org/jira/browse/ATLAS-4845 > Project: Atlas > Issue Type: Bug > Components: atlas-core > Reporter: Priyanshi Shah > Assignee: Priyanshi Shah >Priority: Major > Fix For: 3.0.0, 2.3.0 > > Attachments: ATLAS-4845.patch > > > Repro steps: > # First export + import with changeMarker “0” - worked as expected. > # I tried to run 2nd export + import without any changes in atlas, 2nd > export also worked and gave different changeMarker in output as expected. > *But 2nd import failed* > In other fetch Types this will work. So we also require similar behaviour in > case of Incremental type also because this is very normal scenario and > customer will hit this very frequently. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (ATLAS-4838) Export/Import : changeMarker is not set to entity's lastupdatetime or its closer timestamp value
[
https://issues.apache.org/jira/browse/ATLAS-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Priyanshi Shah updated ATLAS-4838:
--
Fix Version/s: 3.0.0
2.3.0
> Export/Import : changeMarker is not set to entity's lastupdatetime or its
> closer timestamp value
>
>
> Key: ATLAS-4838
> URL: https://issues.apache.org/jira/browse/ATLAS-4838
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
>Reporter: Sharmadha S
> Assignee: Priyanshi Shah
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
> Attachments: ATLAS-4838-V2.patch
>
>
> When an hive table entity is exported using fetchType incremental with
> changeMarker 0 , after exporting , the changeMarker in the export response is
> not set to recent timestamp.
> For example , on exporting iceberg table with fetchType incremental and
> changeMarker as 0 :
> {code:java}
> 2024-02-08 13:28:43,838 INFO - [etp891461509-20 -
> db3f1141-531c-46c0-909a-5e443d4353a2:] ~
> export(item=AtlasObjectId{guid='null', typeName='iceberg_table',
> uniqueAttributes={qualifiedName:default.iceberg_table2@cm}}; matchType=null,
> fetchType=incremental): found 1 entities: options:
> {"itemsToExport":[{"typeName":"iceberg_table","uniqueAttributes":{"qualifiedName":"default.iceberg_table2@cm"}}],"options":{"fetchType":"incremental","changeMarker":0},"fetchTypeOptionValue":"incremental","skipLineageOptionValue":false,"changeTokenFromOptions":0}
> (StartEntityFetchByExportRequest:123)
> 2024-02-08 13:28:48,697 INFO - [etp891461509-20 -
> db3f1141-531c-46c0-909a-5e443d4353a2:] ~ Assigned guid =
> f5e6342a-dfdf-4d54-a1b3-b24f987c4ecb to newly created entity (DataAccess:85)
> 2024-02-08 13:28:48,697 INFO - [etp891461509-20 -
> db3f1141-531c-46c0-909a-5e443d4353a2:] ~ addAuditEntry: user: admin, source:
> cm, target: , operation: EXPORT (ExportImportAuditService:168)
> 2024-02-08 13:28:48,699 INFO - [etp891461509-20 -
> db3f1141-531c-46c0-909a-5e443d4353a2:] ~ <== export(user=admin,
> from=10.19.28.124): status SUCCESS: changeMarker: 1707225183928
> (ExportService:107) {code}
> changeMarker from response is 1707225183928 => Tuesday, February 6, 2024
> 6:43:03.928 PM
>
> But lastUpdatetime of iceberg table is
> {code:java}
> "createTime": 1707398866970 => Thursday, February 8, 2024 6:57:46.970 PM
> "updateTime": 1707398867375. => Thursday, February 8, 2024 6:57:47.375 PM
> {code}
> Value is set to 2 days back. This is not then an incremental export , as it
> will fetch all updates from Tuesday, February 6, 2024 6:43:03.928 PM in next
> export
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (ATLAS-4847) Export/Import : Atlas export fails and throws NullPointerException.
[ https://issues.apache.org/jira/browse/ATLAS-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyanshi Shah updated ATLAS-4847: -- Attachment: NullPointerException_stackTrace.txt > Export/Import : Atlas export fails and throws NullPointerException. > --- > > Key: ATLAS-4847 > URL: https://issues.apache.org/jira/browse/ATLAS-4847 > Project: Atlas > Issue Type: Bug > Components: atlas-core > Reporter: Priyanshi Shah > Assignee: Priyanshi Shah >Priority: Major > Attachments: ATLAS-4847.patch, NullPointerException_stackTrace.txt > > > Steps to reproduce: > 1. Create a hive_table entity and assign a term to it. > 2. Perform 1st incremental export of that entity without using changeMarker. > 3. Without any modification on that entity perform 2nd consecutive > incremental export with changeMarker from previous export. > Problem is it throws NullPointerException and export is failed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (ATLAS-4847) Export/Import : Atlas export fails and throws NullPointerException.
[
https://issues.apache.org/jira/browse/ATLAS-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Priyanshi Shah updated ATLAS-4847:
--
Description:
Steps to reproduce:
1. Create a hive_table entity and assign a term to it.
2. Perform 1st incremental export of that entity without using changeMarker.
3. Without any modification on that entity perform 2nd consecutive incremental
export with changeMarker from previous export.
Problem is it throws NullPointerException and export is failed.
was:
Steps to reproduce:
1. Create a hive_table entity and assign a term to it.
2. Perform 1st incremental export of that entity without using changeMarker.
3. Without any modification on that entity perform 2nd consecutive incremental
export with changeMarker from previous export.
Problem is it throws NullPointerException and export is failed.
Application.log
2024-04-10 17:49:44,148 INFO - [etp370869802-458 -
76b2eeb5-22fb-4631-a99a-2567c307152a:] ~ ==> export(user=admin, from=127.0.0.1)
(ExportService:95)
2024-04-10 17:49:44,219 INFO - [etp370869802-458 -
76b2eeb5-22fb-4631-a99a-2567c307152a:] ~
export(item=AtlasObjectId\{guid='null', typeName='hive_table',
uniqueAttributes={qualifiedName:test_db.abc__hive_impala_table_1@cm}};
matchType=null, fetchType=INCREMENTAL): found 1 entities: options:
\{"itemsToExport":[{"typeName":"hive_table","uniqueAttributes":{"qualifiedName":"test_db.abc__hive_impala_table_1@cm"}}],"options":\{"fetchType":"INCREMENTAL","changeMarker":1712751459762},"fetchTypeOptionValue":"INCREMENTAL","skipLineageOptionValue":false,"omitZipResponseForEmptyExport":false,"changeTokenFromOptions":1712751459762}
(StartEntityFetchByExportRequest:123)
2024-04-10 17:49:44,242 ERROR - [etp370869802-458 -
76b2eeb5-22fb-4631-a99a-2567c307152a:] ~ Operation failed: (ExportService:103)
java.lang.{color:#de350b}NullPointerException{color}
at
org.apache.atlas.repository.impexp.ExportService$ExportContext.getEntitiesWithModifiedTimestamp(ExportService.java:445)
at
org.apache.atlas.repository.impexp.ExportService.addEntity(ExportService.java:332)
at
org.apache.atlas.repository.impexp.ExportService.addGlossaryEntities(ExportService.java:299)
at
org.apache.atlas.repository.impexp.ExportService.processEntity(ExportService.java:274)
> Export/Import : Atlas export fails and throws NullPointerException.
> ---
>
> Key: ATLAS-4847
> URL: https://issues.apache.org/jira/browse/ATLAS-4847
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
>Reporter: Priyanshi Shah
>Assignee: Priyanshi Shah
>Priority: Major
> Attachments: ATLAS-4847.patch, NullPointerException_stackTrace.txt
>
>
> Steps to reproduce:
> 1. Create a hive_table entity and assign a term to it.
> 2. Perform 1st incremental export of that entity without using changeMarker.
> 3. Without any modification on that entity perform 2nd consecutive
> incremental export with changeMarker from previous export.
> Problem is it throws NullPointerException and export is failed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (ATLAS-4847) Export/Import : Atlas export fails and throws NullPointerException.
[
https://issues.apache.org/jira/browse/ATLAS-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Priyanshi Shah updated ATLAS-4847:
--
Description:
Steps to reproduce:
1. Create a hive_table entity and assign a term to it.
2. Perform 1st incremental export of that entity without using changeMarker.
3. Without any modification on that entity perform 2nd consecutive incremental
export with changeMarker from previous export.
Problem is it throws NullPointerException and export is failed.
Application.log
2024-04-10 17:49:44,148 INFO - [etp370869802-458 -
76b2eeb5-22fb-4631-a99a-2567c307152a:] ~ ==> export(user=admin, from=127.0.0.1)
(ExportService:95)
2024-04-10 17:49:44,219 INFO - [etp370869802-458 -
76b2eeb5-22fb-4631-a99a-2567c307152a:] ~
export(item=AtlasObjectId\{guid='null', typeName='hive_table',
uniqueAttributes={qualifiedName:test_db.abc__hive_impala_table_1@cm}};
matchType=null, fetchType=INCREMENTAL): found 1 entities: options:
\{"itemsToExport":[{"typeName":"hive_table","uniqueAttributes":{"qualifiedName":"test_db.abc__hive_impala_table_1@cm"}}],"options":\{"fetchType":"INCREMENTAL","changeMarker":1712751459762},"fetchTypeOptionValue":"INCREMENTAL","skipLineageOptionValue":false,"omitZipResponseForEmptyExport":false,"changeTokenFromOptions":1712751459762}
(StartEntityFetchByExportRequest:123)
2024-04-10 17:49:44,242 ERROR - [etp370869802-458 -
76b2eeb5-22fb-4631-a99a-2567c307152a:] ~ Operation failed: (ExportService:103)
java.lang.{color:#de350b}NullPointerException{color}
at
org.apache.atlas.repository.impexp.ExportService$ExportContext.getEntitiesWithModifiedTimestamp(ExportService.java:445)
at
org.apache.atlas.repository.impexp.ExportService.addEntity(ExportService.java:332)
at
org.apache.atlas.repository.impexp.ExportService.addGlossaryEntities(ExportService.java:299)
at
org.apache.atlas.repository.impexp.ExportService.processEntity(ExportService.java:274)
was:
Steps to reproduce:
1. Create a hive_table entity and assign a term to it.
2. Perform 1st incremental export of that entity without using changeMarker.
3. Without any modification on that entity perform 2nd consecutive incremental
export with changeMarker from previous export.
Problem is it throws NullPointerException and export is failed.
> Export/Import : Atlas export fails and throws NullPointerException.
> ---
>
> Key: ATLAS-4847
> URL: https://issues.apache.org/jira/browse/ATLAS-4847
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
>Reporter: Priyanshi Shah
>Assignee: Priyanshi Shah
>Priority: Major
> Attachments: ATLAS-4847.patch
>
>
> Steps to reproduce:
> 1. Create a hive_table entity and assign a term to it.
> 2. Perform 1st incremental export of that entity without using changeMarker.
> 3. Without any modification on that entity perform 2nd consecutive
> incremental export with changeMarker from previous export.
> Problem is it throws NullPointerException and export is failed.
>
> Application.log
> 2024-04-10 17:49:44,148 INFO - [etp370869802-458 -
> 76b2eeb5-22fb-4631-a99a-2567c307152a:] ~ ==> export(user=admin,
> from=127.0.0.1) (ExportService:95)
> 2024-04-10 17:49:44,219 INFO - [etp370869802-458 -
> 76b2eeb5-22fb-4631-a99a-2567c307152a:] ~
> export(item=AtlasObjectId\{guid='null', typeName='hive_table',
> uniqueAttributes={qualifiedName:test_db.abc__hive_impala_table_1@cm}};
> matchType=null, fetchType=INCREMENTAL): found 1 entities: options:
> \{"itemsToExport":[{"typeName":"hive_table","uniqueAttributes":{"qualifiedName":"test_db.abc__hive_impala_table_1@cm"}}],"options":\{"fetchType":"INCREMENTAL","changeMarker":1712751459762},"fetchTypeOptionValue":"INCREMENTAL","skipLineageOptionValue":false,"omitZipResponseForEmptyExport":false,"changeTokenFromOptions":1712751459762}
> (StartEntityFetchByExportRequest:123)
> 2024-04-10 17:49:44,242 ERROR - [etp370869802-458 -
> 76b2eeb5-22fb-4631-a99a-2567c307152a:] ~ Operation failed:
> (ExportService:103)
> java.lang.{color:#de350b}NullPointerException{color}
> at
> org.apache.atlas.repository.impexp.ExportService$ExportContext.getEntitiesWithModifiedTimestamp(ExportService.java:445)
> at
> org.apache.atlas.repository.impexp.ExportService.addEntity(ExportService.java:332)
> at
> org.apache.atlas.repository.impexp.ExportService.addGlossaryEntities(ExportService.java:299)
> at
> org.apache.atlas.repository.impexp.ExportService.processEntity(ExportService.java:274)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (ATLAS-4847) Export/Import : Atlas export fails and throws NullPointerException.
[ https://issues.apache.org/jira/browse/ATLAS-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyanshi Shah updated ATLAS-4847: -- Attachment: (was: image-2024-04-11-15-15-59-551.png) > Export/Import : Atlas export fails and throws NullPointerException. > --- > > Key: ATLAS-4847 > URL: https://issues.apache.org/jira/browse/ATLAS-4847 > Project: Atlas > Issue Type: Bug > Components: atlas-core > Reporter: Priyanshi Shah > Assignee: Priyanshi Shah >Priority: Major > Attachments: ATLAS-4847.patch > > > Steps to reproduce: > 1. Create a hive_table entity and assign a term to it. > 2. Perform 1st incremental export of that entity without using changeMarker. > 3. Without any modification on that entity perform 2nd consecutive > incremental export with changeMarker from previous export. > Problem is it throws NullPointerException and export is failed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (ATLAS-4847) Export/Import : Atlas export fails and throws NullPointerException.
[ https://issues.apache.org/jira/browse/ATLAS-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyanshi Shah updated ATLAS-4847: -- Attachment: ATLAS-4847.patch > Export/Import : Atlas export fails and throws NullPointerException. > --- > > Key: ATLAS-4847 > URL: https://issues.apache.org/jira/browse/ATLAS-4847 > Project: Atlas > Issue Type: Bug > Components: atlas-core > Reporter: Priyanshi Shah > Assignee: Priyanshi Shah >Priority: Major > Attachments: ATLAS-4847.patch, image-2024-04-11-15-15-59-551.png > > > Steps to reproduce: > 1. Create a hive_table entity and assign a term to it. > 2. Perform 1st incremental export of that entity without using changeMarker. > 3. Without any modification on that entity perform 2nd consecutive > incremental export with changeMarker from previous export. > Problem is it throws NullPointerException and export is failed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: Review Request 74956: ATLAS-4847 : Export/Import : Atlas export fails and throws NullPointerException.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74956/ --- (Updated April 12, 2024, 7:22 a.m.) Review request for atlas, Jayendra Parab, Mandar Ambawane, Pinal Shah, and Sheetal Shah. Bugs: ATLAS-4847 https://issues.apache.org/jira/browse/ATLAS-4847 Repository: atlas Description --- Steps to reproduce: 1. Create a hive_table entity and assign a term to it. 2. Perform 1st incremental export of that entity without using changeMarker. 3. Without any modification on that entity perform 2nd consecutive incremental export with changeMarker from previous export. Result is it that it throws NullPointerException and export is failed. Problem: When an entity which has termed assigned was exported with changeMarker value greater than value of lastModified time of that entity it resulted in throwing NullPointerException. Solution: 1. NullPointerException occured because it was trying to fetch referred entities of AtlasGlossary Object while performing export operation and as AtlasGlossary didn't have any referred entities it gave raise this exception. 2. This problem is solved by handling NullPointer before trying to fetch referred entities. Diffs - repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java 4615c6c2f Diff: https://reviews.apache.org/r/74956/diff/1/ Testing (updated) --- Manual testing has been done for the same. PC link: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1608/ Thanks, Priyanshi Shah
[jira] [Created] (ATLAS-4848) Atlas 'updateTime' parameter is not updated when term is added.
Priyanshi Shah created ATLAS-4848: - Summary: Atlas 'updateTime' parameter is not updated when term is added. Key: ATLAS-4848 URL: https://issues.apache.org/jira/browse/ATLAS-4848 Project: Atlas Issue Type: Bug Components: atlas-core Reporter: Priyanshi Shah Assignee: Priyanshi Shah -- This message was sent by Atlassian Jira (v8.20.10#820010)
Review Request 74956: ATLAS-4847 : Export/Import : Atlas export fails and throws NullPointerException.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74956/ --- Review request for atlas, Jayendra Parab, Mandar Ambawane, Pinal Shah, and Sheetal Shah. Bugs: ATLAS-4847 https://issues.apache.org/jira/browse/ATLAS-4847 Repository: atlas Description --- Steps to reproduce: 1. Create a hive_table entity and assign a term to it. 2. Perform 1st incremental export of that entity without using changeMarker. 3. Without any modification on that entity perform 2nd consecutive incremental export with changeMarker from previous export. Result is it that it throws NullPointerException and export is failed. Problem: When an entity which has termed assigned was exported with changeMarker value greater than value of lastModified time of that entity it resulted in throwing NullPointerException. Solution: 1. NullPointerException occured because it was trying to fetch referred entities of AtlasGlossary Object while performing export operation and as AtlasGlossary didn't have any referred entities it gave raise this exception. 2. This problem is solved by handling NullPointer before trying to fetch referred entities. Diffs - repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java 4615c6c2f Diff: https://reviews.apache.org/r/74956/diff/1/ Testing --- Manual testing has been done for the same. Thanks, Priyanshi Shah
[jira] [Updated] (ATLAS-4847) Export/Import : Atlas export fails and throws NullPointerException.
[ https://issues.apache.org/jira/browse/ATLAS-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyanshi Shah updated ATLAS-4847: -- Description: Steps to reproduce: 1. Create a hive_table entity and assign a term to it. 2. Perform 1st incremental export of that entity without using changeMarker. 3. Without any modification on that entity perform 2nd consecutive incremental export with changeMarker from previous export. Problem is it throws NullPointerException and export is failed. was: Steps to reproduce: 1. Create a hive_table entity and assign a term to it. 2. Perform 1st incremental export of that entity without using changeMarker. 3. Without any modification on that entity perform 2nd consecutive incremental export with changeMarker from previous export. > Export/Import : Atlas export fails and throws NullPointerException. > --- > > Key: ATLAS-4847 > URL: https://issues.apache.org/jira/browse/ATLAS-4847 > Project: Atlas > Issue Type: Bug > Components: atlas-core > Reporter: Priyanshi Shah > Assignee: Priyanshi Shah >Priority: Major > Attachments: image-2024-04-11-15-15-59-551.png > > > Steps to reproduce: > 1. Create a hive_table entity and assign a term to it. > 2. Perform 1st incremental export of that entity without using changeMarker. > 3. Without any modification on that entity perform 2nd consecutive > incremental export with changeMarker from previous export. > Problem is it throws NullPointerException and export is failed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (ATLAS-4847) Export/Import : Atlas export fails and throws NullPointerException.
[ https://issues.apache.org/jira/browse/ATLAS-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyanshi Shah updated ATLAS-4847: -- Description: Steps to reproduce: 1. Create a hive_table entity and assign a term to it. 2. Perform 1st incremental export of that entity without using changeMarker. 3. Without any modification on that entity perform 2nd consecutive incremental export with changeMarker from previous export. was: Steps to reproduce: 1. Create a hive_table entity and assign a term to it. 2. Perform 1st incremental export of that entity without using changeMarker. 3. Without any modification on that entity perform 2nd consecutive incremental export with changeMarker from previous export. !image-2024-04-11-15-15-59-551.png! > Export/Import : Atlas export fails and throws NullPointerException. > --- > > Key: ATLAS-4847 > URL: https://issues.apache.org/jira/browse/ATLAS-4847 > Project: Atlas > Issue Type: Bug > Components: atlas-core > Reporter: Priyanshi Shah > Assignee: Priyanshi Shah >Priority: Major > Attachments: image-2024-04-11-15-15-59-551.png > > > Steps to reproduce: > 1. Create a hive_table entity and assign a term to it. > 2. Perform 1st incremental export of that entity without using changeMarker. > 3. Without any modification on that entity perform 2nd consecutive > incremental export with changeMarker from previous export. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (ATLAS-4847) Export/Import : Atlas export fails and throws NullPointerException.
[ https://issues.apache.org/jira/browse/ATLAS-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyanshi Shah updated ATLAS-4847: -- Attachment: image-2024-04-11-15-15-59-551.png > Export/Import : Atlas export fails and throws NullPointerException. > --- > > Key: ATLAS-4847 > URL: https://issues.apache.org/jira/browse/ATLAS-4847 > Project: Atlas > Issue Type: Bug > Components: atlas-core > Reporter: Priyanshi Shah > Assignee: Priyanshi Shah >Priority: Major > Attachments: image-2024-04-11-15-15-59-551.png > > > Steps to reproduce: > 1. Create a hive_table entity and assign a term to it. > 2. Perform 1st incremental export of that entity without using changeMarker. > 3. Without any modification on that entity perform 2nd consecutive > incremental export with changeMarker from previous export. > Problem is it fails export and throws NullPointerException. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (ATLAS-4847) Export/Import : Atlas export fails and throws NullPointerException.
[ https://issues.apache.org/jira/browse/ATLAS-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyanshi Shah updated ATLAS-4847: -- Description: Steps to reproduce: 1. Create a hive_table entity and assign a term to it. 2. Perform 1st incremental export of that entity without using changeMarker. 3. Without any modification on that entity perform 2nd consecutive incremental export with changeMarker from previous export. !image-2024-04-11-15-15-59-551.png! was: Steps to reproduce: 1. Create a hive_table entity and assign a term to it. 2. Perform 1st incremental export of that entity without using changeMarker. 3. Without any modification on that entity perform 2nd consecutive incremental export with changeMarker from previous export. Problem is it fails export and throws NullPointerException. > Export/Import : Atlas export fails and throws NullPointerException. > --- > > Key: ATLAS-4847 > URL: https://issues.apache.org/jira/browse/ATLAS-4847 > Project: Atlas > Issue Type: Bug > Components: atlas-core > Reporter: Priyanshi Shah > Assignee: Priyanshi Shah >Priority: Major > Attachments: image-2024-04-11-15-15-59-551.png > > > Steps to reproduce: > 1. Create a hive_table entity and assign a term to it. > 2. Perform 1st incremental export of that entity without using changeMarker. > 3. Without any modification on that entity perform 2nd consecutive > incremental export with changeMarker from previous export. > !image-2024-04-11-15-15-59-551.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (ATLAS-4847) Export/Import : Atlas export fails and throws NullPointerException.
[ https://issues.apache.org/jira/browse/ATLAS-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyanshi Shah updated ATLAS-4847: -- Description: Steps to reproduce: 1. Create a hive_table entity and assign a term to it. 2. Perform 1st incremental export of that entity without using changeMarker. 3. Without any modification on that entity perform 2nd consecutive incremental export with changeMarker from previous export. Problem is it fails export and throws NullPointerException. > Export/Import : Atlas export fails and throws NullPointerException. > --- > > Key: ATLAS-4847 > URL: https://issues.apache.org/jira/browse/ATLAS-4847 > Project: Atlas > Issue Type: Bug > Components: atlas-core > Reporter: Priyanshi Shah > Assignee: Priyanshi Shah >Priority: Major > > Steps to reproduce: > 1. Create a hive_table entity and assign a term to it. > 2. Perform 1st incremental export of that entity without using changeMarker. > 3. Without any modification on that entity perform 2nd consecutive > incremental export with changeMarker from previous export. > Problem is it fails export and throws NullPointerException. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (ATLAS-4845) Atlas Import is failing with fetchType: Incremental if there are no changes between two consecutive runs
[ https://issues.apache.org/jira/browse/ATLAS-4845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyanshi Shah updated ATLAS-4845: -- Description: Repro steps: # First export + import with changeMarker “0” - worked as expected. # I tried to run 2nd export + import without any changes in atlas, 2nd export also worked and gave different changeMarker in output as expected. *But 2nd import failed* In other fetch Types this will work. So we also require similar behaviour in case of Incremental type also because this is very normal scenario and customer will hit this very frequently. > Atlas Import is failing with fetchType: Incremental if there are no changes > between two consecutive runs > > > Key: ATLAS-4845 > URL: https://issues.apache.org/jira/browse/ATLAS-4845 > Project: Atlas > Issue Type: Bug > Components: atlas-core > Reporter: Priyanshi Shah > Assignee: Priyanshi Shah >Priority: Major > Attachments: ATLAS-4845.patch > > > Repro steps: > # First export + import with changeMarker “0” - worked as expected. > # I tried to run 2nd export + import without any changes in atlas, 2nd > export also worked and gave different changeMarker in output as expected. > *But 2nd import failed* > In other fetch Types this will work. So we also require similar behaviour in > case of Incremental type also because this is very normal scenario and > customer will hit this very frequently. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (ATLAS-4845) Atlas Import is failing with fetchType: Incremental if there are no changes between two consecutive runs
[ https://issues.apache.org/jira/browse/ATLAS-4845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyanshi Shah updated ATLAS-4845: -- Attachment: ATLAS-4845.patch > Atlas Import is failing with fetchType: Incremental if there are no changes > between two consecutive runs > > > Key: ATLAS-4845 > URL: https://issues.apache.org/jira/browse/ATLAS-4845 > Project: Atlas > Issue Type: Bug > Components: atlas-core > Reporter: Priyanshi Shah > Assignee: Priyanshi Shah >Priority: Major > Attachments: ATLAS-4845.patch > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (ATLAS-4847) Export/Import : Atlas export fails and throws NullPointerException.
[ https://issues.apache.org/jira/browse/ATLAS-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyanshi Shah updated ATLAS-4847: -- Summary: Export/Import : Atlas export fails and throws NullPointerException. (was: Export/Import : Atlas export fails when glossary is assigned to exported entity with fetchtype : Incremental and changeMarker value greater than lastModified time of that entity.) > Export/Import : Atlas export fails and throws NullPointerException. > --- > > Key: ATLAS-4847 > URL: https://issues.apache.org/jira/browse/ATLAS-4847 > Project: Atlas > Issue Type: Bug > Components: atlas-core > Reporter: Priyanshi Shah > Assignee: Priyanshi Shah >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (ATLAS-4847) Export/Import : Atlas export fails when glossary is assigned to exported entity with fetchtype : Incremental and changeMarker value greater than lastModified time of that
Priyanshi Shah created ATLAS-4847: - Summary: Export/Import : Atlas export fails when glossary is assigned to exported entity with fetchtype : Incremental and changeMarker value greater than lastModified time of that entity. Key: ATLAS-4847 URL: https://issues.apache.org/jira/browse/ATLAS-4847 Project: Atlas Issue Type: Bug Components: atlas-core Reporter: Priyanshi Shah Assignee: Priyanshi Shah -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: Review Request 74951: ATLAS-4845 : Atlas Import is failing with fetchType: Incremental if there are no changes between two consecutive runs
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74951/ --- (Updated April 9, 2024, 1:14 p.m.) Review request for atlas, Jayendra Parab, Mandar Ambawane, Pinal Shah, and Sheetal Shah. Bugs: ATLAS-4845 https://issues.apache.org/jira/browse/ATLAS-4845 Repository: atlas Description --- Steps: 1. First export and import with changeMarker “0” - worked as expected. 2. Try to run 2nd export + import without any changes in atlas, 2nd export also worked and gave differentchangeMarker in output as expected. Expectation: a) Empty zip should be created if no entities is fetched. Problem: Export zip file was created with three json files(atlas-typesdef.json, atlas-export-order.json and atlas-export-info.json) even though no entities was fetched and hence, while importing these zip files the import was getting failed. Solution: 1. The flag "omitZipResponseForEmptyExport" has been created which will indicate whether to create an empty zip or not when no entities are fetched during export operation. This flag is introduced to support backward compatibility for export operation. The default value of this flag is false. 2. If the "omitZipResponseForEmptyExport" flag is set to true and if no entities are fetched during export, an empty zip file is created with status code 204. Otherwise, if the flag value is false zip file is created with 3 json files (atlas-typesdef.json, atlas-export-order.json and atlas-export-info.json) with status code as 200. Diffs - intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportRequest.java b03b386c0 repository/src/main/java/org/apache/atlas/repository/impexp/ZipSink.java 5cec39dd6 webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 3aa24 Diff: https://reviews.apache.org/r/74951/diff/2/ Testing (updated) --- Manual testing for the same has been performed. PC is also successfull. https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1603/ Thanks, Priyanshi Shah
Re: Review Request 74951: ATLAS-4845 : Atlas Import is failing with fetchType: Incremental if there are no changes between two consecutive runs
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74951/ --- (Updated April 9, 2024, 5:05 a.m.) Review request for atlas, Jayendra Parab, Mandar Ambawane, Pinal Shah, and Sheetal Shah. Bugs: ATLAS-4845 https://issues.apache.org/jira/browse/ATLAS-4845 Repository: atlas Description --- Steps: 1. First export and import with changeMarker “0” - worked as expected. 2. Try to run 2nd export + import without any changes in atlas, 2nd export also worked and gave differentchangeMarker in output as expected. Expectation: a) Empty zip should be created if no entities is fetched. Problem: Export zip file was created with three json files(atlas-typesdef.json, atlas-export-order.json and atlas-export-info.json) even though no entities was fetched and hence, while importing these zip files the import was getting failed. Solution: 1. The flag "omitZipResponseForEmptyExport" has been created which will indicate whether to create an empty zip or not when no entities are fetched during export operation. This flag is introduced to support backward compatibility for export operation. The default value of this flag is false. 2. If the "omitZipResponseForEmptyExport" flag is set to true and if no entities are fetched during export, an empty zip file is created with status code 204. Otherwise, if the flag value is false zip file is created with 3 json files (atlas-typesdef.json, atlas-export-order.json and atlas-export-info.json) with status code as 200. Diffs (updated) - intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportRequest.java b03b386c0 repository/src/main/java/org/apache/atlas/repository/impexp/ZipSink.java 5cec39dd6 webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 3aa24 Diff: https://reviews.apache.org/r/74951/diff/2/ Changes: https://reviews.apache.org/r/74951/diff/1-2/ Testing --- Manual testing for the same has been performed. PC is also successfull. https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1589/ Thanks, Priyanshi Shah
Re: Review Request 74951: ATLAS-4845 : Atlas Import is failing with fetchType: Incremental if there are no changes between two consecutive runs
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74951/ --- (Updated April 8, 2024, 1:35 p.m.) Review request for atlas, Jayendra Parab, Mandar Ambawane, Pinal Shah, and Sheetal Shah. Bugs: ATLAS-4845 https://issues.apache.org/jira/browse/ATLAS-4845 Repository: atlas Description (updated) --- Steps: 1. First export and import with changeMarker “0” - worked as expected. 2. Try to run 2nd export + import without any changes in atlas, 2nd export also worked and gave differentchangeMarker in output as expected. Expectation: a) Empty zip should be created if no entities is fetched. Problem: Export zip file was created with three json files(atlas-typesdef.json, atlas-export-order.json and atlas-export-info.json) even though no entities was fetched and hence, while importing these zip files the import was getting failed. Solution: 1. The flag "omitZipResponseForEmptyExport" has been created which will indicate whether to create an empty zip or not when no entities are fetched during export operation. This flag is introduced to support backward compatibility for export operation. The default value of this flag is false. 2. If the "omitZipResponseForEmptyExport" flag is set to true and if no entities are fetched during export, an empty zip file is created with status code 204. Otherwise, if the flag value is false zip file is created with 3 json files (atlas-typesdef.json, atlas-export-order.json and atlas-export-info.json) with status code as 200. Diffs - repository/src/main/java/org/apache/atlas/repository/impexp/ZipSink.java 5cec39dd6 webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 3aa24 Diff: https://reviews.apache.org/r/74951/diff/1/ Testing --- Manual testing for the same has been performed. PC is also successfull. https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1589/ Thanks, Priyanshi Shah
[jira] [Updated] (ATLAS-4845) Atlas Import is failing with fetchType: Incremental if there are no changes between two consecutive runs
[ https://issues.apache.org/jira/browse/ATLAS-4845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyanshi Shah updated ATLAS-4845: -- Attachment: (was: ATLAS-4845.patch) > Atlas Import is failing with fetchType: Incremental if there are no changes > between two consecutive runs > > > Key: ATLAS-4845 > URL: https://issues.apache.org/jira/browse/ATLAS-4845 > Project: Atlas > Issue Type: Bug > Components: atlas-core > Reporter: Priyanshi Shah > Assignee: Priyanshi Shah >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (ATLAS-4842) Export/Import: fetchType as "incremental" does full export instead of connected
[ https://issues.apache.org/jira/browse/ATLAS-4842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyanshi Shah updated ATLAS-4842: -- Attachment: ATLAS-4842.patch > Export/Import: fetchType as "incremental" does full export instead of > connected > --- > > Key: ATLAS-4842 > URL: https://issues.apache.org/jira/browse/ATLAS-4842 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Reporter: Sheetal Shah >Assignee: Priyanshi Shah >Priority: Major > Attachments: ATLAS-4842.patch > > > Create two hive_tables in same database: > # create table hive1(id int); > # create table hive1_ctas as select * from hive1; > # create table hive2(name string); > Perform incremental export of hive1 with changeMarker as 0 and skipLineage > value as false. > Import the zip. > Expectation is only the requested hive_table hive1 and its related entities > should be exported, but it does full export and hive2 is also imported. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: Review Request 74939: ATLAS-4842 : Export/Import: fetchType as "incremental" does full export instead of connected
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74939/ --- (Updated April 5, 2024, 8:41 a.m.) Review request for atlas, Jayendra Parab, Pinal Shah, and Sheetal Shah. Bugs: ATLAS-4842 https://issues.apache.org/jira/browse/ATLAS-4842 Repository: atlas Description --- Steps: create two hive_tables in same database: 1. create table hive1(id int); 2. create table hive1_ctas as select * from hive1; 3. create table hive2(name string); 4. Perform incremental export of hive1 with changeMarker as 0 and skipLineage value as false. 5. Import the zip. Expectations: a) Expectation is only the requested hive_table hive1 and its related entities should be exported. Problem: hive2 is also imported which is an independently created entity and not related to hive1. Solution: 1. Earlier, the implementation for fetchType as INCREMENTAL was getting satisfied only if the value of skipLineage was "true" otherwise it was implementing the default fetchType i.e FULL fetch and hence, all the independently created entities in same db was also getting fetched. 2. Now, the problem is fixed by adding additional condition which will handle INCREMENTAL fetchType even if the value for skipLineage is false. Diffs (updated) - repository/src/main/java/org/apache/atlas/repository/impexp/EntitiesExtractor.java da5cf37c4 repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java 65d7a1872 repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java 0e3955dcd repository/src/test/resources/json/stocksDB-Entities/typedef-new-classification-T3.json PRE-CREATION repository/src/test/resources/json/stocksDB-Entities/typesdef-new-classification-T2.json PRE-CREATION Diff: https://reviews.apache.org/r/74939/diff/6/ Changes: https://reviews.apache.org/r/74939/diff/5-6/ Testing --- Manual testing has been done for the same. PC Link : https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1594/ Thanks, Priyanshi Shah
Re: Review Request 74939: ATLAS-4842 : Export/Import: fetchType as "incremental" does full export instead of connected
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74939/ --- (Updated April 4, 2024, 1:16 p.m.) Review request for atlas, Jayendra Parab, Pinal Shah, and Sheetal Shah. Bugs: ATLAS-4842 https://issues.apache.org/jira/browse/ATLAS-4842 Repository: atlas Description --- Steps: create two hive_tables in same database: 1. create table hive1(id int); 2. create table hive1_ctas as select * from hive1; 3. create table hive2(name string); 4. Perform incremental export of hive1 with changeMarker as 0 and skipLineage value as false. 5. Import the zip. Expectations: a) Expectation is only the requested hive_table hive1 and its related entities should be exported. Problem: hive2 is also imported which is an independently created entity and not related to hive1. Solution: 1. Earlier, the implementation for fetchType as INCREMENTAL was getting satisfied only if the value of skipLineage was "true" otherwise it was implementing the default fetchType i.e FULL fetch and hence, all the independently created entities in same db was also getting fetched. 2. Now, the problem is fixed by adding additional condition which will handle INCREMENTAL fetchType even if the value for skipLineage is false. Diffs (updated) - repository/src/main/java/org/apache/atlas/repository/impexp/EntitiesExtractor.java da5cf37c4 repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java 65d7a1872 repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java 0e3955dcd repository/src/test/resources/json/stocksDB-Entities/typedef-new-classification-T3.json PRE-CREATION repository/src/test/resources/json/stocksDB-Entities/typesdef-new-classification-T2.json PRE-CREATION Diff: https://reviews.apache.org/r/74939/diff/5/ Changes: https://reviews.apache.org/r/74939/diff/4-5/ Testing --- Manual testing has been done for the same. PC Link : https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1594/ Thanks, Priyanshi Shah
Re: Review Request 74939: ATLAS-4842 : Export/Import: fetchType as "incremental" does full export instead of connected
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74939/ --- (Updated April 4, 2024, 7:18 a.m.) Review request for atlas, Jayendra Parab, Pinal Shah, and Sheetal Shah. Bugs: ATLAS-4842 https://issues.apache.org/jira/browse/ATLAS-4842 Repository: atlas Description --- Steps: create two hive_tables in same database: 1. create table hive1(id int); 2. create table hive1_ctas as select * from hive1; 3. create table hive2(name string); 4. Perform incremental export of hive1 with changeMarker as 0 and skipLineage value as false. 5. Import the zip. Expectations: a) Expectation is only the requested hive_table hive1 and its related entities should be exported. Problem: hive2 is also imported which is an independently created entity and not related to hive1. Solution: 1. Earlier, the implementation for fetchType as INCREMENTAL was getting satisfied only if the value of skipLineage was "true" otherwise it was implementing the default fetchType i.e FULL fetch and hence, all the independently created entities in same db was also getting fetched. 2. Now, the problem is fixed by adding additional condition which will handle INCREMENTAL fetchType even if the value for skipLineage is false. Diffs (updated) - repository/src/main/java/org/apache/atlas/repository/impexp/EntitiesExtractor.java da5cf37c4 repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java 65d7a1872 repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java 0e3955dcd repository/src/test/resources/json/stocksDB-Entities/typedef-new-classification-T3.json PRE-CREATION repository/src/test/resources/json/stocksDB-Entities/typesdef-new-classification-T2.json PRE-CREATION Diff: https://reviews.apache.org/r/74939/diff/4/ Changes: https://reviews.apache.org/r/74939/diff/3-4/ Testing (updated) --- Manual testing has been done for the same. PC Link : https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1594/ Thanks, Priyanshi Shah
[jira] [Updated] (ATLAS-4845) Atlas Import is failing with fetchType: Incremental if there are no changes between two consecutive runs
[ https://issues.apache.org/jira/browse/ATLAS-4845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyanshi Shah updated ATLAS-4845: -- Attachment: ATLAS-4845.patch > Atlas Import is failing with fetchType: Incremental if there are no changes > between two consecutive runs > > > Key: ATLAS-4845 > URL: https://issues.apache.org/jira/browse/ATLAS-4845 > Project: Atlas > Issue Type: Bug > Components: atlas-core > Reporter: Priyanshi Shah > Assignee: Priyanshi Shah >Priority: Major > Attachments: ATLAS-4845.patch > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Review Request 74951: ATLAS-4845 : Atlas Import is failing with fetchType: Incremental if there are no changes between two consecutive runs
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74951/ --- Review request for atlas, Jayendra Parab, Mandar Ambawane, Pinal Shah, and Sheetal Shah. Bugs: ATLAS-4845 https://issues.apache.org/jira/browse/ATLAS-4845 Repository: atlas Description --- Steps: 1. First export and import with changeMarker “0” - worked as expected. 2. Try to run 2nd export + import without any changes in atlas, 2nd export also worked and gave differentchangeMarker in output as expected. Expectation: a) Empty zip should be created if no entities is fetched. Problem: 2nd Import is getting failed. Solution: 1. Earlier export zip was created with default json files like atlas-export-info.json, atlas-export-order.json and atlas-typesdef.json when though no entities was fetched and hence, while performing import opertaion failure was seen with Empty Zip File exception. 2. Now, the problem is fixed by returning empty zip files if no entities was fetched and the status code is set to 204. Diffs - repository/src/main/java/org/apache/atlas/repository/impexp/ZipSink.java 5cec39dd6 webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 3aa24 Diff: https://reviews.apache.org/r/74951/diff/1/ Testing --- Manual testing for the same has been performed. PC is also successfull. https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1589/ Thanks, Priyanshi Shah
Re: Review Request 74939: ATLAS-4842 : Export/Import: fetchType as "incremental" does full export instead of connected
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74939/ --- (Updated April 2, 2024, 11:34 a.m.) Review request for atlas, Jayendra Parab, Pinal Shah, and Sheetal Shah. Bugs: ATLAS-4842 https://issues.apache.org/jira/browse/ATLAS-4842 Repository: atlas Description --- Steps: create two hive_tables in same database: 1. create table hive1(id int); 2. create table hive1_ctas as select * from hive1; 3. create table hive2(name string); 4. Perform incremental export of hive1 with changeMarker as 0 and skipLineage value as false. 5. Import the zip. Expectations: a) Expectation is only the requested hive_table hive1 and its related entities should be exported. Problem: hive2 is also imported which is an independently created entity and not related to hive1. Solution: 1. Earlier, the implementation for fetchType as INCREMENTAL was getting satisfied only if the value of skipLineage was "true" otherwise it was implementing the default fetchType i.e FULL fetch and hence, all the independently created entities in same db was also getting fetched. 2. Now, the problem is fixed by adding additional condition which will handle INCREMENTAL fetchType even if the value for skipLineage is false. Diffs (updated) - repository/src/main/java/org/apache/atlas/repository/impexp/EntitiesExtractor.java da5cf37c4 repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java 65d7a1872 repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java 0e3955dcd repository/src/test/resources/json/stocksDB-Entities/typesdef-new-classification-T2.json PRE-CREATION Diff: https://reviews.apache.org/r/74939/diff/3/ Changes: https://reviews.apache.org/r/74939/diff/2-3/ Testing (updated) --- Manual testing has been done for the same. PC Link : https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1588/ Thanks, Priyanshi Shah
[jira] [Updated] (ATLAS-4838) Export/Import : changeMarker is not set to entity's lastupdatetime or its closer timestamp value
[
https://issues.apache.org/jira/browse/ATLAS-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Priyanshi Shah updated ATLAS-4838:
--
Attachment: (was: ATLAS-4838.patch)
> Export/Import : changeMarker is not set to entity's lastupdatetime or its
> closer timestamp value
>
>
> Key: ATLAS-4838
> URL: https://issues.apache.org/jira/browse/ATLAS-4838
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
>Reporter: Sharmadha S
> Assignee: Priyanshi Shah
>Priority: Major
> Attachments: ATLAS-4838-V2.patch
>
>
> When an hive table entity is exported using fetchType incremental with
> changeMarker 0 , after exporting , the changeMarker in the export response is
> not set to recent timestamp.
> For example , on exporting iceberg table with fetchType incremental and
> changeMarker as 0 :
> {code:java}
> 2024-02-08 13:28:43,838 INFO - [etp891461509-20 -
> db3f1141-531c-46c0-909a-5e443d4353a2:] ~
> export(item=AtlasObjectId{guid='null', typeName='iceberg_table',
> uniqueAttributes={qualifiedName:default.iceberg_table2@cm}}; matchType=null,
> fetchType=incremental): found 1 entities: options:
> {"itemsToExport":[{"typeName":"iceberg_table","uniqueAttributes":{"qualifiedName":"default.iceberg_table2@cm"}}],"options":{"fetchType":"incremental","changeMarker":0},"fetchTypeOptionValue":"incremental","skipLineageOptionValue":false,"changeTokenFromOptions":0}
> (StartEntityFetchByExportRequest:123)
> 2024-02-08 13:28:48,697 INFO - [etp891461509-20 -
> db3f1141-531c-46c0-909a-5e443d4353a2:] ~ Assigned guid =
> f5e6342a-dfdf-4d54-a1b3-b24f987c4ecb to newly created entity (DataAccess:85)
> 2024-02-08 13:28:48,697 INFO - [etp891461509-20 -
> db3f1141-531c-46c0-909a-5e443d4353a2:] ~ addAuditEntry: user: admin, source:
> cm, target: , operation: EXPORT (ExportImportAuditService:168)
> 2024-02-08 13:28:48,699 INFO - [etp891461509-20 -
> db3f1141-531c-46c0-909a-5e443d4353a2:] ~ <== export(user=admin,
> from=10.19.28.124): status SUCCESS: changeMarker: 1707225183928
> (ExportService:107) {code}
> changeMarker from response is 1707225183928 => Tuesday, February 6, 2024
> 6:43:03.928 PM
>
> But lastUpdatetime of iceberg table is
> {code:java}
> "createTime": 1707398866970 => Thursday, February 8, 2024 6:57:46.970 PM
> "updateTime": 1707398867375. => Thursday, February 8, 2024 6:57:47.375 PM
> {code}
> Value is set to 2 days back. This is not then an incremental export , as it
> will fetch all updates from Tuesday, February 6, 2024 6:43:03.928 PM in next
> export
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (ATLAS-4842) Export/Import: fetchType as "incremental" does full export instead of connected
[ https://issues.apache.org/jira/browse/ATLAS-4842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyanshi Shah updated ATLAS-4842: -- Attachment: (was: ATLAS-4842-V2.patch) > Export/Import: fetchType as "incremental" does full export instead of > connected > --- > > Key: ATLAS-4842 > URL: https://issues.apache.org/jira/browse/ATLAS-4842 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Reporter: Sheetal Shah >Assignee: Priyanshi Shah >Priority: Major > > Create two hive_tables in same database: > # create table hive1(id int); > # create table hive1_ctas as select * from hive1; > # create table hive2(name string); > Perform incremental export of hive1 with changeMarker as 0 and skipLineage > value as false. > Import the zip. > Expectation is only the requested hive_table hive1 and its related entities > should be exported, but it does full export and hive2 is also imported. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (ATLAS-4842) Export/Import: fetchType as "incremental" does full export instead of connected
[ https://issues.apache.org/jira/browse/ATLAS-4842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyanshi Shah updated ATLAS-4842: -- Attachment: (was: 0001-Hive-Incremental.patch) > Export/Import: fetchType as "incremental" does full export instead of > connected > --- > > Key: ATLAS-4842 > URL: https://issues.apache.org/jira/browse/ATLAS-4842 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Reporter: Sheetal Shah >Assignee: Priyanshi Shah >Priority: Major > Attachments: ATLAS-4842-V2.patch > > > Create two hive_tables in same database: > # create table hive1(id int); > # create table hive1_ctas as select * from hive1; > # create table hive2(name string); > Perform incremental export of hive1 with changeMarker as 0 and skipLineage > value as false. > Import the zip. > Expectation is only the requested hive_table hive1 and its related entities > should be exported, but it does full export and hive2 is also imported. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (ATLAS-4845) Atlas Import is failing with fetchType: Incremental if there are no changes between two consecutive runs
Priyanshi Shah created ATLAS-4845: - Summary: Atlas Import is failing with fetchType: Incremental if there are no changes between two consecutive runs Key: ATLAS-4845 URL: https://issues.apache.org/jira/browse/ATLAS-4845 Project: Atlas Issue Type: Bug Components: atlas-core Reporter: Priyanshi Shah Assignee: Priyanshi Shah -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (ATLAS-4844) Atlas - Upgrade Common Configuration2 to 2.10.1
[ https://issues.apache.org/jira/browse/ATLAS-4844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyanshi Shah updated ATLAS-4844: -- Description: Upgrading common configurations2 from 2.8.0 to 2.10.1 > Atlas - Upgrade Common Configuration2 to 2.10.1 > --- > > Key: ATLAS-4844 > URL: https://issues.apache.org/jira/browse/ATLAS-4844 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 3.0.0 > Reporter: Priyanshi Shah > Assignee: Priyanshi Shah >Priority: Major > > Upgrading common configurations2 from 2.8.0 to 2.10.1 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (ATLAS-4844) Atlas - Upgrade Common Configuration2 to 2.10.1
Priyanshi Shah created ATLAS-4844: - Summary: Atlas - Upgrade Common Configuration2 to 2.10.1 Key: ATLAS-4844 URL: https://issues.apache.org/jira/browse/ATLAS-4844 Project: Atlas Issue Type: Improvement Components: atlas-core Affects Versions: 3.0.0 Reporter: Priyanshi Shah Assignee: Priyanshi Shah -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: Review Request 74939: ATLAS-4842 : Export/Import: fetchType as "incremental" does full export instead of connected
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74939/ --- (Updated March 28, 2024, 12:59 p.m.) Review request for atlas, Jayendra Parab, Pinal Shah, and Sheetal Shah. Bugs: ATLAS-4842 https://issues.apache.org/jira/browse/ATLAS-4842 Repository: atlas Description --- Steps: create two hive_tables in same database: 1. create table hive1(id int); 2. create table hive1_ctas as select * from hive1; 3. create table hive2(name string); 4. Perform incremental export of hive1 with changeMarker as 0 and skipLineage value as false. 5. Import the zip. Expectations: a) Expectation is only the requested hive_table hive1 and its related entities should be exported. Problem: hive2 is also imported which is an independently created entity and not related to hive1. Solution: 1. Earlier, the implementation for fetchType as INCREMENTAL was getting satisfied only if the value of skipLineage was "true" otherwise it was implementing the default fetchType i.e FULL fetch and hence, all the independently created entities in same db was also getting fetched. 2. Now, the problem is fixed by adding additional condition which will handle INCREMENTAL fetchType even if the value for skipLineage is false. Diffs - repository/src/main/java/org/apache/atlas/repository/impexp/EntitiesExtractor.java da5cf37c4 repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java 65d7a1872 repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java 0e3955dcd Diff: https://reviews.apache.org/r/74939/diff/2/ Testing (updated) --- Manual testing has been done for the same. PC Link : https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1581/ Thanks, Priyanshi Shah
[jira] [Updated] (ATLAS-4842) Export/Import: fetchType as "incremental" does full export instead of connected
[ https://issues.apache.org/jira/browse/ATLAS-4842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyanshi Shah updated ATLAS-4842: -- Attachment: ATLAS-4842-V2.patch > Export/Import: fetchType as "incremental" does full export instead of > connected > --- > > Key: ATLAS-4842 > URL: https://issues.apache.org/jira/browse/ATLAS-4842 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Reporter: Sheetal Shah >Assignee: Priyanshi Shah >Priority: Major > Attachments: 0001-Hive-Incremental.patch, ATLAS-4842-V2.patch > > > Create two hive_tables in same database: > # create table hive1(id int); > # create table hive1_ctas as select * from hive1; > # create table hive2(name string); > Perform incremental export of hive1 with changeMarker as 0 and skipLineage > value as false. > Import the zip. > Expectation is only the requested hive_table hive1 and its related entities > should be exported, but it does full export and hive2 is also imported. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (ATLAS-4842) Export/Import: fetchType as "incremental" does full export instead of connected
[ https://issues.apache.org/jira/browse/ATLAS-4842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyanshi Shah updated ATLAS-4842: -- Description: Create two hive_tables in same database: # create table hive1(id int); # create table hive1_ctas as select * from hive1; # create table hive2(name string); Perform incremental export of hive1 with changeMarker as 0 and skipLineage value as false. Import the zip. Expectation is only the requested hive_table hive1 and its related entities should be exported, but it does full export and hive2 is also imported. was: Create two hive_tables in same database: # create table hive1(id int); # create table hive2(name string); Perform incremental export of hive1 with changeMarker provided by previous export. Import the zip. Expectation is only hive1 to be exported and imported, but it does full export and hive2 is also imported. > Export/Import: fetchType as "incremental" does full export instead of > connected > --- > > Key: ATLAS-4842 > URL: https://issues.apache.org/jira/browse/ATLAS-4842 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Reporter: Sheetal Shah >Assignee: Priyanshi Shah >Priority: Major > Attachments: 0001-Hive-Incremental.patch, ATLAS-4842-V2.patch > > > Create two hive_tables in same database: > # create table hive1(id int); > # create table hive1_ctas as select * from hive1; > # create table hive2(name string); > Perform incremental export of hive1 with changeMarker as 0 and skipLineage > value as false. > Import the zip. > Expectation is only the requested hive_table hive1 and its related entities > should be exported, but it does full export and hive2 is also imported. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: Review Request 74939: ATLAS-4842 : Export/Import: fetchType as "incremental" does full export instead of connected
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74939/ --- (Updated March 28, 2024, 12:53 p.m.) Review request for atlas, Jayendra Parab, Pinal Shah, and Sheetal Shah. Bugs: ATLAS-4842 https://issues.apache.org/jira/browse/ATLAS-4842 Repository: atlas Description (updated) --- Steps: create two hive_tables in same database: 1. create table hive1(id int); 2. create table hive1_ctas as select * from hive1; 3. create table hive2(name string); 4. Perform incremental export of hive1 with changeMarker as 0 and skipLineage value as false. 5. Import the zip. Expectations: a) Expectation is only the requested hive_table hive1 and its related entities should be exported. Problem: hive2 is also imported which is an independently created entity and not related to hive1. Solution: 1. Earlier, the implementation for fetchType as INCREMENTAL was getting satisfied only if the value of skipLineage was "true" otherwise it was implementing the default fetchType i.e FULL fetch and hence, all the independently created entities in same db was also getting fetched. 2. Now, the problem is fixed by adding additional condition which will handle INCREMENTAL fetchType even if the value for skipLineage is false. Diffs (updated) - repository/src/main/java/org/apache/atlas/repository/impexp/EntitiesExtractor.java da5cf37c4 repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java 65d7a1872 repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java 0e3955dcd Diff: https://reviews.apache.org/r/74939/diff/2/ Changes: https://reviews.apache.org/r/74939/diff/1-2/ Testing --- Manual testing has been done for the same. PC Link : https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1573/ Thanks, Priyanshi Shah
Re: Review Request 74947: ATLAS-4838 : Export/Import : changeMarker is not set to entity's lastupdatetime or its closer timestamp value
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74947/ --- (Updated March 28, 2024, 11:20 a.m.) Review request for atlas, Jayendra Parab, Mandar Ambawane, Pinal Shah, and Sheetal Shah. Bugs: ATLAS-4838 https://issues.apache.org/jira/browse/ATLAS-4838 Repository: atlas Description (updated) --- Steps: When an hive table entity is exported using fetchType incremental with changeMarker 0 , after exporting , the changeMarker in the export response is not set to recent timestamp. Problem: The value of changeMarker was set on the basis of the earliest request recorded on the server. There was some unused pending requests contexts which were not cleared after its usage and hence, changeMarker the value was changeMarker was wrongly set. Solution: Have cleared all the unused pending requests which was found. Diffs - repository/src/main/java/org/apache/atlas/repository/patches/AtlasPatchManager.java e72a87713 webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 0025276bb webapp/src/main/java/org/apache/atlas/web/service/EmbeddedServer.java 4c3a1c569 webapp/src/main/java/org/apache/atlas/web/service/ServiceState.java ea74d21ad Diff: https://reviews.apache.org/r/74947/diff/2/ Testing (updated) --- Performed multiple export operations with incremental fetch type, the change marker values were updated properly on each export operation. PC link: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1578/ Thanks, Priyanshi Shah
[jira] [Updated] (ATLAS-4838) Export/Import : changeMarker is not set to entity's lastupdatetime or its closer timestamp value
[
https://issues.apache.org/jira/browse/ATLAS-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Priyanshi Shah updated ATLAS-4838:
--
Attachment: ATLAS-4838-V2.patch
> Export/Import : changeMarker is not set to entity's lastupdatetime or its
> closer timestamp value
>
>
> Key: ATLAS-4838
> URL: https://issues.apache.org/jira/browse/ATLAS-4838
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
>Reporter: Sharmadha S
> Assignee: Priyanshi Shah
>Priority: Major
> Attachments: ATLAS-4838-V2.patch, ATLAS-4838.patch
>
>
> When an hive table entity is exported using fetchType incremental with
> changeMarker 0 , after exporting , the changeMarker in the export response is
> not set to recent timestamp.
> For example , on exporting iceberg table with fetchType incremental and
> changeMarker as 0 :
> {code:java}
> 2024-02-08 13:28:43,838 INFO - [etp891461509-20 -
> db3f1141-531c-46c0-909a-5e443d4353a2:] ~
> export(item=AtlasObjectId{guid='null', typeName='iceberg_table',
> uniqueAttributes={qualifiedName:default.iceberg_table2@cm}}; matchType=null,
> fetchType=incremental): found 1 entities: options:
> {"itemsToExport":[{"typeName":"iceberg_table","uniqueAttributes":{"qualifiedName":"default.iceberg_table2@cm"}}],"options":{"fetchType":"incremental","changeMarker":0},"fetchTypeOptionValue":"incremental","skipLineageOptionValue":false,"changeTokenFromOptions":0}
> (StartEntityFetchByExportRequest:123)
> 2024-02-08 13:28:48,697 INFO - [etp891461509-20 -
> db3f1141-531c-46c0-909a-5e443d4353a2:] ~ Assigned guid =
> f5e6342a-dfdf-4d54-a1b3-b24f987c4ecb to newly created entity (DataAccess:85)
> 2024-02-08 13:28:48,697 INFO - [etp891461509-20 -
> db3f1141-531c-46c0-909a-5e443d4353a2:] ~ addAuditEntry: user: admin, source:
> cm, target: , operation: EXPORT (ExportImportAuditService:168)
> 2024-02-08 13:28:48,699 INFO - [etp891461509-20 -
> db3f1141-531c-46c0-909a-5e443d4353a2:] ~ <== export(user=admin,
> from=10.19.28.124): status SUCCESS: changeMarker: 1707225183928
> (ExportService:107) {code}
> changeMarker from response is 1707225183928 => Tuesday, February 6, 2024
> 6:43:03.928 PM
>
> But lastUpdatetime of iceberg table is
> {code:java}
> "createTime": 1707398866970 => Thursday, February 8, 2024 6:57:46.970 PM
> "updateTime": 1707398867375. => Thursday, February 8, 2024 6:57:47.375 PM
> {code}
> Value is set to 2 days back. This is not then an incremental export , as it
> will fetch all updates from Tuesday, February 6, 2024 6:43:03.928 PM in next
> export
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: Review Request 74947: ATLAS-4838 : Export/Import : changeMarker is not set to entity's lastupdatetime or its closer timestamp value
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74947/ --- (Updated March 28, 2024, 4:41 a.m.) Review request for atlas, Jayendra Parab, Mandar Ambawane, Pinal Shah, and Sheetal Shah. Bugs: ATLAS-4838 https://issues.apache.org/jira/browse/ATLAS-4838 Repository: atlas Description --- Steps: When an hive table entity is exported using fetchType incremental with changeMarker 0 , after exporting , the changeMarker in the export response is not set to recent timestamp. Problem: The value of changeMarker was set on the basis of the earliest request recorded on the server. There was some unused pending requests which was not cleared after its usage and hence, changeMarker the value was changeMarker was wrongly set. Solution: Have cleared all the unused pending requests which was found. Diffs (updated) - repository/src/main/java/org/apache/atlas/repository/patches/AtlasPatchManager.java e72a87713 webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 0025276bb webapp/src/main/java/org/apache/atlas/web/service/EmbeddedServer.java 4c3a1c569 webapp/src/main/java/org/apache/atlas/web/service/ServiceState.java ea74d21ad Diff: https://reviews.apache.org/r/74947/diff/2/ Changes: https://reviews.apache.org/r/74947/diff/1-2/ Testing (updated) --- Manual testing has been done for the same and also pre-commit got passed. PC link: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1578/ Thanks, Priyanshi Shah
Review Request 74947: ATLAS-4838 : Export/Import : changeMarker is not set to entity's lastupdatetime or its closer timestamp value
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74947/ --- Review request for atlas, Jayendra Parab, Mandar Ambawane, Pinal Shah, and Sheetal Shah. Bugs: ATLAS-4838 https://issues.apache.org/jira/browse/ATLAS-4838 Repository: atlas Description --- Steps: When an hive table entity is exported using fetchType incremental with changeMarker 0 , after exporting , the changeMarker in the export response is not set to recent timestamp. Problem: The value of changeMarker was set on the basis of the earliest request recorded on the server. There was some unused pending requests which was not cleared after its usage and hence, changeMarker the value was changeMarker was wrongly set. Solution: Have cleared all the unused pending requests which was found. Diffs - repository/src/main/java/org/apache/atlas/repository/patches/AtlasPatchManager.java e72a87713 webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 0025276bb webapp/src/main/java/org/apache/atlas/web/service/EmbeddedServer.java 4c3a1c569 webapp/src/main/java/org/apache/atlas/web/service/ServiceState.java ea74d21ad Diff: https://reviews.apache.org/r/74947/diff/1/ Testing --- Manual testing has been done for the same. Thanks, Priyanshi Shah
[jira] [Assigned] (ATLAS-4838) Export/Import : changeMarker is not set to entity's lastupdatetime or its closer timestamp value
[
https://issues.apache.org/jira/browse/ATLAS-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Priyanshi Shah reassigned ATLAS-4838:
-
Assignee: Priyanshi Shah (was: Jayendra Parab)
> Export/Import : changeMarker is not set to entity's lastupdatetime or its
> closer timestamp value
>
>
> Key: ATLAS-4838
> URL: https://issues.apache.org/jira/browse/ATLAS-4838
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
>Reporter: Sharmadha S
> Assignee: Priyanshi Shah
>Priority: Major
> Attachments: ATLAS-4838.patch
>
>
> When an hive table entity is exported using fetchType incremental with
> changeMarker 0 , after exporting , the changeMarker in the export response is
> not set to recent timestamp.
> For example , on exporting iceberg table with fetchType incremental and
> changeMarker as 0 :
> {code:java}
> 2024-02-08 13:28:43,838 INFO - [etp891461509-20 -
> db3f1141-531c-46c0-909a-5e443d4353a2:] ~
> export(item=AtlasObjectId{guid='null', typeName='iceberg_table',
> uniqueAttributes={qualifiedName:default.iceberg_table2@cm}}; matchType=null,
> fetchType=incremental): found 1 entities: options:
> {"itemsToExport":[{"typeName":"iceberg_table","uniqueAttributes":{"qualifiedName":"default.iceberg_table2@cm"}}],"options":{"fetchType":"incremental","changeMarker":0},"fetchTypeOptionValue":"incremental","skipLineageOptionValue":false,"changeTokenFromOptions":0}
> (StartEntityFetchByExportRequest:123)
> 2024-02-08 13:28:48,697 INFO - [etp891461509-20 -
> db3f1141-531c-46c0-909a-5e443d4353a2:] ~ Assigned guid =
> f5e6342a-dfdf-4d54-a1b3-b24f987c4ecb to newly created entity (DataAccess:85)
> 2024-02-08 13:28:48,697 INFO - [etp891461509-20 -
> db3f1141-531c-46c0-909a-5e443d4353a2:] ~ addAuditEntry: user: admin, source:
> cm, target: , operation: EXPORT (ExportImportAuditService:168)
> 2024-02-08 13:28:48,699 INFO - [etp891461509-20 -
> db3f1141-531c-46c0-909a-5e443d4353a2:] ~ <== export(user=admin,
> from=10.19.28.124): status SUCCESS: changeMarker: 1707225183928
> (ExportService:107) {code}
> changeMarker from response is 1707225183928 => Tuesday, February 6, 2024
> 6:43:03.928 PM
>
> But lastUpdatetime of iceberg table is
> {code:java}
> "createTime": 1707398866970 => Thursday, February 8, 2024 6:57:46.970 PM
> "updateTime": 1707398867375. => Thursday, February 8, 2024 6:57:47.375 PM
> {code}
> Value is set to 2 days back. This is not then an incremental export , as it
> will fetch all updates from Tuesday, February 6, 2024 6:43:03.928 PM in next
> export
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (ATLAS-4838) Export/Import : changeMarker is not set to entity's lastupdatetime or its closer timestamp value
[
https://issues.apache.org/jira/browse/ATLAS-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Priyanshi Shah updated ATLAS-4838:
--
Attachment: ATLAS-4838.patch
> Export/Import : changeMarker is not set to entity's lastupdatetime or its
> closer timestamp value
>
>
> Key: ATLAS-4838
> URL: https://issues.apache.org/jira/browse/ATLAS-4838
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
>Reporter: Sharmadha S
>Assignee: Jayendra Parab
>Priority: Major
> Attachments: ATLAS-4838.patch
>
>
> When an hive table entity is exported using fetchType incremental with
> changeMarker 0 , after exporting , the changeMarker in the export response is
> not set to recent timestamp.
> For example , on exporting iceberg table with fetchType incremental and
> changeMarker as 0 :
> {code:java}
> 2024-02-08 13:28:43,838 INFO - [etp891461509-20 -
> db3f1141-531c-46c0-909a-5e443d4353a2:] ~
> export(item=AtlasObjectId{guid='null', typeName='iceberg_table',
> uniqueAttributes={qualifiedName:default.iceberg_table2@cm}}; matchType=null,
> fetchType=incremental): found 1 entities: options:
> {"itemsToExport":[{"typeName":"iceberg_table","uniqueAttributes":{"qualifiedName":"default.iceberg_table2@cm"}}],"options":{"fetchType":"incremental","changeMarker":0},"fetchTypeOptionValue":"incremental","skipLineageOptionValue":false,"changeTokenFromOptions":0}
> (StartEntityFetchByExportRequest:123)
> 2024-02-08 13:28:48,697 INFO - [etp891461509-20 -
> db3f1141-531c-46c0-909a-5e443d4353a2:] ~ Assigned guid =
> f5e6342a-dfdf-4d54-a1b3-b24f987c4ecb to newly created entity (DataAccess:85)
> 2024-02-08 13:28:48,697 INFO - [etp891461509-20 -
> db3f1141-531c-46c0-909a-5e443d4353a2:] ~ addAuditEntry: user: admin, source:
> cm, target: , operation: EXPORT (ExportImportAuditService:168)
> 2024-02-08 13:28:48,699 INFO - [etp891461509-20 -
> db3f1141-531c-46c0-909a-5e443d4353a2:] ~ <== export(user=admin,
> from=10.19.28.124): status SUCCESS: changeMarker: 1707225183928
> (ExportService:107) {code}
> changeMarker from response is 1707225183928 => Tuesday, February 6, 2024
> 6:43:03.928 PM
>
> But lastUpdatetime of iceberg table is
> {code:java}
> "createTime": 1707398866970 => Thursday, February 8, 2024 6:57:46.970 PM
> "updateTime": 1707398867375. => Thursday, February 8, 2024 6:57:47.375 PM
> {code}
> Value is set to 2 days back. This is not then an incremental export , as it
> will fetch all updates from Tuesday, February 6, 2024 6:43:03.928 PM in next
> export
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: Review Request 74939: ATLAS-4842 : Export/Import: fetchType as "incremental" does full export instead of connected
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74939/ --- (Updated March 21, 2024, 8:59 a.m.) Review request for atlas, Jayendra Parab, Pinal Shah, and Sheetal Shah. Bugs: ATLAS-4842 https://issues.apache.org/jira/browse/ATLAS-4842 Repository: atlas Description (updated) --- Steps: create two hive_tables in same database: 1. create table hive1(id int); 2. create table hive2(name string); 3. Perform incremental export of hive1 with changeMarker provided by previous export. 4. Import the zip. Expectations: a) Expectation is only the requested hive_table hive1 should be exported. Problem: hive2 is also imported. Solution: 1. Earlier, the implementation for fetchType as INCREMENTAL was getting satisfied only if the value of skipLineage was "true" otherwise it was implementing the default fetchType i.e FULL fetch and hence, all the unrelated entities was also getting fetched. 2. Now, the problem is fixed by adding additional condition which will handle INCREMENTAL fetchType even if the value for skipLineage is false. Diffs - repository/src/main/java/org/apache/atlas/repository/impexp/EntitiesExtractor.java da5cf37c4 repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java 65d7a1872 Diff: https://reviews.apache.org/r/74939/diff/1/ Testing --- Manual testing has been done for the same. PC Link : https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1573/ Thanks, Priyanshi Shah
Re: Review Request 74939: ATLAS-4842 : Export/Import: fetchType as "incremental" does full export instead of connected
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74939/ --- (Updated March 20, 2024, 12:28 p.m.) Review request for atlas, Jayendra Parab, Pinal Shah, and Sheetal Shah. Bugs: ATLAS-4842 https://issues.apache.org/jira/browse/ATLAS-4842 Repository: atlas Description --- Steps: create two hive_tables in same database: 1. create table hive1(id int); 2. create table hive2(name string); 3. Perform incremental export of hive1 with changeMarker provided by previous export. 4. Import the zip. Expectations: a) Expectation is only the requested hive_table hive1 should be exported. Problem: hive2 is also imported. Diffs - repository/src/main/java/org/apache/atlas/repository/impexp/EntitiesExtractor.java da5cf37c4 repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java 65d7a1872 Diff: https://reviews.apache.org/r/74939/diff/1/ Testing (updated) --- Manual testing has been done for the same. PC Link : https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1573/ Thanks, Priyanshi Shah
[jira] [Updated] (ATLAS-4842) Export/Import: fetchType as "incremental" does full export instead of connected
[ https://issues.apache.org/jira/browse/ATLAS-4842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyanshi Shah updated ATLAS-4842: -- Attachment: 0001-Hive-Incremental.patch > Export/Import: fetchType as "incremental" does full export instead of > connected > --- > > Key: ATLAS-4842 > URL: https://issues.apache.org/jira/browse/ATLAS-4842 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Reporter: Sheetal Shah >Assignee: Priyanshi Shah >Priority: Major > Attachments: 0001-Hive-Incremental.patch > > > Create two hive_tables in same database: > # create table hive1(id int); > # create table hive2(name string); > Perform incremental export of hive1 with changeMarker provided by previous > export. > Import the zip. > Expectation is only hive1 to be exported and imported, but it does full > export and hive2 is also imported. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: Review Request 74939: ATLAS-4842 : Export/Import: fetchType as "incremental" does full export instead of connected
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74939/ --- (Updated March 19, 2024, 9:03 a.m.) Review request for atlas, Jayendra Parab, Pinal Shah, and Sheetal Shah. Summary (updated) - ATLAS-4842 : Export/Import: fetchType as "incremental" does full export instead of connected Bugs: ATLAS-4842 https://issues.apache.org/jira/browse/ATLAS-4842 Repository: atlas Description --- Steps: create two hive_tables in same database: 1. create table hive1(id int); 2. create table hive2(name string); 3. Perform incremental export of hive1 with changeMarker provided by previous export. 4. Import the zip. Expectations: a) Expectation is only the requested hive_table hive1 should be exported. Problem: hive2 is also imported. Diffs - repository/src/main/java/org/apache/atlas/repository/impexp/EntitiesExtractor.java da5cf37c4 repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java 65d7a1872 Diff: https://reviews.apache.org/r/74939/diff/1/ Testing --- Manual testing has been done for the same. Thanks, Priyanshi Shah
Re: Review Request 74939: ATLAS-4839 : Export/Import: fetchType as "incremental" does full export instead of connected
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74939/ --- (Updated March 18, 2024, 7:48 a.m.) Review request for atlas, Jayendra Parab, Pinal Shah, and Sheetal Shah. Bugs: ATLAS-4839 https://issues.apache.org/jira/browse/ATLAS-4839 Repository: atlas Description (updated) --- Steps: create two hive_tables in same database: 1. create table hive1(id int); 2. create table hive2(name string); 3. Perform incremental export of hive1 with changeMarker provided by previous export. 4. Import the zip. Expectations: a) Expectation is only the requested hive_table hive1 should be exported. Problem: hive2 is also imported. Diffs - repository/src/main/java/org/apache/atlas/repository/impexp/EntitiesExtractor.java da5cf37c4 repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java 65d7a1872 Diff: https://reviews.apache.org/r/74939/diff/1/ Testing --- Manual testing has been done for the same. Thanks, Priyanshi Shah
Re: Review Request 74939: ATLAS-4839 : Export/Import: fetchType as "incremental" does full export instead of connected
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74939/ --- (Updated March 18, 2024, 7:41 a.m.) Review request for atlas, Jayendra Parab, Pinal Shah, and Sheetal Shah. Bugs: ATLAS-4839 https://issues.apache.org/jira/browse/ATLAS-4839 Repository: atlas Description (updated) --- Steps: create two hive_tables in same database: 1. create table hive1(id int); 2. create table hive2(name string); 3. Perform incremental export of hive1 with changeMarker provided by previous export. 4. Import the zip. Expectations: a) Expectation is only the requested hive_table hive1 should be exported, but actually hive2 is also imported. Diffs - repository/src/main/java/org/apache/atlas/repository/impexp/EntitiesExtractor.java da5cf37c4 repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java 65d7a1872 Diff: https://reviews.apache.org/r/74939/diff/1/ Testing (updated) --- Manual testing has been done for the same. Thanks, Priyanshi Shah
Re: Review Request 74939: ATLAS-4839 : Export/Import: fetchType as "incremental" does full export instead of connected
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74939/ --- (Updated March 18, 2024, 7:34 a.m.) Review request for atlas, Jayendra Parab, Pinal Shah, and Sheetal Shah. Bugs: https://issues.apache.org/jira/browse/ATLAS-4839 https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/ATLAS-4839 Repository: atlas Description --- Create 2 tables create table hive_table_1(id int) stored as iceberg; create table hive_table_2(id int); Export iceberg table using fetchType incremental and changeMarker 0 Import the zip. Expectation is only hive_table_1 to be exported and imported, but it does full export and hive_table_2 is also imported. Diffs - repository/src/main/java/org/apache/atlas/repository/impexp/EntitiesExtractor.java da5cf37c4 repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java 65d7a1872 Diff: https://reviews.apache.org/r/74939/diff/1/ Testing (updated) --- Steps to repro: create two hive_tables in same database: 1. create table hive1(id int); 2. create table hive2(name string); Expectations: a) Perform incremental export of hive1 with changeMarker provided by previous export. b) Import the zip. c) Expectation is only the requested hive_table hive1 should be exported, but actually hive2 is also imported. Manual testing has been done for the same. Thanks, Priyanshi Shah
Re: Review Request 74939: ATLAS-4839 : Export/Import: fetchType as "incremental" does full export instead of connected
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74939/ --- (Updated March 18, 2024, 7:33 a.m.) Review request for atlas, Jayendra Parab, Pinal Shah, and Sheetal Shah. Bugs: https://issues.apache.org/jira/browse/ATLAS-4839 https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/ATLAS-4839 Repository: atlas Description --- Create 2 tables create table hive_table_1(id int) stored as iceberg; create table hive_table_2(id int); Export iceberg table using fetchType incremental and changeMarker 0 Import the zip. Expectation is only hive_table_1 to be exported and imported, but it does full export and hive_table_2 is also imported. Diffs - repository/src/main/java/org/apache/atlas/repository/impexp/EntitiesExtractor.java da5cf37c4 repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java 65d7a1872 Diff: https://reviews.apache.org/r/74939/diff/1/ Testing (updated) --- Steps to repro: create two hive_tables in same database: 1. create table hive1(id int); 2. create table hive2(name string); Expectations: a) Perform incremental export of hive1 with changeMarker provided by previous export. b) Import the zip. c) Expectation is only the requested hive_table hive1 should be exported, but actually hive2 is also imported. Thanks, Priyanshi Shah
Review Request 74939: ATLAS-4839 : Export/Import: fetchType as "incremental" does full export instead of connected
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74939/
---
Review request for atlas, Jayendra Parab, Pinal Shah, and Sheetal Shah.
Repository: atlas
Description
---
Create 2 tables
create table hive_table_1(id int) stored as iceberg;
create table hive_table_2(id int);
Export iceberg table using fetchType incremental and changeMarker 0
Import the zip.
Expectation is only hive_table_1 to be exported and imported, but it does full
export and hive_table_2 is also imported.
Diffs
-
repository/src/main/java/org/apache/atlas/repository/impexp/EntitiesExtractor.java
da5cf37c4
repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java
65d7a1872
Diff: https://reviews.apache.org/r/74939/diff/1/
Testing
---
Steps to repro:
create two hive_tables in same database:
1. create table hive1(id int);
2. create table hive2(name string);
Expectations:
And perform incremental export of hive1 with changeMarker provided by previous
export.
Import the zip.
Expectation is only the requested hive_table hive1 should be exported, but
actually hive2 is also imported.
Before patch:
Export command:
--
curl -X POST -u admin:admin -H "Content-Type: application/json" -H
"Cache-Control: no-cache" -d '{ "itemsToExport": [ { "typeName": "hive_table",
"uniqueAttributes": { "qualifiedName": "test.hive1@cm" } } ], "options": {
"fetchType": "INCREMENTAL"} }'
"http://ccycloud-2.ppspark37218.root.comops.site:31000/api/atlas/admin/export;
> exportOutput.zip
Import command:
--
curl -g -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H
"Cache-Control: no-cache" -F data=@exportOutput.zip
"http://localhost:21000/api/atlas/admin/import;
After patch:
Export command:
---
curl -X POST -u admin:admin -H "Content-Type: application/json" -H
"Cache-Control: no-cache" -d '{ "itemsToExport": [ { "typeName": "hive_table",
"uniqueAttributes": { "qualifiedName": "test.hive1@cm" } } ], "options": {
"fetchType": "INCREMENTAL"} }' "http://localhost:21000/api/atlas/admin/export;
> exportOutputAfter.zip
Import command:
--
curl -g -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H
"Cache-Control: no-cache" -F data=@exportOutputAfter.zip
"http://ccycloud-1.ps7218052.root.comops.site:31000/api/atlas/admin/import;
Thanks,
Priyanshi Shah
[jira] [Assigned] (ATLAS-4839) Export/Import: fetchType as "incremental" does full export instead of connected
[ https://issues.apache.org/jira/browse/ATLAS-4839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyanshi Shah reassigned ATLAS-4839: - Assignee: Priyanshi Shah > Export/Import: fetchType as "incremental" does full export instead of > connected > --- > > Key: ATLAS-4839 > URL: https://issues.apache.org/jira/browse/ATLAS-4839 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Reporter: Sharmadha S >Assignee: Priyanshi Shah >Priority: Major > > Create 2 tables > # create table iceberg_table_1(id int) stored as iceberg; > # create table hive_table_1(id int); > Export iceberg table using fetchType incremental and changeMarker 0 > > Import the zip. > > Expectation is only iceberg_table to be exported and imported, but it does > full export and hive_table_1 is also imported. -- This message was sent by Atlassian Jira (v8.20.10#820010)
To subscribe on apache atlas
Subscribe.
Re: Request to join Atlas project
No worries, thanks for the update! On Sat, Feb 17, 2024 at 1:09 AM Madhan Neethiraj wrote: > Priyanshi Shah, > > I apologize for the delay in responding to your email. Thank you for your > interest in contributing to Apache Atlas. You now have contributor access. > > Welcome to Apache Atlas. > > Thanks, > Madhan > > > > On 2/16/24, 11:33 AM, "Priyanshi Shah" < > priyanshi.s...@freestoneinfotech.com priyanshi.s...@freestoneinfotech.com>> wrote: > > > Hello Team, > > > A gentle reminder on the above requests. > > > Regards, > Priyanshi Shah > > > On Wed, Feb 7, 2024 at 6:58 PM Priyanshi Shah < > priyanshi.s...@freestoneinfotech.com priyanshi.s...@freestoneinfotech.com>> wrote: > > > > Hello team, > > > > A gentle reminder on the above request. > > > > Regards, > > Priyanshi Shah > > > > On Wed, Nov 1, 2023 at 3:10 PM Priyanshi Shah < > > priyanshi.s...@freestoneinfotech.com priyanshi.s...@freestoneinfotech.com>> wrote: > > > >> Hi team, > >> > >> I would like to contribute to the Atlas project, Can you please add me > as > >> contributor. My Apache id is : priyanshi.shah > >> > >> Regards, > >> Priyanshi Shah > >> > > > > > > >
Re: Request to join Atlas project
Hello Team, A gentle reminder on the above requests. Regards, Priyanshi Shah On Wed, Feb 7, 2024 at 6:58 PM Priyanshi Shah < priyanshi.s...@freestoneinfotech.com> wrote: > Hello team, > > A gentle reminder on the above request. > > Regards, > Priyanshi Shah > > On Wed, Nov 1, 2023 at 3:10 PM Priyanshi Shah < > priyanshi.s...@freestoneinfotech.com> wrote: > >> Hi team, >> >> I would like to contribute to the Atlas project, Can you please add me as >> contributor. My Apache id is : priyanshi.shah >> >> Regards, >> Priyanshi Shah >> >
Re: Request to join Atlas project
Hello team, A gentle reminder on the above request. Regards, Priyanshi Shah On Wed, Nov 1, 2023 at 3:10 PM Priyanshi Shah < priyanshi.s...@freestoneinfotech.com> wrote: > Hi team, > > I would like to contribute to the Atlas project, Can you please add me as > contributor. My Apache id is : priyanshi.shah > > Regards, > Priyanshi Shah >
Request to join Atlas project
Hi team, I would like to contribute to the Atlas project, Can you please add me as contributor. My Apache id is : priyanshi.shah Regards, Priyanshi Shah
