Sharmadha S created ATLAS-4838:
----------------------------------
Summary: Export/Import : changeMarker is not set to entity's
lastupdatetime or its closer timestamp value
Key: ATLAS-4838
URL: https://issues.apache.org/jira/browse/ATLAS-4838
Project: Atlas
Issue Type: Bug
Components: atlas-core
Reporter: Sharmadha S
When an hive table entity is exported using fetchType incremental with
changeMarker 0 , after exporting , the changeMarker in the export response is
not set to recent timestamp.
For example , on exporting iceberg table with fetchType incremental and
changeMarker as 0 :
{code:java}
2024-02-08 13:28:43,838 INFO - [etp891461509-20 -
db3f1141-531c-46c0-909a-5e443d4353a2:] ~ export(item=AtlasObjectId{guid='null',
typeName='iceberg_table',
uniqueAttributes={qualifiedName:default.iceberg_table2@cm}}; matchType=null,
fetchType=incremental): found 1 entities: options:
{"itemsToExport":[{"typeName":"iceberg_table","uniqueAttributes":{"qualifiedName":"default.iceberg_table2@cm"}}],"options":{"fetchType":"incremental","changeMarker":0},"fetchTypeOptionValue":"incremental","skipLineageOptionValue":false,"changeTokenFromOptions":0}
(StartEntityFetchByExportRequest:123)
2024-02-08 13:28:48,697 INFO - [etp891461509-20 -
db3f1141-531c-46c0-909a-5e443d4353a2:] ~ Assigned guid =
f5e6342a-dfdf-4d54-a1b3-b24f987c4ecb to newly created entity (DataAccess:85)
2024-02-08 13:28:48,697 INFO - [etp891461509-20 -
db3f1141-531c-46c0-909a-5e443d4353a2:] ~ addAuditEntry: user: admin, source:
cm, target: , operation: EXPORT (ExportImportAuditService:168)
2024-02-08 13:28:48,699 INFO - [etp891461509-20 -
db3f1141-531c-46c0-909a-5e443d4353a2:] ~ <== export(user=admin,
from=10.19.28.124): status SUCCESS: changeMarker: 1707225183928
(ExportService:107) {code}
changeMarker from response is 1707225183928 => Tuesday, February 6, 2024
6:43:03.928 PM
But lastUpdatetime of iceberg table is
{code:java}
"createTime": 1707398866970 => Thursday, February 8, 2024 6:57:46.970 PM
"updateTime": 1707398867375. => Thursday, February 8, 2024 6:57:47.375 PM {code}
Value is set to 2 days back. This is not then an incremental export , as it
will fetch all updates from Tuesday, February 6, 2024 6:43:03.928 PM in next
export
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
