Umesh Padashetty created ATLAS-4014:
---------------------------------------
Summary: changeMarker in Atlas Export API response does not get
updated
Key: ATLAS-4014
URL: https://issues.apache.org/jira/browse/ATLAS-4014
Project: Atlas
Issue Type: Bug
Components: atlas-core
Affects Versions: 2.1.0
Reporter: Umesh Padashetty
As per [https://atlas.apache.org/index.html#/IncrementalExport,] changeMarker
is used in conjunction with "fetchType": "incremental" to incrementally export
the metadata, instead of exporting everything, thereby reducing the redundancy.
As per this logic, the Atlas Export API returns a new change changeMarker after
every successful Export API call (provided there is a change in Atlas metadata).
I ran the Atlas export on a DB. And then added few tables into the DB and then
again ran the Export. Expectation was changeMarker would be updated. But it is
being observed that the changeMarker never changes and is currently always set
to 1604003211134 in the instance I am testing.
The same changeMarker is set in the atlas-export-info.json too.
The changeMarker is used by Hive Replication Manager for incremental Export.
Since atlas is always returning the same changeMarker, Hive is always passing
back the same changeMarker to atlas in the eventual export calls. This is
leading to full export, instead of incremental export as desired.
Following is the result of grep i did in atlas server logs. As you can see,
changeMarker returned is always 1604003211134.
{code:java}
[root@quasar-ggealw-4 atlas]# grep "<== export" application.log | grep
changeMarker 2020-10-30 05:48:54,451 INFO - [etp1261153343-492 -
bd3b60a4-a2b3-4f5c-96c5-166b09157828:] ~ <== export(user=hrt_qa,
from=10.101.80.202): status SUCCESS: changeMarker: 1604003211134
(ExportService:101) 2020-10-30 06:03:43,282 INFO - [etp1261153343-483 -
3114c858-8f89-4d8f-917c-2c7a3d695d2f:hive:POST/api/atlas/admin/export] ~ <==
export(user=hive, from=172.27.160.133): status SUCCESS: changeMarker:
1604003211134 (ExportService:101) 2020-10-30 06:15:43,880 INFO -
[etp1261153343-489 -
ccb8da65-e433-44a4-a470-898843dccb17:hive:POST/api/atlas/admin/export] ~ <==
export(user=hive, from=172.27.160.133): status SUCCESS: changeMarker:
1604003211134 (ExportService:101) 2020-10-30 06:23:10,810 INFO -
[etp1261153343-51 -
a702086a-984b-43d1-adc8-f9dd0d7950c7:hive:POST/api/atlas/admin/export] ~ <==
export(user=hive, from=172.27.160.133): status SUCCESS: changeMarker:
1604003211134 (ExportService:101) 2020-10-30 06:26:20,227 INFO -
[etp1261153343-493 -
1423275f-b645-42bd-9095-a135707f34ba:hive:POST/api/atlas/admin/export] ~ <==
export(user=hive, from=172.27.160.133): status SUCCESS: changeMarker:
1604003211134 (ExportService:101) 2020-10-30 06:41:47,192 INFO -
[etp1261153343-41 - 2bd5e05f-534f-4a67-b7ef-d8ccc70966ea:] ~ <==
export(user=hrt_qa, from=10.101.80.202): status SUCCESS: changeMarker:
1604003211134 (ExportService:101) 2020-10-30 06:43:39,920 INFO -
[etp1261153343-27 - 812b6c59-24b4-4a4a-8737-008865af5384:] ~ <==
export(user=hrt_qa, from=10.101.80.202): status SUCCESS: changeMarker:
1604003211134 (ExportService:101) 2020-10-30 07:16:19,144 INFO -
[etp1261153343-554 -
e4678848-12b3-4bae-b339-b3766ec5bb9a:hive:POST/api/atlas/admin/export] ~ <==
export(user=hive, from=172.27.160.133): status SUCCESS: changeMarker:
1604003211134 (ExportService:101) 2020-10-30 07:17:55,540 INFO -
[etp1261153343-52 -
95551e0b-98f7-4366-8806-0452d7a49c4c:hive:POST/api/atlas/admin/export] ~ <==
export(user=hive, from=172.27.160.133): status SUCCESS: changeMarker:
1604003211134 (ExportService:101) 2020-10-30 09:49:22,376 INFO -
[etp1261153343-740 -
f94e92cc-4d80-4d61-b19b-c6956b30b987:hive:POST/api/atlas/admin/export] ~ <==
export(user=hive, from=172.27.160.133): status SUCCESS: changeMarker:
1604003211134 (ExportService:101) 2020-10-30 10:11:31,160 INFO -
[etp1261153343-52 -
5e8152c8-d210-4688-914a-06f983fb3898:hive:POST/api/atlas/admin/export] ~ <==
export(user=hive, from=172.27.160.133): status SUCCESS: changeMarker:
1604003211134 (ExportService:101) 2020-10-30 10:24:53,328 INFO -
[etp1261153343-547 -
2cdc6326-4786-4a69-a116-f15c476dc6a0:hive:POST/api/atlas/admin/export] ~ <==
export(user=hive, from=172.27.160.133): status SUCCESS: changeMarker:
1604003211134 (ExportService:101) 2020-10-30 11:20:49,768 INFO -
[etp1261153343-27 -
1c88be93-17e8-4530-a46d-8c28f8bcc4b7:hive:POST/api/atlas/admin/export] ~ <==
export(user=hive, from=172.27.160.133): status SUCCESS: changeMarker:
1604003211134 (ExportService:101) {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
