Umesh Padashetty created ATLAS-4014: ---------------------------------------
Summary: changeMarker in Atlas Export API response does not get updated Key: ATLAS-4014 URL: https://issues.apache.org/jira/browse/ATLAS-4014 Project: Atlas Issue Type: Bug Components: atlas-core Affects Versions: 2.1.0 Reporter: Umesh Padashetty As per [https://atlas.apache.org/index.html#/IncrementalExport,] changeMarker is used in conjunction with "fetchType": "incremental" to incrementally export the metadata, instead of exporting everything, thereby reducing the redundancy. As per this logic, the Atlas Export API returns a new change changeMarker after every successful Export API call (provided there is a change in Atlas metadata). I ran the Atlas export on a DB. And then added few tables into the DB and then again ran the Export. Expectation was changeMarker would be updated. But it is being observed that the changeMarker never changes and is currently always set to 1604003211134 in the instance I am testing. The same changeMarker is set in the atlas-export-info.json too. The changeMarker is used by Hive Replication Manager for incremental Export. Since atlas is always returning the same changeMarker, Hive is always passing back the same changeMarker to atlas in the eventual export calls. This is leading to full export, instead of incremental export as desired. Following is the result of grep i did in atlas server logs. As you can see, changeMarker returned is always 1604003211134. {code:java} [root@quasar-ggealw-4 atlas]# grep "<== export" application.log | grep changeMarker 2020-10-30 05:48:54,451 INFO - [etp1261153343-492 - bd3b60a4-a2b3-4f5c-96c5-166b09157828:] ~ <== export(user=hrt_qa, from=10.101.80.202): status SUCCESS: changeMarker: 1604003211134 (ExportService:101) 2020-10-30 06:03:43,282 INFO - [etp1261153343-483 - 3114c858-8f89-4d8f-917c-2c7a3d695d2f:hive:POST/api/atlas/admin/export] ~ <== export(user=hive, from=172.27.160.133): status SUCCESS: changeMarker: 1604003211134 (ExportService:101) 2020-10-30 06:15:43,880 INFO - [etp1261153343-489 - ccb8da65-e433-44a4-a470-898843dccb17:hive:POST/api/atlas/admin/export] ~ <== export(user=hive, from=172.27.160.133): status SUCCESS: changeMarker: 1604003211134 (ExportService:101) 2020-10-30 06:23:10,810 INFO - [etp1261153343-51 - a702086a-984b-43d1-adc8-f9dd0d7950c7:hive:POST/api/atlas/admin/export] ~ <== export(user=hive, from=172.27.160.133): status SUCCESS: changeMarker: 1604003211134 (ExportService:101) 2020-10-30 06:26:20,227 INFO - [etp1261153343-493 - 1423275f-b645-42bd-9095-a135707f34ba:hive:POST/api/atlas/admin/export] ~ <== export(user=hive, from=172.27.160.133): status SUCCESS: changeMarker: 1604003211134 (ExportService:101) 2020-10-30 06:41:47,192 INFO - [etp1261153343-41 - 2bd5e05f-534f-4a67-b7ef-d8ccc70966ea:] ~ <== export(user=hrt_qa, from=10.101.80.202): status SUCCESS: changeMarker: 1604003211134 (ExportService:101) 2020-10-30 06:43:39,920 INFO - [etp1261153343-27 - 812b6c59-24b4-4a4a-8737-008865af5384:] ~ <== export(user=hrt_qa, from=10.101.80.202): status SUCCESS: changeMarker: 1604003211134 (ExportService:101) 2020-10-30 07:16:19,144 INFO - [etp1261153343-554 - e4678848-12b3-4bae-b339-b3766ec5bb9a:hive:POST/api/atlas/admin/export] ~ <== export(user=hive, from=172.27.160.133): status SUCCESS: changeMarker: 1604003211134 (ExportService:101) 2020-10-30 07:17:55,540 INFO - [etp1261153343-52 - 95551e0b-98f7-4366-8806-0452d7a49c4c:hive:POST/api/atlas/admin/export] ~ <== export(user=hive, from=172.27.160.133): status SUCCESS: changeMarker: 1604003211134 (ExportService:101) 2020-10-30 09:49:22,376 INFO - [etp1261153343-740 - f94e92cc-4d80-4d61-b19b-c6956b30b987:hive:POST/api/atlas/admin/export] ~ <== export(user=hive, from=172.27.160.133): status SUCCESS: changeMarker: 1604003211134 (ExportService:101) 2020-10-30 10:11:31,160 INFO - [etp1261153343-52 - 5e8152c8-d210-4688-914a-06f983fb3898:hive:POST/api/atlas/admin/export] ~ <== export(user=hive, from=172.27.160.133): status SUCCESS: changeMarker: 1604003211134 (ExportService:101) 2020-10-30 10:24:53,328 INFO - [etp1261153343-547 - 2cdc6326-4786-4a69-a116-f15c476dc6a0:hive:POST/api/atlas/admin/export] ~ <== export(user=hive, from=172.27.160.133): status SUCCESS: changeMarker: 1604003211134 (ExportService:101) 2020-10-30 11:20:49,768 INFO - [etp1261153343-27 - 1c88be93-17e8-4530-a46d-8c28f8bcc4b7:hive:POST/api/atlas/admin/export] ~ <== export(user=hive, from=172.27.160.133): status SUCCESS: changeMarker: 1604003211134 (ExportService:101) {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)