Umesh Padashetty created ATLAS-3198:
---------------------------------------
Summary: update/delete/truncate operations ATLAS_HOOK and
ATLAS_ENTITIES improvements
Key: ATLAS-3198
URL: https://issues.apache.org/jira/browse/ATLAS-3198
Project: Atlas
Issue Type: Improvement
Affects Versions: 2.0.0
Reporter: Umesh Padashetty
Fix For: 2.0.0
Attachments: ATLAS_HOOK and ATLAS_ENTITIES messages.xlsx
With the recent changes, we have now stopped ingesting simple insert operations
since with this operation, there is no change in the metadata. Ideally, we
should be doing similar improvements w.r.t. update, delete and truncate
operations too.
Currently, for update/delete/truncate, we are pushing messages in both
ATLAS_HOOK as well as ATLAS_ENTITIES topic, which can be avoided.
If we stop ingesting updates/deletes/truncates, it will further help us reduce
the burden on Kafka and save CPU cycles
I have created and attached an excel sheet, with the list of operations and the
corresponding messages generated in ATLAS_HOOK and ATLAS_ENTITIES topic, along
with suitable comments.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
