Adam Bellemare created ATLAS-4323:
-------------------------------------
Summary: Add GUID key to notification events
Key: ATLAS-4323
URL: https://issues.apache.org/jira/browse/ATLAS-4323
Project: Atlas
Issue Type: Improvement
Components: atlas-core
Reporter: Adam Bellemare
Currently, Atlas event notifications written to Kafka do not have a key. There
are several downsides to this:
1) Data for the same GUID will end up in different Kafka topic partitions (as
the events will be written randomly, or using round-robin). There is no data
locality at the partition level.
2) Consumers may consume events for the same GUID out-of-order. For example, it
may consume all of the events in one partition before moving on to the next
partition (round-robin consumption). Due to the lack of data locality in #1,
there is no guarantee around the ordering in which a consumer will see events
between partitions. This can result in inconsistent consumer behaviour or
materialized state.
3) Unable to apply Topic Compaction. Kafka allows you to compact topics such
that only the most recent event of a given key is maintained. By using a unique
key, we can use Kafka compaction instead of using time-based expiry.
The GUID is unique for each entity and relationship, and is the ideal key for
notification events. The default hash partitioner of the producer will ensure
that all events of the same GUID are delivered to the same topic partition,
remedying all 3 issues discussed above.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)