[
https://issues.apache.org/jira/browse/ATLAS-801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15313766#comment-15313766
]
Hemanth Yamijala commented on ATLAS-801:
----------------------------------------
bq. Given that the current hooks themselves may not capture 100% of all events
for any component due to lack of integration points(eg pig jobs or any job that
creates tables directly to Hcatalog etc) and it is known that we will lose some
events due to the above or due to network partition, client side hooks etc , it
makes more sense to invest on reconciliation than on fault tolerance when Kafka
is down.
This is a good point, and I agree with you.
> Atlas hooks would lose messages if Kafka is down for extended period of time
> ----------------------------------------------------------------------------
>
> Key: ATLAS-801
> URL: https://issues.apache.org/jira/browse/ATLAS-801
> Project: Atlas
> Issue Type: Improvement
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
>
> All integration hooks in Atlas write messages to Kafka which are picked up by
> the Atlas server. If communication to Kafka breaks, then this results in loss
> of metadata messages. This can be mitigated to some extent using multiple
> replicas for Kafka topics (see ATLAS-515). This JIRA is to see if we can make
> this even more robust and have some form of store and forward mechanism for
> increased fault tolerance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
