[
https://issues.apache.org/jira/browse/ATLAS-801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15314532#comment-15314532
]
Daniel Cave commented on ATLAS-801:
-----------------------------------
A minor adaptation on the upstream and downstream ends could also be used to
address your loss of data issue. Incorporating Apache NiFi local to the data
(i.e. the Atlas server(s) in this case) and using it as the conduit between
Atlas and Kafka, rather than Atlas to Kafka directly, would provide the fault
tolerance you are looking for without making major changes in Atlas itself (and
thus extending it to do something that is beyond it's scope). This could also
be implemented in the reverse, i.e. [Hive/Oozie/etc->NiFi]->[Kafka]->[Atlas] or
[Hive/Oozie/etc->NiFi]->[NiFi->Atlas]. For the Atlas outbound, it would look
like [Atlas->NiFi]->Kafka->whatever or [Atlas->NiFi]->[NiFi->whatever].
I would encourage the Atlas team to look into using a NiFi conduit.
> Atlas hooks would lose messages if Kafka is down for extended period of time
> ----------------------------------------------------------------------------
>
> Key: ATLAS-801
> URL: https://issues.apache.org/jira/browse/ATLAS-801
> Project: Atlas
> Issue Type: Improvement
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
>
> All integration hooks in Atlas write messages to Kafka which are picked up by
> the Atlas server. If communication to Kafka breaks, then this results in loss
> of metadata messages. This can be mitigated to some extent using multiple
> replicas for Kafka topics (see ATLAS-515). This JIRA is to see if we can make
> this even more robust and have some form of store and forward mechanism for
> increased fault tolerance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
