Michael Miklavcic created METRON-1594:
-----------------------------------------
Summary: KafkaWriter is asynchronous and may lose data on node
failure
Key: METRON-1594
URL: https://issues.apache.org/jira/browse/METRON-1594
Project: Metron
Issue Type: Bug
Reporter: Michael Miklavcic
Assignee: Michael Miklavcic
Currently, we do not block for data to be sent from kafka writer and we do not
batch. Unfortunately, the send call is asynchronous, so if the node dies before
the data is actually sent from kafka then it will drop data. It is highly
unlikely that we will be able to make kafkawriter synchronous in a performant
way, so we will likely need to batch in the parser and enrichment topologies.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
