Joseph Witt created NIFI-2732:
---------------------------------
Summary: ConsumeKafka 0.9 and 0.10 not handling partition
reassignment case sufficiently
Key: NIFI-2732
URL: https://issues.apache.org/jira/browse/NIFI-2732
Project: Apache NiFi
Issue Type: Bug
Reporter: Joseph Witt
Assignee: Joseph Witt
Priority: Critical
The new ConsumeKafka clients handle the threading model of the consumer api
correctly. However, they are not yet honoring partition reassignment cases
sufficiently which means we could have avoidable cases of duplication. By
registering a partition reassignment listener we can handle it correctly.
Further, the processor is loading subsequent polls of messages into memory
rather than writing directly to the process session/disk. We could write them
to disk and achieve far better performance and efficiency. Early testing shows
easily achieving 100MB/s sustained per thread on a simple laptop setup with
defaults which will scale very nicely on a legit installed content repository.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
