Bryan Bende created NIFI-4133:
---------------------------------
Summary: PublishKafkaRecord_0_10 should allow publishing all
messages from a flow file to the same partition
Key: NIFI-4133
URL: https://issues.apache.org/jira/browse/NIFI-4133
Project: Apache NiFi
Issue Type: Improvement
Affects Versions: 1.3.0, 1.2.0
Reporter: Bryan Bende
Priority: Minor
In some use cases it is required to publish all of the messages from a given
flow file to the same partition so that they can later be consumer in the same
order.
Currently the processor provides an option to choose between the default
partitioner and a round-robin partitioner, and also allows specifying the name
of a field in each record to use as a message key.
The default partitioner has the following behavior:
1) If a partition is specified in the record, use it
2) If no partition is specified but a key is present choose a partition based
on a hash of the key
3) If no partition or key is present choose a partition in a round-robin
fashion
Currently we never pass in a partition to the Kafka record that is created, so
we always fall into #2 or #3, and the message key is really meant to be unique
per-event so we shouldn't be relying on every message using the same message
key.
We should add an option to the processor like "Partition per FlowFile" which
can be used with the default partitioner, and the NiFi side will pass in the
same partition for each message created from the same flow file.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
