[
https://issues.apache.org/jira/browse/NIFI-4133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16825179#comment-16825179
]
Bryan Bende commented on NIFI-4133:
-----------------------------------
Yes I think there should be a new optional property called "Partition" that is
only used when the "Paritioner Class" property is set to "DefaultPartitioner",
and the "Partition" property should support EL from FLOWFILE_ATTRIBUTES.
In the case the "DefaultPartitioner" is used and there is a value for
"Partition", then the value should be passed in when creating the producer
record instead of passing in null like it currently does:
{code:java}
final ProducerRecord<byte[], byte[]> record = new ProducerRecord<>(topic, null,
messageKey, messageContent);{code}
There should also be logic in a customValidate that produces a validation error
if choosing "Round Robin" partitioner and setting the "Partition" property
since that would be unclear which is being used.
I think it should be supported in all the publish kafka processors until we
decide that some are deprecated, I think it is acceptable to leave out PutKafka
since that one is based on significantly different code.
> PublishKafkaRecord_0_10 should allow publishing all messages from a flow file
> to the same partition
> ---------------------------------------------------------------------------------------------------
>
> Key: NIFI-4133
> URL: https://issues.apache.org/jira/browse/NIFI-4133
> Project: Apache NiFi
> Issue Type: Improvement
> Affects Versions: 1.2.0, 1.3.0
> Reporter: Bryan Bende
> Assignee: Sandish Kumar HN
> Priority: Minor
>
> In some use cases it is required to publish all of the messages from a given
> flow file to the same partition so that they can later be consumer in the
> same order.
> Currently the processor provides an option to choose between the default
> partitioner and a round-robin partitioner, and also allows specifying the
> name of a field in each record to use as a message key.
> The default partitioner has the following behavior:
> 1) If a partition is specified in the record, use it
> 2) If no partition is specified but a key is present choose a partition
> based on a hash of the key
> 3) If no partition or key is present choose a partition in a round-robin
> fashion
> Currently we never pass in a partition to the Kafka record that is created,
> so we always fall into #2 or #3, and the message key is really meant to be
> unique per-event so we shouldn't be relying on every message using the same
> message key.
> We should add an option to the processor like "Partition per FlowFile" which
> can be used with the default partitioner, and the NiFi side will pass in the
> same partition for each message created from the same flow file.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
