Thanks again for the report! Based on the information you provided, I was able to identify two problems with the current implementation: (1) the inability to specify "kafka.key" using the FlowFile (non-record) publish strategy, (2) the missing plumbing to pass the "kafka.key" attribute to the code location where the Kafka client library "ProducerRecord" object is crafted.
I've entered a JIRA [1] to track these issues, and a PR [2] is up to address them. During the development of these new components, it was the intent to replicate the features of the Kafka 2 components where it made sense. This omission was an oversight; yours is definitely a valid use case. If you notice other inconsistencies like this, please flag them! It helps everyone in the community for the new components to be feature-rich and performant. You should feel free to open JIRAs yourself, or to raise the issues on the users mailing-list or the community Slack channel [3]. Code / documentation contributions are also welcome. Users are often the best people to describe how a component works (and should work), in a way that is understandable to other users. It sounds like there are some documentation improvements that could be made. Cheers... [1] https://issues.apache.org/jira/browse/NIFI-14053 [2] https://github.com/apache/nifi/pull/9557 [3] https://nifi.apache.org/community/contact/ On Tue, Nov 26, 2024 at 1:04 PM Paul Grey <grey...@gmail.com> wrote: > Hi. > > Thanks for the report. I will have a look, and report back. > > On Tue, Nov 26, 2024 at 12:00 PM <j.allm...@avm.de> wrote: > >> Hi, >> >> I am in the process of migrating a project from Nifi 1.26 to Nifi 2.0, >> and I am running into problems around the PublishKafka processor. >> Since the processor has seen a major redesign, I've had to reconfigure a >> couple of things. I was mostly successful, but I noticed my outgoing Kafka >> messages are missing their key, which used to be set from a flowfile >> attribute ("kafka.key") in 1.26. >> >> It took me a while to find out how to configure the "Kafka Key" property >> of the processor in 2.0, but even after doing so, the key continues to be >> null for all my Kafka messages. According to the documentation, "kafka.key" >> should be the default attribute to derive the message key from, but neither >> setting it explicitly, nor leaving "Kafka Key" empty, nor any other >> configuration so far led to anything other than null as the key of my >> outgoing Kafka messages. >> >> I had a quick look into the source code to understand what the processor >> actually does, and I came to the conclusion that setting the message key >> based on a flowfile attribute requires an impossible to fulfill set of >> conditions, more specifically the Publish Strategy being set to "Use >> Wrapper" and NOT being set to "Use Wrapper". >> >> This conclusion might be wrong, since I am not at all familiar with the >> Nifi source code. I hope this information still helps diagnosing the >> problem, in case it is actually in the source code. >> >> If the problem is in fact me, mis-configuring the processor, any advice >> on how to achieve what 1.26 used to do here is appreciated. >> >> Best regards >> Jonas Allmann >> >
