Joachim, I filed a JIRA [1] and a pull request to add the option of specifying a prefix for headers when added as flowfile attributes.
[1] https://issues.apache.org/jira/browse/NIFI-15550 Thanks, Pierre Le mer. 4 févr. 2026 à 17:37, Emanuel Oliveira <[email protected]> a écrit : > > Personally once developed multi-channel messaging aggregator in nifi pulling > AWS SQS messages (similar pattern as using Kafka) message only contained blob > storage file and 2 or 3 attributes, generated by a 3rd party tool that > connected to all sorts of messaging (email servers, teams, etc) converted > into plain text files uploaded to blob storage + added AWS SQS short limited > message for downstream processing by Apache NiFi that would fetchS3 full file > and do processing etc. > Im sharing because this seems to me normal pattern of operation (small > messages without contents, just pointing to blob storage file location), > sending file contents via kafka seems anti-pattern.. its ok for few > key/values, but not big/files, > > Thanks & Kind Regards, > Emanuel Oliveira > > On Wed, Feb 4, 2026, 15:50 Pierre Villard <[email protected]> wrote: >> >> I see, I guess we could look at an improvement to support this case, >> that should be rather straightforward. >> >> Le mer. 4 févr. 2026 à 16:44, Schuler, Joachim / Kuehne + Nagel / Ham >> MI-DA <[email protected]> a écrit : >> > >> > Our use case is - as far as we can say now - mostly protocol bridging, in >> > the sense that incoming Kafka messages would be delivered as files via >> > SFTP, as a HTTP POST body, and the like. >> > Performance/throughput, in the sense of handling large volumes of small >> > messages, is not expected to be the biggest challenge. Instead, we expect >> > a moderate throughput of XML documents, with rather complex schema >> > definitions, and a typical size of 10kB (but could also be a lot bigger). >> > For that reason, handling one message as one Flowfile would have been our >> > preferred approach, not doing any parsing of the message body if possible. >> > >> > So, thanks again for the insights. Looks like we have to think about >> > alternative options... >> > >> > Best regards, >> > - Joachim >> > >> > >> > Von: Pierre Villard <[email protected]> >> > Gesendet: Dienstag, 3. Februar 2026 16:30 >> > An: Schuler, Joachim / Kuehne + Nagel / Ham MI-DA >> > <[email protected]> >> > Cc: [email protected] >> > Betreff: Re: ConsumeKafka processor does not allow to get access to Kafka >> > message header 'uuid' >> > >> > EXTERNAL EMAIL >> > >> > Not knowing your use case, an option could be something along the lines of: >> > ConsumeKafka -> SplitRecord (to get one message = one flowfile) -> >> > EvaluateJSONPath, assuming JSON Record Writer before, to extract some >> > specific fields into flowfile attributes (like the uuid header), and >> > EvaluateJsonPath again to only keep the message as content of the flowfile. >> > >> > However, while a record based approach can be a bit more complex at first, >> > this is highly recommended from a performance PoV if you have a use case >> > with high throughput. >> > >> > Thanks, >> > Pierre >> > >> > >> > Kühne + Nagel (AG & Co.) KG >> > Rechtsform: Kommanditgesellschaft, Bremen HRA 21928, USt-IdNr.: DE >> > 812773878. >> > Geschäftsleitung Kühne + Nagel (AG & Co.) KG: Tobias Jerschke (Vors.), >> > Sven Bauer, Daniel Becker, Martin Brinkmann, Lars-Olof Grün, Matthias >> > Knicky, Martin Schäfer, Lars Wedel. >> > Persönlich haftende Gesellschafterin: Kühne & Nagel A.G., Rechtsform: >> > Aktiengesellschaft nach luxemburgischem Recht, HR-Nr.: B 18745, >> > Geschäftsführendes Verwaltungsratsmitglied: Markus Blanka-Graff. >> > >> > Wir arbeiten ausschließlich auf Grundlage der Allgemeinen Deutschen >> > Spediteurbedingungen 2017 (ADSp 2017). Hinweis: Die ADSp 2017 weichen in >> > Ziffer 23 hinsichtlich des Haftungshöchstbetrages für Güterschäden (§ 431 >> > HGB) vom Gesetz ab, indem sie die Haftung bei multimodalen Transporten >> > unter Einschluss einer Seebeförderung und bei unbekanntem Schadenort auf 2 >> > SZR/kg und im Übrigen die Regelhaftung von 8,33 SZR/kg zusätzlich auf 1,25 >> > Millionen Euro je Schadenfall sowie 2,5 Millionen Euro je Schadenereignis, >> > mindestens aber 2 SZR/kg, beschränken. Die ADSp sind auf unserer Webseite >> > als Download erhältlich. Auf Anfrage senden wir Ihnen diese auch gerne zu.
