Personally once developed multi-channel messaging aggregator in nifi pulling AWS SQS messages (similar pattern as using Kafka) message only contained blob storage file and 2 or 3 attributes, generated by a 3rd party tool that connected to all sorts of messaging (email servers, teams, etc) converted into plain text files uploaded to blob storage + added AWS SQS short limited message for downstream processing by Apache NiFi that would fetchS3 full file and do processing etc. Im sharing because this seems to me normal pattern of operation (small messages without contents, just pointing to blob storage file location), sending file contents via kafka seems anti-pattern.. its ok for few key/values, but not big/files,
Thanks & Kind Regards, Emanuel Oliveira On Wed, Feb 4, 2026, 15:50 Pierre Villard <[email protected]> wrote: > I see, I guess we could look at an improvement to support this case, > that should be rather straightforward. > > Le mer. 4 févr. 2026 à 16:44, Schuler, Joachim / Kuehne + Nagel / Ham > MI-DA <[email protected]> a écrit : > > > > Our use case is - as far as we can say now - mostly protocol bridging, > in the sense that incoming Kafka messages would be delivered as files via > SFTP, as a HTTP POST body, and the like. > > Performance/throughput, in the sense of handling large volumes of small > messages, is not expected to be the biggest challenge. Instead, we expect a > moderate throughput of XML documents, with rather complex schema > definitions, and a typical size of 10kB (but could also be a lot bigger). > For that reason, handling one message as one Flowfile would have been our > preferred approach, not doing any parsing of the message body if possible. > > > > So, thanks again for the insights. Looks like we have to think about > alternative options... > > > > Best regards, > > - Joachim > > > > > > Von: Pierre Villard <[email protected]> > > Gesendet: Dienstag, 3. Februar 2026 16:30 > > An: Schuler, Joachim / Kuehne + Nagel / Ham MI-DA > <[email protected]> > > Cc: [email protected] > > Betreff: Re: ConsumeKafka processor does not allow to get access to > Kafka message header 'uuid' > > > > EXTERNAL EMAIL > > > > Not knowing your use case, an option could be something along the lines > of: > > ConsumeKafka -> SplitRecord (to get one message = one flowfile) -> > EvaluateJSONPath, assuming JSON Record Writer before, to extract some > specific fields into flowfile attributes (like the uuid header), and > EvaluateJsonPath again to only keep the message as content of the flowfile. > > > > However, while a record based approach can be a bit more complex at > first, this is highly recommended from a performance PoV if you have a use > case with high throughput. > > > > Thanks, > > Pierre > > > > > > Kühne + Nagel (AG & Co.) KG > > Rechtsform: Kommanditgesellschaft, Bremen HRA 21928, USt-IdNr.: DE > 812773878. > > Geschäftsleitung Kühne + Nagel (AG & Co.) KG: Tobias Jerschke (Vors.), > Sven Bauer, Daniel Becker, Martin Brinkmann, Lars-Olof Grün, Matthias > Knicky, Martin Schäfer, Lars Wedel. > > Persönlich haftende Gesellschafterin: Kühne & Nagel A.G., Rechtsform: > Aktiengesellschaft nach luxemburgischem Recht, HR-Nr.: B 18745, > Geschäftsführendes Verwaltungsratsmitglied: Markus Blanka-Graff. > > > > Wir arbeiten ausschließlich auf Grundlage der Allgemeinen Deutschen > Spediteurbedingungen 2017 (ADSp 2017). Hinweis: Die ADSp 2017 weichen in > Ziffer 23 hinsichtlich des Haftungshöchstbetrages für Güterschäden (§ 431 > HGB) vom Gesetz ab, indem sie die Haftung bei multimodalen Transporten > unter Einschluss einer Seebeförderung und bei unbekanntem Schadenort auf 2 > SZR/kg und im Übrigen die Regelhaftung von 8,33 SZR/kg zusätzlich auf 1,25 > Millionen Euro je Schadenfall sowie 2,5 Millionen Euro je Schadenereignis, > mindestens aber 2 SZR/kg, beschränken. Die ADSp sind auf unserer Webseite > als Download erhältlich. Auf Anfrage senden wir Ihnen diese auch gerne zu. >
