Hi Chris, Regarding the PutKafka processor looking at this block[1] of the PutKafka code, it has a default size of 1 MB, but it does not restrict the size. The DATA_SIZE_VALIDATOR does a sanity check and also enforces that the supported value entered is the correct format <value> [B| KB|MB|GB|TB]. Later on in the code at this block[2], the value is set on the Kafka config, again this does not enforce a value maximum.
In regards to the PublishKafka processor I do not see where it accepts a size nor restrict the size at all. Have you adjusted the 'message.max.bytes' config value for your broker(s)? The default value for that is 1 MB [3] (The url references the 0.8 Kafka, however I believe this default has been stable since the early days of the project.) If you really do want to send messages that are larger than 1 MB in size, I would highly recommending reading this post[4] from Gwen Shapira. It does a great job of outlining the things you need to take into consideration. This will also point you to the relevant configs in Kafka that will need to be adjusted if you decide to go this route. Thanks, Andrew [1] https://github.com/apache/nifi/blob/rel/nifi-0.7.0/nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-processors/src/main/java/org/apache/nifi/processors/kafka/PutKafka.java#L174-L180 [2] https://github.com/apache/nifi/blob/rel/nifi-0.7.0/nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-processors/src/main/java/org/apache/nifi/processors/kafka/PutKafka.java#L495 [3] https://kafka.apache.org/08/configuration.html [4] http://ingest.tips/2015/01/21/handling-large-messages-kafka/ On Sat, Aug 20, 2016 at 3:25 PM, McDermott, Chris Kevin (MSDU - STaTS/StorefrontRemote) <[email protected]> wrote: > Hi folks, > > > > From experimentation and looking at the code it seems that the max message > size that can be sent via the PublishKafka and PutKafka processors in 0.7.0 > is 1MB. Can someone please confirm my read on this? > > > > Thanks, > > > > Chris McDermott > > > > Remote Business Analytics > > STaTS/StoreFront Remote > > HPE Storage > > Hewlett Packard Enterprise > > Mobile: +1 978-697-5315 > > > > -- Thanks, Andrew Subscribe to my book: Streaming Data <http://manning.com/psaltis> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306> twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata>
