Re: Apache NiFi Clarification

Hi Bryan, Thanks for your email!! I need to set  NiFi as the main orchestrator for all data movement and transformation in my project .  It would call Sqoop, call an API, call a message bus (like Kafka or whatever) and handle scheduling – or even integrate with an external scheduling program.It

Max Kafka message size

Hi folks, From experimentation and looking at the code it seems that the max message size that can be sent via the PublishKafka and PutKafka processors in 0.7.0 is 1MB. Can someone please confirm my read on this? Thanks, Chris McDermott Remote Business Analytics STaTS/StoreFront Remote HPE

Re: Max Kafka message size

Hi Chris, Regarding the PutKafka processor looking at this block[1] of the PutKafka code, it has a default size of 1 MB, but it does not restrict the size. The DATA_SIZE_VALIDATOR does a sanity check and also enforces that the supported value entered is the correct format [B| KB|MB|GB|TB]. Later

Re: Apache NiFi Clarification

Hello, I think NiFi can do most of the things you are looking to do, although NiFi is not built to be a workflow orchestration tool, many of its features do allow for the type of orchestration you are describing. I suggest taking a look at all of the available processors [1] and reading the

Re: Max Kafka message size

If no jira is raised sooner I'll raise one and get it sorted. On Aug 20, 2016 6:40 PM, "Andrew Psaltis" wrote: > Hi Chris, > Sorry for not catching that code path. I am not sure if it is actually a > regression as I took a look at the 1.0.0-BETA code and it matches the

Re: Max Kafka message size

Thanks, Andrew. I’ve set all of the right broker configs to allow larger messages. Believe me I spent a lot of time banging my head against the wall thinking that the broker and topic configs were wrong. PublisingKafka uses PublishingContext. That class has bean property called

Re: Max Kafka message size

Hi Chris, Sorry for not catching that code path. I am not sure if it is actually a regression as I took a look at the 1.0.0-BETA code and it matches the 0.7.0, specifically this comment block: /* * We're using the default value from Kafka. We are using it to control the * message size before it

Re: Max Kafka message size

Jira is https://issues.apache.org/jira/browse/NIFI-2614. Thanks, Chris McDermott Remote Business Analytics STaTS/StoreFront Remote HPE Storage Hewlett Packard Enterprise Mobile: +1 978-697-5315 On 8/20/16, 6:57 PM, "McDermott, Chris Kevin (MSDU - STaTS/StorefrontRemote)"

Re: Max Kafka message size

I’ll raise a JIRA, Joe. Thanks, Chris McDermott Remote Business Analytics STaTS/StoreFront Remote HPE Storage Hewlett Packard Enterprise Mobile: +1 978-697-5315 On 8/20/16, 6:52 PM, "Joe Witt" wrote: If no jira is raised sooner I'll raise one and get it sorted.

Re: Max Kafka message size

Rgr that. Thanks chris On Aug 20, 2016 7:04 PM, "McDermott, Chris Kevin (MSDU - STaTS/StorefrontRemote)" wrote: > Jira is https://issues.apache.org/jira/browse/NIFI-2614. > > Thanks, > > Chris McDermott > > Remote Business Analytics > STaTS/StoreFront Remote > HPE