Jeremy It is possible that backpressure was being engaged in NiFi and causing our consumer code to handle it poorly. We did fix that a while ago and I think it ended up in NiFi 1.2.0 (off top of mind anyway). Between your current release and the latest 1.3.0 release a few bugs with those processors have been fixed which are quite useful and we've added ones which allow you to consume and publish record objects which if you've read about the record reader/writer stuff at all I bet you'll find really helpful for your flows.
Thanks On Fri, Jun 23, 2017 at 3:31 PM, Jeremy Farbota <[email protected]> wrote: > Hello, > > I'm having issues today with my ConsumeKafka_0_10 processors (Kafka is > 0.10.1 (3 nodes) NiFi is 1.0.0 3 node cluster). They are all throwing this > error seemingly with each new batch (see attached). We are not seeing > errors on other client consumers (clojure, spark). > > My questions are: > > 1) Does this error indicate that some offsets might not be getting > consumed or does the consumer restart and re-read the offset when the > problem occurred? Can I safely ignore this error for the time being since > message seem to keep coming through regardless? > > 2) I reduced the max.poll.records to 10 and I'm still getting this error. > I also increased the heap and restarted the service on each node. I got > this error shortly after I clicked to look at a provenance event on a > processor. I've had an issue in the past where I clicked to look at a > provenance event and one node when down from bufferOverload. Is it possible > that there is some connection between this error and some background > provenance process that I can kill? Is Could this be a memory issue? Is > this a known bug with the consumer? > > We're upgrading to 1.3.0 next week. Is it possible that the upgrade will > fix this issue with ConsumeKafka_0_10? > > > [image: Payoff, Inc.] > *Jeremy Farbota* > Software Engineer, Data > Payoff, Inc. > > [email protected] >
