Thanks Bryan! I will try that and let you know by tomorrow :-)
______________________ *Kind Regards,* *Anshuman Ghosh* *Contact - +49 179 9090964* On Tue, Mar 21, 2017 at 5:49 PM, Bryan Bende <[email protected]> wrote: > According to the Kafka docs [1] the property is "session.timeout.ms". > > Any user defined properties in the Kafka processors are passed along > to the Kafka client as configuration. > > You create a user defined property from the properties screen using > the + icon in the top right when the processor is stopped. > > Add a new property with name as session.timeout.ms and a value in > milliseconds. The docs say the default value is 30000 so I'm concerned > that something else is going on because I find it hard to believe that > the broker hasn't heard from the consumer in NiFi for over 30 seconds > when your processor is schedule for every 2 seconds, and there is no > back-pressure. > > -Bryan > > [1] https://kafka.apache.org/0100/documentation.html#newconsumerconfigs > > On Tue, Mar 21, 2017 at 12:19 PM, Anshuman Ghosh > <[email protected]> wrote: > > Hello Bryan, > > > > One more thing, I have just checked the config/consumer.properties where > the > > consumer timeout is not set > > Does it have to do anything with this? > > > > #consumer timeout > > #consumer.timeout.ms=5000 > > > > # timeout in ms for connecting to zookeeper > > zookeeper.connection.timeout.ms=6000 > > > > > > Thank you! > > ______________________ > > > > Kind Regards, > > Anshuman Ghosh > > Contact - +49 179 9090964 > > > > > > On Tue, Mar 21, 2017 at 5:12 PM, Anshuman Ghosh > > <[email protected]> wrote: > >> > >> Hello Bryan, > >> > >> Thank you so much for your reply! > >> > >> Just to answer your questions - > >> > >> Back Pressure is set to the next processor but that isn't triggered > >> anytime (threshold limit is 1 GB) > >> No other custom properties are set apart from the one I have shared with > >> you. > >> > >> Also it suggests to increase the session timeout value - Where do we do > >> that? > >> Settings for my Kafka cluster is zookeeper.connection.timeout.ms=6000 > >> > >> > >> Thank you! > >> ______________________ > >> > >> Kind Regards, > >> Anshuman Ghosh > >> Contact - +49 179 9090964 > >> > >> > >> On Tue, Mar 21, 2017 at 3:36 PM, Bryan Bende <[email protected]> wrote: > >>> > >>> Anshuman, > >>> > >>> The error message is saying that NiFi tried to use a consumer where > >>> the session already timed-out. So for example, if the session timeout > >>> was 10 seconds, and NiFi had a consumer that hadn't been used in 12 > >>> seconds and then tried to use it, you would get this error because the > >>> session for that consumer already exceeded the 10 seconds and expired. > >>> > >>> The typical reasons you would see that error message are the > following... > >>> > >>> - The processor run schedule exceeds the duration of the > >>> session.timeout (i.e. processor scheduled to run every 30 seconds, > >>> session.timeout is 10 seconds) > >>> - The processor wasn't scheduled to run for some period of time due to > >>> back-pressure, so even though the run schedule might be every 1 > >>> second, it actually didn't execute for longer than the session.timeout > >>> - The amount of time to perform a single poll() from Kafka takes > >>> longer than the session.timeout, this is unlikely for NiFi but > >>> theoretically possible for other consumers depending how they are > >>> implemented > >>> > >>> From your screenshots it looks like ConsumeKafka_0_10 is scheduled to > >>> run every 2 seconds... > >>> > >>> Do you ever have back-pressure between ConsumeKafka and the next > >>> processors? > >>> Do you have any other custom properties set in ConsumeKafka? I can > >>> only see down to Max Uncommitted Time > >>> > >>> The good news is that newer versions of the Kafka 0.10.x client have > >>> some built in heart-beating such that even if a consumer hasn't been > >>> used in a while, it will continue to keep the session alive. > >>> > >>> Koji actually has a PR up to address this: > >>> https://github.com/apache/nifi/pull/1527 > >>> > >>> Thanks, > >>> > >>> Bryan > >>> > >>> > >>> On Tue, Mar 21, 2017 at 9:33 AM, Anshuman Ghosh > >>> <[email protected]> wrote: > >>> > Hello, > >>> > > >>> > Can anyone please help me with this error message "ConsumeKafka_0_10" > >>> > processor. > >>> > Earlier I was not getting these messages, now I am seeing this Error > >>> > and > >>> > Waring message (attached screenshot) and sometimes getting duplicates > >>> > :( > >>> > Processor is running though! > >>> > > >>> > I have attached the configuration parameters screenshots for both the > >>> > Publisher and Consumer processor. > >>> > > >>> > Thank > >>> > ing > >>> > you > >>> > in advance > >>> > ! > >>> > > >>> > ______________________ > >>> > > >>> > Kind Regards, > >>> > Anshuman Ghosh > >>> > Contact - +49 179 9090964 > >>> > > >> > >> > > >
