I was wrong. I think I couldn't increase the timeout value for Kafka spout properly. Therefore, I was wondering how I can increase the timeout value for Kafka spout? What is the right "-esc" property name I need to set in this case? Also, what has changed in the newer version since I didn't have this issue with the previous version?
On Sun, May 14, 2017 at 3:00 AM, Ali Nazemian <[email protected]> wrote: > Hi, > > I have installed the new version of HCP recently. I can see that the > following error has appeared in Storm UI at Kafka spout section related to > Parser topologies: > > org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be > completed since the group has already rebalanced and assigned the > partitions to another member. This means that the time between subsequent > calls to poll() was longer than the configured session.timeout.ms, which > typically implies that the poll loop is spending too much time message > processing. You can address this either by increasing the session timeout > or by reducing the maximum size of batches returned in poll() with > max.poll.records. at org.apache.kafka.clients.consumer.internals. > ConsumerCoordinator$OffsetCommitResponseHandler. > handle(ConsumerCoordinator.java:600) at org.apache.kafka.clients. > consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler. > handle(ConsumerCoordinator.java:541) at org.apache.kafka.clients. > consumer.internals.AbstractCoordinator$CoordinatorResponseHandler. > onSuccess(AbstractCoordinator.java:679) at org.apache.kafka.clients. > consumer.internals.AbstractCoordinator$CoordinatorResponseHandler. > onSuccess(AbstractCoordinator.java:658) at org.apache.kafka.clients. > consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:167) at > org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:133) > at > org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:107) > at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$ > RequestFutureCompletionHandler.onComplete(ConsumerNetworkClient.java:426) > at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:278) at > org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient. > clientPoll(ConsumerNetworkClient.java:360) at org.apache.kafka.clients. > consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:224) > at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll( > ConsumerNetworkClient.java:192) at org.apache.kafka.clients. > consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:163) > at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator. > commitOffsetsSync(ConsumerCoordinator.java:426) at > org.apache.kafka.clients.consumer.KafkaConsumer. > commitSync(KafkaConsumer.java:1059) at org.apache.storm.kafka.spout. > KafkaSpout.commitOffsetsForAckedTuples(KafkaSpout.java:302) at > org.apache.storm.kafka.spout.KafkaSpout.nextTuple(KafkaSpout.java:204) at > org.apache.storm.daemon.executor$fn__6505$fn__6520$fn__6551.invoke(executor.clj:651) > at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) at > clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:748) > > > This error has affected the Parsers throughput significantly! > > I have tried to increase the session timeout, but It didn't affect my > situation. I would be grateful if you can help me to find the source of > this issue. Please be advised that I haven't had this issue with the > previous version of Metron (0.3.1). > > Regards, > Ali > > -- A.Nazemian
