Yes, I explicitly commit via code for this job as an effort to ensure only once processing.
Thanks for taking the time to look into our concerns. Jeremiah Adams Software Engineer www.helixeducation.com Blog | Twitter | Facebook | LinkedIn ________________________________________ From: Yi Pan <nickpa...@gmail.com> Sent: Friday, March 13, 2020 1:27 PM To: dev@samza.apache.org Subject: Re: Got Error Produce Respons with Correlation Id. Hi, Jeremiah, >From what you have answered, it looks to me as a transient error (probably timeout due to some transient network errors as you mentioned) and your job was able to retry/recover and make progress. Just one thing to confirm: I saw your configured task.commit.ms=-1, and you have mentioned that your checkpointed offset metrics DOES increment over time. Are you calling commit in your user code? Thanks! -Yi On Fri, Mar 13, 2020 at 9:46 AM Jeremiah Adams <jad...@helixeducation.com.invalid> wrote: > Do you see the Samza job hanging after that? > The job does not hang. > > > Is the checkpointed offset metrics incrementing in this case? > We do get incremented offsets. > > Not clear on your claiming: "logs stop at that point". No logs are written > after the WARN lines? > My apologies for the confusion - I see no lag messages related to the > warning. I see all of our normal processing logs. I'm assuming this means > the retry worked. > > > What's your Samza configuration? > > job.coordinator.factory=org.apache.samza.standalone.PassthroughJobCoordinatorFactory > job.coordinator.replication.factor=1 > job.default.system=kafka > systems.kafka.producer.bootstrap.servers=<removed>.confluent.cloud:9092 > > task.name.grouper.factory=org.apache.samza.container.grouper.task.GroupByContainerIdsFactory > systems.kafka.producer.ssl.endpoint.identification.algorithm=https > systems.kafka.producer.sasl.mechanism=PLAIN > systems.kafka.producer.request.timeout.ms=20000 > systems.kafka.producer.retry.backoff.ms=500 > systems.kafka.producer.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule > required username="" password=""; > systems.kafka.producer.security.protocol=SASL_SSL > systems.kafka.consumer.ssl.endpoint.identification.algorithm=https > systems.kafka.consumer.sasl.mechanism=PLAIN > systems.kafka.consumer.request.timeout.ms=20000 > systems.kafka.consumer.retry.backoff.ms=500 > systems.kafka.consumer.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule > required username="" password=""; > systems.kafka.consumer.security.protocol=SASL_SSL > processor.id=0 > > # checkpointing > > task.checkpoint.factory=org.apache.samza.checkpoint.kafka.KafkaCheckpointManagerFactory > task.checkpoint.system=kafka > task.checkpoint.replication.factor=3 > task.commit.ms=-1 > > Is the Samza container still running after you see those WARN logs? > Yes. > > > I am thinking this is a timeout issue. We've never seen the issue before. > The warning first appeared after testing Confluent's Cloud kafka offering. > We had no issues when running our own kafka clusters in aws. > > > Jeremiah Adams > Software Engineer > https://url.emailprotection.link/?bM9S-3pRw1lv8pYfwa-TwdjElP4W2K6b9vP5Crz22L_YcgsRJ-13h-OgPZSwFtU7GSNTDi1z-jdaRvWESRhtTVA~~ > Blog | Twitter | Facebook | LinkedIn > > ________________________________________ > From: Yi Pan <nickpa...@gmail.com> > Sent: Wednesday, March 11, 2020 5:48 PM > To: dev@samza.apache.org > Subject: Re: Got Error Produce Respons with Correlation Id. > > Hi, Jeremiah, > > Sorry to reply late. This WARN message indicates that producer failed to > flush to checkpoint topic and would retry. Do you see the Samza job hanging > after that? Is the checkpointed offset metrics incrementing in this case? > Not clear on your claiming: "logs stop at that point". No logs are written > after the WARN lines? What's your Samza configuration? Is the Samza > container still running after you see those WARN logs? > > Thanks! > > -Yi > > On Wed, Mar 11, 2020 at 2:39 PM Jeremiah Adams > <jad...@helixeducation.com.invalid> wrote: > > > Can anyone take a look at the message below? We are trying to gauge our > > risk before moving forward. > > > > > > Jeremiah Adams > > Software Engineer > > > https://url.emailprotection.link/?bM9S-3pRw1lv8pYfwa-TwdjElP4W2K6b9vP5Crz22L_YcgsRJ-13h-OgPZSwFtU7GSNTDi1z-jdaRvWESRhtTVA~~ > > Blog | Twitter | Facebook | LinkedIn > > > > ________________________________________ > > From: Jeremiah Adams <jad...@helixeducation.com.INVALID> > > Sent: Wednesday, March 4, 2020 2:28 PM > > To: dev@samza.apache.org > > Subject: Got Error Produce Response iwth Correlation Id. > > > > Hello devs, > > > > > > I've got a warning showing up in the logs while testing our new Confluent > > Cloud config. Can anyone tell me how concerned I should be about this > > warning? Is there a setting to control timeouts? > > > > > > Also, logs stop at that point, so I can't tell if the "metatdata update" > > was complete. > > > > > > > > 2020-03-04 21:17:51 Sender [WARN] [Producer > > clientId=kafka_producer-application_submission-1] Got error produce > > response with correlation id 144 on topic-partition > > __samza_checkpoint_ver_1_for_application-submission_1-0, retrying > > (2147483646<tel:2147483646> attempts left). Error: NETWORK_EXCEPTION > > 2020-03-04 21:17:51 Sender [WARN] [Producer > > clientId=kafka_producer-application_submission-1] Received invalid > metadata > > error in produce request on partition > > __samza_checkpoint_ver_1_for_application-submission_1-0 due to > > org.apache.kafka.common.errors.NetworkException: The server disconnected > > before a response was received.. Going to request metadata update now > > > > > > Jeremiah Adams > > Software Engineer > > > > > https://url.emailprotection.link/?bM9S-3pRw1lv8pYfwa-TwdjElP4W2K6b9vP5Crz22L_YcgsRJ-13h-OgPZSwFtU7GSNTDi1z-jdaRvWESRhtTVA~~ > > < > > > https://url.emailprotection.link/?basKr9vk92a8vVw0XMnK5bmaSKuBc0AuEZ7YasYc7Df8YVt3SYmcjmLWdKMWzAAINWlUUA33ebGI7pSoTl9cg1g~~ > > > > > Blog< > > > https://url.emailprotection.link/?basKr9vk92a8vVw0XMnK5bmaSKuBc0AuEZ7YasYc7Df-lAcqG1fqHPpNw-wd9z7HtUJeCG5_8UjCf2mHtn6C_zQ~~ > > > > | Twitter< > > > https://url.emailprotection.link/?bVO2q0UXR235wN_yOnM0FjqITPdBYMD3reLGNddq-zPV5ChMQK9JwV4Be-QnrbRoXpJl8IcknAqKzYtA3RABKww~~ > > > > | Facebook< > > > https://url.emailprotection.link/?bUU7m4NfMS_EWGtH1yojBHX9sWZ6uxVdT1eQUkmU5vWY01WFZiS2KJ-c9iLIncdHB7Uw1lRYCprEEpPPQCdiK6Q~~ > > > > | LinkedIn< > > > https://url.emailprotection.link/?b0ZQfJ1pZYnASyoShs9MJI46-r1lxPhA-JS5VSkR7so-DFP0_HxbOo2LsajGOaoYXxb1ZCOMAu7hZscPCnIKWpXz0cpgQ386SnNHjPcwsu4z90mzBkuwoZc6YxOCzMGA0 > > > > > >
