Re: Got Error Produce Respons with Correlation Id.
Cool! Sounds good to me! Happy to be the help! -Yi On Fri, Mar 13, 2020 at 1:13 PM Jeremiah Adams wrote: > Yes, I explicitly commit via code for this job as an effort to ensure only > once processing. > > Thanks for taking the time to look into our concerns. > > Jeremiah Adams > Software Engineer > www.helixeducation.com > Blog | Twitter | Facebook | LinkedIn > > > From: Yi Pan > Sent: Friday, March 13, 2020 1:27 PM > To: dev@samza.apache.org > Subject: Re: Got Error Produce Respons with Correlation Id. > > Hi, Jeremiah, > > From what you have answered, it looks to me as a transient error (probably > timeout due to some transient network errors as you mentioned) and your job > was able to retry/recover and make progress. > > Just one thing to confirm: I saw your configured task.commit.ms=-1, and > you > have mentioned that your checkpointed offset metrics DOES increment over > time. Are you calling commit in your user code? > > Thanks! > > -Yi > > On Fri, Mar 13, 2020 at 9:46 AM Jeremiah Adams > wrote: > > > Do you see the Samza job hanging after that? > > The job does not hang. > > > > > > Is the checkpointed offset metrics incrementing in this case? > > We do get incremented offsets. > > > > Not clear on your claiming: "logs stop at that point". No logs are > written > > after the WARN lines? > > My apologies for the confusion - I see no lag messages related to the > > warning. I see all of our normal processing logs. I'm assuming this means > > the retry worked. > > > > > > What's your Samza configuration? > > > > > job.coordinator.factory=org.apache.samza.standalone.PassthroughJobCoordinatorFactory > > job.coordinator.replication.factor=1 > > job.default.system=kafka > > systems.kafka.producer.bootstrap.servers=.confluent.cloud:9092 > > > > > task.name.grouper.factory=org.apache.samza.container.grouper.task.GroupByContainerIdsFactory > > systems.kafka.producer.ssl.endpoint.identification.algorithm=https > > systems.kafka.producer.sasl.mechanism=PLAIN > > systems.kafka.producer.request.timeout.ms=2 > > systems.kafka.producer.retry.backoff.ms=500 > > > systems.kafka.producer.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule > > required username="" password=""; > > systems.kafka.producer.security.protocol=SASL_SSL > > systems.kafka.consumer.ssl.endpoint.identification.algorithm=https > > systems.kafka.consumer.sasl.mechanism=PLAIN > > systems.kafka.consumer.request.timeout.ms=2 > > systems.kafka.consumer.retry.backoff.ms=500 > > > systems.kafka.consumer.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule > > required username="" password=""; > > systems.kafka.consumer.security.protocol=SASL_SSL > > processor.id=0 > > > > # checkpointing > > > > > task.checkpoint.factory=org.apache.samza.checkpoint.kafka.KafkaCheckpointManagerFactory > > task.checkpoint.system=kafka > > task.checkpoint.replication.factor=3 > > task.commit.ms=-1 > > > > Is the Samza container still running after you see those WARN logs? > > Yes. > > > > > > I am thinking this is a timeout issue. We've never seen the issue before. > > The warning first appeared after testing Confluent's Cloud kafka > offering. > > We had no issues when running our own kafka clusters in aws. > > > > > > Jeremiah Adams > > Software Engineer > > > https://url.emailprotection.link/?bM9S-3pRw1lv8pYfwa-TwdjElP4W2K6b9vP5Crz22L_YcgsRJ-13h-OgPZSwFtU7GSNTDi1z-jdaRvWESRhtTVA~~ > > Blog | Twitter | Facebook | LinkedIn > > > > > > From: Yi Pan > > Sent: Wednesday, March 11, 2020 5:48 PM > > To: dev@samza.apache.org > > Subject: Re: Got Error Produce Respons with Correlation Id. > > > > Hi, Jeremiah, > > > > Sorry to reply late. This WARN message indicates that producer failed to > > flush to checkpoint topic and would retry. Do you see the Samza job > hanging > > after that? Is the checkpointed offset metrics incrementing in this case? > > Not clear on your claiming: "logs stop at that point". No logs are > written > > after the WARN lines? What's your Samza configuration? Is the Samza > > container still running after you see those WARN logs? > > > > Thanks! > > > > -Yi > > > > On Wed, Mar 11, 2020 at 2:39 PM Jeremiah Adams > > wrote: > > > > > Can anyone take a look at the message below? We are trying to gauge our > > > risk before moving forward. > > > > > > > > > Jeremiah Adams > > > Software Engineer > > > > > > https://url.emailprotection.link/?bM9S-3pRw1lv8pYfwa-TwdjElP4W2K6b9vP5Crz22L_YcgsRJ-13h-OgPZSwFtU7GSNTDi1z-jdaRvWESRhtTVA~~ > > > Blog | Twitter | Facebook | LinkedIn > > > > > > > > > From: Jeremiah Adams > > > Sent: Wednesday, March 4, 2020 2:28 PM > > > To: dev@samza.apache.org > > > Subject: Got Error Produce Response iwth Correlation Id. > > > > > > Hello devs, > > > > > > > > > I've got a warning showing up in the logs while testing our new > Confluent > > > Cloud
Re: Got Error Produce Respons with Correlation Id.
Yes, I explicitly commit via code for this job as an effort to ensure only once processing. Thanks for taking the time to look into our concerns. Jeremiah Adams Software Engineer www.helixeducation.com Blog | Twitter | Facebook | LinkedIn From: Yi Pan Sent: Friday, March 13, 2020 1:27 PM To: dev@samza.apache.org Subject: Re: Got Error Produce Respons with Correlation Id. Hi, Jeremiah, >From what you have answered, it looks to me as a transient error (probably timeout due to some transient network errors as you mentioned) and your job was able to retry/recover and make progress. Just one thing to confirm: I saw your configured task.commit.ms=-1, and you have mentioned that your checkpointed offset metrics DOES increment over time. Are you calling commit in your user code? Thanks! -Yi On Fri, Mar 13, 2020 at 9:46 AM Jeremiah Adams wrote: > Do you see the Samza job hanging after that? > The job does not hang. > > > Is the checkpointed offset metrics incrementing in this case? > We do get incremented offsets. > > Not clear on your claiming: "logs stop at that point". No logs are written > after the WARN lines? > My apologies for the confusion - I see no lag messages related to the > warning. I see all of our normal processing logs. I'm assuming this means > the retry worked. > > > What's your Samza configuration? > > job.coordinator.factory=org.apache.samza.standalone.PassthroughJobCoordinatorFactory > job.coordinator.replication.factor=1 > job.default.system=kafka > systems.kafka.producer.bootstrap.servers=.confluent.cloud:9092 > > task.name.grouper.factory=org.apache.samza.container.grouper.task.GroupByContainerIdsFactory > systems.kafka.producer.ssl.endpoint.identification.algorithm=https > systems.kafka.producer.sasl.mechanism=PLAIN > systems.kafka.producer.request.timeout.ms=2 > systems.kafka.producer.retry.backoff.ms=500 > systems.kafka.producer.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule > required username="" password=""; > systems.kafka.producer.security.protocol=SASL_SSL > systems.kafka.consumer.ssl.endpoint.identification.algorithm=https > systems.kafka.consumer.sasl.mechanism=PLAIN > systems.kafka.consumer.request.timeout.ms=2 > systems.kafka.consumer.retry.backoff.ms=500 > systems.kafka.consumer.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule > required username="" password=""; > systems.kafka.consumer.security.protocol=SASL_SSL > processor.id=0 > > # checkpointing > > task.checkpoint.factory=org.apache.samza.checkpoint.kafka.KafkaCheckpointManagerFactory > task.checkpoint.system=kafka > task.checkpoint.replication.factor=3 > task.commit.ms=-1 > > Is the Samza container still running after you see those WARN logs? > Yes. > > > I am thinking this is a timeout issue. We've never seen the issue before. > The warning first appeared after testing Confluent's Cloud kafka offering. > We had no issues when running our own kafka clusters in aws. > > > Jeremiah Adams > Software Engineer > https://url.emailprotection.link/?bM9S-3pRw1lv8pYfwa-TwdjElP4W2K6b9vP5Crz22L_YcgsRJ-13h-OgPZSwFtU7GSNTDi1z-jdaRvWESRhtTVA~~ > Blog | Twitter | Facebook | LinkedIn > > > From: Yi Pan > Sent: Wednesday, March 11, 2020 5:48 PM > To: dev@samza.apache.org > Subject: Re: Got Error Produce Respons with Correlation Id. > > Hi, Jeremiah, > > Sorry to reply late. This WARN message indicates that producer failed to > flush to checkpoint topic and would retry. Do you see the Samza job hanging > after that? Is the checkpointed offset metrics incrementing in this case? > Not clear on your claiming: "logs stop at that point". No logs are written > after the WARN lines? What's your Samza configuration? Is the Samza > container still running after you see those WARN logs? > > Thanks! > > -Yi > > On Wed, Mar 11, 2020 at 2:39 PM Jeremiah Adams > wrote: > > > Can anyone take a look at the message below? We are trying to gauge our > > risk before moving forward. > > > > > > Jeremiah Adams > > Software Engineer > > > https://url.emailprotection.link/?bM9S-3pRw1lv8pYfwa-TwdjElP4W2K6b9vP5Crz22L_YcgsRJ-13h-OgPZSwFtU7GSNTDi1z-jdaRvWESRhtTVA~~ > > Blog | Twitter | Facebook | LinkedIn > > > > > > From: Jeremiah Adams > > Sent: Wednesday, March 4, 2020 2:28 PM > > To: dev@samza.apache.org > > Subject: Got Error Produce Response iwth Correlation Id. > > > > Hello devs, > > > > > > I've got a warning showing up in the logs while testing our new Confluent > > Cloud config. Can anyone tell me how concerned I should be about this > > warning? Is there a setting to control timeouts? > > > > > > Also, logs stop at that point, so I can't tell if the "metatdata update" > > was complete. > > > > > > > > 2020-03-04 21:17:51 Sender [WARN] [Producer > > clientId=kafka_producer-application_submission-1] Got error produce > > response with correlation id 144 on
Re: Got Error Produce Respons with Correlation Id.
Hi, Jeremiah, >From what you have answered, it looks to me as a transient error (probably timeout due to some transient network errors as you mentioned) and your job was able to retry/recover and make progress. Just one thing to confirm: I saw your configured task.commit.ms=-1, and you have mentioned that your checkpointed offset metrics DOES increment over time. Are you calling commit in your user code? Thanks! -Yi On Fri, Mar 13, 2020 at 9:46 AM Jeremiah Adams wrote: > Do you see the Samza job hanging after that? > The job does not hang. > > > Is the checkpointed offset metrics incrementing in this case? > We do get incremented offsets. > > Not clear on your claiming: "logs stop at that point". No logs are written > after the WARN lines? > My apologies for the confusion - I see no lag messages related to the > warning. I see all of our normal processing logs. I'm assuming this means > the retry worked. > > > What's your Samza configuration? > > job.coordinator.factory=org.apache.samza.standalone.PassthroughJobCoordinatorFactory > job.coordinator.replication.factor=1 > job.default.system=kafka > systems.kafka.producer.bootstrap.servers=.confluent.cloud:9092 > > task.name.grouper.factory=org.apache.samza.container.grouper.task.GroupByContainerIdsFactory > systems.kafka.producer.ssl.endpoint.identification.algorithm=https > systems.kafka.producer.sasl.mechanism=PLAIN > systems.kafka.producer.request.timeout.ms=2 > systems.kafka.producer.retry.backoff.ms=500 > systems.kafka.producer.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule > required username="" password=""; > systems.kafka.producer.security.protocol=SASL_SSL > systems.kafka.consumer.ssl.endpoint.identification.algorithm=https > systems.kafka.consumer.sasl.mechanism=PLAIN > systems.kafka.consumer.request.timeout.ms=2 > systems.kafka.consumer.retry.backoff.ms=500 > systems.kafka.consumer.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule > required username="" password=""; > systems.kafka.consumer.security.protocol=SASL_SSL > processor.id=0 > > # checkpointing > > task.checkpoint.factory=org.apache.samza.checkpoint.kafka.KafkaCheckpointManagerFactory > task.checkpoint.system=kafka > task.checkpoint.replication.factor=3 > task.commit.ms=-1 > > Is the Samza container still running after you see those WARN logs? > Yes. > > > I am thinking this is a timeout issue. We've never seen the issue before. > The warning first appeared after testing Confluent's Cloud kafka offering. > We had no issues when running our own kafka clusters in aws. > > > Jeremiah Adams > Software Engineer > www.helixeducation.com > Blog | Twitter | Facebook | LinkedIn > > > From: Yi Pan > Sent: Wednesday, March 11, 2020 5:48 PM > To: dev@samza.apache.org > Subject: Re: Got Error Produce Respons with Correlation Id. > > Hi, Jeremiah, > > Sorry to reply late. This WARN message indicates that producer failed to > flush to checkpoint topic and would retry. Do you see the Samza job hanging > after that? Is the checkpointed offset metrics incrementing in this case? > Not clear on your claiming: "logs stop at that point". No logs are written > after the WARN lines? What's your Samza configuration? Is the Samza > container still running after you see those WARN logs? > > Thanks! > > -Yi > > On Wed, Mar 11, 2020 at 2:39 PM Jeremiah Adams > wrote: > > > Can anyone take a look at the message below? We are trying to gauge our > > risk before moving forward. > > > > > > Jeremiah Adams > > Software Engineer > > > https://url.emailprotection.link/?bM9S-3pRw1lv8pYfwa-TwdjElP4W2K6b9vP5Crz22L_YcgsRJ-13h-OgPZSwFtU7GSNTDi1z-jdaRvWESRhtTVA~~ > > Blog | Twitter | Facebook | LinkedIn > > > > > > From: Jeremiah Adams > > Sent: Wednesday, March 4, 2020 2:28 PM > > To: dev@samza.apache.org > > Subject: Got Error Produce Response iwth Correlation Id. > > > > Hello devs, > > > > > > I've got a warning showing up in the logs while testing our new Confluent > > Cloud config. Can anyone tell me how concerned I should be about this > > warning? Is there a setting to control timeouts? > > > > > > Also, logs stop at that point, so I can't tell if the "metatdata update" > > was complete. > > > > > > > > 2020-03-04 21:17:51 Sender [WARN] [Producer > > clientId=kafka_producer-application_submission-1] Got error produce > > response with correlation id 144 on topic-partition > > __samza_checkpoint_ver_1_for_application-submission_1-0, retrying > > (2147483646 attempts left). Error: NETWORK_EXCEPTION > > 2020-03-04 21:17:51 Sender [WARN] [Producer > > clientId=kafka_producer-application_submission-1] Received invalid > metadata > > error in produce request on partition > > __samza_checkpoint_ver_1_for_application-submission_1-0 due to > > org.apache.kafka.common.errors.NetworkException: The server disconnected > > before a response was received.. Going to request metadata
Re: Got Error Produce Respons with Correlation Id.
Do you see the Samza job hanging after that? The job does not hang. Is the checkpointed offset metrics incrementing in this case? We do get incremented offsets. Not clear on your claiming: "logs stop at that point". No logs are written after the WARN lines? My apologies for the confusion - I see no lag messages related to the warning. I see all of our normal processing logs. I'm assuming this means the retry worked. What's your Samza configuration? job.coordinator.factory=org.apache.samza.standalone.PassthroughJobCoordinatorFactory job.coordinator.replication.factor=1 job.default.system=kafka systems.kafka.producer.bootstrap.servers=.confluent.cloud:9092 task.name.grouper.factory=org.apache.samza.container.grouper.task.GroupByContainerIdsFactory systems.kafka.producer.ssl.endpoint.identification.algorithm=https systems.kafka.producer.sasl.mechanism=PLAIN systems.kafka.producer.request.timeout.ms=2 systems.kafka.producer.retry.backoff.ms=500 systems.kafka.producer.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="" password=""; systems.kafka.producer.security.protocol=SASL_SSL systems.kafka.consumer.ssl.endpoint.identification.algorithm=https systems.kafka.consumer.sasl.mechanism=PLAIN systems.kafka.consumer.request.timeout.ms=2 systems.kafka.consumer.retry.backoff.ms=500 systems.kafka.consumer.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="" password=""; systems.kafka.consumer.security.protocol=SASL_SSL processor.id=0 # checkpointing task.checkpoint.factory=org.apache.samza.checkpoint.kafka.KafkaCheckpointManagerFactory task.checkpoint.system=kafka task.checkpoint.replication.factor=3 task.commit.ms=-1 Is the Samza container still running after you see those WARN logs? Yes. I am thinking this is a timeout issue. We've never seen the issue before. The warning first appeared after testing Confluent's Cloud kafka offering. We had no issues when running our own kafka clusters in aws. Jeremiah Adams Software Engineer www.helixeducation.com Blog | Twitter | Facebook | LinkedIn From: Yi Pan Sent: Wednesday, March 11, 2020 5:48 PM To: dev@samza.apache.org Subject: Re: Got Error Produce Respons with Correlation Id. Hi, Jeremiah, Sorry to reply late. This WARN message indicates that producer failed to flush to checkpoint topic and would retry. Do you see the Samza job hanging after that? Is the checkpointed offset metrics incrementing in this case? Not clear on your claiming: "logs stop at that point". No logs are written after the WARN lines? What's your Samza configuration? Is the Samza container still running after you see those WARN logs? Thanks! -Yi On Wed, Mar 11, 2020 at 2:39 PM Jeremiah Adams wrote: > Can anyone take a look at the message below? We are trying to gauge our > risk before moving forward. > > > Jeremiah Adams > Software Engineer > https://url.emailprotection.link/?bM9S-3pRw1lv8pYfwa-TwdjElP4W2K6b9vP5Crz22L_YcgsRJ-13h-OgPZSwFtU7GSNTDi1z-jdaRvWESRhtTVA~~ > Blog | Twitter | Facebook | LinkedIn > > > From: Jeremiah Adams > Sent: Wednesday, March 4, 2020 2:28 PM > To: dev@samza.apache.org > Subject: Got Error Produce Response iwth Correlation Id. > > Hello devs, > > > I've got a warning showing up in the logs while testing our new Confluent > Cloud config. Can anyone tell me how concerned I should be about this > warning? Is there a setting to control timeouts? > > > Also, logs stop at that point, so I can't tell if the "metatdata update" > was complete. > > > > 2020-03-04 21:17:51 Sender [WARN] [Producer > clientId=kafka_producer-application_submission-1] Got error produce > response with correlation id 144 on topic-partition > __samza_checkpoint_ver_1_for_application-submission_1-0, retrying > (2147483646 attempts left). Error: NETWORK_EXCEPTION > 2020-03-04 21:17:51 Sender [WARN] [Producer > clientId=kafka_producer-application_submission-1] Received invalid metadata > error in produce request on partition > __samza_checkpoint_ver_1_for_application-submission_1-0 due to > org.apache.kafka.common.errors.NetworkException: The server disconnected > before a response was received.. Going to request metadata update now > > > Jeremiah Adams > Software Engineer > > https://url.emailprotection.link/?bM9S-3pRw1lv8pYfwa-TwdjElP4W2K6b9vP5Crz22L_YcgsRJ-13h-OgPZSwFtU7GSNTDi1z-jdaRvWESRhtTVA~~ > < > https://url.emailprotection.link/?basKr9vk92a8vVw0XMnK5bmaSKuBc0AuEZ7YasYc7Df8YVt3SYmcjmLWdKMWzAAINWlUUA33ebGI7pSoTl9cg1g~~ > > > Blog< > https://url.emailprotection.link/?basKr9vk92a8vVw0XMnK5bmaSKuBc0AuEZ7YasYc7Df-lAcqG1fqHPpNw-wd9z7HtUJeCG5_8UjCf2mHtn6C_zQ~~> > | Twitter< > https://url.emailprotection.link/?bVO2q0UXR235wN_yOnM0FjqITPdBYMD3reLGNddq-zPV5ChMQK9JwV4Be-QnrbRoXpJl8IcknAqKzYtA3RABKww~~> > | Facebook< >
