Re: skipped-records-rate vs skippedDueToDeserializationError-rate metric in streams app

Hi Guozhang, You are right. I overlooked the fact that skippedDueToDeserializationError is recorded as DEBUG. That was it. Now that I got it, it feels like an overkill to set metrics level to DEBUG just for this! Thanks for your time! Srikanth On Tue, Jan 30, 2018 at 10:56 PM, Guozhang Wang

Re: skipped-records-rate vs skippedDueToDeserializationError-rate metric in streams app

Your code for setting the handler seems right to me. Another double checking: have you turned on DEBUG level metrics recording in order for this metric? Note skippedDueToDeserializationError is recorded as DEBUG level so you need to set metrics.recording.level accordingly (default is INFO). Lower

Re: skipped-records-rate vs skippedDueToDeserializationError-rate metric in streams app

Guozhang, Here is the snippet. private Properties getProperties() { Properties p = new Properties(); ... p.put(StreamsConfig.NUM_STREAM_THREADS_CONFIG, kafkaConfig.getString(" streamThreads")); p.put(StreamsConfig.DEFAULT_DESERIALIZATION_EXCEPTION_HANDLER_CLASS_CONFIG,

Re: skipped-records-rate vs skippedDueToDeserializationError-rate metric in streams app

Hi Srikanth, How did you set the LogAndContinueExceptionHandler in the configs? Could you copy the code snippet here? Guozhang On Sun, Jan 28, 2018 at 11:26 PM, Srikanth wrote: > Kafka-streams version "1.0.0". > > Thanks, > Srikanth > > On Mon, Jan 29, 2018 at 12:23

Re: skipped-records-rate vs skippedDueToDeserializationError-rate metric in streams app

Kafka-streams version "1.0.0". Thanks, Srikanth On Mon, Jan 29, 2018 at 12:23 AM, Guozhang Wang wrote: > Hello Srikanth, > > Which version of Kafka are you using? I'd like to dig for that particular > branch again. > > Guozhang > > On Sun, Jan 28, 2018 at 8:54 AM, Srikanth

Re: skipped-records-rate vs skippedDueToDeserializationError-rate metric in streams app

Hello Srikanth, Which version of Kafka are you using? I'd like to dig for that particular branch again. Guozhang On Sun, Jan 28, 2018 at 8:54 AM, Srikanth wrote: > Guozhang, > > While I understand that this metric is meaningless when handler is set to > FAIL, in my case

Re: skipped-records-rate vs skippedDueToDeserializationError-rate metric in streams app

Guozhang, While I understand that this metric is meaningless when handler is set to FAIL, in my case I'm actually using LogAndContinueExceptionHandler. In this case, app needs to report such occurrences. What I noticed is that only skipped-records is set. The granularity offered by

Re: skipped-records-rate vs skippedDueToDeserializationError-rate metric in streams app

Guozhang, I read the comments in KAFKA-6376 and realized that we do something similar to handle exception in process(). We register a custom metric "process-skipped" and our alerting system looks for both this and skipped-records. Srikanth On Fri, Jan 26, 2018 at 10:26 PM, Guozhang Wang

Re: skipped-records-rate vs skippedDueToDeserializationError-rate metric in streams app

Hi Srikanth, Looked at the source code once again and discussing with other committer I now remembered why we designed it that way: when you set the HandlerResponse to FAIL, it means that once a "poison record" is received, stop the world by throwing this exception all the way up. And hence at

Re: skipped-records-rate vs skippedDueToDeserializationError-rate metric in streams app

Helo Srikanth, Thanks for reporting this, as I checked the code skippedDueToDeserializationError is effectively only recorded when the DeserializationHandlerResponse is not set to FAIL. I agree it is not exactly matching the documentation's guidance, and will try to file a JIRA and fix it. As