Re: Kafka Streams : TimeoutException Expiring Records

2018-02-22 Thread Tony John
Thanks a lot Bill for looking in to this. I would definitely attempt the suggestions and let you know the outcome. I have gone through KIP-91, but struggling to understand the behavior. So does it mean that these errors are happening due to a failure in the broker? If so why would it kill all the t

Re: Kafka Streams : TimeoutException Expiring Records

2018-02-22 Thread Bill Bejeck
Hi Tony, Looks like you have a known issue that KIP-91( https://cwiki.apache.org/confluence/display/KAFKA/KIP-91+Provide+Intuitive+User+Timeouts+in+The+Producer) will address. In the meantime, as a workaround, you could try setting REQUEST_TIMEOUT_MS_CONFIG to a large value (Integer.MAX_VALUE ?)

Kafka Streams : TimeoutException Expiring Records

2018-02-22 Thread Tony John
Hi All, I am running into an issue with my Kafka Streams application. The application was running fine for almost 2 weeks, then it started throwing the below exception which caused the threads to die. Now when I restart the application, it dies quickly (1-2 hrs) when trying to catch up the lag. T