Re: kafka streams 0.10.2 failing while restoring offsets

2017-03-27 Thread Sachin Mittal
Well lets say previous thread was processing a large block and while processing it got bumped out. So new thread which got this partition may have different offset when it starts the restore and by the time it completes the restore. If that is the case then it should just ignore that task for that

Re: kafka streams 0.10.2 failing while restoring offsets

2017-03-27 Thread Damian Guy
Yes, but we don't know why it is still processing the data. We don't want to have multiple processes acting on the same tasks, hence the exception. What if for some reason the other task is processing a large backlog, how long do we wait before we give up? I think in this case the exception is the

Re: kafka streams 0.10.2 failing while restoring offsets

2017-03-27 Thread Sachin Mittal
Hi, These are the logs https://www.dropbox.com/s/2t4ysfdqbtmcusq/complete_84_85_87_log.zip?dl=0 I think this may not always be the case especially if previous owner is on a different machine. Say it is processing and it takes more than the poll timeout to process or commit the offset. The group

Re: kafka streams 0.10.2 failing while restoring offsets

2017-03-27 Thread Damian Guy
Hi Sachin, This should not happen. The previous owner of the task should have stopped processing before the restoration begins. So if this is happening, then that signals a bug. Do you have more logs? Thanks, Damian On Sat, 25 Mar 2017 at 08:20 Sachin Mittal wrote: > Hi, > So recently we fixed

kafka streams 0.10.2 failing while restoring offsets

2017-03-25 Thread Sachin Mittal
Hi, So recently we fixed the deadlock issue and also built the streams jar by copying the rocks db configs from trunk. So we don't get any deadlock issue now and also we see that the wait time of CPU cores stays around 5% (down from 50% earlier). However we now get a new exception which is not han