Re: Kafka Streams Application crashing on Rebalance

2017-08-01 Thread Eric Lalonde 

> On Aug 1, 2017, at 10:01 AM, Damian Guy  wrote:
> 
> Hi, Yes the issue is in 0.10.2 also.

Hi,

Any chance of a backport to 0.10.2.x? We have lots and lots of state stores. :)

Re: Kafka Streams Application crashing on Rebalance

2017-08-01 Thread Marcus Clendenin 
I'll work on moving to 0.11. Thanks

On Tue, Aug 1, 2017, 1:02 PM Damian Guy  wrote:

> Hi, Yes the issue is in 0.10.2 also.
>
> On Tue, 1 Aug 2017 at 17:37 Eric Lalonde  wrote:
>
> >
> > > On Aug 1, 2017, at 8:00 AM, Damian Guy  wrote:
> > >
> > > It is a bug in 0.10.2 or lower. It has been fixed in 0.11 by
> > > https://issues.apache.org/jira/browse/KAFKA-4494
> >
> > Hi Damien, the Affects Version is set to 0.10.1.0 in KAFKA-4494. Is the
> > issue in 0.10.2.0 as well?
>

Re: Kafka Streams Application crashing on Rebalance

2017-08-01 Thread Damian Guy 
Hi, Yes the issue is in 0.10.2 also.

On Tue, 1 Aug 2017 at 17:37 Eric Lalonde  wrote:

>
> > On Aug 1, 2017, at 8:00 AM, Damian Guy  wrote:
> >
> > It is a bug in 0.10.2 or lower. It has been fixed in 0.11 by
> > https://issues.apache.org/jira/browse/KAFKA-4494
>
> Hi Damien, the Affects Version is set to 0.10.1.0 in KAFKA-4494. Is the
> issue in 0.10.2.0 as well?

Re: Kafka Streams Application crashing on Rebalance

2017-08-01 Thread Eric Lalonde 

> On Aug 1, 2017, at 8:00 AM, Damian Guy  wrote:
> 
> It is a bug in 0.10.2 or lower. It has been fixed in 0.11 by
> https://issues.apache.org/jira/browse/KAFKA-4494

Hi Damien, the Affects Version is set to 0.10.1.0 in KAFKA-4494. Is the issue 
in 0.10.2.0 as well?

Re: Kafka Streams Application crashing on Rebalance

2017-08-01 Thread Damian Guy 
It is a bug in 0.10.2 or lower. It has been fixed in 0.11 by
https://issues.apache.org/jira/browse/KAFKA-4494

On Tue, 1 Aug 2017 at 15:40 Marcus Clendenin  wrote:

> Hi All,
>
>
>
> I have a kafka streams application that is doing a join between a KTable
> and a KStream and it seems that after it starts loading the KTable if I
> either restart the application or start a new jar with the same
> application-id it starts failing. It looks like when it tries to rejoin the
> application-id and do a rebalance of the partitions it throws an error
> regarding a null value coming from RocksDB. Any thoughts on where this is
> coming from? I am running this inside of a docker container if that affects
> anything but the RocksDB folder is mounted as a volume on the host machine.
>
>
>
>
>
> Stacktrace:
>
>
>
> 2017-08-01 13:31:50,309 trackingId=X thread=[StreamThread-1] logType=INFO
>
> module=kafka.streams
> o.a.kafka.streams.processor.internals.StreamThread  - stream-thread
> [StreamThread-1] Starting
>
> 2017-08-01 13:31:50,379 trackingId=X thread=[StreamThread-1] logType=INFO
>
> module=kafka.streams
> o.a.k.c.consumer.internals.AbstractCoordinator  - Discovered coordinator
> .com:9092 (id: 2147483535 <(214)%20748-3535> rack: null) for group
> test-application-id.
>
> 2017-08-01 13:31:50,386 trackingId=X thread=[StreamThread-1] logType=INFO
>
> module=kafka.streams
> o.a.k.c.consumer.internals.ConsumerCoordinator  - Revoking previously
> assigned partitions [] for group test-application-id
>
> 2017-08-01 13:31:50,386 trackingId=X thread=[StreamThread-1] logType=INFO
>
> module=kafka.streams
> o.a.kafka.streams.processor.internals.StreamThread  - stream-thread
> [StreamThread-1] at state RUNNING: partitions [] revoked at the beginning
> of consumer rebalance.
>
> 2017-08-01 13:31:50,387 trackingId=X thread=[StreamThread-1] logType=INFO
>
> module=kafka.streams
> o.a.kafka.streams.processor.internals.StreamThread  - stream-thread
> [StreamThread-1] State transition from RUNNING to PARTITIONS_REVOKED.
>
> 2017-08-01 13:31:50,387 trackingId=X thread=[StreamThread-1] logType=INFO
>
> module=kafka.streams org.apache.kafka.streams.KafkaStreams
> - stream-client [test-application-id-67f96d6e-d1fd-4f31-8ec4-45e82a9cf01c]
> State transition from RUNNING to REBALANCING.
>
> 2017-08-01 13:31:50,388 trackingId=X thread=[StreamThread-1] logType=INFO
>
> module=kafka.streams
> o.a.kafka.streams.processor.internals.StreamThread  - stream-thread
> [StreamThread-1] Updating suspended tasks to contain active tasks []
>
> 2017-08-01 13:31:50,388 trackingId=X thread=[StreamThread-1] logType=INFO
>
> module=kafka.streams
> o.a.kafka.streams.processor.internals.StreamThread  - stream-thread
> [StreamThread-1] Removing all active tasks []
>
> 2017-08-01 13:31:50,388 trackingId=X thread=[StreamThread-1] logType=INFO
>
> module=kafka.streams
> o.a.kafka.streams.processor.internals.StreamThread  - stream-thread
> [StreamThread-1] Removing all standby tasks []
>
> 2017-08-01 13:31:50,389 trackingId=X thread=[StreamThread-1] logType=INFO
>
> module=kafka.streams
> o.a.k.c.consumer.internals.AbstractCoordinator  - (Re-)joining group
> test-application-id
>
> 2017-08-01 13:31:50,416 trackingId=X thread=[StreamThread-1] logType=INFO
>
> module=kafka.streams
> o.a.k.s.p.internals.StreamPartitionAssignor  - stream-thread
> [StreamThread-1] Constructed client metadata
> {67f96d6e-d1fd-4f31-8ec4-45e82a9cf01c=ClientMetadata{hostInfo=null,
>
> consumers=[test-application-id-67f96d6e-d1fd-4f31-8ec4-45e82a9cf01c-StreamThread-1-consumer-f6ed6af8-0aee-4d2e-92a9-00955f7b3441],
> state=[activeTasks: ([]) assignedTasks: ([]) prevActiveTasks: ([])
> prevAssignedTasks: ([]) capacity: 1.0 cost: 0.0]}} from the member
> subscriptions.
>
> 2017-08-01 13:31:50,417 trackingId=X thread=[StreamThread-1] logType=INFO
>
> module=kafka.streams
> o.a.k.s.p.internals.StreamPartitionAssignor  - stream-thread
> [StreamThread-1] Completed validating internal topics in partition assignor
>
> 2017-08-01 13:31:50,417 trackingId=X thread=[StreamThread-1] logType=INFO
>
> module=kafka.streams
> o.a.k.s.p.internals.StreamPartitionAssignor  - stream-thread
> [StreamThread-1] Completed validating internal topics in partition assignor
>
> 2017-08-01 13:31:50,419 trackingId=X thread=[StreamThread-1] logType=INFO
>
> module=kafka.streams
> o.a.k.s.p.internals.StreamPartitionAssignor  - stream-thread
> [StreamThread-1] Assigned tasks to clients as
> {67f96d6e-d1fd-4f31-8ec4-45e82a9cf01c=[activeTasks: ([0_0, 0_1, 0_2, 0_3,
> 0_4, 0_5]) assignedTasks: ([0_0, 0_1, 0_2, 0_3, 0_4, 0_5]) prevActiveTasks:
> ([]) prevAssignedTasks: ([]) capacity: 1.0 cost: 3.0]}.
>
> 2017-08-01 13:31:50,429 trackingId=X thread=[StreamThread-1] logType=INFO
>
>

Kafka Streams Application crashing on Rebalance

2017-08-01 Thread Marcus Clendenin 
Hi All,



I have a kafka streams application that is doing a join between a KTable
and a KStream and it seems that after it starts loading the KTable if I
either restart the application or start a new jar with the same
application-id it starts failing. It looks like when it tries to rejoin the
application-id and do a rebalance of the partitions it throws an error
regarding a null value coming from RocksDB. Any thoughts on where this is
coming from? I am running this inside of a docker container if that affects
anything but the RocksDB folder is mounted as a volume on the host machine.





Stacktrace:



2017-08-01 13:31:50,309 trackingId=X thread=[StreamThread-1] logType=INFO

module=kafka.streams
o.a.kafka.streams.processor.internals.StreamThread  - stream-thread
[StreamThread-1] Starting

2017-08-01 13:31:50,379 trackingId=X thread=[StreamThread-1] logType=INFO

module=kafka.streams
o.a.k.c.consumer.internals.AbstractCoordinator  - Discovered coordinator
.com:9092 (id: 2147483535 rack: null) for group test-application-id.

2017-08-01 13:31:50,386 trackingId=X thread=[StreamThread-1] logType=INFO

module=kafka.streams
o.a.k.c.consumer.internals.ConsumerCoordinator  - Revoking previously
assigned partitions [] for group test-application-id

2017-08-01 13:31:50,386 trackingId=X thread=[StreamThread-1] logType=INFO

module=kafka.streams
o.a.kafka.streams.processor.internals.StreamThread  - stream-thread
[StreamThread-1] at state RUNNING: partitions [] revoked at the beginning
of consumer rebalance.

2017-08-01 13:31:50,387 trackingId=X thread=[StreamThread-1] logType=INFO

module=kafka.streams
o.a.kafka.streams.processor.internals.StreamThread  - stream-thread
[StreamThread-1] State transition from RUNNING to PARTITIONS_REVOKED.

2017-08-01 13:31:50,387 trackingId=X thread=[StreamThread-1] logType=INFO

module=kafka.streams org.apache.kafka.streams.KafkaStreams
- stream-client [test-application-id-67f96d6e-d1fd-4f31-8ec4-45e82a9cf01c]
State transition from RUNNING to REBALANCING.

2017-08-01 13:31:50,388 trackingId=X thread=[StreamThread-1] logType=INFO

module=kafka.streams
o.a.kafka.streams.processor.internals.StreamThread  - stream-thread
[StreamThread-1] Updating suspended tasks to contain active tasks []

2017-08-01 13:31:50,388 trackingId=X thread=[StreamThread-1] logType=INFO

module=kafka.streams
o.a.kafka.streams.processor.internals.StreamThread  - stream-thread
[StreamThread-1] Removing all active tasks []

2017-08-01 13:31:50,388 trackingId=X thread=[StreamThread-1] logType=INFO

module=kafka.streams
o.a.kafka.streams.processor.internals.StreamThread  - stream-thread
[StreamThread-1] Removing all standby tasks []

2017-08-01 13:31:50,389 trackingId=X thread=[StreamThread-1] logType=INFO

module=kafka.streams
o.a.k.c.consumer.internals.AbstractCoordinator  - (Re-)joining group
test-application-id

2017-08-01 13:31:50,416 trackingId=X thread=[StreamThread-1] logType=INFO

module=kafka.streams
o.a.k.s.p.internals.StreamPartitionAssignor  - stream-thread
[StreamThread-1] Constructed client metadata
{67f96d6e-d1fd-4f31-8ec4-45e82a9cf01c=ClientMetadata{hostInfo=null,
consumers=[test-application-id-67f96d6e-d1fd-4f31-8ec4-45e82a9cf01c-StreamThread-1-consumer-f6ed6af8-0aee-4d2e-92a9-00955f7b3441],
state=[activeTasks: ([]) assignedTasks: ([]) prevActiveTasks: ([])
prevAssignedTasks: ([]) capacity: 1.0 cost: 0.0]}} from the member
subscriptions.

2017-08-01 13:31:50,417 trackingId=X thread=[StreamThread-1] logType=INFO

module=kafka.streams
o.a.k.s.p.internals.StreamPartitionAssignor  - stream-thread
[StreamThread-1] Completed validating internal topics in partition assignor

2017-08-01 13:31:50,417 trackingId=X thread=[StreamThread-1] logType=INFO

module=kafka.streams
o.a.k.s.p.internals.StreamPartitionAssignor  - stream-thread
[StreamThread-1] Completed validating internal topics in partition assignor

2017-08-01 13:31:50,419 trackingId=X thread=[StreamThread-1] logType=INFO

module=kafka.streams
o.a.k.s.p.internals.StreamPartitionAssignor  - stream-thread
[StreamThread-1] Assigned tasks to clients as
{67f96d6e-d1fd-4f31-8ec4-45e82a9cf01c=[activeTasks: ([0_0, 0_1, 0_2, 0_3,
0_4, 0_5]) assignedTasks: ([0_0, 0_1, 0_2, 0_3, 0_4, 0_5]) prevActiveTasks:
([]) prevAssignedTasks: ([]) capacity: 1.0 cost: 3.0]}.

2017-08-01 13:31:50,429 trackingId=X thread=[StreamThread-1] logType=INFO

module=kafka.streams
o.a.k.c.consumer.internals.AbstractCoordinator  - Successfully joined group
test-application-id with generation 56

2017-08-01 13:31:50,430 trackingId=X thread=[StreamThread-1] logType=INFO

module=kafka.streams
o.a.k.c.consumer.internals.ConsumerCoordinator  - Setting newly assigned
partitions [stream_topic-0, stream_topic-1, stream_topic-2,