All,
I'm getting CommitFailedExceptions on a small prototype I built using
kafkaStreams. I'm not using the DSL, but the TopologyBuilder with several
processors chained together with a sink in between a few of them. When I try
committing through the ProcessorContext, I see exceptions being thrown about
commit failures due to group rebalance (not to mention delays in processing
during commit attempts). I'm using a single host, with 2 stream threads and a
5-node Kafka cluster. I wouldn't think rebalancing would be occurring after
data starts flowing and I'm committing offsets. This was something I saw with
the new Kafka client APIs as well, and I had to work around by creating static
partition assignments to my consumers in my data flow...otherwise, any dynamic
allocation of new consumers to the group caused this exception to be thrown and
I could never commit my offsets. Are you all not seeing this in your tests? Am
I not supposed to commit through the ProcessorContext? I'm committing once my
interim processor writes its output to the sink point in the flow; is that not
the correct/expected behavior/use?
Mike
