[
https://issues.apache.org/jira/browse/KAFKA-8724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Keith So updated KAFKA-8724:
----------------------------
Description:
We are attempting an upgrade from Kafka 2.0 to 2.3 on a single cluster setup.
We have a mixture of Java/C++ and Python clients (Python clients are using
kafka-python libraries).
After the upgrade, the log cleaner occasionally dies with the attached stack
trace. Using timestamp correlation, we pinned it down to the cleaning of a
__consumer_offsets topic. The config logged at initialization shows that
inter.broker.protocol.version = 2.3-IV1
log.message.format.version = 2.3-IV1
We initially thought this was to do with unclean upgrade from 2.0 to 2.3, but
after resetting the consumer offsets topic (via
[https://medium.com/@nblaye/reset-consumer-offsets-topic-in-kafka-with-zookeeper-5910213284a2])
this still recurs on initially empty consumer offset partitions.
At the moment we are working around by toggling log.cleaner.threads option
using dynamic broker configuration to restore the log cleaner threads
was:
We are attempting an upgrade from Kafka 2.0 to 2.3 on a single cluster setup.
We have a mixture of Java/C++ and Python clients (Python clients are using
kafka-python libraries).
After the upgrade, the log cleaner occasionally dies with the attached stack
trace. Using timestamp correlation, we pinned it down to the cleaning of a
consumer_offsets topic. The config logged at initialization shows that
inter.broker.protocol.version = 2.3-IV1
log.message.format.version = 2.3-IV1
We initially thought this was to do with unclean upgrade from 2.0 to 2.3, but
after resetting the consumer offsets topic (via
https://medium.com/@nblaye/reset-consumer-offsets-topic-in-kafka-with-zookeeper-5910213284a2)
this still recurs on initially empty consumer offset partitions.
At the moment we are working around by toggling log.cleaner.threads option
using dynamic broker configuration to restore the log cleaner threads
> log cleaner thread dies when attempting to clean a __consumer_offsets
> partition after upgrade from 2.0->2.3
> -----------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-8724
> URL: https://issues.apache.org/jira/browse/KAFKA-8724
> Project: Kafka
> Issue Type: Bug
> Components: log cleaner
> Affects Versions: 2.3.0
> Environment: Linux 3.10.0-862.2.3.el7.x86_64 #1 SMP Wed May 9
> 18:05:47 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
> Reporter: Keith So
> Priority: Major
> Attachments: KAFKA-308-stack-trace.txt
>
>
> We are attempting an upgrade from Kafka 2.0 to 2.3 on a single cluster setup.
> We have a mixture of Java/C++ and Python clients (Python clients are using
> kafka-python libraries).
> After the upgrade, the log cleaner occasionally dies with the attached stack
> trace. Using timestamp correlation, we pinned it down to the cleaning of a
> __consumer_offsets topic. The config logged at initialization shows that
> inter.broker.protocol.version = 2.3-IV1
> log.message.format.version = 2.3-IV1
> We initially thought this was to do with unclean upgrade from 2.0 to 2.3, but
> after resetting the consumer offsets topic (via
> [https://medium.com/@nblaye/reset-consumer-offsets-topic-in-kafka-with-zookeeper-5910213284a2])
> this still recurs on initially empty consumer offset partitions.
> At the moment we are working around by toggling log.cleaner.threads option
> using dynamic broker configuration to restore the log cleaner threads
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
