[jira] [Updated] (KAFKA-8876) KafkaBasedLog does not throw exception when some partitions of the topic is offline
[
https://issues.apache.org/jira/browse/KAFKA-8876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Boquan Tang updated KAFKA-8876:
---
Description:
Currently KafkaBasedLog does not check if *all* partitions in the topic is
online or not, this may result it ignoring partitions that's still recovering
and in turn report to KafkaOffsetBackingStore null offset backed by the
concerning partition, while in fact it should either wait or fail the connector
thread to prompt retry, so the offset can be correctly loaded by the connector.
Specifically, we are using debezium mysql connector to replicate mysql binlog
to kafka.
In an attempt of restarting after a cluster downage, we observed following:
{code}
2019-08-29T19:27:32Z INFO
[org.apache.kafka.connect.storage.KafkaOffsetBackingStore] [main] Starting
KafkaOffsetBackingStore
2019-08-29T19:27:32Z INFO [org.apache.kafka.connect.util.KafkaBasedLog] [main]
Starting KafkaBasedLog with topic bobqueue-binlog-shovel-v1-offsets
...skipped client config logs...
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-12 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-10 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-21 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-5 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-20 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-18 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-2 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-13 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-11 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-8 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-23 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-7 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-22 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-6 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-3 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-9 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1,
[jira] [Updated] (KAFKA-8876) KafkaBasedLog does not throw exception when some partitions of the topic is offline
[
https://issues.apache.org/jira/browse/KAFKA-8876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Boquan Tang updated KAFKA-8876:
---
Description:
Currently KafkaBasedLog does not check if *all* partitions in the topic is
online or not, this may result it ignoring partitions that's still recovering
and in turn report to KafkaOffsetBackingStore null offset backed by the
concerning partition, while in fact it should either wait or fail the connector
thread to prompt retry, so the offset can be correctly loaded by the connector.
Specifically, we are using debezium mysql connector to replicate mysql binlog
to kafka.
In an attempt of restarting after a cluster downage, we observed following:
{code}
2019-08-29T19:27:32Z INFO
[org.apache.kafka.connect.storage.KafkaOffsetBackingStore] [main] Starting
KafkaOffsetBackingStore
2019-08-29T19:27:32Z INFO [org.apache.kafka.connect.util.KafkaBasedLog] [main]
Starting KafkaBasedLog with topic bobqueue-binlog-shovel-v1-offsets
...skipped client config logs...
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-12 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-10 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-21 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-5 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-20 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-18 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-2 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-13 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-11 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-8 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-23 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-7 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-22 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-6 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-3 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-9 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1,
[jira] [Assigned] (KAFKA-8875) CreateTopic API should check topic existence before replication factor
[ https://issues.apache.org/jira/browse/KAFKA-8875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huxihx reassigned KAFKA-8875: - Assignee: huxihx > CreateTopic API should check topic existence before replication factor > -- > > Key: KAFKA-8875 > URL: https://issues.apache.org/jira/browse/KAFKA-8875 > Project: Kafka > Issue Type: Bug >Reporter: Jason Gustafson >Assignee: huxihx >Priority: Major > > If you try to create a topic and the replication factor cannot be satisfied, > Kafka will return `INVALID_REPLICATION_FACTOR`. If the topic already exists, > we should probably return `TOPIC_ALREADY_EXISTS` instead. You won't see this > problem if using TopicCommand because we check existence prior to creating > the topic. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (KAFKA-8876) KafkaBasedLog does not throw exception when some partitions of the topic is offline
[
https://issues.apache.org/jira/browse/KAFKA-8876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Boquan Tang updated KAFKA-8876:
---
Affects Version/s: 1.1.0
> KafkaBasedLog does not throw exception when some partitions of the topic is
> offline
> ---
>
> Key: KAFKA-8876
> URL: https://issues.apache.org/jira/browse/KAFKA-8876
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Boquan Tang
>Priority: Major
>
> Currently KafkaBasedLog does not check if *all* partitions in the topic is
> online or not, this may result it ignoring partitions that's still recovering
> and in turn report to KafkaOffsetBackingStore null offset, while in fact it
> should either wait or fail the thread to prompt retry, so the offset can be
> correctly loaded by the connector.
> Specifically, we are using debezium mysql connector to replicate mysql binlog
> to kafka.
> In an attempt of restarting after a cluster downage, we observed following:
> {code}
> 2019-08-29T19:27:32Z INFO
> [org.apache.kafka.connect.storage.KafkaOffsetBackingStore] [main] Starting
> KafkaOffsetBackingStore
> 2019-08-29T19:27:32Z INFO [org.apache.kafka.connect.util.KafkaBasedLog]
> [main] Starting KafkaBasedLog with topic bobqueue-binlog-shovel-v1-offsets
> ...skipped client config logs...
> 2019-08-29T19:27:33Z INFO
> [org.apache.kafka.clients.consumer.internals.Fetcher] [main] [Consumer
> clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1] Resetting offset for
> partition bobqueue-binlog-shovel-v1-offsets-12 to offset 0.
> 2019-08-29T19:27:33Z INFO
> [org.apache.kafka.clients.consumer.internals.Fetcher] [main] [Consumer
> clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1] Resetting offset for
> partition bobqueue-binlog-shovel-v1-offsets-10 to offset 0.
> 2019-08-29T19:27:33Z INFO
> [org.apache.kafka.clients.consumer.internals.Fetcher] [main] [Consumer
> clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1] Resetting offset for
> partition bobqueue-binlog-shovel-v1-offsets-21 to offset 0.
> 2019-08-29T19:27:33Z INFO
> [org.apache.kafka.clients.consumer.internals.Fetcher] [main] [Consumer
> clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1] Resetting offset for
> partition bobqueue-binlog-shovel-v1-offsets-5 to offset 0.
> 2019-08-29T19:27:33Z INFO
> [org.apache.kafka.clients.consumer.internals.Fetcher] [main] [Consumer
> clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1] Resetting offset for
> partition bobqueue-binlog-shovel-v1-offsets-20 to offset 0.
> 2019-08-29T19:27:33Z INFO
> [org.apache.kafka.clients.consumer.internals.Fetcher] [main] [Consumer
> clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1] Resetting offset for
> partition bobqueue-binlog-shovel-v1-offsets-18 to offset 0.
> 2019-08-29T19:27:33Z INFO
> [org.apache.kafka.clients.consumer.internals.Fetcher] [main] [Consumer
> clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1] Resetting offset for
> partition bobqueue-binlog-shovel-v1-offsets-2 to offset 0.
> 2019-08-29T19:27:33Z INFO
> [org.apache.kafka.clients.consumer.internals.Fetcher] [main] [Consumer
> clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1] Resetting offset for
> partition bobqueue-binlog-shovel-v1-offsets-13 to offset 0.
> 2019-08-29T19:27:33Z INFO
> [org.apache.kafka.clients.consumer.internals.Fetcher] [main] [Consumer
> clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1] Resetting offset for
> partition bobqueue-binlog-shovel-v1-offsets-11 to offset 0.
> 2019-08-29T19:27:33Z INFO
> [org.apache.kafka.clients.consumer.internals.Fetcher] [main] [Consumer
> clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1] Resetting offset for
> partition bobqueue-binlog-shovel-v1-offsets-8 to offset 0.
> 2019-08-29T19:27:33Z INFO
> [org.apache.kafka.clients.consumer.internals.Fetcher] [main] [Consumer
> clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1] Resetting offset for
> partition bobqueue-binlog-shovel-v1-offsets-23 to offset 0.
> 2019-08-29T19:27:33Z INFO
> [org.apache.kafka.clients.consumer.internals.Fetcher] [main] [Consumer
> clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1] Resetting offset for
> partition bobqueue-binlog-shovel-v1-offsets-7 to offset 0.
> 2019-08-29T19:27:33Z INFO
> [org.apache.kafka.clients.consumer.internals.Fetcher] [main] [Consumer
> clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1] Resetting offset for
> partition bobqueue-binlog-shovel-v1-offsets-22 to offset 0.
> 2019-08-29T19:27:33Z INFO
> [org.apache.kafka.clients.consumer.internals.Fetcher] [main] [Consumer
> clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1] Resetting offset for
> partition bobqueue-binlog-shovel-v1-offsets-6 to offset 0.
> 2019-08-29T19:27:33Z INFO
>
[jira] [Updated] (KAFKA-8876) KafkaBasedLog does not throw exception when some partitions of the topic is offline
[
https://issues.apache.org/jira/browse/KAFKA-8876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Boquan Tang updated KAFKA-8876:
---
Description:
Currently KafkaBasedLog does not check if *all* partitions in the topic is
online or not, this may result it ignoring partitions that's still recovering
and in turn report to KafkaOffsetBackingStore null offset, while in fact it
should either wait or fail the thread to prompt retry, so the offset can be
correctly loaded by the connector.
Specifically, we are using debezium mysql connector to replicate mysql binlog
to kafka.
In an attempt of restarting after a cluster downage, we observed following:
{code}
2019-08-29T19:27:32Z INFO
[org.apache.kafka.connect.storage.KafkaOffsetBackingStore] [main] Starting
KafkaOffsetBackingStore
2019-08-29T19:27:32Z INFO [org.apache.kafka.connect.util.KafkaBasedLog] [main]
Starting KafkaBasedLog with topic bobqueue-binlog-shovel-v1-offsets
...skipped client config logs...
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-12 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-10 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-21 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-5 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-20 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-18 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-2 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-13 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-11 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-8 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-23 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-7 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-22 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-6 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-3 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-9 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition
[jira] [Created] (KAFKA-8876) KafkaBasedLog does not throw exception when some partitions of the topic is offline
Boquan Tang created KAFKA-8876:
--
Summary: KafkaBasedLog does not throw exception when some
partitions of the topic is offline
Key: KAFKA-8876
URL: https://issues.apache.org/jira/browse/KAFKA-8876
Project: Kafka
Issue Type: Bug
Reporter: Boquan Tang
Currently KafkaBasedLog does not check if *all* partitions in the topic is
online or not, this may result it ignoring partitions that's still recovering
and in turn report to KafkaOffsetBackingStore null offset, while in fact it
should either wait or fail the thread to prompt retry, so the offset can be
correctly loaded by the connector.
Specifically, we are using debezium mysql connector to replicate mysql binlog
to kafka.
In an attempt of restarting after a cluster downage, we observed following:
{code}
2019-08-29T19:27:32Z INFO
[org.apache.kafka.connect.storage.KafkaOffsetBackingStore] [main] Starting
KafkaOffsetBackingStore
2019-08-29T19:27:32Z INFO [org.apache.kafka.connect.util.KafkaBasedLog] [main]
Starting KafkaBasedLog with topic bobqueue-binlog-shovel-v1-offsets
...skipped client config logs...
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-12 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-10 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-21 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-5 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-20 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-18 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-2 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-13 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-11 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-8 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-23 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-7 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-22 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-6 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-3 to offset 0.
2019-08-29T19:27:33Z INFO [org.apache.kafka.clients.consumer.internals.Fetcher]
[main] [Consumer clientId=consumer-1, groupId=bobqueue-binlog-shovel-v1]
Resetting offset for partition bobqueue-binlog-shovel-v1-offsets-9 to offset 0.
2019-08-29T19:27:33Z INFO
[jira] [Created] (KAFKA-8875) CreateTopic API should check topic existence before replication factor
Jason Gustafson created KAFKA-8875: -- Summary: CreateTopic API should check topic existence before replication factor Key: KAFKA-8875 URL: https://issues.apache.org/jira/browse/KAFKA-8875 Project: Kafka Issue Type: Bug Reporter: Jason Gustafson If you try to create a topic and the replication factor cannot be satisfied, Kafka will return `INVALID_REPLICATION_FACTOR`. If the topic already exists, we should probably return `TOPIC_ALREADY_EXISTS` instead. You won't see this problem if using TopicCommand because we check existence prior to creating the topic. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (KAFKA-8874) KIP-517: Add consumer metric indicating time between poll calls
Kevin Lu created KAFKA-8874: --- Summary: KIP-517: Add consumer metric indicating time between poll calls Key: KAFKA-8874 URL: https://issues.apache.org/jira/browse/KAFKA-8874 Project: Kafka Issue Type: New Feature Components: consumer, metrics Reporter: Kevin Lu Assignee: Kevin Lu [https://cwiki.apache.org/confluence/display/KAFKA/KIP-517%3A+Add+consumer+metric+indicating+time+between+poll+calls] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (KAFKA-8873) Implement timeout for Alter/List PartitionReassignment APIs
Stanislav Kozlovski created KAFKA-8873: -- Summary: Implement timeout for Alter/List PartitionReassignment APIs Key: KAFKA-8873 URL: https://issues.apache.org/jira/browse/KAFKA-8873 Project: Kafka Issue Type: Improvement Reporter: Stanislav Kozlovski In the initial implementation of KIP-455, we decided ([https://github.com/apache/kafka/pull/7128#issuecomment-528099402)] to delegate the implementation of the timeout functionality to a separate task. This is in part because the change is not trivial and because there are other controller RPCs which would be good to get updated with timeout functionality -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Resolved] (KAFKA-8568) MirrorMaker 2.0 resource leak
[
https://issues.apache.org/jira/browse/KAFKA-8568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ryanne Dolan resolved KAFKA-8568.
-
Resolution: Fixed
> MirrorMaker 2.0 resource leak
> -
>
> Key: KAFKA-8568
> URL: https://issues.apache.org/jira/browse/KAFKA-8568
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect, mirrormaker
>Affects Versions: 2.2.2
>Reporter: Péter Gergő Barna
>Assignee: Ryanne Dolan
>Priority: Major
>
> This issue produced by the branch KIP-382 (I am not sure which version is
> affected by that branch).
> While MirrorMaker 2.0 is running, the following command returns a number that
> is getting larger and larger.
>
> {noformat}
> lsof -p | grep ESTABLISHED | wc -l{noformat}
>
> Meanwhile, in the error log, NullPointers pop up from the
> MirrorSourceTask.cleanup, because either the consumer or the producer is null
> when the cleanup method tries to close them.
>
> {noformat}
> Exception in thread "Thread-790" java.lang.NullPointerException
> at
> org.apache.kafka.connect.mirror.MirrorSourceTask.cleanup(MirrorSourceTask.java:110)
> at java.lang.Thread.run(Thread.java:748)
> Exception in thread "Thread-792" java.lang.NullPointerException
> at
> org.apache.kafka.connect.mirror.MirrorSourceTask.cleanup(MirrorSourceTask.java:110)
> at java.lang.Thread.run(Thread.java:748)
> Exception in thread "Thread-791" java.lang.NullPointerException
> at
> org.apache.kafka.connect.mirror.MirrorSourceTask.cleanup(MirrorSourceTask.java:116)
> at java.lang.Thread.run(Thread.java:748)
> Exception in thread "Thread-793" java.lang.NullPointerException
> at
> org.apache.kafka.connect.mirror.MirrorSourceTask.cleanup(MirrorSourceTask.java:110)
> at java.lang.Thread.run(Thread.java:748){noformat}
> When the number of the established connections (returned by lsof) reaches a
> certain limit, new exceptions start to pop up in the logs: Too many open files
> {noformat}
> [2019-06-19 12:56:43,949] ERROR
> WorkerSourceTask{id=MirrorHeartbeatConnector-0} failed to send record to
> heartbeats: {} (org.apache.kafka.connect.runtime.WorkerSourceTask)
> org.apache.kafka.common.errors.SaslAuthenticationException: An error:
> (java.security.PrivilegedActionException: javax.security.sasl.SaslException:
> GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Too many open
> files)]) occurred when evaluating SASL token received from the Kafka Broker.
> Kafka Client will go to A
> UTHENTICATION_FAILED state.
> Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Too many open
> files)]
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
> at
> org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.lambda$createSaslToken$1(SaslClientAuthenticator.java:461)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.createSaslToken(SaslClientAuthenticator.java:461)
> at
> org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.sendSaslClientToken(SaslClientAuthenticator.java:370)
> at
> org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.sendInitialToken(SaslClientAuthenticator.java:290)
> at
> org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.authenticate(SaslClientAuthenticator.java:230)
> at
> org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:173)
> at
> org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:536)
> at org.apache.kafka.common.network.Selector.poll(Selector.java:472)
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
> at
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:311)
> at
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:235)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: GSSException: No valid credentials provided (Mechanism level: Too
> many open files)
> at
> sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:775)
> at
> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
> at
> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
> ... 14 more
> Caused by: java.net.SocketException: Too many open files
> at java.net.Socket.createImpl(Socket.java:460)
>
[jira] [Commented] (KAFKA-8568) MirrorMaker 2.0 resource leak
[
https://issues.apache.org/jira/browse/KAFKA-8568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16922927#comment-16922927
]
Ryanne Dolan commented on KAFKA-8568:
-
This was fixed a while back. MM2 was re-using some top-level configuration
properties for internal clients, causing a huge number of extraneous
MetricsReporters, ConfigProviders etc to be created. This was resolved by
limiting which top-level properties are applied to internal clients.
> MirrorMaker 2.0 resource leak
> -
>
> Key: KAFKA-8568
> URL: https://issues.apache.org/jira/browse/KAFKA-8568
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect, mirrormaker
>Affects Versions: 2.2.2
>Reporter: Péter Gergő Barna
>Assignee: Ryanne Dolan
>Priority: Major
>
> This issue produced by the branch KIP-382 (I am not sure which version is
> affected by that branch).
> While MirrorMaker 2.0 is running, the following command returns a number that
> is getting larger and larger.
>
> {noformat}
> lsof -p | grep ESTABLISHED | wc -l{noformat}
>
> Meanwhile, in the error log, NullPointers pop up from the
> MirrorSourceTask.cleanup, because either the consumer or the producer is null
> when the cleanup method tries to close them.
>
> {noformat}
> Exception in thread "Thread-790" java.lang.NullPointerException
> at
> org.apache.kafka.connect.mirror.MirrorSourceTask.cleanup(MirrorSourceTask.java:110)
> at java.lang.Thread.run(Thread.java:748)
> Exception in thread "Thread-792" java.lang.NullPointerException
> at
> org.apache.kafka.connect.mirror.MirrorSourceTask.cleanup(MirrorSourceTask.java:110)
> at java.lang.Thread.run(Thread.java:748)
> Exception in thread "Thread-791" java.lang.NullPointerException
> at
> org.apache.kafka.connect.mirror.MirrorSourceTask.cleanup(MirrorSourceTask.java:116)
> at java.lang.Thread.run(Thread.java:748)
> Exception in thread "Thread-793" java.lang.NullPointerException
> at
> org.apache.kafka.connect.mirror.MirrorSourceTask.cleanup(MirrorSourceTask.java:110)
> at java.lang.Thread.run(Thread.java:748){noformat}
> When the number of the established connections (returned by lsof) reaches a
> certain limit, new exceptions start to pop up in the logs: Too many open files
> {noformat}
> [2019-06-19 12:56:43,949] ERROR
> WorkerSourceTask{id=MirrorHeartbeatConnector-0} failed to send record to
> heartbeats: {} (org.apache.kafka.connect.runtime.WorkerSourceTask)
> org.apache.kafka.common.errors.SaslAuthenticationException: An error:
> (java.security.PrivilegedActionException: javax.security.sasl.SaslException:
> GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Too many open
> files)]) occurred when evaluating SASL token received from the Kafka Broker.
> Kafka Client will go to A
> UTHENTICATION_FAILED state.
> Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Too many open
> files)]
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
> at
> org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.lambda$createSaslToken$1(SaslClientAuthenticator.java:461)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.createSaslToken(SaslClientAuthenticator.java:461)
> at
> org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.sendSaslClientToken(SaslClientAuthenticator.java:370)
> at
> org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.sendInitialToken(SaslClientAuthenticator.java:290)
> at
> org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.authenticate(SaslClientAuthenticator.java:230)
> at
> org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:173)
> at
> org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:536)
> at org.apache.kafka.common.network.Selector.poll(Selector.java:472)
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
> at
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:311)
> at
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:235)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: GSSException: No valid credentials provided (Mechanism level: Too
> many open files)
> at
> sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:775)
> at
> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
> at
>
[jira] [Updated] (KAFKA-7500) MirrorMaker 2.0 (KIP-382)
[ https://issues.apache.org/jira/browse/KAFKA-7500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryanne Dolan updated KAFKA-7500: Labels: pull-request-available ready-to-commit (was: ) > MirrorMaker 2.0 (KIP-382) > - > > Key: KAFKA-7500 > URL: https://issues.apache.org/jira/browse/KAFKA-7500 > Project: Kafka > Issue Type: New Feature > Components: KafkaConnect, mirrormaker >Affects Versions: 2.4.0 >Reporter: Ryanne Dolan >Priority: Major > Labels: pull-request-available, ready-to-commit > Fix For: 2.4.0 > > Attachments: Active-Active XDCR setup.png > > > Implement a drop-in replacement for MirrorMaker leveraging the Connect > framework. > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0] > [https://github.com/apache/kafka/pull/6295] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (KAFKA-7500) MirrorMaker 2.0 (KIP-382)
[ https://issues.apache.org/jira/browse/KAFKA-7500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryanne Dolan updated KAFKA-7500: Priority: Major (was: Minor) > MirrorMaker 2.0 (KIP-382) > - > > Key: KAFKA-7500 > URL: https://issues.apache.org/jira/browse/KAFKA-7500 > Project: Kafka > Issue Type: New Feature > Components: KafkaConnect, mirrormaker >Reporter: Ryanne Dolan >Priority: Major > Fix For: 2.4.0 > > Attachments: Active-Active XDCR setup.png > > > Implement a drop-in replacement for MirrorMaker leveraging the Connect > framework. > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0] > [https://github.com/apache/kafka/pull/6295] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (KAFKA-7500) MirrorMaker 2.0 (KIP-382)
[ https://issues.apache.org/jira/browse/KAFKA-7500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryanne Dolan updated KAFKA-7500: Affects Version/s: 2.4.0 > MirrorMaker 2.0 (KIP-382) > - > > Key: KAFKA-7500 > URL: https://issues.apache.org/jira/browse/KAFKA-7500 > Project: Kafka > Issue Type: New Feature > Components: KafkaConnect, mirrormaker >Affects Versions: 2.4.0 >Reporter: Ryanne Dolan >Priority: Major > Fix For: 2.4.0 > > Attachments: Active-Active XDCR setup.png > > > Implement a drop-in replacement for MirrorMaker leveraging the Connect > framework. > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0] > [https://github.com/apache/kafka/pull/6295] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Reopened] (KAFKA-7940) Flaky Test CustomQuotaCallbackTest#testCustomQuotaCallback
[
https://issues.apache.org/jira/browse/KAFKA-7940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax reopened KAFKA-7940:
Failed again on `trunk`
[https://builds.apache.org/blue/organizations/jenkins/kafka-trunk-jdk11/detail/kafka-trunk-jdk11/790/tests]
with "org.scalatest.exceptions.TestFailedException: Partition
[group1_largeTopic,69] metadata not propagated after 15000 ms"
> Flaky Test CustomQuotaCallbackTest#testCustomQuotaCallback
> --
>
> Key: KAFKA-7940
> URL: https://issues.apache.org/jira/browse/KAFKA-7940
> Project: Kafka
> Issue Type: Bug
> Components: core, unit tests
>Affects Versions: 2.2.0
>Reporter: Matthias J. Sax
>Assignee: Stanislav Kozlovski
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.2.0
>
>
> To get stable nightly builds for `2.2` release, I create tickets for all
> observed test failures.
> [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/14/]
> {quote}java.lang.AssertionError: Too many quotaLimit calls Map(PRODUCE -> 1,
> FETCH -> 1, REQUEST -> 4) at org.junit.Assert.fail(Assert.java:88) at
> org.junit.Assert.assertTrue(Assert.java:41) at
> kafka.api.CustomQuotaCallbackTest.testCustomQuotaCallback(CustomQuotaCallbackTest.scala:105){quote}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (KAFKA-7940) Flaky Test CustomQuotaCallbackTest#testCustomQuotaCallback
[
https://issues.apache.org/jira/browse/KAFKA-7940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-7940:
---
Affects Version/s: 2.4.0
> Flaky Test CustomQuotaCallbackTest#testCustomQuotaCallback
> --
>
> Key: KAFKA-7940
> URL: https://issues.apache.org/jira/browse/KAFKA-7940
> Project: Kafka
> Issue Type: Bug
> Components: core, unit tests
>Affects Versions: 2.2.0, 2.4.0
>Reporter: Matthias J. Sax
>Assignee: Stanislav Kozlovski
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.2.0
>
>
> To get stable nightly builds for `2.2` release, I create tickets for all
> observed test failures.
> [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/14/]
> {quote}java.lang.AssertionError: Too many quotaLimit calls Map(PRODUCE -> 1,
> FETCH -> 1, REQUEST -> 4) at org.junit.Assert.fail(Assert.java:88) at
> org.junit.Assert.assertTrue(Assert.java:41) at
> kafka.api.CustomQuotaCallbackTest.testCustomQuotaCallback(CustomQuotaCallbackTest.scala:105){quote}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (KAFKA-8555) Flaky test ExampleConnectIntegrationTest#testSourceConnector
[ https://issues.apache.org/jira/browse/KAFKA-8555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16922916#comment-16922916 ] Matthias J. Sax commented on KAFKA-8555: [https://builds.apache.org/blue/organizations/jenkins/kafka-trunk-jdk11/detail/kafka-trunk-jdk11/790/tests] > Flaky test ExampleConnectIntegrationTest#testSourceConnector > > > Key: KAFKA-8555 > URL: https://issues.apache.org/jira/browse/KAFKA-8555 > Project: Kafka > Issue Type: Bug >Reporter: Boyang Chen >Priority: Major > Attachments: log-job139.txt, log-job141.txt, log-job23145.txt, > log-job23215.txt, log-job6046.txt > > > [https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/22798/console] > *02:03:21* > org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector > failed, log available in > /home/jenkins/jenkins-slave/workspace/kafka-pr-jdk8-scala2.11/connect/runtime/build/reports/testOutput/org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector.test.stdout*02:03:21* > *02:03:21* > org.apache.kafka.connect.integration.ExampleConnectIntegrationTest > > testSourceConnector FAILED*02:03:21* > org.apache.kafka.connect.errors.DataException: Insufficient records committed > by connector simple-conn in 15000 millis. Records expected=2000, > actual=1013*02:03:21* at > org.apache.kafka.connect.integration.ConnectorHandle.awaitCommits(ConnectorHandle.java:188)*02:03:21* > at > org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector(ExampleConnectIntegrationTest.java:181)*02:03:21* -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (KAFKA-8264) Flaky Test PlaintextConsumerTest#testLowMaxFetchSizeForRequestAndPartition
[
https://issues.apache.org/jira/browse/KAFKA-8264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16922912#comment-16922912
]
Matthias J. Sax commented on KAFKA-8264:
[https://builds.apache.org/blue/organizations/jenkins/kafka-trunk-jdk11/detail/kafka-trunk-jdk11/792/tests]
> Flaky Test PlaintextConsumerTest#testLowMaxFetchSizeForRequestAndPartition
> --
>
> Key: KAFKA-8264
> URL: https://issues.apache.org/jira/browse/KAFKA-8264
> Project: Kafka
> Issue Type: Bug
> Components: core, unit tests
>Affects Versions: 2.0.1, 2.3.0
>Reporter: Matthias J. Sax
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.4.0
>
>
> [https://builds.apache.org/blue/organizations/jenkins/kafka-2.0-jdk8/detail/kafka-2.0-jdk8/252/tests]
> {quote}org.apache.kafka.common.errors.TopicExistsException: Topic 'topic3'
> already exists.{quote}
> STDOUT
>
> {quote}[2019-04-19 03:54:20,080] ERROR [ReplicaFetcher replicaId=1,
> leaderId=0, fetcherId=0] Error for partition __consumer_offsets-0 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:20,080] ERROR [ReplicaFetcher replicaId=2, leaderId=0,
> fetcherId=0] Error for partition __consumer_offsets-0 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:20,312] ERROR [ReplicaFetcher replicaId=2, leaderId=0,
> fetcherId=0] Error for partition topic-0 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:20,313] ERROR [ReplicaFetcher replicaId=2, leaderId=1,
> fetcherId=0] Error for partition topic-1 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:20,994] ERROR [ReplicaFetcher replicaId=1, leaderId=0,
> fetcherId=0] Error for partition topic-0 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:21,727] ERROR [ReplicaFetcher replicaId=0, leaderId=2,
> fetcherId=0] Error for partition topicWithNewMessageFormat-1 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:28,696] ERROR [ReplicaFetcher replicaId=0, leaderId=2,
> fetcherId=0] Error for partition __consumer_offsets-0 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:28,699] ERROR [ReplicaFetcher replicaId=1, leaderId=2,
> fetcherId=0] Error for partition __consumer_offsets-0 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:29,246] ERROR [ReplicaFetcher replicaId=0, leaderId=2,
> fetcherId=0] Error for partition topic-1 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:29,247] ERROR [ReplicaFetcher replicaId=0, leaderId=1,
> fetcherId=0] Error for partition topic-0 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:29,287] ERROR [ReplicaFetcher replicaId=1, leaderId=2,
> fetcherId=0] Error for partition topic-1 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:33,408] ERROR [ReplicaFetcher replicaId=0, leaderId=2,
> fetcherId=0] Error for partition __consumer_offsets-0 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:33,408] ERROR [ReplicaFetcher replicaId=1, leaderId=2,
> fetcherId=0] Error for partition __consumer_offsets-0 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
>
[jira] [Commented] (KAFKA-8264) Flaky Test PlaintextConsumerTest#testLowMaxFetchSizeForRequestAndPartition
[
https://issues.apache.org/jira/browse/KAFKA-8264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16922910#comment-16922910
]
Matthias J. Sax commented on KAFKA-8264:
[https://builds.apache.org/blue/organizations/jenkins/kafka-trunk-jdk8/detail/kafka-trunk-jdk8/3885/tests]
> Flaky Test PlaintextConsumerTest#testLowMaxFetchSizeForRequestAndPartition
> --
>
> Key: KAFKA-8264
> URL: https://issues.apache.org/jira/browse/KAFKA-8264
> Project: Kafka
> Issue Type: Bug
> Components: core, unit tests
>Affects Versions: 2.0.1, 2.3.0
>Reporter: Matthias J. Sax
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.4.0
>
>
> [https://builds.apache.org/blue/organizations/jenkins/kafka-2.0-jdk8/detail/kafka-2.0-jdk8/252/tests]
> {quote}org.apache.kafka.common.errors.TopicExistsException: Topic 'topic3'
> already exists.{quote}
> STDOUT
>
> {quote}[2019-04-19 03:54:20,080] ERROR [ReplicaFetcher replicaId=1,
> leaderId=0, fetcherId=0] Error for partition __consumer_offsets-0 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:20,080] ERROR [ReplicaFetcher replicaId=2, leaderId=0,
> fetcherId=0] Error for partition __consumer_offsets-0 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:20,312] ERROR [ReplicaFetcher replicaId=2, leaderId=0,
> fetcherId=0] Error for partition topic-0 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:20,313] ERROR [ReplicaFetcher replicaId=2, leaderId=1,
> fetcherId=0] Error for partition topic-1 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:20,994] ERROR [ReplicaFetcher replicaId=1, leaderId=0,
> fetcherId=0] Error for partition topic-0 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:21,727] ERROR [ReplicaFetcher replicaId=0, leaderId=2,
> fetcherId=0] Error for partition topicWithNewMessageFormat-1 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:28,696] ERROR [ReplicaFetcher replicaId=0, leaderId=2,
> fetcherId=0] Error for partition __consumer_offsets-0 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:28,699] ERROR [ReplicaFetcher replicaId=1, leaderId=2,
> fetcherId=0] Error for partition __consumer_offsets-0 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:29,246] ERROR [ReplicaFetcher replicaId=0, leaderId=2,
> fetcherId=0] Error for partition topic-1 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:29,247] ERROR [ReplicaFetcher replicaId=0, leaderId=1,
> fetcherId=0] Error for partition topic-0 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:29,287] ERROR [ReplicaFetcher replicaId=1, leaderId=2,
> fetcherId=0] Error for partition topic-1 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:33,408] ERROR [ReplicaFetcher replicaId=0, leaderId=2,
> fetcherId=0] Error for partition __consumer_offsets-0 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:33,408] ERROR [ReplicaFetcher replicaId=1, leaderId=2,
> fetcherId=0] Error for partition __consumer_offsets-0 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
>
[jira] [Commented] (KAFKA-8264) Flaky Test PlaintextConsumerTest#testLowMaxFetchSizeForRequestAndPartition
[
https://issues.apache.org/jira/browse/KAFKA-8264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16922900#comment-16922900
]
Matthias J. Sax commented on KAFKA-8264:
[https://builds.apache.org/blue/organizations/jenkins/kafka-trunk-jdk11/detail/kafka-trunk-jdk11/788/tests]
> Flaky Test PlaintextConsumerTest#testLowMaxFetchSizeForRequestAndPartition
> --
>
> Key: KAFKA-8264
> URL: https://issues.apache.org/jira/browse/KAFKA-8264
> Project: Kafka
> Issue Type: Bug
> Components: core, unit tests
>Affects Versions: 2.0.1, 2.3.0
>Reporter: Matthias J. Sax
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.4.0
>
>
> [https://builds.apache.org/blue/organizations/jenkins/kafka-2.0-jdk8/detail/kafka-2.0-jdk8/252/tests]
> {quote}org.apache.kafka.common.errors.TopicExistsException: Topic 'topic3'
> already exists.{quote}
> STDOUT
>
> {quote}[2019-04-19 03:54:20,080] ERROR [ReplicaFetcher replicaId=1,
> leaderId=0, fetcherId=0] Error for partition __consumer_offsets-0 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:20,080] ERROR [ReplicaFetcher replicaId=2, leaderId=0,
> fetcherId=0] Error for partition __consumer_offsets-0 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:20,312] ERROR [ReplicaFetcher replicaId=2, leaderId=0,
> fetcherId=0] Error for partition topic-0 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:20,313] ERROR [ReplicaFetcher replicaId=2, leaderId=1,
> fetcherId=0] Error for partition topic-1 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:20,994] ERROR [ReplicaFetcher replicaId=1, leaderId=0,
> fetcherId=0] Error for partition topic-0 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:21,727] ERROR [ReplicaFetcher replicaId=0, leaderId=2,
> fetcherId=0] Error for partition topicWithNewMessageFormat-1 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:28,696] ERROR [ReplicaFetcher replicaId=0, leaderId=2,
> fetcherId=0] Error for partition __consumer_offsets-0 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:28,699] ERROR [ReplicaFetcher replicaId=1, leaderId=2,
> fetcherId=0] Error for partition __consumer_offsets-0 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:29,246] ERROR [ReplicaFetcher replicaId=0, leaderId=2,
> fetcherId=0] Error for partition topic-1 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:29,247] ERROR [ReplicaFetcher replicaId=0, leaderId=1,
> fetcherId=0] Error for partition topic-0 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:29,287] ERROR [ReplicaFetcher replicaId=1, leaderId=2,
> fetcherId=0] Error for partition topic-1 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:33,408] ERROR [ReplicaFetcher replicaId=0, leaderId=2,
> fetcherId=0] Error for partition __consumer_offsets-0 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:33,408] ERROR [ReplicaFetcher replicaId=1, leaderId=2,
> fetcherId=0] Error for partition __consumer_offsets-0 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
>
[jira] [Commented] (KAFKA-7937) Flaky Test ResetConsumerGroupOffsetTest.testResetOffsetsNotExistingGroup
[
https://issues.apache.org/jira/browse/KAFKA-7937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16922906#comment-16922906
]
Matthias J. Sax commented on KAFKA-7937:
[~ijuma] Failed again in 2.3
([https://builds.apache.org/blue/organizations/jenkins/kafka-2.3-jdk8/detail/kafka-2.3-jdk8/99/pipeline/])
Can we cherry-pick the fix?
> Flaky Test ResetConsumerGroupOffsetTest.testResetOffsetsNotExistingGroup
>
>
> Key: KAFKA-7937
> URL: https://issues.apache.org/jira/browse/KAFKA-7937
> Project: Kafka
> Issue Type: Bug
> Components: admin, clients, unit tests
>Affects Versions: 2.2.0, 2.1.1, 2.3.0
>Reporter: Matthias J. Sax
>Assignee: Bob Barrett
>Priority: Critical
> Fix For: 2.4.0
>
> Attachments: log-job6122.txt
>
>
> To get stable nightly builds for `2.2` release, I create tickets for all
> observed test failures.
> https://builds.apache.org/blue/organizations/jenkins/kafka-2.2-jdk8/detail/kafka-2.2-jdk8/19/pipeline
> {quote}kafka.admin.ResetConsumerGroupOffsetTest >
> testResetOffsetsNotExistingGroup FAILED
> java.util.concurrent.ExecutionException:
> org.apache.kafka.common.errors.CoordinatorNotAvailableException: The
> coordinator is not available. at
> org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
> at
> kafka.admin.ConsumerGroupCommand$ConsumerGroupService.resetOffsets(ConsumerGroupCommand.scala:306)
> at
> kafka.admin.ResetConsumerGroupOffsetTest.testResetOffsetsNotExistingGroup(ResetConsumerGroupOffsetTest.scala:89)
> Caused by: org.apache.kafka.common.errors.CoordinatorNotAvailableException:
> The coordinator is not available.{quote}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (KAFKA-6461) TableTableJoinIntegrationTest is unstable if caching is enabled
[
https://issues.apache.org/jira/browse/KAFKA-6461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16922907#comment-16922907
]
Matthias J. Sax commented on KAFKA-6461:
[https://builds.apache.org/blue/organizations/jenkins/kafka-trunk-jdk11/detail/kafka-trunk-jdk11/790/tests]
> TableTableJoinIntegrationTest is unstable if caching is enabled
> ---
>
> Key: KAFKA-6461
> URL: https://issues.apache.org/jira/browse/KAFKA-6461
> Project: Kafka
> Issue Type: Improvement
> Components: streams, unit tests
>Reporter: Matthias J. Sax
>Priority: Minor
> Labels: flaky-test
> Fix For: 1.1.0
>
>
> {noformat}
> org.apache.kafka.streams.integration.TableTableJoinIntegrationTest >
> testLeftInner[caching enabled = true] FAILED
> 20:41:05 java.lang.AssertionError: Condition not met within timeout
> 15000. Never received expected final result.
> 20:41:05 at
> org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:276)
> 20:41:05 at
> org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:254)
> 20:41:05 at
> org.apache.kafka.streams.integration.AbstractJoinIntegrationTest.runTest(AbstractJoinIntegrationTest.java:248)
> 20:41:05 at
> org.apache.kafka.streams.integration.TableTableJoinIntegrationTest.testLeftInner(TableTableJoinIntegrationTest.java:313){noformat}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (KAFKA-6461) TableTableJoinIntegrationTest is unstable if caching is enabled
[
https://issues.apache.org/jira/browse/KAFKA-6461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16922908#comment-16922908
]
Matthias J. Sax commented on KAFKA-6461:
[https://builds.apache.org/blue/organizations/jenkins/kafka-trunk-jdk11/detail/kafka-trunk-jdk11/791/tests]
> TableTableJoinIntegrationTest is unstable if caching is enabled
> ---
>
> Key: KAFKA-6461
> URL: https://issues.apache.org/jira/browse/KAFKA-6461
> Project: Kafka
> Issue Type: Improvement
> Components: streams, unit tests
>Reporter: Matthias J. Sax
>Priority: Minor
> Labels: flaky-test
> Fix For: 1.1.0
>
>
> {noformat}
> org.apache.kafka.streams.integration.TableTableJoinIntegrationTest >
> testLeftInner[caching enabled = true] FAILED
> 20:41:05 java.lang.AssertionError: Condition not met within timeout
> 15000. Never received expected final result.
> 20:41:05 at
> org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:276)
> 20:41:05 at
> org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:254)
> 20:41:05 at
> org.apache.kafka.streams.integration.AbstractJoinIntegrationTest.runTest(AbstractJoinIntegrationTest.java:248)
> 20:41:05 at
> org.apache.kafka.streams.integration.TableTableJoinIntegrationTest.testLeftInner(TableTableJoinIntegrationTest.java:313){noformat}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (KAFKA-8264) Flaky Test PlaintextConsumerTest#testLowMaxFetchSizeForRequestAndPartition
[
https://issues.apache.org/jira/browse/KAFKA-8264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16922899#comment-16922899
]
Matthias J. Sax commented on KAFKA-8264:
[https://builds.apache.org/blue/organizations/jenkins/kafka-trunk-jdk11/detail/kafka-trunk-jdk11/787/tests]
> Flaky Test PlaintextConsumerTest#testLowMaxFetchSizeForRequestAndPartition
> --
>
> Key: KAFKA-8264
> URL: https://issues.apache.org/jira/browse/KAFKA-8264
> Project: Kafka
> Issue Type: Bug
> Components: core, unit tests
>Affects Versions: 2.0.1, 2.3.0
>Reporter: Matthias J. Sax
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.4.0
>
>
> [https://builds.apache.org/blue/organizations/jenkins/kafka-2.0-jdk8/detail/kafka-2.0-jdk8/252/tests]
> {quote}org.apache.kafka.common.errors.TopicExistsException: Topic 'topic3'
> already exists.{quote}
> STDOUT
>
> {quote}[2019-04-19 03:54:20,080] ERROR [ReplicaFetcher replicaId=1,
> leaderId=0, fetcherId=0] Error for partition __consumer_offsets-0 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:20,080] ERROR [ReplicaFetcher replicaId=2, leaderId=0,
> fetcherId=0] Error for partition __consumer_offsets-0 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:20,312] ERROR [ReplicaFetcher replicaId=2, leaderId=0,
> fetcherId=0] Error for partition topic-0 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:20,313] ERROR [ReplicaFetcher replicaId=2, leaderId=1,
> fetcherId=0] Error for partition topic-1 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:20,994] ERROR [ReplicaFetcher replicaId=1, leaderId=0,
> fetcherId=0] Error for partition topic-0 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:21,727] ERROR [ReplicaFetcher replicaId=0, leaderId=2,
> fetcherId=0] Error for partition topicWithNewMessageFormat-1 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:28,696] ERROR [ReplicaFetcher replicaId=0, leaderId=2,
> fetcherId=0] Error for partition __consumer_offsets-0 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:28,699] ERROR [ReplicaFetcher replicaId=1, leaderId=2,
> fetcherId=0] Error for partition __consumer_offsets-0 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:29,246] ERROR [ReplicaFetcher replicaId=0, leaderId=2,
> fetcherId=0] Error for partition topic-1 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:29,247] ERROR [ReplicaFetcher replicaId=0, leaderId=1,
> fetcherId=0] Error for partition topic-0 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:29,287] ERROR [ReplicaFetcher replicaId=1, leaderId=2,
> fetcherId=0] Error for partition topic-1 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:33,408] ERROR [ReplicaFetcher replicaId=0, leaderId=2,
> fetcherId=0] Error for partition __consumer_offsets-0 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:33,408] ERROR [ReplicaFetcher replicaId=1, leaderId=2,
> fetcherId=0] Error for partition __consumer_offsets-0 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
>
[jira] [Resolved] (KAFKA-8460) Flaky Test PlaintextConsumerTest#testLowMaxFetchSizeForRequestAndPartition
[ https://issues.apache.org/jira/browse/KAFKA-8460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-8460. Resolution: Duplicate > Flaky Test PlaintextConsumerTest#testLowMaxFetchSizeForRequestAndPartition > --- > > Key: KAFKA-8460 > URL: https://issues.apache.org/jira/browse/KAFKA-8460 > Project: Kafka > Issue Type: Bug >Reporter: Boyang Chen >Priority: Major > > [https://builds.apache.org/job/kafka-pr-jdk11-scala2.12/5168/consoleFull] > *16:17:04* kafka.api.PlaintextConsumerTest > > testLowMaxFetchSizeForRequestAndPartition FAILED*16:17:04* > org.scalatest.exceptions.TestFailedException: Timed out before consuming > expected 2700 records. The number consumed was 1980.*16:17:04* at > org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:530)*16:17:04* > at > org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:529)*16:17:04* > at > org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1389)*16:17:04* > at org.scalatest.Assertions.fail(Assertions.scala:1091)*16:17:04* at > org.scalatest.Assertions.fail$(Assertions.scala:1087)*16:17:04* at > org.scalatest.Assertions$.fail(Assertions.scala:1389)*16:17:04* at > kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:789)*16:17:04* at > kafka.utils.TestUtils$.pollRecordsUntilTrue(TestUtils.scala:765)*16:17:04* at > kafka.api.AbstractConsumerTest.consumeRecords(AbstractConsumerTest.scala:156)*16:17:04* > at > kafka.api.PlaintextConsumerTest.testLowMaxFetchSizeForRequestAndPartition(PlaintextConsumerTest.scala:801)*16:17:04* -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Reopened] (KAFKA-8677) Flakey test GroupEndToEndAuthorizationTest#testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl
[ https://issues.apache.org/jira/browse/KAFKA-8677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax reopened KAFKA-8677: This test is still failing: [https://builds.apache.org/blue/organizations/jenkins/kafka-trunk-jdk11/detail/kafka-trunk-jdk11/788/tests] > Flakey test > GroupEndToEndAuthorizationTest#testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl > > > Key: KAFKA-8677 > URL: https://issues.apache.org/jira/browse/KAFKA-8677 > Project: Kafka > Issue Type: Bug > Components: core, security, unit tests >Affects Versions: 2.4.0 >Reporter: Boyang Chen >Assignee: Anastasia Vela >Priority: Major > Labels: flaky-test > > [https://builds.apache.org/job/kafka-pr-jdk11-scala2.12/6325/console] > > *18:43:39* kafka.api.GroupEndToEndAuthorizationTest > > testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl STARTED*18:44:00* > kafka.api.GroupEndToEndAuthorizationTest.testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl > failed, log available in > /home/jenkins/jenkins-slave/workspace/kafka-pr-jdk11-scala2.12/core/build/reports/testOutput/kafka.api.GroupEndToEndAuthorizationTest.testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl.test.stdout*18:44:00* > *18:44:00* kafka.api.GroupEndToEndAuthorizationTest > > testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl FAILED*18:44:00* > org.scalatest.exceptions.TestFailedException: Consumed 0 records before > timeout instead of the expected 1 records -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (KAFKA-8264) Flaky Test PlaintextConsumerTest#testLowMaxFetchSizeForRequestAndPartition
[
https://issues.apache.org/jira/browse/KAFKA-8264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16922902#comment-16922902
]
Matthias J. Sax commented on KAFKA-8264:
[https://builds.apache.org/blue/organizations/jenkins/kafka-trunk-jdk8/detail/kafka-trunk-jdk8/3883/tests]
> Flaky Test PlaintextConsumerTest#testLowMaxFetchSizeForRequestAndPartition
> --
>
> Key: KAFKA-8264
> URL: https://issues.apache.org/jira/browse/KAFKA-8264
> Project: Kafka
> Issue Type: Bug
> Components: core, unit tests
>Affects Versions: 2.0.1, 2.3.0
>Reporter: Matthias J. Sax
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.4.0
>
>
> [https://builds.apache.org/blue/organizations/jenkins/kafka-2.0-jdk8/detail/kafka-2.0-jdk8/252/tests]
> {quote}org.apache.kafka.common.errors.TopicExistsException: Topic 'topic3'
> already exists.{quote}
> STDOUT
>
> {quote}[2019-04-19 03:54:20,080] ERROR [ReplicaFetcher replicaId=1,
> leaderId=0, fetcherId=0] Error for partition __consumer_offsets-0 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:20,080] ERROR [ReplicaFetcher replicaId=2, leaderId=0,
> fetcherId=0] Error for partition __consumer_offsets-0 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:20,312] ERROR [ReplicaFetcher replicaId=2, leaderId=0,
> fetcherId=0] Error for partition topic-0 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:20,313] ERROR [ReplicaFetcher replicaId=2, leaderId=1,
> fetcherId=0] Error for partition topic-1 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:20,994] ERROR [ReplicaFetcher replicaId=1, leaderId=0,
> fetcherId=0] Error for partition topic-0 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:21,727] ERROR [ReplicaFetcher replicaId=0, leaderId=2,
> fetcherId=0] Error for partition topicWithNewMessageFormat-1 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:28,696] ERROR [ReplicaFetcher replicaId=0, leaderId=2,
> fetcherId=0] Error for partition __consumer_offsets-0 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:28,699] ERROR [ReplicaFetcher replicaId=1, leaderId=2,
> fetcherId=0] Error for partition __consumer_offsets-0 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:29,246] ERROR [ReplicaFetcher replicaId=0, leaderId=2,
> fetcherId=0] Error for partition topic-1 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:29,247] ERROR [ReplicaFetcher replicaId=0, leaderId=1,
> fetcherId=0] Error for partition topic-0 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:29,287] ERROR [ReplicaFetcher replicaId=1, leaderId=2,
> fetcherId=0] Error for partition topic-1 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:33,408] ERROR [ReplicaFetcher replicaId=0, leaderId=2,
> fetcherId=0] Error for partition __consumer_offsets-0 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:33,408] ERROR [ReplicaFetcher replicaId=1, leaderId=2,
> fetcherId=0] Error for partition __consumer_offsets-0 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
>
[jira] [Commented] (KAFKA-8729) Augment ProduceResponse error messaging for specific culprit records
[ https://issues.apache.org/jira/browse/KAFKA-8729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16922793#comment-16922793 ] ASF GitHub Bot commented on KAFKA-8729: --- guozhangwang commented on pull request #7142: KAFKA-8729, pt 1: Add 4 new metrics to keep track of various types of invalid record rejections URL: https://github.com/apache/kafka/pull/7142 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Augment ProduceResponse error messaging for specific culprit records > > > Key: KAFKA-8729 > URL: https://issues.apache.org/jira/browse/KAFKA-8729 > Project: Kafka > Issue Type: Improvement > Components: clients, core >Reporter: Guozhang Wang >Assignee: Tu Tran >Priority: Major > > 1. We should replace the misleading CORRUPT_RECORD error code with a new > INVALID_RECORD. > 2. We should augment the ProduceResponse with customizable error message and > indicators of culprit records. > 3. We should change the client-side handling logic of non-retriable > INVALID_RECORD to re-batch the records. > Details see: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-467%3A+Augment+ProduceResponse+error+messaging+for+specific+culprit+records -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (KAFKA-8872) Improvements to controller "deleting" state / topic Identifiers
Lucas Bradstreet created KAFKA-8872: --- Summary: Improvements to controller "deleting" state / topic Identifiers Key: KAFKA-8872 URL: https://issues.apache.org/jira/browse/KAFKA-8872 Project: Kafka Issue Type: Improvement Reporter: Lucas Bradstreet Kafka currently uniquely identifies a topic by its name. This is generally sufficient, but there are flaws in this scheme if a topic is deleted and recreated with the same name. As a result, Kafka attempts to prevent these classes of issues by ensuring a topic is deleted from all replicas before completing a deletion. This solution is not perfect, as it is possible for partitions to be reassigned from brokers while they are down, and there are no guarantees that this state will ever be cleaned up and will not cause issues in the future. As the controller must wait for all replicas to delete their local partitions, deletes can also become blocked, preventing topics from being created with the same name until the deletion is complete on all replicas. This can mean that downtime for a single broker can effectively cause a complete outage for everyone producing/consuming to that topic name, as the topic cannot be recreated without manual intervention. Unique topic IDs could help address this issue by associating a unique ID with each topic, ensuring a newly created topic with a previously used name cannot be confused with a previous topic with that name. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (KAFKA-8871) Allow timestamp manipulation in ValueTransformerWithKey
[ https://issues.apache.org/jira/browse/KAFKA-8871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Levani Kokhreidze updated KAFKA-8871: - Issue Type: Improvement (was: New Feature) > Allow timestamp manipulation in ValueTransformerWithKey > --- > > Key: KAFKA-8871 > URL: https://issues.apache.org/jira/browse/KAFKA-8871 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Levani Kokhreidze >Priority: Minor > > h3. Motivation > When using `KStream#transform` in Kafka Streams DSL to manipulate the > timestamp, `KStreamImpl#transform` implementation marks *repartitionRequired* > as *true,* which isn't necessarily okay when one may just want to manipulate > with timestamp without affecting the key. It would be great if DSL user could > manipulate the timestamp in `ValueTransformerWithKey`. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Comment Edited] (KAFKA-8870) Prevent dirty reads of Streams state store from Interactive queries
[ https://issues.apache.org/jira/browse/KAFKA-8870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16922705#comment-16922705 ] Vinoth Chandar edited comment on KAFKA-8870 at 9/4/19 5:40 PM: --- Initial thoughts from [~mjsax] [~guozhang] & me below Approach 1 : We create a “staging in-memory buffer” for each state store – each write to the staging buffer will also be a write into the store changelog topic. After a transaction is committed, we replay/merge all buffered writes into the state store to expose them to IQ. During processing, the Processors would also read from the “staging buffer” to ensure read-your-own-writes semantics. Additionally, we re-introduce the local .checkpoint file. On commit, we first commit the Kafka transaction, second, merged the staging buffer into the local store, and third update the checkpoint file with the changelog end-offsets. This algorithm ensures that the checkpoint file only indicates committed data. On failure, we only need to re-read the changelog topic starting at the checkpoint file offsets to then log-end-offset in read-committed mode. If we crash and lose the staging-buffer updates, we will abort the transaction anyway. If we fail after the commit but before we merged the staging buffer into the local state store, we know that all staging-buffer updates are written into the changelog topic already and hence, we use the changelog topic for recovery. Approach 2 : Implement the buffer using rocksdb transactions [https://github.com/facebook/rocksdb/wiki/Transactions] , where we only commit rocksdb transaction when kafka commits. If we fail before comitting to rocksdb, we restore from Kafka again anyway since that is the source of truth. Approach 1 generalizes to different state stores better. Would like to pick up this work unless someone objects was (Author: vc): Initial thoughts from [~mjsax] [~guozhang] & me below Approach 1 : We create a “staging in-memory buffer” for each state store – each write to the staging buffer will also be a write into the store changelog topic. After a transaction is committed, we replay/merge all buffered writes into the state store to expose them to IQ. During processing, the Processors would also read from the “staging buffer” to ensure read-your-own-writes semantics. Additionally, we re-introduce the local .checkpoint file. On commit, we first commit the Kafka transaction, second, merged the staging buffer into the local store, and third update the checkpoint file with the changelog end-offsets. This algorithm ensures that the checkpoint file only indicates committed data. On failure, we only need to re-read the changelog topic starting at the checkpoint file offsets to then log-end-offset in read-committed mode. If we crash and lose the staging-buffer updates, we will abort the transaction anyway. If we fail after the commit but before we merged the staging buffer into the local state store, we know that all staging-buffer updates are written into the changelog topic already and hence, we use the changelog topic for recovery. Approach 2 : Implement the buffer using rocksdb transactions [https://github.com/facebook/rocksdb/wiki/Transactions] , where we only commit rocksdb transaction when kafka commits. If we fail before comitting to rocksdb, we restore from Kafka again anyway since that is the source of truth. Approach 1 generalizes to different state stores better. > Prevent dirty reads of Streams state store from Interactive queries > --- > > Key: KAFKA-8870 > URL: https://issues.apache.org/jira/browse/KAFKA-8870 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Vinoth Chandar >Priority: Major > > Today, Interactive Queries (IQ) against Streams state store could see > uncommitted data, even with EOS processing guarantees (these are actually > orthogonal, but clarifying since EOS may give the impression that everything > is dandy). This is causes primarily because state updates in rocksdb are > visible even before the kafka transaction is committed. Thus, if the instance > fails, then the failed over instance will redo the uncommited old transaction > and the following could be possible during recovery,. > Value for key K can go from *V0 → V1 → V2* on active instance A, IQ reads V1, > instance A fails and any failure/rebalancing will leave the standy instance B > rewinding offsets and reprocessing, during which time IQ can again see V0 or > V1 or any number of previous values for the same key. > In this issue, we will plan work towards providing consistency for IQ, for a > single row in a single state store. i.e once a query sees V1, it can only see > either V1 or V2. > > >
[jira] [Created] (KAFKA-8869) Map taskConfigs in KafkaConfigBackingStore grows monotonically despite of task removals
Konstantine Karantasis created KAFKA-8869:
-
Summary: Map taskConfigs in KafkaConfigBackingStore grows
monotonically despite of task removals
Key: KAFKA-8869
URL: https://issues.apache.org/jira/browse/KAFKA-8869
Project: Kafka
Issue Type: Bug
Components: KafkaConnect
Affects Versions: 2.3.0
Reporter: Konstantine Karantasis
Assignee: Konstantine Karantasis
Fix For: 2.3.1
Investigation of https://issues.apache.org/jira/browse/KAFKA-8676 revealed
another issue:
a map in {{KafkaConfigBackingStore}} keeps growing despite of connectors and
tasks getting removed eventually.
This bug does not affect directly rebalancing protocols but it'd good to
resolve and use in a way similar to how {{connectorConfigs}} is used.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (KAFKA-8867) Kafka Connect JDBC fails to create PostgreSQL table with default boolean value in schema
[
https://issues.apache.org/jira/browse/KAFKA-8867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tudor updated KAFKA-8867:
-
Summary: Kafka Connect JDBC fails to create PostgreSQL table with default
boolean value in schema (was: Kafka Connect JDBC fails to create PostgreSQL
table with default boolean value)
> Kafka Connect JDBC fails to create PostgreSQL table with default boolean
> value in schema
>
>
> Key: KAFKA-8867
> URL: https://issues.apache.org/jira/browse/KAFKA-8867
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 2.3.0
>Reporter: Tudor
>Priority: Major
>
> The `CREATE TABLE ..` statement generated for JDBC sink connectors when
> configured with `auto.create: true` generates field declarations that do not
> conform to allowed PostgreSQL syntax when considering fields of type boolean
> with default values.
> Example record value Avro schema:
> {code:java}
> {
> "namespace": "com.test.avro.schema.v1",
> "type": "record",
> "name": "SomeEvent",
> "fields": [
> {
> "name": "boolean_field",
> "type": "boolean",
> "default": false
> }
> ]
> }
> {code}
> The connector task fails with:
> {code:java}
> ERROR WorkerSinkTask{id=test-events-sink-0} RetriableException from SinkTask:
> (org.apache.kafka.connect.runtime.WorkerSinkTask:551)
> org.apache.kafka.connect.errors.RetriableException: java.sql.SQLException:
> org.postgresql.util.PSQLException: ERROR: column "boolean_field" is of type
> boolean but default expression is of type integer
> Hint: You will need to rewrite or cast the expression.
> at io.confluent.connect.jdbc.sink.JdbcSinkTask.put(JdbcSinkTask.java:93)
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:538)
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:321)
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:224)
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:192)
> at
> org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:175)
> at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:219)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748){code}
>
> The generated SQL statement is:
> {code:java}
> CREATE TABLE "test_data" ("boolean_field" BOOLEAN DEFAULT 0){code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Created] (KAFKA-8868) Consider auto-generating Streams binary protocol messages
John Roesler created KAFKA-8868: --- Summary: Consider auto-generating Streams binary protocol messages Key: KAFKA-8868 URL: https://issues.apache.org/jira/browse/KAFKA-8868 Project: Kafka Issue Type: Improvement Components: streams Reporter: John Roesler Assignee: John Roesler Rather than maintain hand coded protocol serialization code, Streams could use the same code-generation framework as Clients/Core. There isn't a perfect match, since the code generation framework includes an assumption that you're generating "protocol messages", rather than just arbitrary blobs, but I think it's close enough to justify using it, and improving it over time. Using the code generation allows us to drop a lot of detail-oriented, brittle, and hard-to-maintain serialization logic in favor of a schema spec. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (KAFKA-8861) Fix flaky RegexSourceIntegrationTest.testMultipleConsumersCanReadFromPartitionedTopic
[
https://issues.apache.org/jira/browse/KAFKA-8861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bill Bejeck updated KAFKA-8861:
---
Fix Version/s: 2.3.1
2.4.0
2.2.2
2.1.2
2.0.2
> Fix flaky
> RegexSourceIntegrationTest.testMultipleConsumersCanReadFromPartitionedTopic
> -
>
> Key: KAFKA-8861
> URL: https://issues.apache.org/jira/browse/KAFKA-8861
> Project: Kafka
> Issue Type: Bug
> Components: streams, unit tests
>Affects Versions: 2.0.1, 2.1.1, 2.3.0, 2.2.1, 2.4.0
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Minor
> Labels: flaky-test
> Fix For: 2.0.2, 2.1.2, 2.2.2, 2.4.0, 2.3.1
>
>
> This is similar to KAFKA-8011 and KAFKA-8026. The error stack is shown below.
> {code:java}
> java.util.ConcurrentModificationException
> at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909)
> at java.util.ArrayList$Itr.next(ArrayList.java:859)
> at java.util.AbstractList.equals(AbstractList.java:521)
> at
> org.apache.kafka.streams.integration.RegexSourceIntegrationTest.lambda$testMultipleConsumersCanReadFromPartitionedTopic$5(RegexSourceIntegrationTest.java:351)
> at
> org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:366)
> at
> org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:336)
> at
> org.apache.kafka.streams.integration.RegexSourceIntegrationTest.testMultipleConsumersCanReadFromPartitionedTopic(RegexSourceIntegrationTest.java:351)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:365)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:330)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:78)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:328)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (KAFKA-8861) Fix flaky RegexSourceIntegrationTest.testMultipleConsumersCanReadFromPartitionedTopic
[
https://issues.apache.org/jira/browse/KAFKA-8861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bill Bejeck updated KAFKA-8861:
---
Affects Version/s: 2.4.0
2.0.1
2.1.1
2.3.0
2.2.1
> Fix flaky
> RegexSourceIntegrationTest.testMultipleConsumersCanReadFromPartitionedTopic
> -
>
> Key: KAFKA-8861
> URL: https://issues.apache.org/jira/browse/KAFKA-8861
> Project: Kafka
> Issue Type: Bug
> Components: streams, unit tests
>Affects Versions: 2.0.1, 2.1.1, 2.3.0, 2.2.1, 2.4.0
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Minor
> Labels: flaky-test
>
> This is similar to KAFKA-8011 and KAFKA-8026. The error stack is shown below.
> {code:java}
> java.util.ConcurrentModificationException
> at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909)
> at java.util.ArrayList$Itr.next(ArrayList.java:859)
> at java.util.AbstractList.equals(AbstractList.java:521)
> at
> org.apache.kafka.streams.integration.RegexSourceIntegrationTest.lambda$testMultipleConsumersCanReadFromPartitionedTopic$5(RegexSourceIntegrationTest.java:351)
> at
> org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:366)
> at
> org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:336)
> at
> org.apache.kafka.streams.integration.RegexSourceIntegrationTest.testMultipleConsumersCanReadFromPartitionedTopic(RegexSourceIntegrationTest.java:351)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:365)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:330)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:78)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:328)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (KAFKA-8867) Kafka Connect JDBC fails to create PostgreSQL table with default boolean value
[
https://issues.apache.org/jira/browse/KAFKA-8867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tudor updated KAFKA-8867:
-
Description:
The `CREATE TABLE ..` statement generated for JDBC sink connectors when
configured with `auto.create: true` generates field declarations that do not
conform to allowed PostgreSQL syntax when considering fields of type boolean
with default values.
Example record value Avro schema:
{code:java}
{
"namespace": "com.test.avro.schema.v1",
"type": "record",
"name": "SomeEvent",
"fields": [
{
"name": "boolean_field",
"type": "boolean",
"default": false
}
]
}
{code}
The connector task fails with:
{code:java}
ERROR WorkerSinkTask{id=test-events-sink-0} RetriableException from SinkTask:
(org.apache.kafka.connect.runtime.WorkerSinkTask:551)
org.apache.kafka.connect.errors.RetriableException: java.sql.SQLException:
org.postgresql.util.PSQLException: ERROR: column "boolean_field" is of type
boolean but default expression is of type integer
Hint: You will need to rewrite or cast the expression.
at io.confluent.connect.jdbc.sink.JdbcSinkTask.put(JdbcSinkTask.java:93)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:538)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:321)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:224)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:192)
at
org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:175)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:219)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748){code}
The generated SQL statement is:
{code:java}
CREATE TABLE "test_data" ("boolean_field" BOOLEAN DEFAULT 0){code}
was:
The `CREATE TABLE ..` statement generated for sink connectors when configured
with `auto.create: true` generates field declarations that do not conform to
allowed PostgreSQL syntax when considering fields of type boolean with default
values.
Example record value Avro schema:
{code:java}
{
"namespace": "com.test.avro.schema.v1",
"type": "record",
"name": "SomeEvent",
"fields": [
{
"name": "boolean_field",
"type": "boolean",
"default": false
}
]
}
{code}
The connector task fails with:
{code:java}
ERROR WorkerSinkTask{id=test-events-sink-0} RetriableException from SinkTask:
(org.apache.kafka.connect.runtime.WorkerSinkTask:551)
org.apache.kafka.connect.errors.RetriableException: java.sql.SQLException:
org.postgresql.util.PSQLException: ERROR: column "boolean_field" is of type
boolean but default expression is of type integer
Hint: You will need to rewrite or cast the expression.
at io.confluent.connect.jdbc.sink.JdbcSinkTask.put(JdbcSinkTask.java:93)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:538)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:321)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:224)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:192)
at
org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:175)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:219)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748){code}
The generated SQL statement is:
{code:java}
CREATE TABLE "test_data" ("boolean_field" BOOLEAN DEFAULT 0){code}
> Kafka Connect JDBC fails to create PostgreSQL table with default boolean value
> --
>
> Key: KAFKA-8867
> URL: https://issues.apache.org/jira/browse/KAFKA-8867
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 2.3.0
>Reporter: Tudor
>Priority: Major
>
> The `CREATE TABLE ..` statement generated for JDBC sink connectors when
> configured with `auto.create: true` generates field declarations that do not
> conform to allowed PostgreSQL syntax when
[jira] [Updated] (KAFKA-8867) Kafka Connect JDBC fails to create PostgreSQL table with default boolean value
[
https://issues.apache.org/jira/browse/KAFKA-8867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tudor-Alexandru Voicu updated KAFKA-8867:
-
Description:
The `CREATE TABLE ..` statement generated for sink connectors when configured
with `auto.create: true` generates field declarations that do not conform to
allowed PostgreSQL syntax when considering fields of type boolean with default
values.
Example record value Avro schema:
{code:java}
{
"namespace": "com.test.avro.schema.v1",
"type": "record",
"name": "SomeEvent",
"fields": [
{
"name": "boolean_field",
"type": "boolean",
"default": false
}
]
}
{code}
The connector task fails with:
{code:java}
ERROR WorkerSinkTask{id=test-events-sink-0} RetriableException from SinkTask:
(org.apache.kafka.connect.runtime.WorkerSinkTask:551)
org.apache.kafka.connect.errors.RetriableException: java.sql.SQLException:
org.postgresql.util.PSQLException: ERROR: column "boolean_field" is of type
boolean but default expression is of type integer
Hint: You will need to rewrite or cast the expression.
at io.confluent.connect.jdbc.sink.JdbcSinkTask.put(JdbcSinkTask.java:93)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:538)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:321)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:224)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:192)
at
org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:175)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:219)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748){code}
The generated SQL statement is:
{code:java}
CREATE TABLE "test_data" ("boolean_field" BOOLEAN DEFAULT 0){code}
was:
The `CREATE TABLE ..` statement generated for sink connectors when configured
with `auto.create: true` generates field declarations that do not conform to
allowed PostgreSQL syntax when considering fields of type boolean with default
values:
Example of source message values Avro schema for input topic:
{code:java}
{
"namespace": "com.test.avro.schema.v1",
"type": "record",
"name": "SomeEvent",
"fields": [
{
"name": "boolean_field",
"type": "boolean",
"default": false
}
]
}
{code}
The connector task fails with:
{code:java}
ERROR WorkerSinkTask{id=test-events-sink-0} RetriableException from SinkTask:
(org.apache.kafka.connect.runtime.WorkerSinkTask:551)
org.apache.kafka.connect.errors.RetriableException: java.sql.SQLException:
org.postgresql.util.PSQLException: ERROR: column "boolean_field" is of type
boolean but default expression is of type integer
Hint: You will need to rewrite or cast the expression.
at io.confluent.connect.jdbc.sink.JdbcSinkTask.put(JdbcSinkTask.java:93)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:538)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:321)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:224)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:192)
at
org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:175)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:219)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748){code}
The generated SQL statement is:
{code:java}
CREATE TABLE "test_data" ("boolean_field" BOOLEAN DEFAULT 0){code}
> Kafka Connect JDBC fails to create PostgreSQL table with default boolean value
> --
>
> Key: KAFKA-8867
> URL: https://issues.apache.org/jira/browse/KAFKA-8867
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 2.3.0
>Reporter: Tudor-Alexandru Voicu
>Priority: Major
>
> The `CREATE TABLE ..` statement generated for sink connectors when configured
> with `auto.create: true` generates
[jira] [Updated] (KAFKA-8867) Kafka Connect JDBC fails to create PostgreSQL table with default boolean value
[
https://issues.apache.org/jira/browse/KAFKA-8867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tudor-Alexandru Voicu updated KAFKA-8867:
-
Description:
The `CREATE TABLE ..` statement generated for sink connectors when configured
with `auto.create: true` generates field declarations that do not conform to
allowed PostgreSQL syntax when considering fields of type boolean with default
values:
Example of source message values Avro schema for input topic:
{code:java}
{
"namespace": "com.test.avro.schema.v1",
"type": "record",
"name": "SomeEvent",
"fields": [
{
"name": "boolean_field",
"type": "boolean",
"default": false
}
]
}
{code}
The connector task fails with:
{code:java}
ERROR WorkerSinkTask{id=test-events-sink-0} RetriableException from SinkTask:
(org.apache.kafka.connect.runtime.WorkerSinkTask:551)
org.apache.kafka.connect.errors.RetriableException: java.sql.SQLException:
org.postgresql.util.PSQLException: ERROR: column "boolean_field" is of type
boolean but default expression is of type integer
Hint: You will need to rewrite or cast the expression.
at io.confluent.connect.jdbc.sink.JdbcSinkTask.put(JdbcSinkTask.java:93)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:538)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:321)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:224)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:192)
at
org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:175)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:219)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748){code}
The generated SQL statement is:
{code:java}
CREATE TABLE "test_data" ("boolean_field" BOOLEAN DEFAULT 0){code}
was:
The `CREATE TABLE ..` statement generated for sink connectors when configured
with `auto.table.create=true` generates field declarations that do not conform
to allowed PostgreSQL syntax when considering fields of type boolean with
default values:
Example of source message values Avro schema for input topic:
{code:java}
{
"namespace": "com.test.avro.schema.v1",
"type": "record",
"name": "SomeEvent",
"fields": [
{
"name": "boolean_field",
"type": "boolean",
"default": false
}
]
}
{code}
The connector task fails with:
{code:java}
ERROR WorkerSinkTask{id=test-events-sink-0} RetriableException from SinkTask:
(org.apache.kafka.connect.runtime.WorkerSinkTask:551)
org.apache.kafka.connect.errors.RetriableException: java.sql.SQLException:
org.postgresql.util.PSQLException: ERROR: column "boolean_field" is of type
boolean but default expression is of type integer
Hint: You will need to rewrite or cast the expression.
at io.confluent.connect.jdbc.sink.JdbcSinkTask.put(JdbcSinkTask.java:93)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:538)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:321)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:224)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:192)
at
org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:175)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:219)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748){code}
The generated SQL statement is:
{code:java}
CREATE TABLE "test_data" ("boolean_field" BOOLEAN DEFAULT 0){code}
> Kafka Connect JDBC fails to create PostgreSQL table with default boolean value
> --
>
> Key: KAFKA-8867
> URL: https://issues.apache.org/jira/browse/KAFKA-8867
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 2.3.0
>Reporter: Tudor-Alexandru Voicu
>Priority: Major
>
> The `CREATE TABLE ..` statement generated for sink connectors when configured
[jira] [Created] (KAFKA-8867) Kafka Connect JDBC fails to create PostgreSQL table with default boolean value
Tudor-Alexandru Voicu created KAFKA-8867:
Summary: Kafka Connect JDBC fails to create PostgreSQL table with
default boolean value
Key: KAFKA-8867
URL: https://issues.apache.org/jira/browse/KAFKA-8867
Project: Kafka
Issue Type: Bug
Components: KafkaConnect
Affects Versions: 2.3.0
Reporter: Tudor-Alexandru Voicu
The `CREATE TABLE ..` statement generated for sink connectors when configured
with `auto.table.create=true` generates field declarations that do not conform
to allowed PostgreSQL syntax when considering fields of type boolean with
default values:
Example of source message values Avro schema for input topic:
{code:java}
{
"namespace": "com.test.avro.schema.v1",
"type": "record",
"name": "SomeEvent",
"fields": [
{
"name": "boolean_field",
"type": "boolean",
"default": false
}
]
}
{code}
The connector task fails with:
{code:java}
ERROR WorkerSinkTask{id=test-events-sink-0} RetriableException from SinkTask:
(org.apache.kafka.connect.runtime.WorkerSinkTask:551)
org.apache.kafka.connect.errors.RetriableException: java.sql.SQLException:
org.postgresql.util.PSQLException: ERROR: column "boolean_field" is of type
boolean but default expression is of type integer
Hint: You will need to rewrite or cast the expression.
at io.confluent.connect.jdbc.sink.JdbcSinkTask.put(JdbcSinkTask.java:93)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:538)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:321)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:224)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:192)
at
org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:175)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:219)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748){code}
The generated SQL statement is:
{code:java}
CREATE TABLE "test_data" ("boolean_field" BOOLEAN DEFAULT 0){code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (KAFKA-8861) Fix flaky RegexSourceIntegrationTest.testMultipleConsumersCanReadFromPartitionedTopic
[
https://issues.apache.org/jira/browse/KAFKA-8861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16922540#comment-16922540
]
ASF GitHub Bot commented on KAFKA-8861:
---
bbejeck commented on pull request #7281: KAFKA-8861 Fix flaky
RegexSourceIntegrationTest.testMultipleConsumers…
URL: https://github.com/apache/kafka/pull/7281
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Fix flaky
> RegexSourceIntegrationTest.testMultipleConsumersCanReadFromPartitionedTopic
> -
>
> Key: KAFKA-8861
> URL: https://issues.apache.org/jira/browse/KAFKA-8861
> Project: Kafka
> Issue Type: Bug
> Components: streams, unit tests
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Minor
> Labels: flaky-test
>
> This is similar to KAFKA-8011 and KAFKA-8026. The error stack is shown below.
> {code:java}
> java.util.ConcurrentModificationException
> at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909)
> at java.util.ArrayList$Itr.next(ArrayList.java:859)
> at java.util.AbstractList.equals(AbstractList.java:521)
> at
> org.apache.kafka.streams.integration.RegexSourceIntegrationTest.lambda$testMultipleConsumersCanReadFromPartitionedTopic$5(RegexSourceIntegrationTest.java:351)
> at
> org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:366)
> at
> org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:336)
> at
> org.apache.kafka.streams.integration.RegexSourceIntegrationTest.testMultipleConsumersCanReadFromPartitionedTopic(RegexSourceIntegrationTest.java:351)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:365)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:330)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:78)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:328)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Resolved] (KAFKA-8676) Avoid Stopping Unnecessary Connectors and Tasks
[ https://issues.apache.org/jira/browse/KAFKA-8676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Randall Hauch resolved KAFKA-8676. -- Fix Version/s: (was: 2.3.0) 2.3.1 2.4.0 Reviewer: Konstantine Karantasis Resolution: Fixed Merged the proposed fix (https://github.com/apache/kafka/pull/7097) and [~kkonstantine]'s unit test (https://github.com/apache/kafka/pull/7287). > Avoid Stopping Unnecessary Connectors and Tasks > > > Key: KAFKA-8676 > URL: https://issues.apache.org/jira/browse/KAFKA-8676 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Affects Versions: 2.3.0 > Environment: centOS >Reporter: Luying Liu >Priority: Major > Labels: ready-to-commit > Fix For: 2.4.0, 2.3.1 > > Original Estimate: 1h > Remaining Estimate: 1h > > When adding a new connector or changing a connector configuration, Kafka > Connect 2.3.0 will stop all existing tasks and start all the tasks, including > the new tasks and the existing ones. However, it is not necessary at all. > Only the new connector and tasks need to be started. As the rebalancing can > be applied for both running and suspended tasks.The following patch will fix > this problem and starts only the new tasks and connectors. > The problem lies in the > KafkaConfigBackingStore.ConsumeCallback.onCompletion() function (line 623 in > KafkaConfigBackingStore.java). When record.key() startsWith "commit-", the > tasks are being committed, and the deferred tasks are processed, Some new > tasks are added to the 'updatedTasks'(line 623 in > KafkaConfigBackingStore.java), and the 'updatedTasks' are sent to > updateListener to complete the task configuration update(line 638 in > KafkaConfigBackingStore.java). In the updateListener.onTaskConfigUpdate() > function, the 'updatedTasks' are added to the member variable, > 'taskConfigUpdates', of class DistributedHerder(line 1295 in > DistributedHerder.java). > In another thread, 'taskConfigUpdates' is copied to 'taskConfigUpdatesCopy' > in updateConfigsWithIncrementalCooperative() (line 445 in > DistributedHerder.java). The 'taskConfigUpdatesCopy' is subsequently used in > processTaskConfigUpdatesWithIncrementalCooperative() (line 345 in > DistributedHerder.java). This function then uses 'taskConfigUpdatesCopy' to > find connectors to stop(line 492 in DistributedHerder.java), and finally get > the tasks to stop, which are all the tasks. The worker thread does the actual > job of stop(line 499 in DistributedHerder.java). > In the original code, all the tasks are added to the 'updatedTasks' (line 623 > in KafkaConfigBackingStore.java), which means all the active connectors are > in the 'connectorsWhoseTasksToStop' set, and all the tasks are in the > 'tasksToStop' list. This causes the stops, and of course the subsequent > restarts, of all the tasks. > So, adding only the 'deferred' tasks to the 'updatedTasks' can avoid the > stops and restarts of unnecessary tasks. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (KAFKA-8866) Make Authorizer create/delete exceptions Optional
[ https://issues.apache.org/jira/browse/KAFKA-8866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16922441#comment-16922441 ] ASF GitHub Bot commented on KAFKA-8866: --- rajinisivaram commented on pull request #7294: KAFKA-8866; Return exceptions as Optional in authorizer API URL: https://github.com/apache/kafka/pull/7294 Follow up on review comment in PR #7268 ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Make Authorizer create/delete exceptions Optional > - > > Key: KAFKA-8866 > URL: https://issues.apache.org/jira/browse/KAFKA-8866 > Project: Kafka > Issue Type: Sub-task > Components: security >Reporter: Rajini Sivaram >Assignee: Rajini Sivaram >Priority: Major > Fix For: 2.4.0 > > > From PR review of: > [https://github.com/apache/kafka/pull/7268|https://github.com/apache/kafka/pull/7268:] > We currently return possibly null ApiException in AclCreateResult and > AclDeleteResult. It would be better to return Optional. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (KAFKA-8866) Make Authorizer create/delete exceptions Optional
Rajini Sivaram created KAFKA-8866: - Summary: Make Authorizer create/delete exceptions Optional Key: KAFKA-8866 URL: https://issues.apache.org/jira/browse/KAFKA-8866 Project: Kafka Issue Type: Sub-task Components: security Reporter: Rajini Sivaram Assignee: Rajini Sivaram Fix For: 2.4.0 >From PR review of: >[https://github.com/apache/kafka/pull/7268|https://github.com/apache/kafka/pull/7268:] We currently return possibly null ApiException in AclCreateResult and AclDeleteResult. It would be better to return Optional. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Resolved] (KAFKA-8760) KIP-504: Add new Java Authorizer API
[ https://issues.apache.org/jira/browse/KAFKA-8760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajini Sivaram resolved KAFKA-8760. --- Resolution: Fixed > KIP-504: Add new Java Authorizer API > - > > Key: KAFKA-8760 > URL: https://issues.apache.org/jira/browse/KAFKA-8760 > Project: Kafka > Issue Type: Sub-task > Components: security >Reporter: Rajini Sivaram >Assignee: Rajini Sivaram >Priority: Major > Fix For: 2.4.0 > > > See > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-504+-+Add+new+Java+Authorizer+Interface] > for details. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (KAFKA-8848) Update system test to use new authorizer
[ https://issues.apache.org/jira/browse/KAFKA-8848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajini Sivaram updated KAFKA-8848: -- Parent Issue: KAFKA-8865 (was: KAFKA-8760) > Update system test to use new authorizer > > > Key: KAFKA-8848 > URL: https://issues.apache.org/jira/browse/KAFKA-8848 > Project: Kafka > Issue Type: Sub-task > Components: security >Reporter: Rajini Sivaram >Assignee: Rajini Sivaram >Priority: Major > Fix For: 2.4.0 > > > We should run system tests with the new authorizer. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (KAFKA-8760) KIP-504: Add new Java Authorizer API
[ https://issues.apache.org/jira/browse/KAFKA-8760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajini Sivaram updated KAFKA-8760: -- Parent: KAFKA-8865 Issue Type: Sub-task (was: New Feature) > KIP-504: Add new Java Authorizer API > - > > Key: KAFKA-8760 > URL: https://issues.apache.org/jira/browse/KAFKA-8760 > Project: Kafka > Issue Type: Sub-task > Components: security >Reporter: Rajini Sivaram >Assignee: Rajini Sivaram >Priority: Major > Fix For: 2.4.0 > > > See > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-504+-+Add+new+Java+Authorizer+Interface] > for details. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (KAFKA-8847) Deprecate and remove usage of supporting classes in kafka.security.auth
[ https://issues.apache.org/jira/browse/KAFKA-8847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajini Sivaram updated KAFKA-8847: -- Parent Issue: KAFKA-8865 (was: KAFKA-8760) > Deprecate and remove usage of supporting classes in kafka.security.auth > --- > > Key: KAFKA-8847 > URL: https://issues.apache.org/jira/browse/KAFKA-8847 > Project: Kafka > Issue Type: Sub-task > Components: security >Reporter: Rajini Sivaram >Assignee: Rajini Sivaram >Priority: Major > Fix For: 2.4.0 > > > Deprecate Acl, Resource etc. from `kafka.security.auth` and replace > references to these with the equivalent Java classes. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (KAFKA-8865) KIP-504: New Java Authorizer API
[ https://issues.apache.org/jira/browse/KAFKA-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajini Sivaram updated KAFKA-8865: -- Issue Type: New Feature (was: Improvement) > KIP-504: New Java Authorizer API > > > Key: KAFKA-8865 > URL: https://issues.apache.org/jira/browse/KAFKA-8865 > Project: Kafka > Issue Type: New Feature > Components: security >Reporter: Rajini Sivaram >Assignee: Rajini Sivaram >Priority: Major > > Parent task for sub-tasks related to KIP-504 -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (KAFKA-8865) KIP-504: New Java Authorizer API
Rajini Sivaram created KAFKA-8865: - Summary: KIP-504: New Java Authorizer API Key: KAFKA-8865 URL: https://issues.apache.org/jira/browse/KAFKA-8865 Project: Kafka Issue Type: Improvement Components: security Reporter: Rajini Sivaram Assignee: Rajini Sivaram Parent task for sub-tasks related to KIP-504 -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (KAFKA-6745) kafka consumer rebalancing takes long time (from 3 secs to 5 minutes)
[ https://issues.apache.org/jira/browse/KAFKA-6745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16922375#comment-16922375 ] Xin commented on KAFKA-6745: same problem，kafka version：1.1.0，spark streaming application Received successful join group response after 5min > kafka consumer rebalancing takes long time (from 3 secs to 5 minutes) > - > > Key: KAFKA-6745 > URL: https://issues.apache.org/jira/browse/KAFKA-6745 > Project: Kafka > Issue Type: Improvement > Components: clients, core >Affects Versions: 0.11.0.0 >Reporter: Ramkumar >Priority: Major > > Hi, We had an HTTP service 3 nodes around Kafka 0.8 . This http service acts > as a REST api for the publishers and consumers to use middleware intead of > using kafka client api. Here the when the consumers rebalance is not a major > issue. > We wanted to upgrade to kafka 0.11 , we have updated our http services (3 > node cluster) to use new Kafka consumer API , but it takes rebalancing of > consumer (multiple consumer under same Group) between secs to 5 mins > (max.poll.interval.ms). Because of this time our http clients are timing out > and do failover. This rebalancing time is major issue. It is not clear from > the documentation ,that rebalance activity for the group takes place after > max.poll.interval.ms or it starts after 3 secs and complete any time with in > 5 minutes. We tried to reduce max.poll.interval.ms to 15 seconds. but this > also triggers rebalance internally. > Below are the other parameters we have set In our service > max.poll.interval.ms = 30 sec > seconds heartbeat.interval.ms = 1 > minute session.timeout.ms = 4 > minutes consumer.cache.timeout = 2 min > > > below is the log > ""2018-03-26 12:53:23,009 [qtp1404928347-11556] INFO > org.apache.kafka.clients.consumer.internals.AbstractCoordinator - > (Re-)joining group firstnetportal_001 > ""2018-03-26 12:57:52,793 [qtp1404928347-11556] INFO > org.apache.kafka.clients.consumer.internals.AbstractCoordinator - > Successfully joined group firstnetportal_001 with generation 7475 > Please let me know if there are any other application/client use http > interace in 3 nodes with out any having this issue > > > > > > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Comment Edited] (KAFKA-8738) Cleaning thread blocked when more than one ALTER_REPLICA_LOG_DIRS requests sent
[
https://issues.apache.org/jira/browse/KAFKA-8738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16922366#comment-16922366
]
dingsainan edited comment on KAFKA-8738 at 9/4/19 10:41 AM:
Sorry for my late reply.
Below are the detail of this case.
h2. 3.1 command
{panel}
./kafka-reassign-partitions.sh --zookeeper 127.0.1.1:2181/local-cluster
--bootstrap-server 127.0.1.1:9092 --reassignment-json-file
/mnt/storage00/Nora/reassignment211.json --execute
{panel}
h2. 3.2 content of 1st reassign
{code:java}
{"partitions":
[{"topic": "lancer_ops_billions_all_log_json_billions",
"partition": 1,
"replicas": [6,15],
"log_dirs": ["any","/data/mnt/storage02/datum/kafka_data"]}]
}
{code}
h2. 3.3 content of 2nd reassign
{code:java}
{"partitions":
[{"topic": "lancer_ops_billions_all_log_json_billions",
"partition": 1,
"replicas": [6,15],
"log_dirs": ["any","/data/mnt/storage03/datum/kafka_data"]}]
}
Map(lancer_ops_billions_all_log_json_billions-1 -> LogCleaningPaused(1))
{code}
h2. 3.4 result
lancer_ops_billions_all_log_json_billions-1 ：this log cleaner does not work for
this partition
h2. 3.5 the status of the log cleaner
the status of the log cleaner
{code:java}
1.None
2.Map(lancer_ops_billions_all_log_json_billions-1 -> LogCleaningPaused(1))
3.Map(lancer_ops_billions_all_log_json_billions-1 -> LogCleaningPaused(2))
4.Map(lancer_ops_billions_all_log_json_billions-1 -> LogCleaningPaused(1))
5.keep in LogCleaningPaused(1){code}
h2. 3.6 issue code
|{{//}}
{{if}} {{(cleaner != null && !isFuture) {}}
{{ }}{{trace(s}}{{"the cleaner is not null and the isfure is false "}}{{)}}
{{ }}{{cleaner.abortCleaning(topicPartition)}}
{{ }}{{cleaner.updateCheckpoints(removedLog.}}{{dir}}{{.getParentFile)}}
{{}}}|
h2. 3.7 fix to do
[^migrationCase.pdf]
was (Author: norading):
Sorry for my late reply.
Below are the detail of this case.
h1. 三、执行过程
h2. 3.1执行的命令
{panel}
./kafka-reassign-partitions.sh --zookeeper 127.0.1.1:2181/local-cluster
--bootstrap-server 127.0.1.1:9092 --reassignment-json-file
/mnt/storage00/Nora/reassignment211.json --execute{panel}
h2. 3.2第一次迁移内容
{code:java}
// code placeholder
{"partitions":
[{"topic": "lancer_ops_billions_all_log_json_billions",
"partition": 1,
"replicas": [6,15],
"log_dirs": ["any","/data/mnt/storage02/datum/kafka_data"]}]
}
{code}
h2. 3.3第二次迁移内容
{code:java}
{"partitions":
[{"topic": "lancer_ops_billions_all_log_json_billions",
"partition": 1,
"replicas": [6,15],
"log_dirs": ["any","/data/mnt/storage03/datum/kafka_data"]}]
}
Map(lancer_ops_billions_all_log_json_billions-1 -> LogCleaningPaused(1))
{code}
h2. 3.4结果
lancer_ops_billions_all_log_json_billions-1这个分区的日志保留时间不再起作用
该TP的状态改变情况
从
None
--->Map(lancer_ops_billions_all_log_json_billions-1 -> LogCleaningPaused(1))
--->Map(lancer_ops_billions_all_log_json_billions-1 -> LogCleaningPaused(2))
--->Map(lancer_ops_billions_all_log_json_billions-1 -> LogCleaningPaused(1))
最后保持在 LogCleaningPaused(1)状态。
h2. 3.5导致问题出现的代码块
|{{//}}{{只有当不是future文件的时候才会执行resume操作}}
{{if}} {{(cleaner != null && !isFuture) {}}
{{ }}{{trace(s}}{{"the cleaner is not null and the isfure is false "}}{{)}}
{{ }}{{cleaner.abortCleaning(topicPartition)}}
{{ }}{{cleaner.updateCheckpoints(removedLog.}}{{dir}}{{.getParentFile)}}
{{}}}|
h2. 3.6fix方式
|{{//}}{{针对future文件也需要进行tp的resume clean的操作}}|
[^migrationCase.pdf]
> Cleaning thread blocked when more than one ALTER_REPLICA_LOG_DIRS requests
> sent
>
>
> Key: KAFKA-8738
> URL: https://issues.apache.org/jira/browse/KAFKA-8738
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 2.1.1
>Reporter: dingsainan
>Priority: Major
> Attachments: migrationCase.pdf
>
>
> Hi,
>
> I am experiencing one situation that the log cleaner dose not work for the
> related topic-partition when using --kafka-reassign-partitions.sh tool for
> V2.1.1 for more than one time frequently.
>
> My operation:
> submitting one task for migration replica in one same broker first, when
> the previous task still in progress, we submit one new task for the same
> topic-partition.
>
> {code:java}
> // the first task:
> {"partitions":
> [{"topic": "lancer_ops_billions_all_log_json_billions",
> "partition": 1,
> "replicas": [6,15],
> "log_dirs": ["any","/data/mnt/storage02/datum/kafka_data"]}]
> }
> //the second task
> {"partitions":
> [{"topic": "lancer_ops_billions_all_log_json_billions",
> "partition": 1,
> "replicas": [6,15],
>
[jira] [Comment Edited] (KAFKA-8738) Cleaning thread blocked when more than one ALTER_REPLICA_LOG_DIRS requests sent
[
https://issues.apache.org/jira/browse/KAFKA-8738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16922366#comment-16922366
]
dingsainan edited comment on KAFKA-8738 at 9/4/19 10:34 AM:
Sorry for my late reply.
Below are the detail of this case.
h1. 三、执行过程
h2. 3.1执行的命令
{panel}
./kafka-reassign-partitions.sh --zookeeper 127.0.1.1:2181/local-cluster
--bootstrap-server 127.0.1.1:9092 --reassignment-json-file
/mnt/storage00/Nora/reassignment211.json --execute{panel}
h2. 3.2第一次迁移内容
{code:java}
// code placeholder
{"partitions":
[{"topic": "lancer_ops_billions_all_log_json_billions",
"partition": 1,
"replicas": [6,15],
"log_dirs": ["any","/data/mnt/storage02/datum/kafka_data"]}]
}
{code}
h2. 3.3第二次迁移内容
{code:java}
{"partitions":
[{"topic": "lancer_ops_billions_all_log_json_billions",
"partition": 1,
"replicas": [6,15],
"log_dirs": ["any","/data/mnt/storage03/datum/kafka_data"]}]
}
Map(lancer_ops_billions_all_log_json_billions-1 -> LogCleaningPaused(1))
{code}
h2. 3.4结果
lancer_ops_billions_all_log_json_billions-1这个分区的日志保留时间不再起作用
该TP的状态改变情况
从
None
--->Map(lancer_ops_billions_all_log_json_billions-1 -> LogCleaningPaused(1))
--->Map(lancer_ops_billions_all_log_json_billions-1 -> LogCleaningPaused(2))
--->Map(lancer_ops_billions_all_log_json_billions-1 -> LogCleaningPaused(1))
最后保持在 LogCleaningPaused(1)状态。
h2. 3.5导致问题出现的代码块
|{{//}}{{只有当不是future文件的时候才会执行resume操作}}
{{if}} {{(cleaner != null && !isFuture) {}}
{{ }}{{trace(s}}{{"the cleaner is not null and the isfure is false "}}{{)}}
{{ }}{{cleaner.abortCleaning(topicPartition)}}
{{ }}{{cleaner.updateCheckpoints(removedLog.}}{{dir}}{{.getParentFile)}}
{{}}}|
h2. 3.6fix方式
|{{//}}{{针对future文件也需要进行tp的resume clean的操作}}|
[^migrationCase.pdf]
was (Author: norading):
Sorry for my late reply.
Below are the detail of this case.
h1. 三、执行过程
h2. 3.1执行的命令
{panel}
{panel}
|{{.}}{{/kafka-reassign-partitions}}{{.sh --zookeeper zkAddress
--bootstrap-server bootstrap --reassignment-json-}}{{file}}
{{/mnt/storage00/Nora/reassignment211}}{{.json --execute}}|
h2. 3.2第一次迁移内容
{panel}
{panel}
|{{{}}{{"partitions"}}{{:}}
{{}}{{[{}}{{"topic"}}{{:
}}{{"lancer_ops_billions_all_log_json_billions"}}{{,}}
{{ }}{{"partition"}}{{: 1,}}
{{ }}{{"replicas"}}{{: [6,15],}}
{{ }}{{"log_dirs"}}{{:
[}}{{"any"}}{{,}}{{"/data/mnt/storage02/datum/kafka_data"}}{{]}]}}
{{}}}|
h2. 3.3第二次迁移内容
{panel}
{panel}
|{{{}}{{"partitions"}}{{:}}
{{}}{{[{}}{{"topic"}}{{:
}}{{"lancer_ops_billions_all_log_json_billions"}}{{,}}
{{ }}{{"partition"}}{{: 1,}}
{{ }}{{"replicas"}}{{: [6,15],}}
{{ }}{{"log_dirs"}}{{:
[}}{{"any"}}{{,}}{{"/data/mnt/storage03/datum/kafka_data"}}{{]}]}}
{{}}}
{{Map(lancer_ops_billions_all_log_json_billions-1 -> LogCleaningPaused(1))}}|
h2. 3.4结果
lancer_ops_billions_all_log_json_billions-1这个分区的日志保留时间不再起作用
该TP的状态改变情况
从
None
--->Map(lancer_ops_billions_all_log_json_billions-1 -> LogCleaningPaused(1))
--->Map(lancer_ops_billions_all_log_json_billions-1 -> LogCleaningPaused(2))
--->Map(lancer_ops_billions_all_log_json_billions-1 -> LogCleaningPaused(1))
最后保持在 LogCleaningPaused(1)状态。
h2. 3.5导致问题出现的代码块
{panel}
{panel}
|{{//}}{{只有当不是future文件的时候才会执行resume操作}}
{{if}} {{(cleaner != null && !isFuture) {}}
{{ }}{{trace(s}}{{"the cleaner is not null and the isfure is false "}}{{)}}
{{ }}{{cleaner.abortCleaning(topicPartition)}}
{{ }}{{cleaner.updateCheckpoints(removedLog.}}{{dir}}{{.getParentFile)}}
{{}}}|
h2. 3.6fix方式
{panel}
{panel}
|{{//}}{{针对future文件也需要进行tp的resume clean的操作}}|
[^migrationCase.pdf]
> Cleaning thread blocked when more than one ALTER_REPLICA_LOG_DIRS requests
> sent
>
>
> Key: KAFKA-8738
> URL: https://issues.apache.org/jira/browse/KAFKA-8738
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 2.1.1
>Reporter: dingsainan
>Priority: Major
> Attachments: migrationCase.pdf
>
>
> Hi,
>
> I am experiencing one situation that the log cleaner dose not work for the
> related topic-partition when using --kafka-reassign-partitions.sh tool for
> V2.1.1 for more than one time frequently.
>
> My operation:
> submitting one task for migration replica in one same broker first, when
> the previous task still in progress, we submit one new task for the same
> topic-partition.
>
> {code:java}
> // the first task:
> {"partitions":
> [{"topic": "lancer_ops_billions_all_log_json_billions",
> "partition": 1,
> "replicas": [6,15],
> "log_dirs": ["any","/data/mnt/storage02/datum/kafka_data"]}]
> }
> //the second
[jira] [Commented] (KAFKA-8738) Cleaning thread blocked when more than one ALTER_REPLICA_LOG_DIRS requests sent
[
https://issues.apache.org/jira/browse/KAFKA-8738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16922366#comment-16922366
]
dingsainan commented on KAFKA-8738:
---
Sorry for my late reply.
Below are the detail of this case.
h1. 三、执行过程
h2. 3.1执行的命令
{panel}
{panel}
|{{.}}{{/kafka-reassign-partitions}}{{.sh --zookeeper zkAddress
--bootstrap-server bootstrap --reassignment-json-}}{{file}}
{{/mnt/storage00/Nora/reassignment211}}{{.json --execute}}|
h2. 3.2第一次迁移内容
{panel}
{panel}
|{{{}}{{"partitions"}}{{:}}
{{}}{{[{}}{{"topic"}}{{:
}}{{"lancer_ops_billions_all_log_json_billions"}}{{,}}
{{ }}{{"partition"}}{{: 1,}}
{{ }}{{"replicas"}}{{: [6,15],}}
{{ }}{{"log_dirs"}}{{:
[}}{{"any"}}{{,}}{{"/data/mnt/storage02/datum/kafka_data"}}{{]}]}}
{{}}}|
h2. 3.3第二次迁移内容
{panel}
{panel}
|{{{}}{{"partitions"}}{{:}}
{{}}{{[{}}{{"topic"}}{{:
}}{{"lancer_ops_billions_all_log_json_billions"}}{{,}}
{{ }}{{"partition"}}{{: 1,}}
{{ }}{{"replicas"}}{{: [6,15],}}
{{ }}{{"log_dirs"}}{{:
[}}{{"any"}}{{,}}{{"/data/mnt/storage03/datum/kafka_data"}}{{]}]}}
{{}}}
{{Map(lancer_ops_billions_all_log_json_billions-1 -> LogCleaningPaused(1))}}|
h2. 3.4结果
lancer_ops_billions_all_log_json_billions-1这个分区的日志保留时间不再起作用
该TP的状态改变情况
从
None
--->Map(lancer_ops_billions_all_log_json_billions-1 -> LogCleaningPaused(1))
--->Map(lancer_ops_billions_all_log_json_billions-1 -> LogCleaningPaused(2))
--->Map(lancer_ops_billions_all_log_json_billions-1 -> LogCleaningPaused(1))
最后保持在 LogCleaningPaused(1)状态。
h2. 3.5导致问题出现的代码块
{panel}
{panel}
|{{//}}{{只有当不是future文件的时候才会执行resume操作}}
{{if}} {{(cleaner != null && !isFuture) {}}
{{ }}{{trace(s}}{{"the cleaner is not null and the isfure is false "}}{{)}}
{{ }}{{cleaner.abortCleaning(topicPartition)}}
{{ }}{{cleaner.updateCheckpoints(removedLog.}}{{dir}}{{.getParentFile)}}
{{}}}|
h2. 3.6fix方式
{panel}
{panel}
|{{//}}{{针对future文件也需要进行tp的resume clean的操作}}|
[^migrationCase.pdf]
> Cleaning thread blocked when more than one ALTER_REPLICA_LOG_DIRS requests
> sent
>
>
> Key: KAFKA-8738
> URL: https://issues.apache.org/jira/browse/KAFKA-8738
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 2.1.1
>Reporter: dingsainan
>Priority: Major
> Attachments: migrationCase.pdf
>
>
> Hi,
>
> I am experiencing one situation that the log cleaner dose not work for the
> related topic-partition when using --kafka-reassign-partitions.sh tool for
> V2.1.1 for more than one time frequently.
>
> My operation:
> submitting one task for migration replica in one same broker first, when
> the previous task still in progress, we submit one new task for the same
> topic-partition.
>
> {code:java}
> // the first task:
> {"partitions":
> [{"topic": "lancer_ops_billions_all_log_json_billions",
> "partition": 1,
> "replicas": [6,15],
> "log_dirs": ["any","/data/mnt/storage02/datum/kafka_data"]}]
> }
> //the second task
> {"partitions":
> [{"topic": "lancer_ops_billions_all_log_json_billions",
> "partition": 1,
> "replicas": [6,15],
> "log_dirs": ["any","/data/mnt/storage03/datum/kafka_data"]}]
> }
>
> {code}
>
> My search:
> Kafka executes abortAndPauseCleaning() once task is submitted, shortly,
> another task is submitted for the same topic-partition, so the clean thread
> status is {color:#ff}LogCleaningPaused(2){color} currently. When the
> second task completed, the clean thread will be resumed for this
> topic-partition once. In my case, the previous task is killed directly, no
> resumeClean() is executed for the first task, so when the second task is
> completed, the clean status for the topic-partition is still
> {color:#ff}LogCleaningPaused(1){color}, which blocks the clean thread for
> the topic-partition.
>
> _That's all my search, please confirm._
>
> _Thanks_
> _Nora_
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (KAFKA-8738) Cleaning thread blocked when more than one ALTER_REPLICA_LOG_DIRS requests sent
[
https://issues.apache.org/jira/browse/KAFKA-8738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
dingsainan updated KAFKA-8738:
--
Attachment: migrationCase.pdf
> Cleaning thread blocked when more than one ALTER_REPLICA_LOG_DIRS requests
> sent
>
>
> Key: KAFKA-8738
> URL: https://issues.apache.org/jira/browse/KAFKA-8738
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 2.1.1
>Reporter: dingsainan
>Priority: Major
> Attachments: migrationCase.pdf
>
>
> Hi,
>
> I am experiencing one situation that the log cleaner dose not work for the
> related topic-partition when using --kafka-reassign-partitions.sh tool for
> V2.1.1 for more than one time frequently.
>
> My operation:
> submitting one task for migration replica in one same broker first, when
> the previous task still in progress, we submit one new task for the same
> topic-partition.
>
> {code:java}
> // the first task:
> {"partitions":
> [{"topic": "lancer_ops_billions_all_log_json_billions",
> "partition": 1,
> "replicas": [6,15],
> "log_dirs": ["any","/data/mnt/storage02/datum/kafka_data"]}]
> }
> //the second task
> {"partitions":
> [{"topic": "lancer_ops_billions_all_log_json_billions",
> "partition": 1,
> "replicas": [6,15],
> "log_dirs": ["any","/data/mnt/storage03/datum/kafka_data"]}]
> }
>
> {code}
>
> My search:
> Kafka executes abortAndPauseCleaning() once task is submitted, shortly,
> another task is submitted for the same topic-partition, so the clean thread
> status is {color:#ff}LogCleaningPaused(2){color} currently. When the
> second task completed, the clean thread will be resumed for this
> topic-partition once. In my case, the previous task is killed directly, no
> resumeClean() is executed for the first task, so when the second task is
> completed, the clean status for the topic-partition is still
> {color:#ff}LogCleaningPaused(1){color}, which blocks the clean thread for
> the topic-partition.
>
> _That's all my search, please confirm._
>
> _Thanks_
> _Nora_
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (KAFKA-6188) Broker fails with FATAL Shutdown - log dirs have failed
[ https://issues.apache.org/jira/browse/KAFKA-6188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16922300#comment-16922300 ] jack handey commented on KAFKA-6188: [~lindong] [~TeilaRei] I'm using kafka 2.2.0 download from apache.org. and i have the similar problem as below: org.apache.kafka.common.errors.KafkaStorageException: Error while deleting segments for visitlog-0 in dir /tmp/kafka-logsorg.apache.kafka.common.errors.KafkaStorageException: Error while deleting segments for visitlog-0 in dir /tmp/kafka-logsCaused by: java.nio.file.NoSuchFileException: /tmp/kafka-logs/visitlog-0/3838.log at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:409) at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262) at java.nio.file.Files.move(Files.java:1395) at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:805) at org.apache.kafka.common.record.FileRecords.renameTo(FileRecords.java:224) at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:488) at kafka.log.Log.asyncDeleteSegment(Log.scala:1924) at kafka.log.Log.deleteSegment(Log.scala:1909) at kafka.log.Log.$anonfun$deleteSegments$3(Log.scala:1455) at kafka.log.Log.$anonfun$deleteSegments$3$adapted(Log.scala:1455) at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) at kafka.log.Log.$anonfun$deleteSegments$2(Log.scala:1455) at scala.runtime.java8.JFunction0$mcI$sp.apply(JFunction0$mcI$sp.java:23) at kafka.log.Log.maybeHandleIOException(Log.scala:2013) at kafka.log.Log.deleteSegments(Log.scala:1446) at kafka.log.Log.deleteOldSegments(Log.scala:1441) at kafka.log.Log.deleteRetentionMsBreachedSegments(Log.scala:1519) at kafka.log.Log.deleteOldSegments(Log.scala:1509) at kafka.log.LogManager.$anonfun$cleanupLogs$3(LogManager.scala:913) at kafka.log.LogManager.$anonfun$cleanupLogs$3$adapted(LogManager.scala:910) at scala.collection.immutable.List.foreach(List.scala:392) at kafka.log.LogManager.cleanupLogs(LogManager.scala:910) at kafka.log.LogManager.$anonfun$startup$2(LogManager.scala:395) at kafka.utils.KafkaScheduler.$anonfun$schedule$2(KafkaScheduler.scala:114) at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:63) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Suppressed: java.nio.file.NoSuchFileException: /tmp/kafka-logs/visitlog-0/3838.log -> /tmp/kafka-logs/visitlog-0/3838.log.deleted at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:396) at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262) at java.nio.file.Files.move(Files.java:1395) at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:802) ... 30 more[2019-09-02 04:48:44,391] INFO [ReplicaManager broker=0] Stopping serving replicas in dir /tmp/kafka-logs (kafka.server.ReplicaManager)[2019-09-02 04:48:44,395] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions Set(__consumer_offsets-22, __consumer_offsets-30, __consumer_offsets-8, __consumer_offsets-21, __consumer_offsets-4, __consumer_offsets-27, __consumer_offsets-7, __consumer_offsets-9, __consumer_offsets-46, __consumer_offsets-25, __consumer_offsets-35, __consumer_offsets-41, __consumer_offsets-33, __consumer_offsets-23, __consumer_offsets-49, __consumer_offsets-47, __consumer_offsets-16, __consumer_offsets-28, __consumer_offsets-31, __consumer_offsets-36, __consumer_offsets-42, __consumer_offsets-3, __consumer_offsets-18, __consumer_offsets-37, __consumer_offsets-15, __consumer_offsets-24, systemlog-0, __consumer_offsets-38, __consumer_offsets-17, __consumer_offsets-48, __consumer_offsets-19, __consumer_offsets-11, __consumer_offsets-13, __consumer_offsets-2, __consumer_offsets-43, visitlog-0, __consumer_offsets-6, __consumer_offsets-14, __consumer_offsets-20, __consumer_offsets-0, __consumer_offsets-44,
[jira] [Commented] (KAFKA-7849) Warning when adding GlobalKTable
[ https://issues.apache.org/jira/browse/KAFKA-7849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16921983#comment-16921983 ] ASF GitHub Bot commented on KAFKA-7849: --- mjsax commented on pull request #7104: KAFKA-7849: Fix the warning when using GlobalKTable URL: https://github.com/apache/kafka/pull/7104 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Warning when adding GlobalKTable > > > Key: KAFKA-7849 > URL: https://issues.apache.org/jira/browse/KAFKA-7849 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.1.0 >Reporter: Dmitry Minkovsky >Assignee: Omar Al-Safi >Priority: Minor > Labels: newbie > > Per > https://lists.apache.org/thread.html/59c119be8a2723c501e0653fa3ed571e8c09be40d5b5170c151528b5@%3Cusers.kafka.apache.org%3E > > When I add a GlobalKTable for topic "message-write-service-user-ids-by-email" > to my topology, I get this warning: > > [2019-01-19 12:18:14,008] WARN > (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:421) > [Consumer > clientId=message-write-service-55f2ca4d-0efc-4344-90d3-955f9f5a65fd-StreamThread-2-consumer, > groupId=message-write-service] The following subscribed topics are not > assigned to any members: [message-write-service-user-ids-by-email] -- This message was sent by Atlassian Jira (v8.3.2#803003)
