[jira] [Assigned] (KAFKA-9258) Connect ConnectorStatusMetricsGroup sometimes NPE
[ https://issues.apache.org/jira/browse/KAFKA-9258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton reassigned KAFKA-9258: Assignee: (was: Chris Egerton) > Connect ConnectorStatusMetricsGroup sometimes NPE > - > > Key: KAFKA-9258 > URL: https://issues.apache.org/jira/browse/KAFKA-9258 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Reporter: Cyrus Vafadari >Priority: Major > > java.lang.NullPointerException > at > org.apache.kafka.connect.runtime.Worker$ConnectorStatusMetricsGroup.recordTaskRemoved(Worker.java:901) > at > org.apache.kafka.connect.runtime.Worker.awaitStopTask(Worker.java:720) > at > org.apache.kafka.connect.runtime.Worker.awaitStopTasks(Worker.java:740) > at > org.apache.kafka.connect.runtime.Worker.stopAndAwaitTasks(Worker.java:758) > at > org.apache.kafka.connect.runtime.distributed.DistributedHerder.processTaskConfigUpdatesWithIncrementalCooperative(DistributedHerder.java:575) > at > org.apache.kafka.connect.runtime.distributed.DistributedHerder.tick(DistributedHerder.java:396) > at > org.apache.kafka.connect.runtime.distributed.DistributedHerder.run(DistributedHerder.java:282) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266) > at java.util.concurrent.FutureTask.run(FutureTask.java) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-9258) Connect ConnectorStatusMetricsGroup sometimes NPE
[ https://issues.apache.org/jira/browse/KAFKA-9258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton updated KAFKA-9258: - Affects Version/s: 2.4.0 > Connect ConnectorStatusMetricsGroup sometimes NPE > - > > Key: KAFKA-9258 > URL: https://issues.apache.org/jira/browse/KAFKA-9258 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Affects Versions: 2.4.0 >Reporter: Cyrus Vafadari >Priority: Major > > java.lang.NullPointerException > at > org.apache.kafka.connect.runtime.Worker$ConnectorStatusMetricsGroup.recordTaskRemoved(Worker.java:901) > at > org.apache.kafka.connect.runtime.Worker.awaitStopTask(Worker.java:720) > at > org.apache.kafka.connect.runtime.Worker.awaitStopTasks(Worker.java:740) > at > org.apache.kafka.connect.runtime.Worker.stopAndAwaitTasks(Worker.java:758) > at > org.apache.kafka.connect.runtime.distributed.DistributedHerder.processTaskConfigUpdatesWithIncrementalCooperative(DistributedHerder.java:575) > at > org.apache.kafka.connect.runtime.distributed.DistributedHerder.tick(DistributedHerder.java:396) > at > org.apache.kafka.connect.runtime.distributed.DistributedHerder.run(DistributedHerder.java:282) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266) > at java.util.concurrent.FutureTask.run(FutureTask.java) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KAFKA-9258) Connect ConnectorStatusMetricsGroup sometimes NPE
[ https://issues.apache.org/jira/browse/KAFKA-9258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton reassigned KAFKA-9258: Assignee: (was: Chris Egerton) > Connect ConnectorStatusMetricsGroup sometimes NPE > - > > Key: KAFKA-9258 > URL: https://issues.apache.org/jira/browse/KAFKA-9258 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Reporter: Cyrus Vafadari >Priority: Major > > java.lang.NullPointerException > at > org.apache.kafka.connect.runtime.Worker$ConnectorStatusMetricsGroup.recordTaskRemoved(Worker.java:901) > at > org.apache.kafka.connect.runtime.Worker.awaitStopTask(Worker.java:720) > at > org.apache.kafka.connect.runtime.Worker.awaitStopTasks(Worker.java:740) > at > org.apache.kafka.connect.runtime.Worker.stopAndAwaitTasks(Worker.java:758) > at > org.apache.kafka.connect.runtime.distributed.DistributedHerder.processTaskConfigUpdatesWithIncrementalCooperative(DistributedHerder.java:575) > at > org.apache.kafka.connect.runtime.distributed.DistributedHerder.tick(DistributedHerder.java:396) > at > org.apache.kafka.connect.runtime.distributed.DistributedHerder.run(DistributedHerder.java:282) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266) > at java.util.concurrent.FutureTask.run(FutureTask.java) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KAFKA-9258) Connect ConnectorStatusMetricsGroup sometimes NPE
[ https://issues.apache.org/jira/browse/KAFKA-9258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton reassigned KAFKA-9258: Assignee: Chris Egerton > Connect ConnectorStatusMetricsGroup sometimes NPE > - > > Key: KAFKA-9258 > URL: https://issues.apache.org/jira/browse/KAFKA-9258 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Reporter: Cyrus Vafadari >Assignee: Chris Egerton >Priority: Major > > java.lang.NullPointerException > at > org.apache.kafka.connect.runtime.Worker$ConnectorStatusMetricsGroup.recordTaskRemoved(Worker.java:901) > at > org.apache.kafka.connect.runtime.Worker.awaitStopTask(Worker.java:720) > at > org.apache.kafka.connect.runtime.Worker.awaitStopTasks(Worker.java:740) > at > org.apache.kafka.connect.runtime.Worker.stopAndAwaitTasks(Worker.java:758) > at > org.apache.kafka.connect.runtime.distributed.DistributedHerder.processTaskConfigUpdatesWithIncrementalCooperative(DistributedHerder.java:575) > at > org.apache.kafka.connect.runtime.distributed.DistributedHerder.tick(DistributedHerder.java:396) > at > org.apache.kafka.connect.runtime.distributed.DistributedHerder.run(DistributedHerder.java:282) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266) > at java.util.concurrent.FutureTask.run(FutureTask.java) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-9258) Connect ConnectorStatusMetricsGroup sometimes NPE
[ https://issues.apache.org/jira/browse/KAFKA-9258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16986557#comment-16986557 ] Chris Egerton commented on KAFKA-9258: -- This issue prevents users from restarting tasks that have failed during startup, and causes the REST API to throw a 500 error if they try to do so. This persists even after the connector has been reconfigured. > Connect ConnectorStatusMetricsGroup sometimes NPE > - > > Key: KAFKA-9258 > URL: https://issues.apache.org/jira/browse/KAFKA-9258 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Reporter: Cyrus Vafadari >Priority: Major > > java.lang.NullPointerException > at > org.apache.kafka.connect.runtime.Worker$ConnectorStatusMetricsGroup.recordTaskRemoved(Worker.java:901) > at > org.apache.kafka.connect.runtime.Worker.awaitStopTask(Worker.java:720) > at > org.apache.kafka.connect.runtime.Worker.awaitStopTasks(Worker.java:740) > at > org.apache.kafka.connect.runtime.Worker.stopAndAwaitTasks(Worker.java:758) > at > org.apache.kafka.connect.runtime.distributed.DistributedHerder.processTaskConfigUpdatesWithIncrementalCooperative(DistributedHerder.java:575) > at > org.apache.kafka.connect.runtime.distributed.DistributedHerder.tick(DistributedHerder.java:396) > at > org.apache.kafka.connect.runtime.distributed.DistributedHerder.run(DistributedHerder.java:282) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266) > at java.util.concurrent.FutureTask.run(FutureTask.java) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KAFKA-9258) Connect ConnectorStatusMetricsGroup sometimes NPE
[ https://issues.apache.org/jira/browse/KAFKA-9258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton reassigned KAFKA-9258: Assignee: Chris Egerton > Connect ConnectorStatusMetricsGroup sometimes NPE > - > > Key: KAFKA-9258 > URL: https://issues.apache.org/jira/browse/KAFKA-9258 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Reporter: Cyrus Vafadari >Assignee: Chris Egerton >Priority: Major > > java.lang.NullPointerException > at > org.apache.kafka.connect.runtime.Worker$ConnectorStatusMetricsGroup.recordTaskRemoved(Worker.java:901) > at > org.apache.kafka.connect.runtime.Worker.awaitStopTask(Worker.java:720) > at > org.apache.kafka.connect.runtime.Worker.awaitStopTasks(Worker.java:740) > at > org.apache.kafka.connect.runtime.Worker.stopAndAwaitTasks(Worker.java:758) > at > org.apache.kafka.connect.runtime.distributed.DistributedHerder.processTaskConfigUpdatesWithIncrementalCooperative(DistributedHerder.java:575) > at > org.apache.kafka.connect.runtime.distributed.DistributedHerder.tick(DistributedHerder.java:396) > at > org.apache.kafka.connect.runtime.distributed.DistributedHerder.run(DistributedHerder.java:282) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266) > at java.util.concurrent.FutureTask.run(FutureTask.java) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-9259) suppress() for windowed-Serdes does not work with default serdes
[ https://issues.apache.org/jira/browse/KAFKA-9259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16986546#comment-16986546 ] John Roesler commented on KAFKA-9259: - Thanks for the report, [~mjsax]! Just a quick additional note for whoever picks this up: To properly parameterize windowed serdes, you need extra information about the windows (eg the window size). Theoretically, the suppress operator could find this out by traversing the topology upstream to find it on its window-definition ancestor, but it might get a bit brittle. If this turns out to be a mess, it would probably be cleaner just to implement https://issues.apache.org/jira/browse/KAFKA-9260 instead, which would also fix this issue. > suppress() for windowed-Serdes does not work with default serdes > > > Key: KAFKA-9259 > URL: https://issues.apache.org/jira/browse/KAFKA-9259 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.1.0 >Reporter: Matthias J. Sax >Priority: Major > > The suppress() operator either inherits serdes from its upstream operator or > falls back to default serdes from the config. > If the upstream operator is an windowed aggregation, the window-aggregation > operator wraps the user passed-in serde with a window-serde and pushed it > into suppress() – however, if default serdes are used, the window-aggregation > operator cannot push anything into suppress(). At runtime, it just creates a > default serde and wraps it according. For this case, suppress() also falls > back to default serdes; however, it does not wrap the serde and thus a > ClassCastException is thrown when the serde is used later. > suppress() is already aware if the upstream aggregation is time/session > windowed or not and thus should use this information to wrap default serdes > accordingly. > The current workaround for windowed-suppress is to overwrite the default > serde upstream to suppress(), such that suppress() inherits serdes and does > not fall back to default serdes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (KAFKA-9254) Topic level configuration failed
[
https://issues.apache.org/jira/browse/KAFKA-9254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16986025#comment-16986025
]
fenghong edited comment on KAFKA-9254 at 12/3/19 1:48 AM:
--
We can reproduce this bug with the latest version (2.3.1) and this version
(2.0.1)
Write data continuously during testing
producer config
{code:java}
acks=all
{code}
OS
{code:java}
Linux 4.15.0-70-generic #79~16.04.1-Ubuntu SMP Tue Nov 12 14:01:10 UTC 2019
x86_64 x86_64 x86_64 GNU/Linux
{code}
JAVA
{code:java}
java version "1.8.0_192"
Java(TM) SE Runtime Environment (build 1.8.0_192-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.192-b12, mixed mode)
{code}
server.properties
{code:java}
broker.id=4
delete.topic.enable=true
listeners=PLAINTEXT://xxx:9092
num.network.threads=9
num.io.threads=12
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/data/data/kafka,/data2/data/kafka,/data3/data/kafka,/data4/data/kafka
num.partitions=1
num.recovery.threads.per.data.dir=2
offsets.topic.replication.factor=5
transaction.state.log.replication.factor=5
transaction.state.log.min.isr=3
log.retention.hours=26280
log.segment.bytes=10737418
log.retention.check.interval.ms=30
zookeeper.connect=xxx1:2181,xxx2:2181,xxx3:2181,xxx4:2181,xxx5:2181
zookeeper.connection.timeout.ms=6000
zookeeper.session.timeout.ms=6000
group.initial.rebalance.delay.ms=500
default.replication.factor=3
min.insync.replicas=3
auto.create.topics.enable=false
unclean.leader.election.enable=false
log.cleanup.policy=delete
num.replica.fetchers=2
replica.lag.time.max.ms=1
replica.fetch.wait.max.ms=500
{code}
was (Author: fenghong):
We can reproduce this bug with the latest version (2.3.1) and this version
(2.0.1)
Write data continuously during testing
OS
{code:java}
Linux 4.15.0-70-generic #79~16.04.1-Ubuntu SMP Tue Nov 12 14:01:10 UTC 2019
x86_64 x86_64 x86_64 GNU/Linux
{code}
JAVA
{code:java}
java version "1.8.0_192"
Java(TM) SE Runtime Environment (build 1.8.0_192-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.192-b12, mixed mode)
{code}
server.properties
{code:java}
broker.id=4
delete.topic.enable=true
listeners=PLAINTEXT://xxx:9092
num.network.threads=9
num.io.threads=12
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/data/data/kafka,/data2/data/kafka,/data3/data/kafka,/data4/data/kafka
num.partitions=1
num.recovery.threads.per.data.dir=2
offsets.topic.replication.factor=5
transaction.state.log.replication.factor=5
transaction.state.log.min.isr=3
log.retention.hours=26280
log.segment.bytes=10737418
log.retention.check.interval.ms=30
zookeeper.connect=xxx1:2181,xxx2:2181,xxx3:2181,xxx4:2181,xxx5:2181
zookeeper.connection.timeout.ms=6000
zookeeper.session.timeout.ms=6000
group.initial.rebalance.delay.ms=500
default.replication.factor=3
min.insync.replicas=3
auto.create.topics.enable=false
unclean.leader.election.enable=false
log.cleanup.policy=delete
num.replica.fetchers=2
replica.lag.time.max.ms=1
replica.fetch.wait.max.ms=500
{code}
> Topic level configuration failed
>
>
> Key: KAFKA-9254
> URL: https://issues.apache.org/jira/browse/KAFKA-9254
> Project: Kafka
> Issue Type: Bug
> Components: config, log, replication
>Affects Versions: 2.0.1
>Reporter: fenghong
>Priority: Critical
>
> We are engineers at Huobi and now encounter Kafka BUG
> Modifying DynamicBrokerConfig more than 2 times will invalidate the topic
> level unrelated configuration
> The bug reproduction method as follows：
> # Set Kafka Broker config server.properties min.insync.replicas=3
> # Create topic test-1 and set topic‘s level config min.insync.replicas=2
> # Dynamically modify the configuration twice as shown below
> {code:java}
> bin/kafka-configs.sh --bootstrap-server xxx:9092 --entity-type brokers
> --entity-default --alter --add-config log.message.timestamp.type=LogAppendTime
> bin/kafka-configs.sh --bootstrap-server xxx:9092 --entity-type brokers
> --entity-default --alter --add-config log.retention.ms=60480
> {code}
> # stop a Kafka Server and found the Exception as shown below
> org.apache.kafka.common.errors.NotEnoughReplicasException: Number of insync
> replicas for partition test-1-0 is [2], below required minimum [3]
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KAFKA-9261) NPE when updating client metadata
Jason Gustafson created KAFKA-9261:
--
Summary: NPE when updating client metadata
Key: KAFKA-9261
URL: https://issues.apache.org/jira/browse/KAFKA-9261
Project: Kafka
Issue Type: Bug
Reporter: Jason Gustafson
Assignee: Jason Gustafson
We have seen the following exception recently:
{code}
java.lang.NullPointerException
at java.base/java.util.Objects.requireNonNull(Objects.java:221)
at org.apache.kafka.common.Cluster.(Cluster.java:134)
at org.apache.kafka.common.Cluster.(Cluster.java:89)
at
org.apache.kafka.clients.MetadataCache.computeClusterView(MetadataCache.java:120)
at org.apache.kafka.clients.MetadataCache.(MetadataCache.java:82)
at org.apache.kafka.clients.MetadataCache.(MetadataCache.java:58)
at
org.apache.kafka.clients.Metadata.handleMetadataResponse(Metadata.java:325)
at org.apache.kafka.clients.Metadata.update(Metadata.java:252)
at
org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.handleCompletedMetadataResponse(NetworkClient.java:1059)
at
org.apache.kafka.clients.NetworkClient.handleCompletedReceives(NetworkClient.java:845)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:548)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:262)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:233)
at
org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1281)
at
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1225)
at
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1201)
{code}
The client assumes that if a leader is included in the response, then node
information must also be available. There are at least a couple possible
reasons this assumption can fail:
1. The client is able to detect stale partition metadata using leader epoch
information available. If stale partition metadata is detected, the client
ignores it and uses the last known metadata. However, it cannot detect stale
broker information and will always accept the latest update. This means that
the latest metadata may be a mix of multiple metadata responses and therefore
the invariant will not generally hold.
2. There is no lock which protects both the fetching of partition metadata and
the live broker when handling a Metadata request. This means an UpdateMetadata
request can arrive concurrently and break the intended invariant.
It seems case 2 has been possible for a long time, but it should be extremely
rare. Case 1 was only made possible with KIP-320, which added the leader epoch
tracking. It should also be rare, but the window for inconsistent metadata is
probably a bit bigger than the window for a concurrent update.
To fix this, we should make the client more defensive about metadata updates
and not assume that the leader is among the live endpoints.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9210) kafka stream loss data
[
https://issues.apache.org/jira/browse/KAFKA-9210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16986466#comment-16986466
]
Sophie Blee-Goldman commented on KAFKA-9210:
Hi [~panpan.liu], it seems you might be hitting KAFKA-7443. That issue has been
fixed in 2.1.1 so I suggest upgrading your Streams client, and verifying that
you are getting the expected results then
> kafka stream loss data
> --
>
> Key: KAFKA-9210
> URL: https://issues.apache.org/jira/browse/KAFKA-9210
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.0.1
>Reporter: panpan.liu
>Priority: Major
> Attachments: app.log, screenshot-1.png
>
>
> kafka broker: 2.0.1
> kafka stream client: 2.1.0
> # two applications run at the same time
> # after some days,I stop one application(in k8s)
> # The flollowing log occured and I check the data and find that value is
> less than what I expected.
> {quote}Partitions
> [flash-app-xmc-worker-share-store-minute-repartition-1]Partitions
> [flash-app-xmc-worker-share-store-minute-repartition-1] for changelog
> flash-app-xmc-worker-share-store-minute-changelog-12019-11-19
> 05:50:49.816|WARN
> |flash-client-xmc-StreamThread-3|o.a.k.s.p.i.StoreChangelogReader|101|stream-thread
> [flash-client-xmc-StreamThread-3] Restoring StreamTasks failed. Deleting
> StreamTasks stores to recreate from
> scratch.org.apache.kafka.clients.consumer.OffsetOutOfRangeException: Offsets
> out of range with no configured reset policy for partitions:
> \{flash-app-xmc-worker-share-store-minute-changelog-1=6128684} at
> org.apache.kafka.clients.consumer.internals.Fetcher.parseCompletedFetch(Fetcher.java:987)2019-11-19
> 05:50:49.817|INFO
> |flash-client-xmc-StreamThread-3|o.a.k.s.p.i.StoreChangelogReader|105|stream-thread
> [flash-client-xmc-StreamThread-3] Reinitializing StreamTask TaskId: 10_1
> ProcessorTopology: KSTREAM-SOURCE-70: topics:
> [flash-app-xmc-worker-share-store-minute-repartition] children:
> [KSTREAM-AGGREGATE-67] KSTREAM-AGGREGATE-67: states:
> [worker-share-store-minute] children: [KTABLE-TOSTREAM-71]
> KTABLE-TOSTREAM-71: children: [KSTREAM-SINK-72]
> KSTREAM-SINK-72: topic:
> StaticTopicNameExtractor(xmc-worker-share-minute)Partitions
> [flash-app-xmc-worker-share-store-minute-repartition-1] for changelog
> flash-app-xmc-worker-share-store-minute-changelog-12019-11-19
> 05:50:49.842|WARN
> |flash-client-xmc-StreamThread-3|o.a.k.s.p.i.StoreChangelogReader|101|stream-thread
> [flash-client-xmc-StreamThread-3] Restoring StreamTasks failed. Deleting
> StreamTasks stores to recreate from
> scratch.org.apache.kafka.clients.consumer.OffsetOutOfRangeException: Offsets
> out of range with no configured reset policy for partitions:
> \{flash-app-xmc-worker-share-store-minute-changelog-1=6128684} at
> org.apache.kafka.clients.consumer.internals.Fetcher.parseCompletedFetch(Fetcher.java:987)2019-11-19
> 05:50:49.842|INFO
> |flash-client-xmc-StreamThread-3|o.a.k.s.p.i.StoreChangelogReader|105|stream-thread
> [flash-client-xmc-StreamThread-3] Reinitializing StreamTask TaskId: 10_1
> ProcessorTopology: KSTREAM-SOURCE-70: topics:
> [flash-app-xmc-worker-share-store-minute-repartition] children:
> [KSTREAM-AGGREGATE-67] KSTREAM-AGGREGATE-67: states:
> [worker-share-store-minute] children: [KTABLE-TOSTREAM-71]
> KTABLE-TOSTREAM-71: children: [KSTREAM-SINK-72]
> KSTREAM-SINK-72: topic:
> StaticTopicNameExtractor(xmc-worker-share-minute)Partitions
> [flash-app-xmc-worker-share-store-minute-repartition-1] for changelog
> flash-app-xmc-worker-share-store-minute-changelog-12019-11-19
> 05:50:49.905|WARN
> |flash-client-xmc-StreamThread-3|o.a.k.s.p.i.StoreChangelogReader|101|stream-thread
> [flash-client-xmc-StreamThread-3] Restoring StreamTasks failed. Deleting
> StreamTasks stores to recreate from
> scratch.org.apache.kafka.clients.consumer.OffsetOutOfRangeException: Offsets
> out of range with no configured reset policy for partitions:
> \{flash-app-xmc-worker-share-store-minute-changelog-1=6128684} at
> org.apache.kafka.clients.consumer.internals.Fetcher.parseCompletedFetch(Fetcher.java:987)2019-11-19
> 05:50:49.906|INFO
> |flash-client-xmc-StreamThread-3|o.a.k.s.p.i.StoreChangelogReader|105|stream-thread
> [flash-client-xmc-StreamThread-3] Reinitializing StreamTask TaskId: 10_1
> ProcessorTopology: KSTREAM-SOURCE-70: topics:
> [flash-app-xmc-worker-share-store-minute-repartition] children:
> [KSTREAM-AGGREGATE-67] KSTREAM-AGGREGATE-67: states:
> [worker-share-store-minute]
> {quote}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9173) StreamsPartitionAssignor assigns partitions to only one worker
[ https://issues.apache.org/jira/browse/KAFKA-9173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16986460#comment-16986460 ] Sophie Blee-Goldman commented on KAFKA-9173: I agree we should aim to fix this, but the workaround here is pretty simple – you just need to decrease the number of threads to better match the actual workload. If your app only has 10 tasks and you want to run 10 instances, each instance only needs one thread. Even if we did "fix" the assignor so that it spread the 10 tasks evenly across the 10 instances, 19 of the 20 threads would have no tasks assigned and nothing to do. You can change num.threads to 1 for each instance, and you will see each be assigned one of the ten tasks > StreamsPartitionAssignor assigns partitions to only one worker > -- > > Key: KAFKA-9173 > URL: https://issues.apache.org/jira/browse/KAFKA-9173 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.3.0, 2.2.1 >Reporter: Oleg Muravskiy >Priority: Major > Labels: user-experience > Attachments: StreamsPartitionAssignor.log > > > I'm running a distributed KafkaStreams application on 10 worker nodes, > subscribed to 21 topics with 10 partitions in each. I'm only using a > Processor interface, and a persistent state store. > However, only one worker gets assigned partitions, all other workers get > nothing. Restarting the application, or cleaning local state stores does not > help. StreamsPartitionAssignor migrates to other nodes, and eventually picks > up other node to assign partitions to, but still only one node. > It's difficult to figure out where to look for the signs of problems, I'm > attaching the log messages from the StreamsPartitionAssignor. Let me know > what else I could provide to help resolve this. > [^StreamsPartitionAssignor.log] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-9225) kafka fail to run on linux-aarch64
[
https://issues.apache.org/jira/browse/KAFKA-9225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sophie Blee-Goldman updated KAFKA-9225:
---
Fix Version/s: 3.0.0
> kafka fail to run on linux-aarch64
> --
>
> Key: KAFKA-9225
> URL: https://issues.apache.org/jira/browse/KAFKA-9225
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Reporter: jiamei xie
>Priority: Major
> Labels: incompatible
> Fix For: 3.0.0
>
> Attachments: compat_report.html
>
>
> *Steps to reproduce:*
> 1. Download Kafka latest source code
> 2. Build it with gradle
> 3. Run
> [streamDemo|[https://kafka.apache.org/23/documentation/streams/quickstart]]
> when running bin/kafka-run-class.sh
> org.apache.kafka.streams.examples.wordcount.WordCountDemo, it crashed with
> the following error message
> {code:java}
> xjm@ubuntu-arm01:~/kafka$bin/kafka-run-class.sh
> org.apache.kafka.streams.examples.wordcount.WordCountDemo
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/core/build/dependant-libs-2.12.10/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/tools/build/dependant-libs-2.12.10/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/connect/api/build/dependant-libs/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/connect/transforms/build/dependant-libs/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/connect/runtime/build/dependant-libs/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/connect/file/build/dependant-libs/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/connect/mirror/build/dependant-libs/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/connect/mirror-client/build/dependant-libs/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/connect/json/build/dependant-libs/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/connect/basic-auth-extension/build/dependant-libs/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> [2019-11-19 15:42:23,277] WARN The configuration 'admin.retries' was supplied
> but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig)
> [2019-11-19 15:42:23,278] WARN The configuration 'admin.retry.backoff.ms' was
> supplied but isn't a known config.
> (org.apache.kafka.clients.consumer.ConsumerConfig)
> [2019-11-19 15:42:24,278] ERROR stream-client
> [streams-wordcount-0f3cf88b-e2c4-4fb6-b7a3-9754fad5cd48] All stream threads
> have died. The instance will be in error state and should be closed.
> (org.apach e.kafka.streams.KafkaStreams)
> Exception in thread
> "streams-wordcount-0f3cf88b-e2c4-4fb6-b7a3-9754fad5cd48-StreamThread-1"
> java.lang.UnsatisfiedLinkError: /tmp/librocksdbjni1377754636857652484.so:
> /tmp/librocksdbjni13777546368576524 84.so:
> cannot open shared object file: No such file or directory (Possible cause:
> can't load AMD 64-bit .so on a AARCH64-bit platform)
> {code}
> *Analyze:*
> This issue is caused by rocksdbjni-5.18.3.jar which doesn't come with aarch64
> native support. Replace rocksdbjni-5.18.3.jar with rocksdbjni-6.3.6.jar from
> [https://mvnrepository.com/artifact/org.rocksdb/rocksdbjni/6.3.6] can fix
> this problem.
> Attached is the binary compatibility report of rocksdbjni.jar between 5.18.3
> and 6.3.6. The result is 81.8%. So is it possible to upgrade rocksdbjni to
> 6.3.6 in upstream? Should there be any kind of tests to execute, please
> kindly point me. Thanks a lot.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-9225) kafka fail to run on linux-aarch64
[
https://issues.apache.org/jira/browse/KAFKA-9225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sophie Blee-Goldman updated KAFKA-9225:
---
Priority: Blocker (was: Major)
> kafka fail to run on linux-aarch64
> --
>
> Key: KAFKA-9225
> URL: https://issues.apache.org/jira/browse/KAFKA-9225
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Reporter: jiamei xie
>Priority: Blocker
> Labels: incompatible
> Fix For: 3.0.0
>
> Attachments: compat_report.html
>
>
> *Steps to reproduce:*
> 1. Download Kafka latest source code
> 2. Build it with gradle
> 3. Run
> [streamDemo|[https://kafka.apache.org/23/documentation/streams/quickstart]]
> when running bin/kafka-run-class.sh
> org.apache.kafka.streams.examples.wordcount.WordCountDemo, it crashed with
> the following error message
> {code:java}
> xjm@ubuntu-arm01:~/kafka$bin/kafka-run-class.sh
> org.apache.kafka.streams.examples.wordcount.WordCountDemo
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/core/build/dependant-libs-2.12.10/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/tools/build/dependant-libs-2.12.10/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/connect/api/build/dependant-libs/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/connect/transforms/build/dependant-libs/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/connect/runtime/build/dependant-libs/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/connect/file/build/dependant-libs/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/connect/mirror/build/dependant-libs/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/connect/mirror-client/build/dependant-libs/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/connect/json/build/dependant-libs/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/connect/basic-auth-extension/build/dependant-libs/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> [2019-11-19 15:42:23,277] WARN The configuration 'admin.retries' was supplied
> but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig)
> [2019-11-19 15:42:23,278] WARN The configuration 'admin.retry.backoff.ms' was
> supplied but isn't a known config.
> (org.apache.kafka.clients.consumer.ConsumerConfig)
> [2019-11-19 15:42:24,278] ERROR stream-client
> [streams-wordcount-0f3cf88b-e2c4-4fb6-b7a3-9754fad5cd48] All stream threads
> have died. The instance will be in error state and should be closed.
> (org.apach e.kafka.streams.KafkaStreams)
> Exception in thread
> "streams-wordcount-0f3cf88b-e2c4-4fb6-b7a3-9754fad5cd48-StreamThread-1"
> java.lang.UnsatisfiedLinkError: /tmp/librocksdbjni1377754636857652484.so:
> /tmp/librocksdbjni13777546368576524 84.so:
> cannot open shared object file: No such file or directory (Possible cause:
> can't load AMD 64-bit .so on a AARCH64-bit platform)
> {code}
> *Analyze:*
> This issue is caused by rocksdbjni-5.18.3.jar which doesn't come with aarch64
> native support. Replace rocksdbjni-5.18.3.jar with rocksdbjni-6.3.6.jar from
> [https://mvnrepository.com/artifact/org.rocksdb/rocksdbjni/6.3.6] can fix
> this problem.
> Attached is the binary compatibility report of rocksdbjni.jar between 5.18.3
> and 6.3.6. The result is 81.8%. So is it possible to upgrade rocksdbjni to
> 6.3.6 in upstream? Should there be any kind of tests to execute, please
> kindly point me. Thanks a lot.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KAFKA-9260) Improve Serde "push down" and "wrapping"
Matthias J. Sax created KAFKA-9260: -- Summary: Improve Serde "push down" and "wrapping" Key: KAFKA-9260 URL: https://issues.apache.org/jira/browse/KAFKA-9260 Project: Kafka Issue Type: Improvement Components: streams Reporter: Matthias J. Sax Kafka Streams DSL supports "serde push down" feature to let downstream operators inherit upstream serdes if key and/or value are not modified. Furthermore, some operators "wrap" user specified serdes internally (eg, windowed aggregations wrap the user specified key-serde with a time/session window serde – some stores use `ValueAndTimestampSerde` and foreign-key joins also uses some internal wrappers). The current implementation faces couple of issues, because the "serde push down" feature is a DSL level feature that is used when the Topology is generated. Furthermore, "serde wrapping" is an operator specific feature, not a DSL concept per-se. At runtime, neither "push down" nor "wrapping" are know concepts. This design implies that if users specify serdes, wrapping and push down works as expected. However, if we fall back to default serdes, special care needs to be taken: for example, some operators not apply the wrapping logic during translation time, and there is additional code that does the wrapping of default serdes as runtime. Another approach would be to wrap a null-Serde, and update the wrapper later (ie, overwrite `null` with the default serde from the config). Overall, the current design leads to bugs (eg, KAFKA-9248 and KAFKA-9259), and user confusion how it actually works and when/where to specify serdes. Hence, we should consider to rework how we do serde push down and/or wrapping. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-9259) suppress() for windowed-Serdes does not work with default serdes
Matthias J. Sax created KAFKA-9259: -- Summary: suppress() for windowed-Serdes does not work with default serdes Key: KAFKA-9259 URL: https://issues.apache.org/jira/browse/KAFKA-9259 Project: Kafka Issue Type: Bug Components: streams Affects Versions: 2.1.0 Reporter: Matthias J. Sax The suppress() operator either inherits serdes from its upstream operator or falls back to default serdes from the config. If the upstream operator is an windowed aggregation, the window-aggregation operator wraps the user passed-in serde with a window-serde and pushed it into suppress() – however, if default serdes are used, the window-aggregation operator cannot push anything into suppress(). At runtime, it just creates a default serde and wraps it according. For this case, suppress() also falls back to default serdes; however, it does not wrap the serde and thus a ClassCastException is thrown when the serde is used later. suppress() is already aware if the upstream aggregation is time/session windowed or not and thus should use this information to wrap default serdes accordingly. The current workaround for windowed-suppress is to overwrite the default serde upstream to suppress(), such that suppress() inherits serdes and does not fall back to default serdes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-9071) transactional.id.expiration.ms config value should be implemented as a Long
[ https://issues.apache.org/jira/browse/KAFKA-9071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16986434#comment-16986434 ] Mario Georgiev commented on KAFKA-9071: --- Can we move the ticket to review? > transactional.id.expiration.ms config value should be implemented as a Long > --- > > Key: KAFKA-9071 > URL: https://issues.apache.org/jira/browse/KAFKA-9071 > Project: Kafka > Issue Type: Improvement > Components: config >Affects Versions: 2.3.0 >Reporter: Joost van de Wijgerd >Assignee: Mario Georgiev >Priority: Major > > Currently the value of this config parameter is limited to MAX_INT > effectively limiting the transactional id expiration to ~ 25 days. This is > causing some issues for us on our Acceptance environment (which is not used > that often / heavily) where our transactional services will start failing > because if this issue. > I believe best practice for millisecond values should be to implement them as > a Long and not as an Integer > this is currently the max value: transactional.id.expiration.ms=2147483647 > while I would like to set it to: transactional.id.expiration.ms=3154000 > (i.e. 1 year) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-9258) Connect ConnectorStatusMetricsGroup sometimes NPE
[ https://issues.apache.org/jira/browse/KAFKA-9258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16986386#comment-16986386 ] ASF GitHub Bot commented on KAFKA-9258: --- cyrusv commented on pull request #7768: KAFKA-9258 Check Connect Metrics non-null in task stop URL: https://github.com/apache/kafka/pull/7768 Connect sometimes will stop task when start() has failed. In these cases, we must guard against NPE as noted in https://issues.apache.org/jira/browse/KAFKA-9258 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Connect ConnectorStatusMetricsGroup sometimes NPE > - > > Key: KAFKA-9258 > URL: https://issues.apache.org/jira/browse/KAFKA-9258 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Reporter: Cyrus Vafadari >Priority: Major > > java.lang.NullPointerException > at > org.apache.kafka.connect.runtime.Worker$ConnectorStatusMetricsGroup.recordTaskRemoved(Worker.java:901) > at > org.apache.kafka.connect.runtime.Worker.awaitStopTask(Worker.java:720) > at > org.apache.kafka.connect.runtime.Worker.awaitStopTasks(Worker.java:740) > at > org.apache.kafka.connect.runtime.Worker.stopAndAwaitTasks(Worker.java:758) > at > org.apache.kafka.connect.runtime.distributed.DistributedHerder.processTaskConfigUpdatesWithIncrementalCooperative(DistributedHerder.java:575) > at > org.apache.kafka.connect.runtime.distributed.DistributedHerder.tick(DistributedHerder.java:396) > at > org.apache.kafka.connect.runtime.distributed.DistributedHerder.run(DistributedHerder.java:282) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266) > at java.util.concurrent.FutureTask.run(FutureTask.java) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-9258) Connect ConnectorStatusMetricsGroup sometimes NPE
Cyrus Vafadari created KAFKA-9258: - Summary: Connect ConnectorStatusMetricsGroup sometimes NPE Key: KAFKA-9258 URL: https://issues.apache.org/jira/browse/KAFKA-9258 Project: Kafka Issue Type: Bug Components: KafkaConnect Reporter: Cyrus Vafadari java.lang.NullPointerException at org.apache.kafka.connect.runtime.Worker$ConnectorStatusMetricsGroup.recordTaskRemoved(Worker.java:901) at org.apache.kafka.connect.runtime.Worker.awaitStopTask(Worker.java:720) at org.apache.kafka.connect.runtime.Worker.awaitStopTasks(Worker.java:740) at org.apache.kafka.connect.runtime.Worker.stopAndAwaitTasks(Worker.java:758) at org.apache.kafka.connect.runtime.distributed.DistributedHerder.processTaskConfigUpdatesWithIncrementalCooperative(DistributedHerder.java:575) at org.apache.kafka.connect.runtime.distributed.DistributedHerder.tick(DistributedHerder.java:396) at org.apache.kafka.connect.runtime.distributed.DistributedHerder.run(DistributedHerder.java:282) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266) at java.util.concurrent.FutureTask.run(FutureTask.java) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-9071) transactional.id.expiration.ms config value should be implemented as a Long
[ https://issues.apache.org/jira/browse/KAFKA-9071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16986351#comment-16986351 ] ASF GitHub Bot commented on KAFKA-9071: --- despondency commented on pull request #7767: KAFKA-9071 transactional.id.expiration.ms config value should be impl… URL: https://github.com/apache/kafka/pull/7767 transactional.id.expiration.ms config value should be implemented as a Long ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > transactional.id.expiration.ms config value should be implemented as a Long > --- > > Key: KAFKA-9071 > URL: https://issues.apache.org/jira/browse/KAFKA-9071 > Project: Kafka > Issue Type: Improvement > Components: config >Affects Versions: 2.3.0 >Reporter: Joost van de Wijgerd >Assignee: Mario Georgiev >Priority: Major > > Currently the value of this config parameter is limited to MAX_INT > effectively limiting the transactional id expiration to ~ 25 days. This is > causing some issues for us on our Acceptance environment (which is not used > that often / heavily) where our transactional services will start failing > because if this issue. > I believe best practice for millisecond values should be to implement them as > a Long and not as an Integer > this is currently the max value: transactional.id.expiration.ms=2147483647 > while I would like to set it to: transactional.id.expiration.ms=3154000 > (i.e. 1 year) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KAFKA-9071) transactional.id.expiration.ms config value should be implemented as a Long
[ https://issues.apache.org/jira/browse/KAFKA-9071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mario Georgiev reassigned KAFKA-9071: - Assignee: Mario Georgiev > transactional.id.expiration.ms config value should be implemented as a Long > --- > > Key: KAFKA-9071 > URL: https://issues.apache.org/jira/browse/KAFKA-9071 > Project: Kafka > Issue Type: Improvement > Components: config >Affects Versions: 2.3.0 >Reporter: Joost van de Wijgerd >Assignee: Mario Georgiev >Priority: Major > > Currently the value of this config parameter is limited to MAX_INT > effectively limiting the transactional id expiration to ~ 25 days. This is > causing some issues for us on our Acceptance environment (which is not used > that often / heavily) where our transactional services will start failing > because if this issue. > I believe best practice for millisecond values should be to implement them as > a Long and not as an Integer > this is currently the max value: transactional.id.expiration.ms=2147483647 > while I would like to set it to: transactional.id.expiration.ms=3154000 > (i.e. 1 year) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-9156) LazyTimeIndex & LazyOffsetIndex may cause niobufferoverflow in concurrent state
[ https://issues.apache.org/jira/browse/KAFKA-9156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16986321#comment-16986321 ] ASF GitHub Bot commented on KAFKA-9156: --- ijuma commented on pull request #7760: KAFKA-9156: fix LazyTimeIndex & LazyOffsetIndex concurrency URL: https://github.com/apache/kafka/pull/7760 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > LazyTimeIndex & LazyOffsetIndex may cause niobufferoverflow in concurrent > state > --- > > Key: KAFKA-9156 > URL: https://issues.apache.org/jira/browse/KAFKA-9156 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 2.3.0, 2.3.1 >Reporter: shilin Lu >Assignee: Alex Mironov >Priority: Blocker > Labels: regression > Fix For: 2.4.0, 2.3.2 > > Attachments: image-2019-11-07-17-42-13-852.png, > image-2019-11-07-17-44-05-357.png, image-2019-11-07-17-46-53-650.png > > > !image-2019-11-07-17-42-13-852.png! > this timeindex get function is not thread safe ,may cause create some > timeindex. > !image-2019-11-07-17-44-05-357.png! > When create timeindex not exactly one ,may cause mappedbytebuffer position to > end. Then write index entry to this mmap file will cause > java.nio.BufferOverflowException. > > !image-2019-11-07-17-46-53-650.png! > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-5034) Connect workers don't discover new Connector Plugins without Restart
[ https://issues.apache.org/jira/browse/KAFKA-5034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16986252#comment-16986252 ] ASF GitHub Bot commented on KAFKA-5034: --- C0urante commented on pull request #4905: KAFKA-5034: Enable plugins to be added to the plugin path at runtime URL: https://github.com/apache/kafka/pull/4905 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Connect workers don't discover new Connector Plugins without Restart > > > Key: KAFKA-5034 > URL: https://issues.apache.org/jira/browse/KAFKA-5034 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Reporter: Gwen Shapira >Priority: Major > > If I want to add a new Connector Plugin to a running distributed Connect > cluster, I need to copy the JAR to the classpath and then restart all the > workers so they will pick up the new plugin before I can create a connector. > This is both un-intuitive (most modern up can pick up changes dynamically) > and can get difficult when a connect cluster is shared between teams. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KAFKA-9156) LazyTimeIndex & LazyOffsetIndex may cause niobufferoverflow in concurrent state
[ https://issues.apache.org/jira/browse/KAFKA-9156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Mironov reassigned KAFKA-9156: --- Assignee: Alex Mironov > LazyTimeIndex & LazyOffsetIndex may cause niobufferoverflow in concurrent > state > --- > > Key: KAFKA-9156 > URL: https://issues.apache.org/jira/browse/KAFKA-9156 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 2.3.0, 2.3.1 >Reporter: shilin Lu >Assignee: Alex Mironov >Priority: Blocker > Labels: regression > Fix For: 2.4.0, 2.3.2 > > Attachments: image-2019-11-07-17-42-13-852.png, > image-2019-11-07-17-44-05-357.png, image-2019-11-07-17-46-53-650.png > > > !image-2019-11-07-17-42-13-852.png! > this timeindex get function is not thread safe ,may cause create some > timeindex. > !image-2019-11-07-17-44-05-357.png! > When create timeindex not exactly one ,may cause mappedbytebuffer position to > end. Then write index entry to this mmap file will cause > java.nio.BufferOverflowException. > > !image-2019-11-07-17-46-53-650.png! > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-9164) Don't evict active topics' metadata from the producer's cache
[ https://issues.apache.org/jira/browse/KAFKA-9164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brian Byrne resolved KAFKA-9164. Resolution: Invalid Marking issue invalid since the producer logic was actually handling this correctly. > Don't evict active topics' metadata from the producer's cache > - > > Key: KAFKA-9164 > URL: https://issues.apache.org/jira/browse/KAFKA-9164 > Project: Kafka > Issue Type: Bug > Components: producer >Reporter: Brian Byrne >Assignee: Brian Byrne >Priority: Minor > > The producer's metadata currently marks a topic as "not needing to be > retained" if it has been 5 minutes since it was first considered, regardless > of whether records were being actively produced for the topic. This shouldn't > happen and should be fixed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KAFKA-9255) MessageSet v1 protocol wrong specification
[ https://issues.apache.org/jira/browse/KAFKA-9255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mickael Maison reassigned KAFKA-9255: - Assignee: Fábio Silva > MessageSet v1 protocol wrong specification > -- > > Key: KAFKA-9255 > URL: https://issues.apache.org/jira/browse/KAFKA-9255 > Project: Kafka > Issue Type: Bug > Components: documentation >Reporter: Fábio Silva >Assignee: Fábio Silva >Priority: Major > > The documentation contains a BNF specification missing the timestamp field on > 'message' field entry. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-9255) MessageSet v1 protocol wrong specification
[ https://issues.apache.org/jira/browse/KAFKA-9255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mickael Maison resolved KAFKA-9255. --- Fix Version/s: 2.5.0 Resolution: Fixed > MessageSet v1 protocol wrong specification > -- > > Key: KAFKA-9255 > URL: https://issues.apache.org/jira/browse/KAFKA-9255 > Project: Kafka > Issue Type: Bug > Components: documentation >Reporter: Fábio Silva >Assignee: Fábio Silva >Priority: Major > Fix For: 2.5.0 > > > The documentation contains a BNF specification missing the timestamp field on > 'message' field entry. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-9255) MessageSet v1 protocol wrong specification
[ https://issues.apache.org/jira/browse/KAFKA-9255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16986099#comment-16986099 ] ASF GitHub Bot commented on KAFKA-9255: --- mimaison commented on pull request #7764: KAFKA-9255: MessageSet v1 protocol wrong specification URL: https://github.com/apache/kafka/pull/7764 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > MessageSet v1 protocol wrong specification > -- > > Key: KAFKA-9255 > URL: https://issues.apache.org/jira/browse/KAFKA-9255 > Project: Kafka > Issue Type: Bug > Components: documentation >Reporter: Fábio Silva >Priority: Major > > The documentation contains a BNF specification missing the timestamp field on > 'message' field entry. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (KAFKA-9254) Topic level configuration failed
[
https://issues.apache.org/jira/browse/KAFKA-9254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16986025#comment-16986025
]
fenghong edited comment on KAFKA-9254 at 12/2/19 12:43 PM:
---
We can reproduce this bug with the latest version (2.3.1) and this version
(2.0.1)
Write data continuously during testing
OS
{code:java}
Linux 4.15.0-70-generic #79~16.04.1-Ubuntu SMP Tue Nov 12 14:01:10 UTC 2019
x86_64 x86_64 x86_64 GNU/Linux
{code}
JAVA
{code:java}
java version "1.8.0_192"
Java(TM) SE Runtime Environment (build 1.8.0_192-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.192-b12, mixed mode)
{code}
server.properties
{code:java}
broker.id=4
delete.topic.enable=true
listeners=PLAINTEXT://xxx:9092
num.network.threads=9
num.io.threads=12
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/data/data/kafka,/data2/data/kafka,/data3/data/kafka,/data4/data/kafka
num.partitions=1
num.recovery.threads.per.data.dir=2
offsets.topic.replication.factor=5
transaction.state.log.replication.factor=5
transaction.state.log.min.isr=3
log.retention.hours=26280
log.segment.bytes=10737418
log.retention.check.interval.ms=30
zookeeper.connect=xxx1:2181,xxx2:2181,xxx3:2181,xxx4:2181,xxx5:2181
zookeeper.connection.timeout.ms=6000
zookeeper.session.timeout.ms=6000
group.initial.rebalance.delay.ms=500
default.replication.factor=3
min.insync.replicas=3
auto.create.topics.enable=false
unclean.leader.election.enable=false
log.cleanup.policy=delete
num.replica.fetchers=2
replica.lag.time.max.ms=1
replica.fetch.wait.max.ms=500
{code}
was (Author: fenghong):
We can reproduce this bug with the latest version (2.3.1) and this version
(2.0.1)
Need to continue writing data during testing
OS
{code:java}
Linux 4.15.0-70-generic #79~16.04.1-Ubuntu SMP Tue Nov 12 14:01:10 UTC 2019
x86_64 x86_64 x86_64 GNU/Linux
{code}
JAVA
{code:java}
java version "1.8.0_192"
Java(TM) SE Runtime Environment (build 1.8.0_192-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.192-b12, mixed mode)
{code}
server.properties
{code:java}
broker.id=4
delete.topic.enable=true
listeners=PLAINTEXT://xxx:9092
num.network.threads=9
num.io.threads=12
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/data/data/kafka,/data2/data/kafka,/data3/data/kafka,/data4/data/kafka
num.partitions=1
num.recovery.threads.per.data.dir=2
offsets.topic.replication.factor=5
transaction.state.log.replication.factor=5
transaction.state.log.min.isr=3
log.retention.hours=26280
log.segment.bytes=10737418
log.retention.check.interval.ms=30
zookeeper.connect=xxx1:2181,xxx2:2181,xxx3:2181,xxx4:2181,xxx5:2181
zookeeper.connection.timeout.ms=6000
zookeeper.session.timeout.ms=6000
group.initial.rebalance.delay.ms=500
default.replication.factor=3
min.insync.replicas=3
auto.create.topics.enable=false
unclean.leader.election.enable=false
log.cleanup.policy=delete
num.replica.fetchers=2
replica.lag.time.max.ms=1
replica.fetch.wait.max.ms=500
{code}
> Topic level configuration failed
>
>
> Key: KAFKA-9254
> URL: https://issues.apache.org/jira/browse/KAFKA-9254
> Project: Kafka
> Issue Type: Bug
> Components: config, log, replication
>Affects Versions: 2.0.1
>Reporter: fenghong
>Priority: Critical
>
> We are engineers at Huobi and now encounter Kafka BUG
> Modifying DynamicBrokerConfig more than 2 times will invalidate the topic
> level unrelated configuration
> The bug reproduction method as follows：
> # Set Kafka Broker config server.properties min.insync.replicas=3
> # Create topic test-1 and set topic‘s level config min.insync.replicas=2
> # Dynamically modify the configuration twice as shown below
> {code:java}
> bin/kafka-configs.sh --bootstrap-server xxx:9092 --entity-type brokers
> --entity-default --alter --add-config log.message.timestamp.type=LogAppendTime
> bin/kafka-configs.sh --bootstrap-server xxx:9092 --entity-type brokers
> --entity-default --alter --add-config log.retention.ms=60480
> {code}
> # stop a Kafka Server and found the Exception as shown below
> org.apache.kafka.common.errors.NotEnoughReplicasException: Number of insync
> replicas for partition test-1-0 is [2], below required minimum [3]
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (KAFKA-9254) Topic level configuration failed
[
https://issues.apache.org/jira/browse/KAFKA-9254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16986025#comment-16986025
]
fenghong edited comment on KAFKA-9254 at 12/2/19 12:39 PM:
---
We can reproduce this bug with the latest version (2.3.1) and this version
(2.0.1)
Need to continue writing data during testing
OS
{code:java}
Linux 4.15.0-70-generic #79~16.04.1-Ubuntu SMP Tue Nov 12 14:01:10 UTC 2019
x86_64 x86_64 x86_64 GNU/Linux
{code}
JAVA
{code:java}
java version "1.8.0_192"
Java(TM) SE Runtime Environment (build 1.8.0_192-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.192-b12, mixed mode)
{code}
server.properties
{code:java}
broker.id=4
delete.topic.enable=true
listeners=PLAINTEXT://xxx:9092
num.network.threads=9
num.io.threads=12
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/data/data/kafka,/data2/data/kafka,/data3/data/kafka,/data4/data/kafka
num.partitions=1
num.recovery.threads.per.data.dir=2
offsets.topic.replication.factor=5
transaction.state.log.replication.factor=5
transaction.state.log.min.isr=3
log.retention.hours=26280
log.segment.bytes=10737418
log.retention.check.interval.ms=30
zookeeper.connect=xxx1:2181,xxx2:2181,xxx3:2181,xxx4:2181,xxx5:2181
zookeeper.connection.timeout.ms=6000
zookeeper.session.timeout.ms=6000
group.initial.rebalance.delay.ms=500
default.replication.factor=3
min.insync.replicas=3
auto.create.topics.enable=false
unclean.leader.election.enable=false
log.cleanup.policy=delete
num.replica.fetchers=2
replica.lag.time.max.ms=1
replica.fetch.wait.max.ms=500
{code}
was (Author: fenghong):
We can reproduce this bug with the latest version (2.3.1) and this version
(2.0.1)
Need to continue writing data during testing
OS
{code:java}
Linux 4.15.0-70-generic #79~16.04.1-Ubuntu SMP Tue Nov 12 14:01:10 UTC 2019
x86_64 x86_64 x86_64 GNU/Linux
{code}
JAVA
{code:java}
java version "1.8.0_192"
Java(TM) SE Runtime Environment (build 1.8.0_192-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.192-b12, mixed mode)
{code}
Server.propertity
{code:java}
broker.id=4
delete.topic.enable=true
listeners=PLAINTEXT://xxx:9092
num.network.threads=9
num.io.threads=12
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/data/data/kafka,/data2/data/kafka,/data3/data/kafka,/data4/data/kafka
num.partitions=1
num.recovery.threads.per.data.dir=2
offsets.topic.replication.factor=5
transaction.state.log.replication.factor=5
transaction.state.log.min.isr=3
log.retention.hours=26280
log.segment.bytes=10737418
log.retention.check.interval.ms=30
zookeeper.connect=xxx1:2181,xxx2:2181,xxx3:2181,xxx4:2181,xxx5:2181
zookeeper.connection.timeout.ms=6000
zookeeper.session.timeout.ms=6000
group.initial.rebalance.delay.ms=500
default.replication.factor=3
min.insync.replicas=3
auto.create.topics.enable=false
unclean.leader.election.enable=false
log.cleanup.policy=delete
num.replica.fetchers=2
replica.lag.time.max.ms=1
replica.fetch.wait.max.ms=500
{code}
> Topic level configuration failed
>
>
> Key: KAFKA-9254
> URL: https://issues.apache.org/jira/browse/KAFKA-9254
> Project: Kafka
> Issue Type: Bug
> Components: config, log, replication
>Affects Versions: 2.0.1
>Reporter: fenghong
>Priority: Critical
>
> We are engineers at Huobi and now encounter Kafka BUG
> Modifying DynamicBrokerConfig more than 2 times will invalidate the topic
> level unrelated configuration
> The bug reproduction method as follows：
> # Set Kafka Broker config server.properties min.insync.replicas=3
> # Create topic test-1 and set topic‘s level config min.insync.replicas=2
> # Dynamically modify the configuration twice as shown below
> {code:java}
> bin/kafka-configs.sh --bootstrap-server xxx:9092 --entity-type brokers
> --entity-default --alter --add-config log.message.timestamp.type=LogAppendTime
> bin/kafka-configs.sh --bootstrap-server xxx:9092 --entity-type brokers
> --entity-default --alter --add-config log.retention.ms=60480
> {code}
> # stop a Kafka Server and found the Exception as shown below
> org.apache.kafka.common.errors.NotEnoughReplicasException: Number of insync
> replicas for partition test-1-0 is [2], below required minimum [3]
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (KAFKA-9254) Topic level configuration failed
[
https://issues.apache.org/jira/browse/KAFKA-9254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16986025#comment-16986025
]
fenghong edited comment on KAFKA-9254 at 12/2/19 12:38 PM:
---
We can reproduce this bug with the latest version (2.3.1) and this version
(2.0.1)
Need to continue writing data during testing
OS
{code:java}
Linux 4.15.0-70-generic #79~16.04.1-Ubuntu SMP Tue Nov 12 14:01:10 UTC 2019
x86_64 x86_64 x86_64 GNU/Linux
{code}
JAVA
{code:java}
java version "1.8.0_192"
Java(TM) SE Runtime Environment (build 1.8.0_192-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.192-b12, mixed mode)
{code}
Server.propertity
{code:java}
broker.id=4
delete.topic.enable=true
listeners=PLAINTEXT://xxx:9092
num.network.threads=9
num.io.threads=12
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/data/data/kafka,/data2/data/kafka,/data3/data/kafka,/data4/data/kafka
num.partitions=1
num.recovery.threads.per.data.dir=2
offsets.topic.replication.factor=5
transaction.state.log.replication.factor=5
transaction.state.log.min.isr=3
log.retention.hours=26280
log.segment.bytes=10737418
log.retention.check.interval.ms=30
zookeeper.connect=xxx1:2181,xxx2:2181,xxx3:2181,xxx4:2181,xxx5:2181
zookeeper.connection.timeout.ms=6000
zookeeper.session.timeout.ms=6000
group.initial.rebalance.delay.ms=500
default.replication.factor=3
min.insync.replicas=3
auto.create.topics.enable=false
unclean.leader.election.enable=false
log.cleanup.policy=delete
num.replica.fetchers=2
replica.lag.time.max.ms=1
replica.fetch.wait.max.ms=500
{code}
was (Author: fenghong):
We can reproduce this bug with the latest version (2.3.1)
Need to continue writing data during testing
OS
{code:java}
Linux 4.15.0-70-generic #79~16.04.1-Ubuntu SMP Tue Nov 12 14:01:10 UTC 2019
x86_64 x86_64 x86_64 GNU/Linux
{code}
JAVA
{code:java}
java version "1.8.0_192"
Java(TM) SE Runtime Environment (build 1.8.0_192-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.192-b12, mixed mode)
{code}
Server.propertity
{code:java}
broker.id=4
delete.topic.enable=true
listeners=PLAINTEXT://xxx:9092
num.network.threads=9
num.io.threads=12
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/data/data/kafka,/data2/data/kafka,/data3/data/kafka,/data4/data/kafka
num.partitions=1
num.recovery.threads.per.data.dir=2
offsets.topic.replication.factor=5
transaction.state.log.replication.factor=5
transaction.state.log.min.isr=3
log.retention.hours=26280
log.segment.bytes=10737418
log.retention.check.interval.ms=30
zookeeper.connect=xxx1:2181,xxx2:2181,xxx3:2181,xxx4:2181,xxx5:2181
zookeeper.connection.timeout.ms=6000
zookeeper.session.timeout.ms=6000
group.initial.rebalance.delay.ms=500
default.replication.factor=3
min.insync.replicas=3
auto.create.topics.enable=false
unclean.leader.election.enable=false
log.cleanup.policy=delete
num.replica.fetchers=2
replica.lag.time.max.ms=1
replica.fetch.wait.max.ms=500
{code}
> Topic level configuration failed
>
>
> Key: KAFKA-9254
> URL: https://issues.apache.org/jira/browse/KAFKA-9254
> Project: Kafka
> Issue Type: Bug
> Components: config, log, replication
>Affects Versions: 2.0.1
>Reporter: fenghong
>Priority: Critical
>
> We are engineers at Huobi and now encounter Kafka BUG
> Modifying DynamicBrokerConfig more than 2 times will invalidate the topic
> level unrelated configuration
> The bug reproduction method as follows：
> # Set Kafka Broker config server.properties min.insync.replicas=3
> # Create topic test-1 and set topic‘s level config min.insync.replicas=2
> # Dynamically modify the configuration twice as shown below
> {code:java}
> bin/kafka-configs.sh --bootstrap-server xxx:9092 --entity-type brokers
> --entity-default --alter --add-config log.message.timestamp.type=LogAppendTime
> bin/kafka-configs.sh --bootstrap-server xxx:9092 --entity-type brokers
> --entity-default --alter --add-config log.retention.ms=60480
> {code}
> # stop a Kafka Server and found the Exception as shown below
> org.apache.kafka.common.errors.NotEnoughReplicasException: Number of insync
> replicas for partition test-1-0 is [2], below required minimum [3]
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9254) Topic level configuration failed
[
https://issues.apache.org/jira/browse/KAFKA-9254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16986025#comment-16986025
]
fenghong commented on KAFKA-9254:
-
We can reproduce this bug with the latest version (2.3.1)
Need to continue writing data during testing
OS
{code:java}
Linux 4.15.0-70-generic #79~16.04.1-Ubuntu SMP Tue Nov 12 14:01:10 UTC 2019
x86_64 x86_64 x86_64 GNU/Linux
{code}
JAVA
{code:java}
java version "1.8.0_192"
Java(TM) SE Runtime Environment (build 1.8.0_192-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.192-b12, mixed mode)
{code}
Server.propertity
{code:java}
broker.id=4
delete.topic.enable=true
listeners=PLAINTEXT://xxx:9092
num.network.threads=9
num.io.threads=12
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/data/data/kafka,/data2/data/kafka,/data3/data/kafka,/data4/data/kafka
num.partitions=1
num.recovery.threads.per.data.dir=2
offsets.topic.replication.factor=5
transaction.state.log.replication.factor=5
transaction.state.log.min.isr=3
log.retention.hours=26280
log.segment.bytes=10737418
log.retention.check.interval.ms=30
zookeeper.connect=xxx1:2181,xxx2:2181,xxx3:2181,xxx4:2181,xxx5:2181
zookeeper.connection.timeout.ms=6000
zookeeper.session.timeout.ms=6000
group.initial.rebalance.delay.ms=500
default.replication.factor=3
min.insync.replicas=3
auto.create.topics.enable=false
unclean.leader.election.enable=false
log.cleanup.policy=delete
num.replica.fetchers=2
replica.lag.time.max.ms=1
replica.fetch.wait.max.ms=500
{code}
> Topic level configuration failed
>
>
> Key: KAFKA-9254
> URL: https://issues.apache.org/jira/browse/KAFKA-9254
> Project: Kafka
> Issue Type: Bug
> Components: config, log, replication
>Affects Versions: 2.0.1
>Reporter: fenghong
>Priority: Critical
>
> We are engineers at Huobi and now encounter Kafka BUG
> Modifying DynamicBrokerConfig more than 2 times will invalidate the topic
> level unrelated configuration
> The bug reproduction method as follows：
> # Set Kafka Broker config server.properties min.insync.replicas=3
> # Create topic test-1 and set topic‘s level config min.insync.replicas=2
> # Dynamically modify the configuration twice as shown below
> {code:java}
> bin/kafka-configs.sh --bootstrap-server xxx:9092 --entity-type brokers
> --entity-default --alter --add-config log.message.timestamp.type=LogAppendTime
> bin/kafka-configs.sh --bootstrap-server xxx:9092 --entity-type brokers
> --entity-default --alter --add-config log.retention.ms=60480
> {code}
> # stop a Kafka Server and found the Exception as shown below
> org.apache.kafka.common.errors.NotEnoughReplicasException: Number of insync
> replicas for partition test-1-0 is [2], below required minimum [3]
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9187) kafka.api.SaslGssapiSslEndToEndAuthorizationTest.testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl
[
https://issues.apache.org/jira/browse/KAFKA-9187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16986016#comment-16986016
]
Bruno Cadonna commented on KAFKA-9187:
--
https://builds.apache.org/job/kafka-pr-jdk11-scala2.12/9656/testReport/junit/kafka.api/SaslPlainSslEndToEndAuthorizationTest/testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl/
> kafka.api.SaslGssapiSslEndToEndAuthorizationTest.testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl
> --
>
> Key: KAFKA-9187
> URL: https://issues.apache.org/jira/browse/KAFKA-9187
> Project: Kafka
> Issue Type: Test
> Components: core
>Reporter: Bill Bejeck
>Priority: Major
> Labels: flaky-test
>
> Failed in [https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/26593/]
>
> {noformat}
> Error Messageorg.scalatest.exceptions.TestFailedException: Consumed 0 records
> before timeout instead of the expected 1
> recordsStacktraceorg.scalatest.exceptions.TestFailedException: Consumed 0
> records before timeout instead of the expected 1 records
> at
> org.scalatest.Assertions$class.newAssertionFailedException(Assertions.scala:530)
> at
> org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1389)
> at org.scalatest.Assertions$class.fail(Assertions.scala:1091)
> at org.scalatest.Assertions$.fail(Assertions.scala:1389)
> at kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:842)
> at kafka.utils.TestUtils$.pollRecordsUntilTrue(TestUtils.scala:793)
> at
> kafka.utils.TestUtils$.pollUntilAtLeastNumRecords(TestUtils.scala:1334)
> at kafka.utils.TestUtils$.consumeRecords(TestUtils.scala:1343)
> at
> kafka.api.EndToEndAuthorizationTest.consumeRecords(EndToEndAuthorizationTest.scala:530)
> at
> kafka.api.EndToEndAuthorizationTest.testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl(EndToEndAuthorizationTest.scala:369)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:365)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:330)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:78)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:328)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:65)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:292)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:412)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
> at
> org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
> at
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
> at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImp
[jira] [Created] (KAFKA-9257) Allow listeners to bind on same port but different interfaces
Gérald Quintana created KAFKA-9257:
--
Summary: Allow listeners to bind on same port but different
interfaces
Key: KAFKA-9257
URL: https://issues.apache.org/jira/browse/KAFKA-9257
Project: Kafka
Issue Type: Improvement
Components: network
Affects Versions: 2.3.1
Reporter: Gérald Quintana
It's not possible to bind two listeners to two distinct network interfaces
using the same port
{code:java}
listeners=ONE://:9092,TWO://:9092
advertised.listeners=ONE://:9092,TWO://:9092
listener.security.protocol.map=ONE:PLAINTEXT,TWO:PLAINTEXT{code}
Kafka server returns an error:
{noformat}
[2019-12-02 13:07:18,256] ERROR Exiting Kafka due to fatal exception
(kafka.Kafka$)
java.lang.IllegalArgumentException: requirement failed: Each listener must
have a different port, listeners: ONE://:9092,TWO://:9092
at scala.Predef$.require(Predef.scala:281)
at kafka.utils.CoreUtils$.validate$1(CoreUtils.scala:305)
at kafka.utils.CoreUtils$.listenerListToEndPoints(CoreUtils.scala:316)
at kafka.server.KafkaConfig.advertisedListeners(KafkaConfig.scala:1407)
at kafka.server.KafkaConfig.validateValues(KafkaConfig.scala:1482)
at kafka.server.KafkaConfig.(KafkaConfig.scala:1460)
at kafka.server.KafkaConfig.(KafkaConfig.scala:1114)
at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:1094)
at kafka.server.KafkaServerStartable$.fromProps(KafkaServerStartable.scala:28)
at kafka.Kafka$.main(Kafka.scala:68)
at kafka.Kafka.main(Kafka.scala){noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-6291) Cannot close EmbeddedZookeeper on Windows
[
https://issues.apache.org/jira/browse/KAFKA-6291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16986003#comment-16986003
]
Wahid Gazzah commented on KAFKA-6291:
-
Any update on this issue please ?
I'm also using Windows os,
> Cannot close EmbeddedZookeeper on Windows
> -
>
> Key: KAFKA-6291
> URL: https://issues.apache.org/jira/browse/KAFKA-6291
> Project: Kafka
> Issue Type: Bug
> Components: zkclient
>Affects Versions: 0.11.0.0, 1.0.0
> Environment: Windows 10 (doesn't reproduce on Linux)
> JDK 8
>Reporter: Viliam Durina
>Priority: Major
> Labels: windows
>
> We created {{EmbeddedZookeeper}} and {{ZkClient}} for various tests using
> this code:
> {code:java}
> EmbeddedZookeeper zkServer = new EmbeddedZookeeper();
> ZkClient zkClient = new ZkClient("127.0.0.1:" + zkServer.port(),
> 3, 3, ZKStringSerializer$.MODULE$);
> zkClient.close();
> zkServer.shutdown();
> {code}
> This works fine on Linux, but on Windows, the {{zkServer.shutdown()}} call
> fails with this exception:
> {code}
> [Thread-1] ERROR org.apache.kafka.test.TestUtils - Error deleting
> C:\Users\vilo\AppData\Local\Temp\kafka-5521621147877426083
> java.nio.file.FileSystemException:
> C:\Users\vilo\AppData\Local\Temp\kafka-5521621147877426083\version-2\log.1:
> The process cannot access the file because it is being used by another
> process.
> at
> sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
> at
> sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
> at
> sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
> at
> sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:269)
> at
> sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
> at java.nio.file.Files.delete(Files.java:1126)
> at org.apache.kafka.common.utils.Utils$1.visitFile(Utils.java:630)
> at org.apache.kafka.common.utils.Utils$1.visitFile(Utils.java:619)
> at java.nio.file.Files.walkFileTree(Files.java:2670)
> at java.nio.file.Files.walkFileTree(Files.java:2742)
> at org.apache.kafka.common.utils.Utils.delete(Utils.java:619)
> at org.apache.kafka.test.TestUtils$1.run(TestUtils.java:184)
> java.nio.file.FileSystemException:
> C:\Users\vilo\AppData\Local\Temp\kafka-5521621147877426083\version-2\log.1:
> The process cannot access the file because it is being used by another
> process.
> at
> sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
> at
> sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
> at
> sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
> at
> sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:269)
> at
> sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
> at java.nio.file.Files.delete(Files.java:1126)
> at org.apache.kafka.common.utils.Utils$1.visitFile(Utils.java:630)
> at org.apache.kafka.common.utils.Utils$1.visitFile(Utils.java:619)
> at java.nio.file.Files.walkFileTree(Files.java:2670)
> at java.nio.file.Files.walkFileTree(Files.java:2742)
> at org.apache.kafka.common.utils.Utils.delete(Utils.java:619)
> at kafka.zk.EmbeddedZookeeper.shutdown(EmbeddedZookeeper.scala:53)
> at com.hazelcast.jet.KafkaTest.test(KafkaTest.java:32)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(
[jira] [Updated] (KAFKA-9256) mirrorMaker + kafka = java.lang.NoSuchMethodError: org.apache.kafka.common.requests.MetadataResponse.prepareResponse
[
https://issues.apache.org/jira/browse/KAFKA-9256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rudik updated KAFKA-9256:
-
Description:
kafkaBroker - 2.3.0
java - openjdk version "11.0.2" 2019-01-15
kafka cluster from = \{broker - os-4603:9092,os-4604:9092,os-4605:9092, auth -
plain } \{zookeeper - os-4595:2181,os-4596:2181,os-4599:2181}
kafka cluster to = \{broker - os-4804:9092,os-4805:9092,os-4806:9092, auth -
plain} \{zookeeper - os-4804:2181,os-4805:2181,os-4806:2181}
[kafka@os-4804 kafka]$ cat config/mirrormaker_consumer_pdc.properties
zookeeper.connect=os-4595:2181,os-4596:2181,os-4599:2181
bootstrap.servers=os-4603:9092,os-4604:9092,os-4605:9092
group.id=dp-MirrorMaker-group
exclude.internal.topics=true
#mirror.topics.whitelist=.*
client.id=mirror_maker_consumer
[kafka@os-4804 kafka]$ cat config/mirrormaker_producer.properties
zookeeper.connect=os-4804:2181,os-4805:2181,os-4806:2181
bootstrap.servers=os-4804:9092,os-4805:9092,os-4806:9092
acks=1
batch.size=100
client.id=mirror_maker_producer
Start MirrorMaker:
[kafka@os-4804 kafka]$ bin/kafka-mirror-maker.sh --consumer.config
./config/mirrormaker_consumer_pdc.properties --num.streams 2 --producer.config
./config/mirrormaker_producer.properties --whitelist="connect-configs,
connect-offsets, connect-status, last_topic"
WARNING: The default partition assignment strategy of the mirror maker will
change from 'range' to 'roundrobin' in an upcoming release (so that better load
balancing can be achiev ed). If you prefer to make this switch in advance of
that release add the following to the corresponding config:
'partition.assignment.strategy=org.apache.kafka.clients.consumer.R
oundRobinAssignor'
[2019-12-02 13:50:45,391] WARN The configuration 'zookeeper.connect' was
supplied but isn't a known config.
(org.apache.kafka.clients.producer.ProducerConfig)
[2019-12-02 13:50:45,483] WARN The configuration 'zookeeper.connect' was
supplied but isn't a known config.
(org.apache.kafka.clients.consumer.ConsumerConfig)
[2019-12-02 13:50:45,508] WARN The configuration 'zookeeper.connect' was
supplied but isn't a known config.
(org.apache.kafka.clients.consumer.ConsumerConfig)
[2019-12-02 13:50:45,749] WARN [Consumer clientId=dp-MirrorMaker-group-0,
groupId=dp-MirrorMaker-group] Connection to node -2 (os-4604/10.6.107.93:9092)
could not be established. Broker may not be available.
(org.apache.kafka.clients.NetworkClient)
[2019-12-02 13:50:45,750] WARN [Consumer clientId=dp-MirrorMaker-group-1,
groupId=dp-MirrorMaker-group] Connection to node -2 (os-4604/10.6.107.93:9092)
could not be established. Broker may not be available.
(org.apache.kafka.clients.NetworkClient)
[2019-12-02 13:50:45,758] WARN [Consumer clientId=dp-MirrorMaker-group-0,
groupId=dp-MirrorMaker-group] Connection to node -3 (os-4605/10.6.107.94:9092)
could not be established. Broker may not be available.
(org.apache.kafka.clients.NetworkClient)
logs in KafkaBroker:
[2019-12-02 13:51:54,020] ERROR [KafkaApi-1] Error when handling request:
clientId=dp-MirrorMaker-group-1, correlationId=269, api=METADATA,
body=\{topics=null,allow_auto_topic_crea
tion=true,include_cluster_authorized_operations=false,include_topic_authorized_operations=false}
(kafka.server.KafkaApis)
java.lang.NoSuchMethodError:
org.apache.kafka.common.requests.MetadataResponse.prepareResponse(ILjava/util/List;Ljava/lang/String;ILjava/util/List;I)Lorg/apache/kafka/common/reque
sts/MetadataResponse;
at
kafka.server.KafkaApis.$anonfun$handleTopicMetadataRequest$7(KafkaApis.scala:1103)
at
kafka.server.KafkaApis.$anonfun$handleTopicMetadataRequest$7$adapted(KafkaApis.scala:1096)
at kafka.server.KafkaApis.sendResponseMaybeThrottle(KafkaApis.scala:2539)
at kafka.server.KafkaApis.handleTopicMetadataRequest(KafkaApis.scala:1096)
at kafka.server.KafkaApis.handle(KafkaApis.scala:116)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:69)
at java.base/java.lang.Thread.run(Thread.java:834)
was:
kafkaBroker - 2.3.0
java - openjdk version "11.0.2" 2019-01-15
kafka cluster from = \{broker - os-4603:9092,os-4604:9092,os-4605:9092, auth -
plain } \{zookeeper - os-4595:2181,os-4596:2181,os-4599:2181}
kafka cluster to = \{broker - os-4804:9092,os-4805:9092,os-4806:9092, auth -
plain} \{zookeeper - os-4804:2181,os-4805:2181,os-4806:2181}
Start MirrorMaker:
[kafka@os-4804 kafka]$ bin/kafka-mirror-maker.sh --consumer.config
./config/mirrormaker_consumer_pdc.properties --num.streams 2 --producer.config
./config/mirrormaker_producer.pr operties --whitelist="connect-configs,
connect-offsets, connect-status, last_topic"
WARNING: The default partition assignment strategy of the mirror maker will
change from 'range' to 'roundrobin' in an upcoming release (so that better load
balancing can be achiev ed). If you prefer to make this switch in advance of
that release add the following to the correspond
[jira] [Created] (KAFKA-9256) mirrorMaker + kafka = java.lang.NoSuchMethodError: org.apache.kafka.common.requests.MetadataResponse.prepareResponse
Rudik created KAFKA-9256:
Summary: mirrorMaker + kafka = java.lang.NoSuchMethodError:
org.apache.kafka.common.requests.MetadataResponse.prepareResponse
Key: KAFKA-9256
URL: https://issues.apache.org/jira/browse/KAFKA-9256
Project: Kafka
Issue Type: Bug
Components: mirrormaker
Affects Versions: 2.3.0
Reporter: Rudik
kafkaBroker - 2.3.0
java - openjdk version "11.0.2" 2019-01-15
kafka cluster from = \{broker - os-4603:9092,os-4604:9092,os-4605:9092, auth -
plain } \{zookeeper - os-4595:2181,os-4596:2181,os-4599:2181}
kafka cluster to = \{broker - os-4804:9092,os-4805:9092,os-4806:9092, auth -
plain} \{zookeeper - os-4804:2181,os-4805:2181,os-4806:2181}
Start MirrorMaker:
[kafka@os-4804 kafka]$ bin/kafka-mirror-maker.sh --consumer.config
./config/mirrormaker_consumer_pdc.properties --num.streams 2 --producer.config
./config/mirrormaker_producer.pr operties --whitelist="connect-configs,
connect-offsets, connect-status, last_topic"
WARNING: The default partition assignment strategy of the mirror maker will
change from 'range' to 'roundrobin' in an upcoming release (so that better load
balancing can be achiev ed). If you prefer to make this switch in advance of
that release add the following to the corresponding config:
'partition.assignment.strategy=org.apache.kafka.clients.consumer.R
oundRobinAssignor'
[2019-12-02 13:50:45,391] WARN The configuration 'zookeeper.connect' was
supplied but isn't a known config.
(org.apache.kafka.clients.producer.ProducerConfig)
[2019-12-02 13:50:45,483] WARN The configuration 'zookeeper.connect' was
supplied but isn't a known config.
(org.apache.kafka.clients.consumer.ConsumerConfig)
[2019-12-02 13:50:45,508] WARN The configuration 'zookeeper.connect' was
supplied but isn't a known config.
(org.apache.kafka.clients.consumer.ConsumerConfig)
[2019-12-02 13:50:45,749] WARN [Consumer clientId=dp-MirrorMaker-group-0,
groupId=dp-MirrorMaker-group] Connection to node -2 (os-4604/10.6.107.93:9092)
could not be established. Broker may not be available.
(org.apache.kafka.clients.NetworkClient)
[2019-12-02 13:50:45,750] WARN [Consumer clientId=dp-MirrorMaker-group-1,
groupId=dp-MirrorMaker-group] Connection to node -2 (os-4604/10.6.107.93:9092)
could not be established. Broker may not be available.
(org.apache.kafka.clients.NetworkClient)
[2019-12-02 13:50:45,758] WARN [Consumer clientId=dp-MirrorMaker-group-0,
groupId=dp-MirrorMaker-group] Connection to node -3 (os-4605/10.6.107.94:9092)
could not be established. Broker may not be available.
(org.apache.kafka.clients.NetworkClient)
logs in KafkaBroker:
[2019-12-02 13:51:54,020] ERROR [KafkaApi-1] Error when handling request:
clientId=dp-MirrorMaker-group-1, correlationId=269, api=METADATA,
body=\{topics=null,allow_auto_topic_crea
tion=true,include_cluster_authorized_operations=false,include_topic_authorized_operations=false}
(kafka.server.KafkaApis)
java.lang.NoSuchMethodError:
org.apache.kafka.common.requests.MetadataResponse.prepareResponse(ILjava/util/List;Ljava/lang/String;ILjava/util/List;I)Lorg/apache/kafka/common/reque
sts/MetadataResponse;
at
kafka.server.KafkaApis.$anonfun$handleTopicMetadataRequest$7(KafkaApis.scala:1103)
at
kafka.server.KafkaApis.$anonfun$handleTopicMetadataRequest$7$adapted(KafkaApis.scala:1096)
at kafka.server.KafkaApis.sendResponseMaybeThrottle(KafkaApis.scala:2539)
at kafka.server.KafkaApis.handleTopicMetadataRequest(KafkaApis.scala:1096)
at kafka.server.KafkaApis.handle(KafkaApis.scala:116)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:69)
at java.base/java.lang.Thread.run(Thread.java:834)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-8264) Flaky Test PlaintextConsumerTest#testLowMaxFetchSizeForRequestAndPartition
[
https://issues.apache.org/jira/browse/KAFKA-8264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16985973#comment-16985973
]
Bruno Cadonna commented on KAFKA-8264:
--
{code:java}
org.scalatest.exceptions.TestFailedException: Timed out before consuming
expected 2700 records. The number consumed was 1440.
{code}
https://builds.apache.org/job/kafka-pr-jdk11-scala2.12/9656/
> Flaky Test PlaintextConsumerTest#testLowMaxFetchSizeForRequestAndPartition
> --
>
> Key: KAFKA-8264
> URL: https://issues.apache.org/jira/browse/KAFKA-8264
> Project: Kafka
> Issue Type: Bug
> Components: core, unit tests
>Affects Versions: 2.0.1, 2.3.0
>Reporter: Matthias J. Sax
>Assignee: Stanislav Kozlovski
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.5.0
>
>
> [https://builds.apache.org/blue/organizations/jenkins/kafka-2.0-jdk8/detail/kafka-2.0-jdk8/252/tests]
> {quote}org.apache.kafka.common.errors.TopicExistsException: Topic 'topic3'
> already exists.{quote}
> STDOUT
>
> {quote}[2019-04-19 03:54:20,080] ERROR [ReplicaFetcher replicaId=1,
> leaderId=0, fetcherId=0] Error for partition __consumer_offsets-0 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:20,080] ERROR [ReplicaFetcher replicaId=2, leaderId=0,
> fetcherId=0] Error for partition __consumer_offsets-0 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:20,312] ERROR [ReplicaFetcher replicaId=2, leaderId=0,
> fetcherId=0] Error for partition topic-0 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:20,313] ERROR [ReplicaFetcher replicaId=2, leaderId=1,
> fetcherId=0] Error for partition topic-1 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:20,994] ERROR [ReplicaFetcher replicaId=1, leaderId=0,
> fetcherId=0] Error for partition topic-0 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:21,727] ERROR [ReplicaFetcher replicaId=0, leaderId=2,
> fetcherId=0] Error for partition topicWithNewMessageFormat-1 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:28,696] ERROR [ReplicaFetcher replicaId=0, leaderId=2,
> fetcherId=0] Error for partition __consumer_offsets-0 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:28,699] ERROR [ReplicaFetcher replicaId=1, leaderId=2,
> fetcherId=0] Error for partition __consumer_offsets-0 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:29,246] ERROR [ReplicaFetcher replicaId=0, leaderId=2,
> fetcherId=0] Error for partition topic-1 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:29,247] ERROR [ReplicaFetcher replicaId=0, leaderId=1,
> fetcherId=0] Error for partition topic-0 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:29,287] ERROR [ReplicaFetcher replicaId=1, leaderId=2,
> fetcherId=0] Error for partition topic-1 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:33,408] ERROR [ReplicaFetcher replicaId=0, leaderId=2,
> fetcherId=0] Error for partition __consumer_offsets-0 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-19 03:54:33,408] ERROR [ReplicaFetcher replicaId=1, leaderId=2,
> fetcherId=0] Error for partition __consumer_offsets-0 at offset 0
> (kafka.s
