[jira] [Updated] (KAFKA-15659) Flaky test RestoreIntegrationTest.shouldInvokeUserDefinedGlobalStateRestoreListener
[
https://issues.apache.org/jira/browse/KAFKA-15659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Levani Kokhreidze updated KAFKA-15659:
--
Fix Version/s: 3.7.0
> Flaky test
> RestoreIntegrationTest.shouldInvokeUserDefinedGlobalStateRestoreListener
>
>
> Key: KAFKA-15659
> URL: https://issues.apache.org/jira/browse/KAFKA-15659
> Project: Kafka
> Issue Type: Test
>Reporter: Divij Vaidya
>Assignee: Levani Kokhreidze
>Priority: Major
> Labels: flaky-test, streams
> Fix For: 3.7.0
>
> Attachments: Screenshot 2023-10-20 at 13.19.20.png
>
>
> The test added in the PR [https://github.com/apache/kafka/pull/14519]
> {{shouldInvokeUserDefinedGlobalStateRestoreListener}} has been flaky since it
> was added. You can find the flaky build on trunk using the link
> [https://ge.apache.org/scans/tests?search.relativeStartTime=P28D=kafka=trunk=Europe%2FBerlin=org.apache.kafka.streams.integration.RestoreIntegrationTest=shouldInvokeUserDefinedGlobalStateRestoreListener()]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KAFKA-15659) Flaky test RestoreIntegrationTest.shouldInvokeUserDefinedGlobalStateRestoreListener
[
https://issues.apache.org/jira/browse/KAFKA-15659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Levani Kokhreidze resolved KAFKA-15659.
---
Resolution: Fixed
> Flaky test
> RestoreIntegrationTest.shouldInvokeUserDefinedGlobalStateRestoreListener
>
>
> Key: KAFKA-15659
> URL: https://issues.apache.org/jira/browse/KAFKA-15659
> Project: Kafka
> Issue Type: Test
>Reporter: Divij Vaidya
>Assignee: Levani Kokhreidze
>Priority: Major
> Labels: flaky-test, streams
> Attachments: Screenshot 2023-10-20 at 13.19.20.png
>
>
> The test added in the PR [https://github.com/apache/kafka/pull/14519]
> {{shouldInvokeUserDefinedGlobalStateRestoreListener}} has been flaky since it
> was added. You can find the flaky build on trunk using the link
> [https://ge.apache.org/scans/tests?search.relativeStartTime=P28D=kafka=trunk=Europe%2FBerlin=org.apache.kafka.streams.integration.RestoreIntegrationTest=shouldInvokeUserDefinedGlobalStateRestoreListener()]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (KAFKA-15659) Flaky test RestoreIntegrationTest.shouldInvokeUserDefinedGlobalStateRestoreListener
[
https://issues.apache.org/jira/browse/KAFKA-15659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Levani Kokhreidze reassigned KAFKA-15659:
-
Assignee: Levani Kokhreidze
> Flaky test
> RestoreIntegrationTest.shouldInvokeUserDefinedGlobalStateRestoreListener
>
>
> Key: KAFKA-15659
> URL: https://issues.apache.org/jira/browse/KAFKA-15659
> Project: Kafka
> Issue Type: Test
>Reporter: Divij Vaidya
>Assignee: Levani Kokhreidze
>Priority: Major
> Labels: flaky-test, streams
> Attachments: Screenshot 2023-10-20 at 13.19.20.png
>
>
> The test added in the PR [https://github.com/apache/kafka/pull/14519]
> {{shouldInvokeUserDefinedGlobalStateRestoreListener}} has been flaky since it
> was added. You can find the flaky build on trunk using the link
> [https://ge.apache.org/scans/tests?search.relativeStartTime=P28D=kafka=trunk=Europe%2FBerlin=org.apache.kafka.streams.integration.RestoreIntegrationTest=shouldInvokeUserDefinedGlobalStateRestoreListener()]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-15571) StateRestoreListener#onRestoreSuspended is never called because wrapper DelegatingStateRestoreListener doesn't implement onRestoreSuspended
[ https://issues.apache.org/jira/browse/KAFKA-15571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Levani Kokhreidze updated KAFKA-15571: -- Affects Version/s: 3.6.0 > StateRestoreListener#onRestoreSuspended is never called because wrapper > DelegatingStateRestoreListener doesn't implement onRestoreSuspended > --- > > Key: KAFKA-15571 > URL: https://issues.apache.org/jira/browse/KAFKA-15571 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 3.5.0, 3.6.0, 3.5.1 >Reporter: Levani Kokhreidze >Assignee: Levani Kokhreidze >Priority: Major > > With https://issues.apache.org/jira/browse/KAFKA-10575 > `StateRestoreListener#onRestoreSuspended` was added. But local tests show > that it is never called because `DelegatingStateRestoreListener` was not > updated to call a new method. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-15571) StateRestoreListener#onRestoreSuspended is never called because wrapper DelegatingStateRestoreListener doesn't implement onRestoreSuspended
[ https://issues.apache.org/jira/browse/KAFKA-15571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Levani Kokhreidze updated KAFKA-15571: -- Description: With https://issues.apache.org/jira/browse/KAFKA-10575 `StateRestoreListener#onRestoreSuspended` was added. But local tests show that it is never called because `DelegatingStateRestoreListener` was not updated to call a new method. (was: With https://issues.apache.org/jira/browse/KAFKA-10575 `StateRestoreListener#onRestoreSuspended` was added. But local tests show that it is never called because `DelegatingStateRestoreListener` was not updated to call a new method) > StateRestoreListener#onRestoreSuspended is never called because wrapper > DelegatingStateRestoreListener doesn't implement onRestoreSuspended > --- > > Key: KAFKA-15571 > URL: https://issues.apache.org/jira/browse/KAFKA-15571 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 3.5.0, 3.5.1 >Reporter: Levani Kokhreidze >Assignee: Levani Kokhreidze >Priority: Major > > With https://issues.apache.org/jira/browse/KAFKA-10575 > `StateRestoreListener#onRestoreSuspended` was added. But local tests show > that it is never called because `DelegatingStateRestoreListener` was not > updated to call a new method. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-15571) StateRestoreListener#onRestoreSuspended is never called because wrapper DelegatingStateRestoreListener doesn't implement onRestoreSuspended
[ https://issues.apache.org/jira/browse/KAFKA-15571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Levani Kokhreidze reassigned KAFKA-15571: - Assignee: Levani Kokhreidze > StateRestoreListener#onRestoreSuspended is never called because wrapper > DelegatingStateRestoreListener doesn't implement onRestoreSuspended > --- > > Key: KAFKA-15571 > URL: https://issues.apache.org/jira/browse/KAFKA-15571 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 3.5.0, 3.5.1 >Reporter: Levani Kokhreidze >Assignee: Levani Kokhreidze >Priority: Major > > With https://issues.apache.org/jira/browse/KAFKA-10575 > `StateRestoreListener#onRestoreSuspended` was added. But local tests show > that it is never called because `DelegatingStateRestoreListener` was not > updated to call a new method -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-15571) StateRestoreListener#onRestoreSuspended is never called because wrapper DelegatingStateRestoreListener doesn't implement onRestoreSuspended
Levani Kokhreidze created KAFKA-15571: - Summary: StateRestoreListener#onRestoreSuspended is never called because wrapper DelegatingStateRestoreListener doesn't implement onRestoreSuspended Key: KAFKA-15571 URL: https://issues.apache.org/jira/browse/KAFKA-15571 Project: Kafka Issue Type: Bug Components: streams Affects Versions: 3.5.1, 3.5.0 Reporter: Levani Kokhreidze With https://issues.apache.org/jira/browse/KAFKA-10575 `StateRestoreListener#onRestoreSuspended` was added. But local tests show that it is never called because `DelegatingStateRestoreListener` was not updated to call a new method -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-13877) Flaky RackAwarenessIntegrationTest.shouldDistributeStandbyReplicasOverMultipleClientTags
[
https://issues.apache.org/jira/browse/KAFKA-13877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Levani Kokhreidze reassigned KAFKA-13877:
-
Assignee: Levani Kokhreidze
> Flaky
> RackAwarenessIntegrationTest.shouldDistributeStandbyReplicasOverMultipleClientTags
>
>
> Key: KAFKA-13877
> URL: https://issues.apache.org/jira/browse/KAFKA-13877
> Project: Kafka
> Issue Type: Bug
> Components: streams, unit tests
>Reporter: Guozhang Wang
>Assignee: Levani Kokhreidze
>Priority: Major
> Labels: newbie
>
> The following test fails on local testbeds about once per 10-15 runs:
> {code}
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:87)
> at org.junit.Assert.assertTrue(Assert.java:42)
> at org.junit.Assert.assertTrue(Assert.java:53)
> at
> org.apache.kafka.streams.integration.RackAwarenessIntegrationTest.shouldDistributeStandbyReplicasOverMultipleClientTags(RackAwarenessIntegrationTest.java:192)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
> at
> com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:53)
> at
> com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:230)
> at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:58)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (KAFKA-13877) Flaky RackAwarenessIntegrationTest.shouldDistributeStandbyReplicasOverMultipleClientTags
[
https://issues.apache.org/jira/browse/KAFKA-13877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17532474#comment-17532474
]
Levani Kokhreidze commented on KAFKA-13877:
---
I will take this on.
> Flaky
> RackAwarenessIntegrationTest.shouldDistributeStandbyReplicasOverMultipleClientTags
>
>
> Key: KAFKA-13877
> URL: https://issues.apache.org/jira/browse/KAFKA-13877
> Project: Kafka
> Issue Type: Bug
> Components: streams, unit tests
>Reporter: Guozhang Wang
>Assignee: Levani Kokhreidze
>Priority: Major
> Labels: newbie
>
> The following test fails on local testbeds about once per 10-15 runs:
> {code}
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:87)
> at org.junit.Assert.assertTrue(Assert.java:42)
> at org.junit.Assert.assertTrue(Assert.java:53)
> at
> org.apache.kafka.streams.integration.RackAwarenessIntegrationTest.shouldDistributeStandbyReplicasOverMultipleClientTags(RackAwarenessIntegrationTest.java:192)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
> at
> com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:53)
> at
> com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:230)
> at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:58)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Comment Edited] (KAFKA-13272) KStream offset stuck after brokers outage
[
https://issues.apache.org/jira/browse/KAFKA-13272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17521653#comment-17521653
]
Levani Kokhreidze edited comment on KAFKA-13272 at 4/13/22 12:30 PM:
-
Thanks [~nicktelford]
I was not aware that such tool existed. Good reason to upgrade to the 3.X
version.
was (Author: lkokhreidze):
Thanks [~nicktelford]
I was not aware that such tool existed. We are still at 2.8.X version. More
reason to upgrade to the 3.X versions.
> KStream offset stuck after brokers outage
> -
>
> Key: KAFKA-13272
> URL: https://issues.apache.org/jira/browse/KAFKA-13272
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 2.8.0
> Environment: Kafka running on Kubernetes
> centos
>Reporter: F Méthot
>Priority: Major
>
> Our KStream app offset stay stuck on 1 partition after outage possibly when
> exactly_once is enabled.
> Running with KStream 2.8, kafka broker 2.8,
> 3 brokers.
> commands topic is 10 partitions (replication 2, min-insync 2)
> command-expiry-store-changelog topic is 10 partitions (replication 2,
> min-insync 2)
> events topic is 10 partitions (replication 2, min-insync 2)
> with this topology
> Topologies:
>
> {code:java}
> Sub-topology: 0
> Source: KSTREAM-SOURCE-00 (topics: [commands])
> --> KSTREAM-TRANSFORM-01
> Processor: KSTREAM-TRANSFORM-01 (stores: [])
> --> KSTREAM-TRANSFORM-02
> <-- KSTREAM-SOURCE-00
> Processor: KSTREAM-TRANSFORM-02 (stores: [command-expiry-store])
> --> KSTREAM-SINK-03
> <-- KSTREAM-TRANSFORM-01
> Sink: KSTREAM-SINK-03 (topic: events)
> <-- KSTREAM-TRANSFORM-02
> {code}
> h3.
> h3. Attempt 1 at reproducing this issue
>
> Our stream app runs with processing.guarantee *exactly_once*
> After a Kafka test outage where all 3 brokers pod were deleted at the same
> time,
> Brokers restarted and initialized succesfuly.
> When restarting the topology above, one of the tasks would never initialize
> fully, the restore phase would keep outputting this messages every few
> minutes:
>
> {code:java}
> 2021-08-16 14:20:33,421 INFO stream-thread
> [commands-processor-51b0a534-34b6-47a4-bd9c-c57b6ecb8665-StreamThread-1]
> Restoration in progress for 1 partitions.
> {commands-processor-expiry-store-changelog-8: position=11775908,
> end=11775911, totalRestored=2002076}
> [commands-processor-51b0a534-34b6-47a4-bd9c-c57b6ecb8665-StreamThread-1]
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader)
> {code}
> Task for partition 8 would never initialize, no more data would be read from
> the source commands topic for that partition.
>
> In an attempt to recover, we restarted the stream app with stream
> processing.guarantee back to at_least_once, than it proceed with reading the
> changelog and restoring partition 8 fully.
> But we noticed afterward, for the next hour until we rebuilt the system, that
> partition 8 from command-expiry-store-changelog would not be
> cleaned/compacted by the log cleaner/compacter compared to other partitions.
> (could be unrelated, because we have seen that before)
> So we resorted to delete/recreate our command-expiry-store-changelog topic
> and events topic and regenerate it from the commands, reading from beginning.
> Things went back to normal
> h3. Attempt 2 at reproducing this issue
> kstream runs with *exactly-once*
> We force-deleted all 3 pod running kafka.
> After that, one of the partition can’t be restored. (like reported in
> previous attempt)
> For that partition, we noticed these logs on the broker
> {code:java}
> [2021-08-27 17:45:32,799] INFO [Transaction Marker Channel Manager 1002]:
> Couldn’t find leader endpoint for partitions Set(__consumer_offsets-11,
> command-expiry-store-changelog-9) while trying to send transaction markers
> for commands-processor-0_9, these partitions are likely deleted already and
> hence can be skipped
> (kafka.coordinator.transaction.TransactionMarkerChannelManager){code}
> Then
> - we stop the kstream app,
> - restarted kafka brokers cleanly
> - Restarting the Kstream app,
> Those logs messages showed up on the kstream app log:
>
> {code:java}
> 2021-08-27 18:34:42,413 INFO [Consumer
> clientId=commands-processor-76602c87-f682-4648-859b-8fa9b6b937f3-StreamThread-1-consumer,
> groupId=commands-processor] The following partitions still have unstable
> offsets which are not cleared on the broker side: [commands-9], this could be
> either transactional offsets waiting for completion, or normal offsets
> waiting for replication after appending to local log
> [commands-processor-76602c87-f682-4648-859b-8fa9b6b937f3-StreamThread-1]
>
[jira] [Commented] (KAFKA-13272) KStream offset stuck after brokers outage
[
https://issues.apache.org/jira/browse/KAFKA-13272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17521653#comment-17521653
]
Levani Kokhreidze commented on KAFKA-13272:
---
Thanks [~nicktelford]
I was not aware that such tool existed. We are still at 2.8.X version. More
reason to upgrade to the 3.X versions.
> KStream offset stuck after brokers outage
> -
>
> Key: KAFKA-13272
> URL: https://issues.apache.org/jira/browse/KAFKA-13272
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 2.8.0
> Environment: Kafka running on Kubernetes
> centos
>Reporter: F Méthot
>Priority: Major
>
> Our KStream app offset stay stuck on 1 partition after outage possibly when
> exactly_once is enabled.
> Running with KStream 2.8, kafka broker 2.8,
> 3 brokers.
> commands topic is 10 partitions (replication 2, min-insync 2)
> command-expiry-store-changelog topic is 10 partitions (replication 2,
> min-insync 2)
> events topic is 10 partitions (replication 2, min-insync 2)
> with this topology
> Topologies:
>
> {code:java}
> Sub-topology: 0
> Source: KSTREAM-SOURCE-00 (topics: [commands])
> --> KSTREAM-TRANSFORM-01
> Processor: KSTREAM-TRANSFORM-01 (stores: [])
> --> KSTREAM-TRANSFORM-02
> <-- KSTREAM-SOURCE-00
> Processor: KSTREAM-TRANSFORM-02 (stores: [command-expiry-store])
> --> KSTREAM-SINK-03
> <-- KSTREAM-TRANSFORM-01
> Sink: KSTREAM-SINK-03 (topic: events)
> <-- KSTREAM-TRANSFORM-02
> {code}
> h3.
> h3. Attempt 1 at reproducing this issue
>
> Our stream app runs with processing.guarantee *exactly_once*
> After a Kafka test outage where all 3 brokers pod were deleted at the same
> time,
> Brokers restarted and initialized succesfuly.
> When restarting the topology above, one of the tasks would never initialize
> fully, the restore phase would keep outputting this messages every few
> minutes:
>
> {code:java}
> 2021-08-16 14:20:33,421 INFO stream-thread
> [commands-processor-51b0a534-34b6-47a4-bd9c-c57b6ecb8665-StreamThread-1]
> Restoration in progress for 1 partitions.
> {commands-processor-expiry-store-changelog-8: position=11775908,
> end=11775911, totalRestored=2002076}
> [commands-processor-51b0a534-34b6-47a4-bd9c-c57b6ecb8665-StreamThread-1]
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader)
> {code}
> Task for partition 8 would never initialize, no more data would be read from
> the source commands topic for that partition.
>
> In an attempt to recover, we restarted the stream app with stream
> processing.guarantee back to at_least_once, than it proceed with reading the
> changelog and restoring partition 8 fully.
> But we noticed afterward, for the next hour until we rebuilt the system, that
> partition 8 from command-expiry-store-changelog would not be
> cleaned/compacted by the log cleaner/compacter compared to other partitions.
> (could be unrelated, because we have seen that before)
> So we resorted to delete/recreate our command-expiry-store-changelog topic
> and events topic and regenerate it from the commands, reading from beginning.
> Things went back to normal
> h3. Attempt 2 at reproducing this issue
> kstream runs with *exactly-once*
> We force-deleted all 3 pod running kafka.
> After that, one of the partition can’t be restored. (like reported in
> previous attempt)
> For that partition, we noticed these logs on the broker
> {code:java}
> [2021-08-27 17:45:32,799] INFO [Transaction Marker Channel Manager 1002]:
> Couldn’t find leader endpoint for partitions Set(__consumer_offsets-11,
> command-expiry-store-changelog-9) while trying to send transaction markers
> for commands-processor-0_9, these partitions are likely deleted already and
> hence can be skipped
> (kafka.coordinator.transaction.TransactionMarkerChannelManager){code}
> Then
> - we stop the kstream app,
> - restarted kafka brokers cleanly
> - Restarting the Kstream app,
> Those logs messages showed up on the kstream app log:
>
> {code:java}
> 2021-08-27 18:34:42,413 INFO [Consumer
> clientId=commands-processor-76602c87-f682-4648-859b-8fa9b6b937f3-StreamThread-1-consumer,
> groupId=commands-processor] The following partitions still have unstable
> offsets which are not cleared on the broker side: [commands-9], this could be
> either transactional offsets waiting for completion, or normal offsets
> waiting for replication after appending to local log
> [commands-processor-76602c87-f682-4648-859b-8fa9b6b937f3-StreamThread-1]
> (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
>
> {code}
> This would cause our processor to not consume from that specific source
> topic-partition.
> Deleting downstream topic and replaying data would
[jira] [Commented] (KAFKA-13272) KStream offset stuck after brokers outage
[
https://issues.apache.org/jira/browse/KAFKA-13272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17516798#comment-17516798
]
Levani Kokhreidze commented on KAFKA-13272:
---
Hi [~guozhang] , [~ableegoldman]
Do you have any workarounds in mind that could help us to mitigate the issue if
it happens again?
> KStream offset stuck after brokers outage
> -
>
> Key: KAFKA-13272
> URL: https://issues.apache.org/jira/browse/KAFKA-13272
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 2.8.0
> Environment: Kafka running on Kubernetes
> centos
>Reporter: F Méthot
>Priority: Major
>
> Our KStream app offset stay stuck on 1 partition after outage possibly when
> exactly_once is enabled.
> Running with KStream 2.8, kafka broker 2.8,
> 3 brokers.
> commands topic is 10 partitions (replication 2, min-insync 2)
> command-expiry-store-changelog topic is 10 partitions (replication 2,
> min-insync 2)
> events topic is 10 partitions (replication 2, min-insync 2)
> with this topology
> Topologies:
>
> {code:java}
> Sub-topology: 0
> Source: KSTREAM-SOURCE-00 (topics: [commands])
> --> KSTREAM-TRANSFORM-01
> Processor: KSTREAM-TRANSFORM-01 (stores: [])
> --> KSTREAM-TRANSFORM-02
> <-- KSTREAM-SOURCE-00
> Processor: KSTREAM-TRANSFORM-02 (stores: [command-expiry-store])
> --> KSTREAM-SINK-03
> <-- KSTREAM-TRANSFORM-01
> Sink: KSTREAM-SINK-03 (topic: events)
> <-- KSTREAM-TRANSFORM-02
> {code}
> h3.
> h3. Attempt 1 at reproducing this issue
>
> Our stream app runs with processing.guarantee *exactly_once*
> After a Kafka test outage where all 3 brokers pod were deleted at the same
> time,
> Brokers restarted and initialized succesfuly.
> When restarting the topology above, one of the tasks would never initialize
> fully, the restore phase would keep outputting this messages every few
> minutes:
>
> {code:java}
> 2021-08-16 14:20:33,421 INFO stream-thread
> [commands-processor-51b0a534-34b6-47a4-bd9c-c57b6ecb8665-StreamThread-1]
> Restoration in progress for 1 partitions.
> {commands-processor-expiry-store-changelog-8: position=11775908,
> end=11775911, totalRestored=2002076}
> [commands-processor-51b0a534-34b6-47a4-bd9c-c57b6ecb8665-StreamThread-1]
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader)
> {code}
> Task for partition 8 would never initialize, no more data would be read from
> the source commands topic for that partition.
>
> In an attempt to recover, we restarted the stream app with stream
> processing.guarantee back to at_least_once, than it proceed with reading the
> changelog and restoring partition 8 fully.
> But we noticed afterward, for the next hour until we rebuilt the system, that
> partition 8 from command-expiry-store-changelog would not be
> cleaned/compacted by the log cleaner/compacter compared to other partitions.
> (could be unrelated, because we have seen that before)
> So we resorted to delete/recreate our command-expiry-store-changelog topic
> and events topic and regenerate it from the commands, reading from beginning.
> Things went back to normal
> h3. Attempt 2 at reproducing this issue
> kstream runs with *exactly-once*
> We force-deleted all 3 pod running kafka.
> After that, one of the partition can’t be restored. (like reported in
> previous attempt)
> For that partition, we noticed these logs on the broker
> {code:java}
> [2021-08-27 17:45:32,799] INFO [Transaction Marker Channel Manager 1002]:
> Couldn’t find leader endpoint for partitions Set(__consumer_offsets-11,
> command-expiry-store-changelog-9) while trying to send transaction markers
> for commands-processor-0_9, these partitions are likely deleted already and
> hence can be skipped
> (kafka.coordinator.transaction.TransactionMarkerChannelManager){code}
> Then
> - we stop the kstream app,
> - restarted kafka brokers cleanly
> - Restarting the Kstream app,
> Those logs messages showed up on the kstream app log:
>
> {code:java}
> 2021-08-27 18:34:42,413 INFO [Consumer
> clientId=commands-processor-76602c87-f682-4648-859b-8fa9b6b937f3-StreamThread-1-consumer,
> groupId=commands-processor] The following partitions still have unstable
> offsets which are not cleared on the broker side: [commands-9], this could be
> either transactional offsets waiting for completion, or normal offsets
> waiting for replication after appending to local log
> [commands-processor-76602c87-f682-4648-859b-8fa9b6b937f3-StreamThread-1]
> (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
>
> {code}
> This would cause our processor to not consume from that specific source
> topic-partition.
> Deleting downstream topic and replaying data would NOT fix
[jira] [Comment Edited] (KAFKA-13272) KStream offset stuck after brokers outage
[
https://issues.apache.org/jira/browse/KAFKA-13272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17515235#comment-17515235
]
Levani Kokhreidze edited comment on KAFKA-13272 at 3/31/22, 11:11 AM:
--
Hi,
We've also encountered this issue, but it's not clear what triggered it. We
haven't experienced any broker outage but Kafka Streams application did crash
before the issue appeared.
We were not able to solve it by any means. Even after application reset, issue
still appeared. Few partitions were essentially stuck.
The only thing that solved the problem is changing the application ID of the
Kafka Streams application and re-playing all the data.
was (Author: lkokhreidze):
Hi,
We've also encountered this issue, but it's not clear what triggered it. We
haven't experienced any broker outage but Kafka Streams application did crash
before the issue appeared.
We were not able to solve it by any means. Even after application reset, issue
still appeared. Few partitions were essentially stuck.
The only thing that solved the problem is changing the consumer group name and
re-playing all the data.
> KStream offset stuck after brokers outage
> -
>
> Key: KAFKA-13272
> URL: https://issues.apache.org/jira/browse/KAFKA-13272
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 2.8.0
> Environment: Kafka running on Kubernetes
> centos
>Reporter: F Méthot
>Priority: Major
>
> Our KStream app offset stay stuck on 1 partition after outage possibly when
> exactly_once is enabled.
> Running with KStream 2.8, kafka broker 2.8,
> 3 brokers.
> commands topic is 10 partitions (replication 2, min-insync 2)
> command-expiry-store-changelog topic is 10 partitions (replication 2,
> min-insync 2)
> events topic is 10 partitions (replication 2, min-insync 2)
> with this topology
> Topologies:
>
> {code:java}
> Sub-topology: 0
> Source: KSTREAM-SOURCE-00 (topics: [commands])
> --> KSTREAM-TRANSFORM-01
> Processor: KSTREAM-TRANSFORM-01 (stores: [])
> --> KSTREAM-TRANSFORM-02
> <-- KSTREAM-SOURCE-00
> Processor: KSTREAM-TRANSFORM-02 (stores: [command-expiry-store])
> --> KSTREAM-SINK-03
> <-- KSTREAM-TRANSFORM-01
> Sink: KSTREAM-SINK-03 (topic: events)
> <-- KSTREAM-TRANSFORM-02
> {code}
> h3.
> h3. Attempt 1 at reproducing this issue
>
> Our stream app runs with processing.guarantee *exactly_once*
> After a Kafka test outage where all 3 brokers pod were deleted at the same
> time,
> Brokers restarted and initialized succesfuly.
> When restarting the topology above, one of the tasks would never initialize
> fully, the restore phase would keep outputting this messages every few
> minutes:
>
> {code:java}
> 2021-08-16 14:20:33,421 INFO stream-thread
> [commands-processor-51b0a534-34b6-47a4-bd9c-c57b6ecb8665-StreamThread-1]
> Restoration in progress for 1 partitions.
> {commands-processor-expiry-store-changelog-8: position=11775908,
> end=11775911, totalRestored=2002076}
> [commands-processor-51b0a534-34b6-47a4-bd9c-c57b6ecb8665-StreamThread-1]
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader)
> {code}
> Task for partition 8 would never initialize, no more data would be read from
> the source commands topic for that partition.
>
> In an attempt to recover, we restarted the stream app with stream
> processing.guarantee back to at_least_once, than it proceed with reading the
> changelog and restoring partition 8 fully.
> But we noticed afterward, for the next hour until we rebuilt the system, that
> partition 8 from command-expiry-store-changelog would not be
> cleaned/compacted by the log cleaner/compacter compared to other partitions.
> (could be unrelated, because we have seen that before)
> So we resorted to delete/recreate our command-expiry-store-changelog topic
> and events topic and regenerate it from the commands, reading from beginning.
> Things went back to normal
> h3. Attempt 2 at reproducing this issue
> kstream runs with *exactly-once*
> We force-deleted all 3 pod running kafka.
> After that, one of the partition can’t be restored. (like reported in
> previous attempt)
> For that partition, we noticed these logs on the broker
> {code:java}
> [2021-08-27 17:45:32,799] INFO [Transaction Marker Channel Manager 1002]:
> Couldn’t find leader endpoint for partitions Set(__consumer_offsets-11,
> command-expiry-store-changelog-9) while trying to send transaction markers
> for commands-processor-0_9, these partitions are likely deleted already and
> hence can be skipped
> (kafka.coordinator.transaction.TransactionMarkerChannelManager){code}
> Then
> - we stop the kstream app,
> - restarted kafka brokers cleanly
> -
[jira] [Commented] (KAFKA-13272) KStream offset stuck after brokers outage
[
https://issues.apache.org/jira/browse/KAFKA-13272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17515235#comment-17515235
]
Levani Kokhreidze commented on KAFKA-13272:
---
Hi,
We've also encountered this issue, but it's not clear what triggered it. We
haven't experienced any broker outage but Kafka Streams application did crash
before the issue appeared.
We were not able to solve it by any means. Even after application reset, issue
still appeared. Few partitions were essentially stuck.
The only thing that solved the problem is changing the consumer group name and
re-playing all the data.
> KStream offset stuck after brokers outage
> -
>
> Key: KAFKA-13272
> URL: https://issues.apache.org/jira/browse/KAFKA-13272
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 2.8.0
> Environment: Kafka running on Kubernetes
> centos
>Reporter: F Méthot
>Priority: Major
>
> Our KStream app offset stay stuck on 1 partition after outage possibly when
> exactly_once is enabled.
> Running with KStream 2.8, kafka broker 2.8,
> 3 brokers.
> commands topic is 10 partitions (replication 2, min-insync 2)
> command-expiry-store-changelog topic is 10 partitions (replication 2,
> min-insync 2)
> events topic is 10 partitions (replication 2, min-insync 2)
> with this topology
> Topologies:
>
> {code:java}
> Sub-topology: 0
> Source: KSTREAM-SOURCE-00 (topics: [commands])
> --> KSTREAM-TRANSFORM-01
> Processor: KSTREAM-TRANSFORM-01 (stores: [])
> --> KSTREAM-TRANSFORM-02
> <-- KSTREAM-SOURCE-00
> Processor: KSTREAM-TRANSFORM-02 (stores: [command-expiry-store])
> --> KSTREAM-SINK-03
> <-- KSTREAM-TRANSFORM-01
> Sink: KSTREAM-SINK-03 (topic: events)
> <-- KSTREAM-TRANSFORM-02
> {code}
> h3.
> h3. Attempt 1 at reproducing this issue
>
> Our stream app runs with processing.guarantee *exactly_once*
> After a Kafka test outage where all 3 brokers pod were deleted at the same
> time,
> Brokers restarted and initialized succesfuly.
> When restarting the topology above, one of the tasks would never initialize
> fully, the restore phase would keep outputting this messages every few
> minutes:
>
> {code:java}
> 2021-08-16 14:20:33,421 INFO stream-thread
> [commands-processor-51b0a534-34b6-47a4-bd9c-c57b6ecb8665-StreamThread-1]
> Restoration in progress for 1 partitions.
> {commands-processor-expiry-store-changelog-8: position=11775908,
> end=11775911, totalRestored=2002076}
> [commands-processor-51b0a534-34b6-47a4-bd9c-c57b6ecb8665-StreamThread-1]
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader)
> {code}
> Task for partition 8 would never initialize, no more data would be read from
> the source commands topic for that partition.
>
> In an attempt to recover, we restarted the stream app with stream
> processing.guarantee back to at_least_once, than it proceed with reading the
> changelog and restoring partition 8 fully.
> But we noticed afterward, for the next hour until we rebuilt the system, that
> partition 8 from command-expiry-store-changelog would not be
> cleaned/compacted by the log cleaner/compacter compared to other partitions.
> (could be unrelated, because we have seen that before)
> So we resorted to delete/recreate our command-expiry-store-changelog topic
> and events topic and regenerate it from the commands, reading from beginning.
> Things went back to normal
> h3. Attempt 2 at reproducing this issue
> kstream runs with *exactly-once*
> We force-deleted all 3 pod running kafka.
> After that, one of the partition can’t be restored. (like reported in
> previous attempt)
> For that partition, we noticed these logs on the broker
> {code:java}
> [2021-08-27 17:45:32,799] INFO [Transaction Marker Channel Manager 1002]:
> Couldn’t find leader endpoint for partitions Set(__consumer_offsets-11,
> command-expiry-store-changelog-9) while trying to send transaction markers
> for commands-processor-0_9, these partitions are likely deleted already and
> hence can be skipped
> (kafka.coordinator.transaction.TransactionMarkerChannelManager){code}
> Then
> - we stop the kstream app,
> - restarted kafka brokers cleanly
> - Restarting the Kstream app,
> Those logs messages showed up on the kstream app log:
>
> {code:java}
> 2021-08-27 18:34:42,413 INFO [Consumer
> clientId=commands-processor-76602c87-f682-4648-859b-8fa9b6b937f3-StreamThread-1-consumer,
> groupId=commands-processor] The following partitions still have unstable
> offsets which are not cleared on the broker side: [commands-9], this could be
> either transactional offsets waiting for completion, or normal offsets
> waiting for replication after appending to local log
>
[jira] [Assigned] (KAFKA-10686) Pluggable standby tasks assignor for Kafka Streams
[ https://issues.apache.org/jira/browse/KAFKA-10686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Levani Kokhreidze reassigned KAFKA-10686: - Assignee: (was: Levani Kokhreidze) > Pluggable standby tasks assignor for Kafka Streams > -- > > Key: KAFKA-10686 > URL: https://issues.apache.org/jira/browse/KAFKA-10686 > Project: Kafka > Issue Type: New Feature > Components: streams >Reporter: Levani Kokhreidze >Priority: Major > Labels: needs-kip > > In production, Kafka Streams instances often run across different clusters > and availability zones. In order to guarantee high availability of the Kafka > Streams deployments, users would need more granular control over which > instances standby tasks can be created. > Idea of this ticket is to expose interface for Kafka Streams which can be > implemented by the users to control where standby tasks can be created. > Kafka Streams can have RackAware assignment as a default implementation that > will take into account `rack.id` of the application and make sure that > standby tasks are created on different racks. > Point of this ticket though is to give more flexibility to users on standby > task creation, in cases where just rack awareness is not enough. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (KAFKA-6718) Rack Aware Stand-by Task Assignment for Kafka Streams
[ https://issues.apache.org/jira/browse/KAFKA-6718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17500540#comment-17500540 ] Levani Kokhreidze commented on KAFKA-6718: -- Relevant PRs for this task are: [https://github.com/apache/kafka/pull/10851] [https://github.com/apache/kafka/pull/10802] [https://github.com/apache/kafka/pull/11837] > Rack Aware Stand-by Task Assignment for Kafka Streams > - > > Key: KAFKA-6718 > URL: https://issues.apache.org/jira/browse/KAFKA-6718 > Project: Kafka > Issue Type: New Feature > Components: streams >Reporter: Deepak Goyal >Assignee: Levani Kokhreidze >Priority: Major > Labels: kip > Fix For: 3.2.0 > > > |Machines in data centre are sometimes grouped in racks. Racks provide > isolation as each rack may be in a different physical location and has its > own power source. When tasks are properly replicated across racks, it > provides fault tolerance in that if a rack goes down, the remaining racks can > continue to serve traffic. > > This feature is already implemented at Kafka > [KIP-36|https://cwiki.apache.org/confluence/display/KAFKA/KIP-36+Rack+aware+replica+assignment] > but we needed similar for task assignments at Kafka Streams Application > layer. > > This features enables replica tasks to be assigned on different racks for > fault-tolerance. > NUM_STANDBY_REPLICAS = x > totalTasks = x+1 (replica + active) > # If there are no rackID provided: Cluster will behave rack-unaware > # If same rackId is given to all the nodes: Cluster will behave rack-unaware > # If (totalTasks <= number of racks), then Cluster will be rack aware i.e. > each replica task is each assigned to a different rack. > # Id (totalTasks > number of racks), then it will first assign tasks on > different racks, further tasks will be assigned to least loaded node, cluster > wide.| > We have added another config in StreamsConfig called "RACK_ID_CONFIG" which > helps StickyPartitionAssignor to assign tasks in such a way that no two > replica tasks are on same rack if possible. > Post that it also helps to maintain stickyness with-in the rack.| -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (KAFKA-6718) Rack Aware Stand-by Task Assignment for Kafka Streams
[ https://issues.apache.org/jira/browse/KAFKA-6718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Levani Kokhreidze updated KAFKA-6718: - Labels: kip (was: needs-kip) > Rack Aware Stand-by Task Assignment for Kafka Streams > - > > Key: KAFKA-6718 > URL: https://issues.apache.org/jira/browse/KAFKA-6718 > Project: Kafka > Issue Type: New Feature > Components: streams >Reporter: Deepak Goyal >Assignee: Levani Kokhreidze >Priority: Major > Labels: kip > > |Machines in data centre are sometimes grouped in racks. Racks provide > isolation as each rack may be in a different physical location and has its > own power source. When tasks are properly replicated across racks, it > provides fault tolerance in that if a rack goes down, the remaining racks can > continue to serve traffic. > > This feature is already implemented at Kafka > [KIP-36|https://cwiki.apache.org/confluence/display/KAFKA/KIP-36+Rack+aware+replica+assignment] > but we needed similar for task assignments at Kafka Streams Application > layer. > > This features enables replica tasks to be assigned on different racks for > fault-tolerance. > NUM_STANDBY_REPLICAS = x > totalTasks = x+1 (replica + active) > # If there are no rackID provided: Cluster will behave rack-unaware > # If same rackId is given to all the nodes: Cluster will behave rack-unaware > # If (totalTasks <= number of racks), then Cluster will be rack aware i.e. > each replica task is each assigned to a different rack. > # Id (totalTasks > number of racks), then it will first assign tasks on > different racks, further tasks will be assigned to least loaded node, cluster > wide.| > We have added another config in StreamsConfig called "RACK_ID_CONFIG" which > helps StickyPartitionAssignor to assign tasks in such a way that no two > replica tasks are on same rack if possible. > Post that it also helps to maintain stickyness with-in the rack.| -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KAFKA-6718) Rack Aware Stand-by Task Assignment for Kafka Streams
[ https://issues.apache.org/jira/browse/KAFKA-6718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Levani Kokhreidze reassigned KAFKA-6718: Assignee: Levani Kokhreidze > Rack Aware Stand-by Task Assignment for Kafka Streams > - > > Key: KAFKA-6718 > URL: https://issues.apache.org/jira/browse/KAFKA-6718 > Project: Kafka > Issue Type: New Feature > Components: streams >Reporter: Deepak Goyal >Assignee: Levani Kokhreidze >Priority: Major > Labels: needs-kip > > |Machines in data centre are sometimes grouped in racks. Racks provide > isolation as each rack may be in a different physical location and has its > own power source. When tasks are properly replicated across racks, it > provides fault tolerance in that if a rack goes down, the remaining racks can > continue to serve traffic. > > This feature is already implemented at Kafka > [KIP-36|https://cwiki.apache.org/confluence/display/KAFKA/KIP-36+Rack+aware+replica+assignment] > but we needed similar for task assignments at Kafka Streams Application > layer. > > This features enables replica tasks to be assigned on different racks for > fault-tolerance. > NUM_STANDBY_REPLICAS = x > totalTasks = x+1 (replica + active) > # If there are no rackID provided: Cluster will behave rack-unaware > # If same rackId is given to all the nodes: Cluster will behave rack-unaware > # If (totalTasks <= number of racks), then Cluster will be rack aware i.e. > each replica task is each assigned to a different rack. > # Id (totalTasks > number of racks), then it will first assign tasks on > different racks, further tasks will be assigned to least loaded node, cluster > wide.| > We have added another config in StreamsConfig called "RACK_ID_CONFIG" which > helps StickyPartitionAssignor to assign tasks in such a way that no two > replica tasks are on same rack if possible. > Post that it also helps to maintain stickyness with-in the rack.| -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-10772) java.lang.IllegalStateException: There are insufficient bytes available to read assignment from the sync-group response (actual byte size 0)
[
https://issues.apache.org/jira/browse/KAFKA-10772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247164#comment-17247164
]
Levani Kokhreidze commented on KAFKA-10772:
---
Thanks a lot [~cadonna] for the thorough investigation.
I'll test 2.5+ brokers and see how it goes.
> java.lang.IllegalStateException: There are insufficient bytes available to
> read assignment from the sync-group response (actual byte size 0)
>
>
> Key: KAFKA-10772
> URL: https://issues.apache.org/jira/browse/KAFKA-10772
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.6.0
>Reporter: Levani Kokhreidze
>Assignee: Bruno Cadonna
>Priority: Blocker
> Attachments: KAFKA-10772.log
>
>
> From time to time we encounter the following exception that results in Kafka
> Streams threads dying.
> Broker version 2.4.1, Client version 2.6.0
> {code:java}
> Nov 27 00:59:53.681 streaming-app service: prod | streaming-app-2 |
> stream-client [cluster1-profile-stats-pipeline-client-id] State transition
> from REBALANCING to ERROR Nov 27 00:59:53.681 streaming-app service: prod |
> streaming-app-2 | stream-client [cluster1-profile-stats-pipeline-client-id]
> State transition from REBALANCING to ERROR Nov 27 00:59:53.682 streaming-app
> service: prod | streaming-app-2 | 2020-11-27 00:59:53.681 ERROR 105 ---
> [-StreamThread-1] .KafkaStreamsBasedStreamProcessingEngine : Stream
> processing pipeline: [profile-stats] encountered unrecoverable exception.
> Thread: [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is
> completely dead. If all worker threads die, Kafka Streams will be moved to
> permanent ERROR state. Nov 27 00:59:53.682 streaming-app service: prod |
> streaming-app-2 | Stream processing pipeline: [profile-stats] encountered
> unrecoverable exception. Thread:
> [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is completely
> dead. If all worker threads die, Kafka Streams will be moved to permanent
> ERROR state. java.lang.IllegalStateException: There are insufficient bytes
> available to read assignment from the sync-group response (actual byte size
> 0) , this is not expected; it is possible that the leader's assign function
> is buggy and did not return any assignment for this member, or because static
> member is configured and the protocol is buggy hence did not get the
> assignment for this member at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:367)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:440)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:359)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:513)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1268)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1230)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1210)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:766)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:624)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:551)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:510)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-10772) java.lang.IllegalStateException: There are insufficient bytes available to read assignment from the sync-group response (actual byte size 0)
[
https://issues.apache.org/jira/browse/KAFKA-10772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17245780#comment-17245780
]
Levani Kokhreidze commented on KAFKA-10772:
---
Hi [~ableegoldman],
I doubled checked the logs, there's no log statement that says "Assigned tasks
to clients as" during the incident.
That log statement appears only after we have restarted Kafka streams process
and service started processing things successfully.
Does this help? I'll provide the logs by end of the day.
> java.lang.IllegalStateException: There are insufficient bytes available to
> read assignment from the sync-group response (actual byte size 0)
>
>
> Key: KAFKA-10772
> URL: https://issues.apache.org/jira/browse/KAFKA-10772
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.6.0
>Reporter: Levani Kokhreidze
>Priority: Blocker
> Attachments: KAFKA-10772.log
>
>
> From time to time we encounter the following exception that results in Kafka
> Streams threads dying.
> Broker version 2.4.1, Client version 2.6.0
> {code:java}
> Nov 27 00:59:53.681 streaming-app service: prod | streaming-app-2 |
> stream-client [cluster1-profile-stats-pipeline-client-id] State transition
> from REBALANCING to ERROR Nov 27 00:59:53.681 streaming-app service: prod |
> streaming-app-2 | stream-client [cluster1-profile-stats-pipeline-client-id]
> State transition from REBALANCING to ERROR Nov 27 00:59:53.682 streaming-app
> service: prod | streaming-app-2 | 2020-11-27 00:59:53.681 ERROR 105 ---
> [-StreamThread-1] .KafkaStreamsBasedStreamProcessingEngine : Stream
> processing pipeline: [profile-stats] encountered unrecoverable exception.
> Thread: [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is
> completely dead. If all worker threads die, Kafka Streams will be moved to
> permanent ERROR state. Nov 27 00:59:53.682 streaming-app service: prod |
> streaming-app-2 | Stream processing pipeline: [profile-stats] encountered
> unrecoverable exception. Thread:
> [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is completely
> dead. If all worker threads die, Kafka Streams will be moved to permanent
> ERROR state. java.lang.IllegalStateException: There are insufficient bytes
> available to read assignment from the sync-group response (actual byte size
> 0) , this is not expected; it is possible that the leader's assign function
> is buggy and did not return any assignment for this member, or because static
> member is configured and the protocol is buggy hence did not get the
> assignment for this member at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:367)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:440)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:359)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:513)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1268)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1230)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1210)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:766)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:624)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:551)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:510)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-10772) java.lang.IllegalStateException: There are insufficient bytes available to read assignment from the sync-group response (actual byte size 0)
[
https://issues.apache.org/jira/browse/KAFKA-10772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17245744#comment-17245744
]
Levani Kokhreidze commented on KAFKA-10772:
---
Hi [~ableegoldman],
Sorry, got a bit distracted. I'll see what I can find and try to add additional
logs.
> java.lang.IllegalStateException: There are insufficient bytes available to
> read assignment from the sync-group response (actual byte size 0)
>
>
> Key: KAFKA-10772
> URL: https://issues.apache.org/jira/browse/KAFKA-10772
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.6.0
>Reporter: Levani Kokhreidze
>Priority: Blocker
> Attachments: KAFKA-10772.log
>
>
> From time to time we encounter the following exception that results in Kafka
> Streams threads dying.
> Broker version 2.4.1, Client version 2.6.0
> {code:java}
> Nov 27 00:59:53.681 streaming-app service: prod | streaming-app-2 |
> stream-client [cluster1-profile-stats-pipeline-client-id] State transition
> from REBALANCING to ERROR Nov 27 00:59:53.681 streaming-app service: prod |
> streaming-app-2 | stream-client [cluster1-profile-stats-pipeline-client-id]
> State transition from REBALANCING to ERROR Nov 27 00:59:53.682 streaming-app
> service: prod | streaming-app-2 | 2020-11-27 00:59:53.681 ERROR 105 ---
> [-StreamThread-1] .KafkaStreamsBasedStreamProcessingEngine : Stream
> processing pipeline: [profile-stats] encountered unrecoverable exception.
> Thread: [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is
> completely dead. If all worker threads die, Kafka Streams will be moved to
> permanent ERROR state. Nov 27 00:59:53.682 streaming-app service: prod |
> streaming-app-2 | Stream processing pipeline: [profile-stats] encountered
> unrecoverable exception. Thread:
> [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is completely
> dead. If all worker threads die, Kafka Streams will be moved to permanent
> ERROR state. java.lang.IllegalStateException: There are insufficient bytes
> available to read assignment from the sync-group response (actual byte size
> 0) , this is not expected; it is possible that the leader's assign function
> is buggy and did not return any assignment for this member, or because static
> member is configured and the protocol is buggy hence did not get the
> assignment for this member at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:367)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:440)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:359)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:513)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1268)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1230)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1210)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:766)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:624)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:551)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:510)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-10772) java.lang.IllegalStateException: There are insufficient bytes available to read assignment from the sync-group response (actual byte size 0)
[
https://issues.apache.org/jira/browse/KAFKA-10772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17241865#comment-17241865
]
Levani Kokhreidze commented on KAFKA-10772:
---
Maybe this helps – we noticed that this problem happens mostly when there's
leadership change in the cluster.
> java.lang.IllegalStateException: There are insufficient bytes available to
> read assignment from the sync-group response (actual byte size 0)
>
>
> Key: KAFKA-10772
> URL: https://issues.apache.org/jira/browse/KAFKA-10772
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.6.0
>Reporter: Levani Kokhreidze
>Priority: Major
> Attachments: KAFKA-10772.log
>
>
> From time to time we encounter the following exception that results in Kafka
> Streams threads dying.
> Broker version 2.4.1, Client version 2.6.0
> {code:java}
> Nov 27 00:59:53.681 streaming-app service: prod | streaming-app-2 |
> stream-client [cluster1-profile-stats-pipeline-client-id] State transition
> from REBALANCING to ERROR Nov 27 00:59:53.681 streaming-app service: prod |
> streaming-app-2 | stream-client [cluster1-profile-stats-pipeline-client-id]
> State transition from REBALANCING to ERROR Nov 27 00:59:53.682 streaming-app
> service: prod | streaming-app-2 | 2020-11-27 00:59:53.681 ERROR 105 ---
> [-StreamThread-1] .KafkaStreamsBasedStreamProcessingEngine : Stream
> processing pipeline: [profile-stats] encountered unrecoverable exception.
> Thread: [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is
> completely dead. If all worker threads die, Kafka Streams will be moved to
> permanent ERROR state. Nov 27 00:59:53.682 streaming-app service: prod |
> streaming-app-2 | Stream processing pipeline: [profile-stats] encountered
> unrecoverable exception. Thread:
> [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is completely
> dead. If all worker threads die, Kafka Streams will be moved to permanent
> ERROR state. java.lang.IllegalStateException: There are insufficient bytes
> available to read assignment from the sync-group response (actual byte size
> 0) , this is not expected; it is possible that the leader's assign function
> is buggy and did not return any assignment for this member, or because static
> member is configured and the protocol is buggy hence did not get the
> assignment for this member at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:367)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:440)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:359)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:513)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1268)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1230)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1210)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:766)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:624)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:551)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:510)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-10772) java.lang.IllegalStateException: There are insufficient bytes available to read assignment from the sync-group response (actual byte size 0)
[
https://issues.apache.org/jira/browse/KAFKA-10772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17241863#comment-17241863
]
Levani Kokhreidze commented on KAFKA-10772:
---
Hi [~ableegoldman], yes we are using static membership.
I've attached logs for the incident.[^KAFKA-10772.log]
> java.lang.IllegalStateException: There are insufficient bytes available to
> read assignment from the sync-group response (actual byte size 0)
>
>
> Key: KAFKA-10772
> URL: https://issues.apache.org/jira/browse/KAFKA-10772
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.6.0
>Reporter: Levani Kokhreidze
>Priority: Major
> Attachments: KAFKA-10772.log
>
>
> From time to time we encounter the following exception that results in Kafka
> Streams threads dying.
> Broker version 2.4.1, Client version 2.6.0
> {code:java}
> Nov 27 00:59:53.681 streaming-app service: prod | streaming-app-2 |
> stream-client [cluster1-profile-stats-pipeline-client-id] State transition
> from REBALANCING to ERROR Nov 27 00:59:53.681 streaming-app service: prod |
> streaming-app-2 | stream-client [cluster1-profile-stats-pipeline-client-id]
> State transition from REBALANCING to ERROR Nov 27 00:59:53.682 streaming-app
> service: prod | streaming-app-2 | 2020-11-27 00:59:53.681 ERROR 105 ---
> [-StreamThread-1] .KafkaStreamsBasedStreamProcessingEngine : Stream
> processing pipeline: [profile-stats] encountered unrecoverable exception.
> Thread: [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is
> completely dead. If all worker threads die, Kafka Streams will be moved to
> permanent ERROR state. Nov 27 00:59:53.682 streaming-app service: prod |
> streaming-app-2 | Stream processing pipeline: [profile-stats] encountered
> unrecoverable exception. Thread:
> [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is completely
> dead. If all worker threads die, Kafka Streams will be moved to permanent
> ERROR state. java.lang.IllegalStateException: There are insufficient bytes
> available to read assignment from the sync-group response (actual byte size
> 0) , this is not expected; it is possible that the leader's assign function
> is buggy and did not return any assignment for this member, or because static
> member is configured and the protocol is buggy hence did not get the
> assignment for this member at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:367)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:440)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:359)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:513)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1268)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1230)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1210)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:766)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:624)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:551)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:510)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (KAFKA-10772) java.lang.IllegalStateException: There are insufficient bytes available to read assignment from the sync-group response (actual byte size 0)
[
https://issues.apache.org/jira/browse/KAFKA-10772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17241863#comment-17241863
]
Levani Kokhreidze edited comment on KAFKA-10772 at 12/1/20, 8:50 PM:
-
Hi [~ableegoldman], yes we are using static membership.
I've attached logs for the incident.
[^KAFKA-10772.log]
was (Author: lkokhreidze):
Hi [~ableegoldman], yes we are using static membership.
I've attached logs for the incident.[^KAFKA-10772.log]
> java.lang.IllegalStateException: There are insufficient bytes available to
> read assignment from the sync-group response (actual byte size 0)
>
>
> Key: KAFKA-10772
> URL: https://issues.apache.org/jira/browse/KAFKA-10772
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.6.0
>Reporter: Levani Kokhreidze
>Priority: Major
> Attachments: KAFKA-10772.log
>
>
> From time to time we encounter the following exception that results in Kafka
> Streams threads dying.
> Broker version 2.4.1, Client version 2.6.0
> {code:java}
> Nov 27 00:59:53.681 streaming-app service: prod | streaming-app-2 |
> stream-client [cluster1-profile-stats-pipeline-client-id] State transition
> from REBALANCING to ERROR Nov 27 00:59:53.681 streaming-app service: prod |
> streaming-app-2 | stream-client [cluster1-profile-stats-pipeline-client-id]
> State transition from REBALANCING to ERROR Nov 27 00:59:53.682 streaming-app
> service: prod | streaming-app-2 | 2020-11-27 00:59:53.681 ERROR 105 ---
> [-StreamThread-1] .KafkaStreamsBasedStreamProcessingEngine : Stream
> processing pipeline: [profile-stats] encountered unrecoverable exception.
> Thread: [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is
> completely dead. If all worker threads die, Kafka Streams will be moved to
> permanent ERROR state. Nov 27 00:59:53.682 streaming-app service: prod |
> streaming-app-2 | Stream processing pipeline: [profile-stats] encountered
> unrecoverable exception. Thread:
> [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is completely
> dead. If all worker threads die, Kafka Streams will be moved to permanent
> ERROR state. java.lang.IllegalStateException: There are insufficient bytes
> available to read assignment from the sync-group response (actual byte size
> 0) , this is not expected; it is possible that the leader's assign function
> is buggy and did not return any assignment for this member, or because static
> member is configured and the protocol is buggy hence did not get the
> assignment for this member at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:367)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:440)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:359)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:513)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1268)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1230)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1210)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:766)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:624)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:551)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:510)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-10772) java.lang.IllegalStateException: There are insufficient bytes available to read assignment from the sync-group response (actual byte size 0)
[
https://issues.apache.org/jira/browse/KAFKA-10772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Levani Kokhreidze updated KAFKA-10772:
--
Attachment: KAFKA-10772.log
> java.lang.IllegalStateException: There are insufficient bytes available to
> read assignment from the sync-group response (actual byte size 0)
>
>
> Key: KAFKA-10772
> URL: https://issues.apache.org/jira/browse/KAFKA-10772
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.6.0
>Reporter: Levani Kokhreidze
>Priority: Major
> Attachments: KAFKA-10772.log
>
>
> From time to time we encounter the following exception that results in Kafka
> Streams threads dying.
> Broker version 2.4.1, Client version 2.6.0
> {code:java}
> Nov 27 00:59:53.681 streaming-app service: prod | streaming-app-2 |
> stream-client [cluster1-profile-stats-pipeline-client-id] State transition
> from REBALANCING to ERROR Nov 27 00:59:53.681 streaming-app service: prod |
> streaming-app-2 | stream-client [cluster1-profile-stats-pipeline-client-id]
> State transition from REBALANCING to ERROR Nov 27 00:59:53.682 streaming-app
> service: prod | streaming-app-2 | 2020-11-27 00:59:53.681 ERROR 105 ---
> [-StreamThread-1] .KafkaStreamsBasedStreamProcessingEngine : Stream
> processing pipeline: [profile-stats] encountered unrecoverable exception.
> Thread: [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is
> completely dead. If all worker threads die, Kafka Streams will be moved to
> permanent ERROR state. Nov 27 00:59:53.682 streaming-app service: prod |
> streaming-app-2 | Stream processing pipeline: [profile-stats] encountered
> unrecoverable exception. Thread:
> [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is completely
> dead. If all worker threads die, Kafka Streams will be moved to permanent
> ERROR state. java.lang.IllegalStateException: There are insufficient bytes
> available to read assignment from the sync-group response (actual byte size
> 0) , this is not expected; it is possible that the leader's assign function
> is buggy and did not return any assignment for this member, or because static
> member is configured and the protocol is buggy hence did not get the
> assignment for this member at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:367)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:440)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:359)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:513)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1268)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1230)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1210)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:766)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:624)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:551)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:510)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-10772) java.lang.IllegalStateException: There are insufficient bytes available to read assignment from the sync-group response (actual byte size 0)
[
https://issues.apache.org/jira/browse/KAFKA-10772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Levani Kokhreidze updated KAFKA-10772:
--
Description:
>From time to time we encounter the following exception that results in Kafka
>Streams threads dying.
Broker version 2.4.1, Client version 2.6.0
{code:java}
Nov 27 00:59:53.681 streaming-app service: prod | streaming-app-2 |
stream-client [cluster1-profile-stats-pipeline-client-id] State transition from
REBALANCING to ERROR Nov 27 00:59:53.681 streaming-app service: prod |
streaming-app-2 | stream-client [cluster1-profile-stats-pipeline-client-id]
State transition from REBALANCING to ERROR Nov 27 00:59:53.682 streaming-app
service: prod | streaming-app-2 | 2020-11-27 00:59:53.681 ERROR 105 ---
[-StreamThread-1] .KafkaStreamsBasedStreamProcessingEngine : Stream processing
pipeline: [profile-stats] encountered unrecoverable exception. Thread:
[cluster1-profile-stats-pipeline-client-id-StreamThread-1] is completely dead.
If all worker threads die, Kafka Streams will be moved to permanent ERROR
state. Nov 27 00:59:53.682 streaming-app service: prod | streaming-app-2 |
Stream processing pipeline: [profile-stats] encountered unrecoverable
exception. Thread: [cluster1-profile-stats-pipeline-client-id-StreamThread-1]
is completely dead. If all worker threads die, Kafka Streams will be moved to
permanent ERROR state. java.lang.IllegalStateException: There are insufficient
bytes available to read assignment from the sync-group response (actual byte
size 0) , this is not expected; it is possible that the leader's assign
function is buggy and did not return any assignment for this member, or because
static member is configured and the protocol is buggy hence did not get the
assignment for this member at
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:367)
at
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:440)
at
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:359)
at
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:513)
at
org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1268)
at
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1230)
at
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1210)
at
org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:766)
at
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:624)
at
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:551)
at
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:510)
{code}
was:
>From time to time we encounter the following exception that results in Kafka
>Streams threads dying.
{code:java}
Nov 27 00:59:53.681 streaming-app service: prod | streaming-app-2 |
stream-client [cluster1-profile-stats-pipeline-client-id] State transition from
REBALANCING to ERROR Nov 27 00:59:53.681 streaming-app service: prod |
streaming-app-2 | stream-client [cluster1-profile-stats-pipeline-client-id]
State transition from REBALANCING to ERROR Nov 27 00:59:53.682 streaming-app
service: prod | streaming-app-2 | 2020-11-27 00:59:53.681 ERROR 105 ---
[-StreamThread-1] .KafkaStreamsBasedStreamProcessingEngine : Stream processing
pipeline: [profile-stats] encountered unrecoverable exception. Thread:
[cluster1-profile-stats-pipeline-client-id-StreamThread-1] is completely dead.
If all worker threads die, Kafka Streams will be moved to permanent ERROR
state. Nov 27 00:59:53.682 streaming-app service: prod | streaming-app-2 |
Stream processing pipeline: [profile-stats] encountered unrecoverable
exception. Thread: [cluster1-profile-stats-pipeline-client-id-StreamThread-1]
is completely dead. If all worker threads die, Kafka Streams will be moved to
permanent ERROR state. java.lang.IllegalStateException: There are insufficient
bytes available to read assignment from the sync-group response (actual byte
size 0) , this is not expected; it is possible that the leader's assign
function is buggy and did not return any assignment for this member, or because
static member is configured and the protocol is buggy hence did not get the
assignment for this member at
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:367)
at
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:440)
at
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:359)
at
[jira] [Updated] (KAFKA-10772) java.lang.IllegalStateException: There are insufficient bytes available to read assignment from the sync-group response (actual byte size 0)
[
https://issues.apache.org/jira/browse/KAFKA-10772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Levani Kokhreidze updated KAFKA-10772:
--
Affects Version/s: 2.6.0
> java.lang.IllegalStateException: There are insufficient bytes available to
> read assignment from the sync-group response (actual byte size 0)
>
>
> Key: KAFKA-10772
> URL: https://issues.apache.org/jira/browse/KAFKA-10772
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.6.0
>Reporter: Levani Kokhreidze
>Priority: Major
>
> From time to time we encounter the following exception that results in Kafka
> Streams threads dying.
> {code:java}
> Nov 27 00:59:53.681 streaming-app service: prod | streaming-app-2 |
> stream-client [cluster1-profile-stats-pipeline-client-id] State transition
> from REBALANCING to ERROR Nov 27 00:59:53.681 streaming-app service: prod |
> streaming-app-2 | stream-client [cluster1-profile-stats-pipeline-client-id]
> State transition from REBALANCING to ERROR Nov 27 00:59:53.682 streaming-app
> service: prod | streaming-app-2 | 2020-11-27 00:59:53.681 ERROR 105 ---
> [-StreamThread-1] .KafkaStreamsBasedStreamProcessingEngine : Stream
> processing pipeline: [profile-stats] encountered unrecoverable exception.
> Thread: [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is
> completely dead. If all worker threads die, Kafka Streams will be moved to
> permanent ERROR state. Nov 27 00:59:53.682 streaming-app service: prod |
> streaming-app-2 | Stream processing pipeline: [profile-stats] encountered
> unrecoverable exception. Thread:
> [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is completely
> dead. If all worker threads die, Kafka Streams will be moved to permanent
> ERROR state. java.lang.IllegalStateException: There are insufficient bytes
> available to read assignment from the sync-group response (actual byte size
> 0) , this is not expected; it is possible that the leader's assign function
> is buggy and did not return any assignment for this member, or because static
> member is configured and the protocol is buggy hence did not get the
> assignment for this member at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:367)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:440)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:359)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:513)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1268)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1230)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1210)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:766)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:624)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:551)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:510)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-10772) java.lang.IllegalStateException: There are insufficient bytes available to read assignment from the sync-group response (actual byte size 0)
[
https://issues.apache.org/jira/browse/KAFKA-10772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17239528#comment-17239528
]
Levani Kokhreidze commented on KAFKA-10772:
---
cc [~ableegoldman] [~guozhang]
There have been some discussions around this exception in
https://issues.apache.org/jira/browse/KAFKA-10134 but I couldn't find report
around this bug. Not sure if 2.6.1 release addresses this problem, but we see
this exception occurring that causes Kafka Streams to move to ERROR state.
> java.lang.IllegalStateException: There are insufficient bytes available to
> read assignment from the sync-group response (actual byte size 0)
>
>
> Key: KAFKA-10772
> URL: https://issues.apache.org/jira/browse/KAFKA-10772
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Reporter: Levani Kokhreidze
>Priority: Major
>
> From time to time we encounter the following exception that results in Kafka
> Streams threads dying.
> {code:java}
> Nov 27 00:59:53.681 streaming-app service: prod | streaming-app-2 |
> stream-client [cluster1-profile-stats-pipeline-client-id] State transition
> from REBALANCING to ERROR Nov 27 00:59:53.681 streaming-app service: prod |
> streaming-app-2 | stream-client [cluster1-profile-stats-pipeline-client-id]
> State transition from REBALANCING to ERROR Nov 27 00:59:53.682 streaming-app
> service: prod | streaming-app-2 | 2020-11-27 00:59:53.681 ERROR 105 ---
> [-StreamThread-1] .KafkaStreamsBasedStreamProcessingEngine : Stream
> processing pipeline: [profile-stats] encountered unrecoverable exception.
> Thread: [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is
> completely dead. If all worker threads die, Kafka Streams will be moved to
> permanent ERROR state. Nov 27 00:59:53.682 streaming-app service: prod |
> streaming-app-2 | Stream processing pipeline: [profile-stats] encountered
> unrecoverable exception. Thread:
> [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is completely
> dead. If all worker threads die, Kafka Streams will be moved to permanent
> ERROR state. java.lang.IllegalStateException: There are insufficient bytes
> available to read assignment from the sync-group response (actual byte size
> 0) , this is not expected; it is possible that the leader's assign function
> is buggy and did not return any assignment for this member, or because static
> member is configured and the protocol is buggy hence did not get the
> assignment for this member at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:367)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:440)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:359)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:513)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1268)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1230)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1210)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:766)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:624)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:551)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:510)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KAFKA-10772) java.lang.IllegalStateException: There are insufficient bytes available to read assignment from the sync-group response (actual byte size 0)
Levani Kokhreidze created KAFKA-10772:
-
Summary: java.lang.IllegalStateException: There are insufficient
bytes available to read assignment from the sync-group response (actual byte
size 0)
Key: KAFKA-10772
URL: https://issues.apache.org/jira/browse/KAFKA-10772
Project: Kafka
Issue Type: Bug
Components: streams
Reporter: Levani Kokhreidze
>From time to time we encounter the following exception that results in Kafka
>Streams threads dying.
{code:java}
Nov 27 00:59:53.681 streaming-app service: prod | streaming-app-2 |
stream-client [cluster1-profile-stats-pipeline-client-id] State transition from
REBALANCING to ERROR Nov 27 00:59:53.681 streaming-app service: prod |
streaming-app-2 | stream-client [cluster1-profile-stats-pipeline-client-id]
State transition from REBALANCING to ERROR Nov 27 00:59:53.682 streaming-app
service: prod | streaming-app-2 | 2020-11-27 00:59:53.681 ERROR 105 ---
[-StreamThread-1] .KafkaStreamsBasedStreamProcessingEngine : Stream processing
pipeline: [profile-stats] encountered unrecoverable exception. Thread:
[cluster1-profile-stats-pipeline-client-id-StreamThread-1] is completely dead.
If all worker threads die, Kafka Streams will be moved to permanent ERROR
state. Nov 27 00:59:53.682 streaming-app service: prod | streaming-app-2 |
Stream processing pipeline: [profile-stats] encountered unrecoverable
exception. Thread: [cluster1-profile-stats-pipeline-client-id-StreamThread-1]
is completely dead. If all worker threads die, Kafka Streams will be moved to
permanent ERROR state. java.lang.IllegalStateException: There are insufficient
bytes available to read assignment from the sync-group response (actual byte
size 0) , this is not expected; it is possible that the leader's assign
function is buggy and did not return any assignment for this member, or because
static member is configured and the protocol is buggy hence did not get the
assignment for this member at
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:367)
at
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:440)
at
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:359)
at
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:513)
at
org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1268)
at
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1230)
at
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1210)
at
org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:766)
at
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:624)
at
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:551)
at
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:510)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (KAFKA-10686) Pluggable standby tasks assignor for Kafka Streams
[ https://issues.apache.org/jira/browse/KAFKA-10686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Levani Kokhreidze reassigned KAFKA-10686: - Assignee: Levani Kokhreidze > Pluggable standby tasks assignor for Kafka Streams > -- > > Key: KAFKA-10686 > URL: https://issues.apache.org/jira/browse/KAFKA-10686 > Project: Kafka > Issue Type: New Feature > Components: streams >Reporter: Levani Kokhreidze >Assignee: Levani Kokhreidze >Priority: Major > Labels: needs-kip > > In production, Kafka Streams instances often run across different clusters > and availability zones. In order to guarantee high availability of the Kafka > Streams deployments, users would need more granular control over which > instances standby tasks can be created. > Idea of this ticket is to expose interface for Kafka Streams which can be > implemented by the users to control where standby tasks can be created. > Kafka Streams can have RackAware assignment as a default implementation that > will take into account `rack.id` of the application and make sure that > standby tasks are created on different racks. > Point of this ticket though is to give more flexibility to users on standby > task creation, in cases where just rack awareness is not enough. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-10686) Pluggable standby tasks assignor for Kafka Streams
[ https://issues.apache.org/jira/browse/KAFKA-10686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Levani Kokhreidze updated KAFKA-10686: -- Issue Type: New Feature (was: Improvement) > Pluggable standby tasks assignor for Kafka Streams > -- > > Key: KAFKA-10686 > URL: https://issues.apache.org/jira/browse/KAFKA-10686 > Project: Kafka > Issue Type: New Feature > Components: streams >Reporter: Levani Kokhreidze >Priority: Major > Labels: needs-kip > > In production, Kafka Streams instances often run across different clusters > and availability zones. In order to guarantee high availability of the Kafka > Streams deployments, users would need more granular control over which > instances standby tasks can be created. > Idea of this ticket is to expose interface for Kafka Streams which can be > implemented by the users to control where standby tasks can be created. > Kafka Streams can have RackAware assignment as a default implementation that > will take into account `rack.id` of the application and make sure that > standby tasks are created on different racks. > Point of this ticket though is to give more flexibility to users on standby > task creation, in cases where just rack awareness is not enough. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-10686) Pluggable standby tasks assignor for Kafka Streams
[ https://issues.apache.org/jira/browse/KAFKA-10686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17226832#comment-17226832 ] Levani Kokhreidze commented on KAFKA-10686: --- Hi [~cadonna]! Yes, that's exactly write. I'll definitely look into the KIP-441 to get some inspiration. Thanks for the tip! > Pluggable standby tasks assignor for Kafka Streams > -- > > Key: KAFKA-10686 > URL: https://issues.apache.org/jira/browse/KAFKA-10686 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Levani Kokhreidze >Priority: Major > Labels: needs-kip > > In production, Kafka Streams instances often run across different clusters > and availability zones. In order to guarantee high availability of the Kafka > Streams deployments, users would need more granular control over which > instances standby tasks can be created. > Idea of this ticket is to expose interface for Kafka Streams which can be > implemented by the users to control where standby tasks can be created. > Kafka Streams can have RackAware assignment as a default implementation that > will take into account `rack.id` of the application and make sure that > standby tasks are created on different racks. > Point of this ticket though is to give more flexibility to users on standby > task creation, in cases where just rack awareness is not enough. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (KAFKA-10686) Pluggable standby tasks assignor for Kafka Streams
[ https://issues.apache.org/jira/browse/KAFKA-10686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17226832#comment-17226832 ] Levani Kokhreidze edited comment on KAFKA-10686 at 11/5/20, 4:55 PM: - Hi [~cadonna]! Yes, that's exactly right. I'll definitely look into the KIP-441 to get some inspiration. Thanks for the tip! was (Author: lkokhreidze): Hi [~cadonna]! Yes, that's exactly write. I'll definitely look into the KIP-441 to get some inspiration. Thanks for the tip! > Pluggable standby tasks assignor for Kafka Streams > -- > > Key: KAFKA-10686 > URL: https://issues.apache.org/jira/browse/KAFKA-10686 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Levani Kokhreidze >Priority: Major > Labels: needs-kip > > In production, Kafka Streams instances often run across different clusters > and availability zones. In order to guarantee high availability of the Kafka > Streams deployments, users would need more granular control over which > instances standby tasks can be created. > Idea of this ticket is to expose interface for Kafka Streams which can be > implemented by the users to control where standby tasks can be created. > Kafka Streams can have RackAware assignment as a default implementation that > will take into account `rack.id` of the application and make sure that > standby tasks are created on different racks. > Point of this ticket though is to give more flexibility to users on standby > task creation, in cases where just rack awareness is not enough. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-10686) Pluggable standby tasks assignor for Kafka Streams
[ https://issues.apache.org/jira/browse/KAFKA-10686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Levani Kokhreidze updated KAFKA-10686: -- Description: In production, Kafka Streams instances often run across different clusters and availability zones. In order to guarantee high availability of the Kafka Streams deployments, users would need more granular control over which instances standby tasks can be created. Idea of this ticket is to expose interface for Kafka Streams which can be implemented by the users to control where standby tasks can be created. Kafka Streams can have RackAware assignment as a default implementation that will take into account `rack.id` of the application and make sure that standby tasks are created on different racks. Point of this ticket though is to give more flexibility to users on standby task creation, in cases where just rack awareness is not enough. was: In production, Kafka Streams instances often run across different clusters and availability zones. In order to guarantee high availability of the Kafka Streams deployments, users would need more granular control over which instances standby tasks can be created. Idea of this ticket is to expose interface for Kafka Streams which can be implemented by users to control where standby tasks can be created. Kafka Streams can have RackAware assignment as a default implementation that will take into account `rack.id` of the application and make sure that standby tasks are created on different racks. Point of this ticket though is to give more flexibility to users on standby task creation, in cases where just rack awareness is not enough. > Pluggable standby tasks assignor for Kafka Streams > -- > > Key: KAFKA-10686 > URL: https://issues.apache.org/jira/browse/KAFKA-10686 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Levani Kokhreidze >Priority: Major > Labels: needs-kip > > In production, Kafka Streams instances often run across different clusters > and availability zones. In order to guarantee high availability of the Kafka > Streams deployments, users would need more granular control over which > instances standby tasks can be created. > Idea of this ticket is to expose interface for Kafka Streams which can be > implemented by the users to control where standby tasks can be created. > Kafka Streams can have RackAware assignment as a default implementation that > will take into account `rack.id` of the application and make sure that > standby tasks are created on different racks. > Point of this ticket though is to give more flexibility to users on standby > task creation, in cases where just rack awareness is not enough. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-10686) Pluggable standby tasks assignor for Kafka Streams
[ https://issues.apache.org/jira/browse/KAFKA-10686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Levani Kokhreidze updated KAFKA-10686: -- Labels: needs-kip (was: ) > Pluggable standby tasks assignor for Kafka Streams > -- > > Key: KAFKA-10686 > URL: https://issues.apache.org/jira/browse/KAFKA-10686 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Levani Kokhreidze >Priority: Major > Labels: needs-kip > > In production, Kafka Streams instances often run across different clusters > and availability zones. In order to guarantee high availability of the Kafka > Streams deployments, users would need more granular control over which > instances standby tasks can be created. > Idea of this ticket is to expose interface for Kafka Streams which can be > implemented by users to control where standby tasks can be created. > Kafka Streams can have RackAware assignment as a default implementation that > will take into account `rack.id` of the application and make sure that > standby tasks are created on different racks. > Point of this ticket though is to give more flexibility to users on standby > task creation, in cases where just rack awareness is not enough. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-10686) Pluggable standby tasks assignor for Kafka Streams
[ https://issues.apache.org/jira/browse/KAFKA-10686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Levani Kokhreidze updated KAFKA-10686: -- Description: In production, Kafka Streams instances often run across different clusters and availability zones. In order to guarantee high availability of the Kafka Streams deployments, users would need more granular control over which instances standby tasks can be created. Idea of this ticket is to expose interface for Kafka Streams which can be implemented by users to control where standby tasks can be created. Kafka Streams can have RackAware assignment as a default implementation that will take into account `rack.id` of the application and make sure that standby tasks are created on different racks. Point of this ticket though is to give more flexibility to users on standby task creation, in cases where just rack awareness is not enough. was: In production, Kafka Streams instances often run across different clusters and availability zones. In order to guarantee high availability of the Kafka Streams deployments, users would need more granular control over which on instances standby tasks can be created. Idea of this ticket is to expose interface for Kafka Streams which can be implemented by users to control where standby tasks can be created. Kafka Streams can have RackAware assignment as a default implementation that will take into account `rack.id` of the application and make sure that standby tasks are created on different racks. Point of this ticket though is to give more flexibility to users on standby task creation, in cases where just rack awareness is not enough. > Pluggable standby tasks assignor for Kafka Streams > -- > > Key: KAFKA-10686 > URL: https://issues.apache.org/jira/browse/KAFKA-10686 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Levani Kokhreidze >Priority: Major > > In production, Kafka Streams instances often run across different clusters > and availability zones. In order to guarantee high availability of the Kafka > Streams deployments, users would need more granular control over which > instances standby tasks can be created. > Idea of this ticket is to expose interface for Kafka Streams which can be > implemented by users to control where standby tasks can be created. > Kafka Streams can have RackAware assignment as a default implementation that > will take into account `rack.id` of the application and make sure that > standby tasks are created on different racks. > Point of this ticket though is to give more flexibility to users on standby > task creation, in cases where just rack awareness is not enough. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-10686) Pluggable standby tasks assignor for Kafka Streams
[ https://issues.apache.org/jira/browse/KAFKA-10686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Levani Kokhreidze updated KAFKA-10686: -- Description: In production, Kafka Streams instances often run across different clusters and availability zones. In order to guarantee high availability of the Kafka Streams deployments, users would need more granular control over which on instances standby tasks can be created. Idea of this ticket is to expose interface for Kafka Streams which can be implemented by users to control where standby tasks can be created. Kafka Streams can have RackAware assignment as a default implementation that will take into account `rack.id` of the application and make sure that standby tasks are created on different racks. Point of this ticket though is to give more flexibility to users on standby task creation, in cases where just rack awareness is not enough. was: In production, Kafka Streams instances often run across different clusters and availability zones. In order to guarantee high availability of the Kafka Streams deployments, users would need more granular control over which on instances standby tasks can be created. Idea of this ticket is to expose interface for Kafka Streams which can be implemented by users to control where standby tasks can be created. Kafka Streams can have RackAware assignment as a default implementation that will take into account `rack.id` of the application and make sure that standby tasks are created on different racks. Point of this ticket though is to more flexibility to users on standby task creation, in cases where just rack awareness is not enough. > Pluggable standby tasks assignor for Kafka Streams > -- > > Key: KAFKA-10686 > URL: https://issues.apache.org/jira/browse/KAFKA-10686 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Levani Kokhreidze >Priority: Major > > In production, Kafka Streams instances often run across different clusters > and availability zones. In order to guarantee high availability of the Kafka > Streams deployments, users would need more granular control over which on > instances standby tasks can be created. > Idea of this ticket is to expose interface for Kafka Streams which can be > implemented by users to control where standby tasks can be created. > Kafka Streams can have RackAware assignment as a default implementation that > will take into account `rack.id` of the application and make sure that > standby tasks are created on different racks. > Point of this ticket though is to give more flexibility to users on standby > task creation, in cases where just rack awareness is not enough. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-10686) Pluggable standby tasks assignor for Kafka Streams
[ https://issues.apache.org/jira/browse/KAFKA-10686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17226719#comment-17226719 ] Levani Kokhreidze commented on KAFKA-10686: --- If this ticket makes sense, I'd be keen to work on this and provide KIP. Would love to incorporate https://issues.apache.org/jira/browse/KAFKA-6718 as part of this KIP. > Pluggable standby tasks assignor for Kafka Streams > -- > > Key: KAFKA-10686 > URL: https://issues.apache.org/jira/browse/KAFKA-10686 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Levani Kokhreidze >Priority: Major > > In production, Kafka Streams instances often run across different clusters > and availability zones. In order to guarantee high availability of the Kafka > Streams deployments, users would need more granular control over which on > instances standby tasks can be created. > Idea of this ticket is to expose interface for Kafka Streams which can be > implemented by users to control where standby tasks can be created. > Kafka Streams can have RackAware assignment as a default implementation that > will take into account `rack.id` of the application and make sure that > standby tasks are created on different racks. > Point of this ticket though is to more flexibility to users on standby task > creation, in cases where just rack awareness is not enough. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-10686) Pluggable standby tasks assignor for Kafka Streams
Levani Kokhreidze created KAFKA-10686: - Summary: Pluggable standby tasks assignor for Kafka Streams Key: KAFKA-10686 URL: https://issues.apache.org/jira/browse/KAFKA-10686 Project: Kafka Issue Type: Improvement Components: streams Reporter: Levani Kokhreidze In production, Kafka Streams instances often run across different clusters and availability zones. In order to guarantee high availability of the Kafka Streams deployments, users would need more granular control over which on instances standby tasks can be created. Idea of this ticket is to expose interface for Kafka Streams which can be implemented by users to control where standby tasks can be created. Kafka Streams can have RackAware assignment as a default implementation that will take into account `rack.id` of the application and make sure that standby tasks are created on different racks. Point of this ticket though is to more flexibility to users on standby task creation, in cases where just rack awareness is not enough. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-10454) Kafka Streams Stuck in infinite REBALANCING loop when stream <> table join partitions don't match
[
https://issues.apache.org/jira/browse/KAFKA-10454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17193155#comment-17193155
]
Levani Kokhreidze commented on KAFKA-10454:
---
Hey [~vvcephei] thanks for the feedback.
Finally got time today to look into this issue. I've managed to find the cause
and chose slightly different approach then what you suggested.
Would love you to get your feedback on PR
[https://github.com/apache/kafka/pull/9237]
> Kafka Streams Stuck in infinite REBALANCING loop when stream <> table join
> partitions don't match
> -
>
> Key: KAFKA-10454
> URL: https://issues.apache.org/jira/browse/KAFKA-10454
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.6.0
>Reporter: Levani Kokhreidze
>Assignee: Levani Kokhreidze
>Priority: Major
>
> Here's integration test: [https://github.com/apache/kafka/pull/9237]
>
> From the first glance, issue is that when one joins stream to table, and
> table source topic doesn't have same number of partitions as stream topic,
> `StateChangelogReader` tries to recover state from changelog (which in this
> case is the same as source topic) for table from partitions that don't exist.
> Logs are spammed with:
>
> {code:java}
> [2020-09-01 12:33:07,508] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,508] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,508] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-3 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,510] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-0 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,510] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,510] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,510] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-3 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,513] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-0 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,513] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,513] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
>
[jira] [Assigned] (KAFKA-10454) Kafka Streams Stuck in infinite REBALANCING loop when stream <> table join partitions don't match
[
https://issues.apache.org/jira/browse/KAFKA-10454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Levani Kokhreidze reassigned KAFKA-10454:
-
Assignee: Levani Kokhreidze
> Kafka Streams Stuck in infinite REBALANCING loop when stream <> table join
> partitions don't match
> -
>
> Key: KAFKA-10454
> URL: https://issues.apache.org/jira/browse/KAFKA-10454
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.6.0
>Reporter: Levani Kokhreidze
>Assignee: Levani Kokhreidze
>Priority: Major
>
> Here's integration test: [https://github.com/apache/kafka/pull/9237]
>
> From the first glance, issue is that when one joins stream to table, and
> table source topic doesn't have same number of partitions as stream topic,
> `StateChangelogReader` tries to recover state from changelog (which in this
> case is the same as source topic) for table from partitions that don't exist.
> Logs are spammed with:
>
> {code:java}
> [2020-09-01 12:33:07,508] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,508] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,508] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-3 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,510] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-0 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,510] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,510] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,510] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-3 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,513] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-0 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,513] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,513] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,513] INFO stream-thread
>
[jira] [Comment Edited] (KAFKA-10454) Kafka Streams Stuck in infinite REBALANCING loop when stream <> table join partitions don't match
[
https://issues.apache.org/jira/browse/KAFKA-10454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17188319#comment-17188319
]
Levani Kokhreidze edited comment on KAFKA-10454 at 9/1/20, 10:05 AM:
-
I guess, in this scenario, Kafka Streams should do co-partitioning check and
enforce num of partitions inherited from table source topic?
was (Author: lkokhreidze):
I guess, in this scenario, Kafka Streams should do co-partitioning check and
should fail early on with some meaningful error message to end user.
> Kafka Streams Stuck in infinite REBALANCING loop when stream <> table join
> partitions don't match
> -
>
> Key: KAFKA-10454
> URL: https://issues.apache.org/jira/browse/KAFKA-10454
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.6.0
>Reporter: Levani Kokhreidze
>Priority: Major
>
> Here's integration test: [https://github.com/apache/kafka/pull/9237]
>
> From the first glance, issue is that when one joins stream to table, and
> table source topic doesn't have same number of partitions as stream topic,
> `StateChangelogReader` tries to recover state from changelog (which in this
> case is the same as source topic) for table from partitions that don't exist.
> Logs are spammed with:
>
> {code:java}
> [2020-09-01 12:33:07,508] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,508] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,508] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-3 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,510] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-0 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,510] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,510] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,510] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-3 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,513] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-0 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,513] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,513] INFO stream-thread
>
[jira] [Comment Edited] (KAFKA-10454) Kafka Streams Stuck in infinite REBALANCING loop when stream <> table join partitions don't match
[
https://issues.apache.org/jira/browse/KAFKA-10454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17188334#comment-17188334
]
Levani Kokhreidze edited comment on KAFKA-10454 at 9/1/20, 9:58 AM:
Workaround we have used so far is to force repartitioning with num of
partitions that equal to table source topic. Based on example from the
integration test posted above, it would look like this:
{code:java}
stream
.selectKey((key, value) -> key)
.repartition(Repartitioned.numberOfPartitions(2))
.join(table, (value1, value2) -> value2)
.to(outputTopic);
{code}
was (Author: lkokhreidze):
Workaround we have used so far is to force repartitioning with num of
partitions that equal to table source topic. Based on example from the
integration test posted above, it would look like this:
{code:java}
/stream
.selectKey((key, value) -> key)
.repartition(Repartitioned.numberOfPartitions(2))
.join(table, (value1, value2) -> value2)
.to(outputTopic);
{code}
> Kafka Streams Stuck in infinite REBALANCING loop when stream <> table join
> partitions don't match
> -
>
> Key: KAFKA-10454
> URL: https://issues.apache.org/jira/browse/KAFKA-10454
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.6.0
>Reporter: Levani Kokhreidze
>Priority: Major
>
> Here's integration test: [https://github.com/apache/kafka/pull/9237]
>
> From the first glance, issue is that when one joins stream to table, and
> table source topic doesn't have same number of partitions as stream topic,
> `StateChangelogReader` tries to recover state from changelog (which in this
> case is the same as source topic) for table from partitions that don't exist.
> Logs are spammed with:
>
> {code:java}
> [2020-09-01 12:33:07,508] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,508] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,508] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-3 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,510] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-0 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,510] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,510] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,510] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-3 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,513] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-0 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01
[jira] [Commented] (KAFKA-10454) Kafka Streams Stuck in infinite REBALANCING loop when stream <> table join partitions don't match
[
https://issues.apache.org/jira/browse/KAFKA-10454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17188334#comment-17188334
]
Levani Kokhreidze commented on KAFKA-10454:
---
Workaround we have used so far is to force repartitioning with num of
partitions that equal to table source topic. Based on example from the
integration test posted above, it would look like this:
{code:java}
/stream
.selectKey((key, value) -> key)
.repartition(Repartitioned.numberOfPartitions(2))
.join(table, (value1, value2) -> value2)
.to(outputTopic);
{code}
> Kafka Streams Stuck in infinite REBALANCING loop when stream <> table join
> partitions don't match
> -
>
> Key: KAFKA-10454
> URL: https://issues.apache.org/jira/browse/KAFKA-10454
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.6.0
>Reporter: Levani Kokhreidze
>Priority: Major
>
> Here's integration test: [https://github.com/apache/kafka/pull/9237]
>
> From the first glance, issue is that when one joins stream to table, and
> table source topic doesn't have same number of partitions as stream topic,
> `StateChangelogReader` tries to recover state from changelog (which in this
> case is the same as source topic) for table from partitions that don't exist.
> Logs are spammed with:
>
> {code:java}
> [2020-09-01 12:33:07,508] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,508] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,508] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-3 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,510] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-0 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,510] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,510] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,510] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-3 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,513] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-0 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,513] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,513] INFO stream-thread
>
[jira] [Updated] (KAFKA-10454) Kafka Streams Stuck in infinite REBALANCING loop when stream <> table join partitions don't match
[
https://issues.apache.org/jira/browse/KAFKA-10454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Levani Kokhreidze updated KAFKA-10454:
--
Affects Version/s: 2.6.0
> Kafka Streams Stuck in infinite REBALANCING loop when stream <> table join
> partitions don't match
> -
>
> Key: KAFKA-10454
> URL: https://issues.apache.org/jira/browse/KAFKA-10454
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.6.0
>Reporter: Levani Kokhreidze
>Priority: Major
>
> Here's integration test: [https://github.com/apache/kafka/pull/9237]
>
> From the first glance, issue is that when one joins stream to table, and
> table source topic doesn't have same number of partitions as stream topic,
> `StateChangelogReader` tries to recover state from changelog (which in this
> case is the same as source topic) for table from partitions that don't exist.
> Logs are spammed with:
>
> {code:java}
> [2020-09-01 12:33:07,508] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,508] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,508] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-3 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,510] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-0 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,510] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,510] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,510] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-3 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,513] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-0 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,513] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,513] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,513] INFO stream-thread
>
[jira] [Updated] (KAFKA-10454) Kafka Streams Stuck in infinite REBALANCING loop when stream <> table join partitions don't match
[
https://issues.apache.org/jira/browse/KAFKA-10454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Levani Kokhreidze updated KAFKA-10454:
--
Fix Version/s: (was: 2.6.0)
> Kafka Streams Stuck in infinite REBALANCING loop when stream <> table join
> partitions don't match
> -
>
> Key: KAFKA-10454
> URL: https://issues.apache.org/jira/browse/KAFKA-10454
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Reporter: Levani Kokhreidze
>Priority: Major
>
> Here's integration test: [https://github.com/apache/kafka/pull/9237]
>
> From the first glance, issue is that when one joins stream to table, and
> table source topic doesn't have same number of partitions as stream topic,
> `StateChangelogReader` tries to recover state from changelog (which in this
> case is the same as source topic) for table from partitions that don't exist.
> Logs are spammed with:
>
> {code:java}
> [2020-09-01 12:33:07,508] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,508] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,508] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-3 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,510] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-0 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,510] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,510] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,510] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-3 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,513] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-0 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,513] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,513] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,513] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset
[jira] [Commented] (KAFKA-10454) Kafka Streams Stuck in infinite REBALANCING loop when stream <> table join partitions don't match
[
https://issues.apache.org/jira/browse/KAFKA-10454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17188319#comment-17188319
]
Levani Kokhreidze commented on KAFKA-10454:
---
I guess, in this scenario, Kafka Streams should do co-partitioning check and
should fail early on with some meaningful error message to end user.
> Kafka Streams Stuck in infinite REBALANCING loop when stream <> table join
> partitions don't match
> -
>
> Key: KAFKA-10454
> URL: https://issues.apache.org/jira/browse/KAFKA-10454
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Reporter: Levani Kokhreidze
>Priority: Major
> Fix For: 2.6.0
>
>
> Here's integration test: [https://github.com/apache/kafka/pull/9237]
>
> From the first glance, issue is that when one joins stream to table, and
> table source topic doesn't have same number of partitions as stream topic,
> `StateChangelogReader` tries to recover state from changelog (which in this
> case is the same as source topic) for table from partitions that don't exist.
> Logs are spammed with:
>
> {code:java}
> [2020-09-01 12:33:07,508] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,508] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,508] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-3 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,510] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-0 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,510] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,510] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,510] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-3 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,513] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-0 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,513] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,513] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found;
> will retry in the next time.
>
[jira] [Comment Edited] (KAFKA-10454) Kafka Streams Stuck in infinite REBALANCING loop when stream <> table join partitions don't match
[
https://issues.apache.org/jira/browse/KAFKA-10454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17188313#comment-17188313
]
Levani Kokhreidze edited comment on KAFKA-10454 at 9/1/20, 9:43 AM:
CC [~mjsax] [~vvcephei] [~guozhang]
I may try to fix this as soon as I can - but if by any chance somebody wants to
pick this up before me, feel free to do so.
was (Author: lkokhreidze):
CC [~mjsax] [~vvcephei] [~guozhang]
I may try to fix this as soon as I can - but if any chance somebody wants to
pick it up before me, feel free to do so.
> Kafka Streams Stuck in infinite REBALANCING loop when stream <> table join
> partitions don't match
> -
>
> Key: KAFKA-10454
> URL: https://issues.apache.org/jira/browse/KAFKA-10454
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Reporter: Levani Kokhreidze
>Priority: Major
> Fix For: 2.6.0
>
>
> Here's integration test: [https://github.com/apache/kafka/pull/9237]
>
> From the first glance, issue is that when one joins stream to table, and
> table source topic doesn't have same number of partitions as stream topic,
> `StateChangelogReader` tries to recover state from changelog (which in this
> case is the same as source topic) for table from partitions that don't exist.
> Logs are spammed with:
>
> {code:java}
> [2020-09-01 12:33:07,508] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,508] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,508] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-3 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,510] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-0 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,510] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,510] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,510] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-3 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,513] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-0 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,513] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,513] INFO stream-thread
>
[jira] [Commented] (KAFKA-10454) Kafka Streams Stuck in infinite REBALANCING loop when stream <> table join partitions don't match
[
https://issues.apache.org/jira/browse/KAFKA-10454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17188313#comment-17188313
]
Levani Kokhreidze commented on KAFKA-10454:
---
CC [~mjsax] [~vvcephei] [~guozhang]
I may try to fix this as soon as I can - but if any chance somebody wants to
pick it up before me, feel free to do so.
> Kafka Streams Stuck in infinite REBALANCING loop when stream <> table join
> partitions don't match
> -
>
> Key: KAFKA-10454
> URL: https://issues.apache.org/jira/browse/KAFKA-10454
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Reporter: Levani Kokhreidze
>Priority: Major
> Fix For: 2.6.0
>
>
> Here's integration test: [https://github.com/apache/kafka/pull/9237]
>
> From the first glance, issue is that when one joins stream to table, and
> table source topic doesn't have same number of partitions as stream topic,
> `StateChangelogReader` tries to recover state from changelog (which in this
> case is the same as source topic) for table from partitions that don't exist.
> Logs are spammed with:
>
> {code:java}
> [2020-09-01 12:33:07,508] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,508] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,508] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-3 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,510] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-0 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,510] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,510] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,510] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-3 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,513] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-0 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,513] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found;
> will retry in the next time.
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> [2020-09-01 12:33:07,513] INFO stream-thread
> [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
> End offset for changelog
> topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found;
> will retry in the next time.
>
[jira] [Updated] (KAFKA-10454) Kafka Streams Stuck in infinite REBALANCING loop when stream <> table join partitions don't match
[
https://issues.apache.org/jira/browse/KAFKA-10454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Levani Kokhreidze updated KAFKA-10454:
--
Description:
Here's integration test: [https://github.com/apache/kafka/pull/9237]
>From the first glance, issue is that when one joins stream to table, and table
>source topic doesn't have same number of partitions as stream topic,
>`StateChangelogReader` tries to recover state from changelog (which in this
>case is the same as source topic) for table from partitions that don't exist.
>Logs are spammed with:
{code:java}
[2020-09-01 12:33:07,508] INFO stream-thread
[app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
End offset for changelog
topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found; will
retry in the next time.
(org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
[2020-09-01 12:33:07,508] INFO stream-thread
[app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
End offset for changelog
topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found; will
retry in the next time.
(org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
[2020-09-01 12:33:07,508] INFO stream-thread
[app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
End offset for changelog
topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-3 cannot be found; will
retry in the next time.
(org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
[2020-09-01 12:33:07,510] INFO stream-thread
[app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
End offset for changelog
topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-0 cannot be found; will
retry in the next time.
(org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
[2020-09-01 12:33:07,510] INFO stream-thread
[app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
End offset for changelog
topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found; will
retry in the next time.
(org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
[2020-09-01 12:33:07,510] INFO stream-thread
[app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
End offset for changelog
topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found; will
retry in the next time.
(org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
[2020-09-01 12:33:07,510] INFO stream-thread
[app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
End offset for changelog
topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-3 cannot be found; will
retry in the next time.
(org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
[2020-09-01 12:33:07,513] INFO stream-thread
[app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
End offset for changelog
topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-0 cannot be found; will
retry in the next time.
(org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
[2020-09-01 12:33:07,513] INFO stream-thread
[app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
End offset for changelog
topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found; will
retry in the next time.
(org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
[2020-09-01 12:33:07,513] INFO stream-thread
[app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
End offset for changelog
topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found; will
retry in the next time.
(org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
[2020-09-01 12:33:07,513] INFO stream-thread
[app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
End offset for changelog
topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-3 cannot be found; will
retry in the next time.
(org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
[2020-09-01 12:33:07,515] INFO stream-thread
[app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1]
End offset for changelog
topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-0 cannot be found; will
retry in the next time.
(org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
[2020-09-01 12:33:07,515] INFO stream-thread
[jira] [Created] (KAFKA-10454) Kafka Streams Stuck in infinite REBALANCING loop when stream <> table join partitions don't match
Levani Kokhreidze created KAFKA-10454: - Summary: Kafka Streams Stuck in infinite REBALANCING loop when stream <> table join partitions don't match Key: KAFKA-10454 URL: https://issues.apache.org/jira/browse/KAFKA-10454 Project: Kafka Issue Type: Bug Components: streams Reporter: Levani Kokhreidze Fix For: 2.6.0 TBD -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-10375) Restore consumer fails with SSL handshake fail exception
[
https://issues.apache.org/jira/browse/KAFKA-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Levani Kokhreidze updated KAFKA-10375:
--
Attachment: stacktrace.txt
> Restore consumer fails with SSL handshake fail exception
>
>
> Key: KAFKA-10375
> URL: https://issues.apache.org/jira/browse/KAFKA-10375
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.6.0
>Reporter: Levani Kokhreidze
>Priority: Major
> Attachments: stacktrace.txt
>
>
> After upgrading to 2.6, we started getting "SSL handshake fail" exceptions.
> Curios thing is that it seems to affect only restore consumers. For mTLS, we
> use dynamic certificates that are being reloaded automatically every X
> minutes.
> We didn't have any issues with it, up until upgrading 2.6 and other stream
> processing jobs running Kafka 2.4 don't have similar problems.
> After restarting the Kafka Streams instance, issue goes away.
>
> From the stacktrace, it's visible that problem is:
> {code:java}
> Aug 07 10:36:12.478 | Caused by:
> java.security.cert.CertificateExpiredException: NotAfter: Fri Aug 07 07:45:16
> GMT 2020
> {code}
> Seems like somehow restore consumer gets stuck with old certificate and it's
> not refreshed.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-10375) Restore consumer fails with SSL handshake fail exception
[
https://issues.apache.org/jira/browse/KAFKA-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Levani Kokhreidze updated KAFKA-10375:
--
Attachment: (was: stacktrace.txt)
> Restore consumer fails with SSL handshake fail exception
>
>
> Key: KAFKA-10375
> URL: https://issues.apache.org/jira/browse/KAFKA-10375
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.6.0
>Reporter: Levani Kokhreidze
>Priority: Major
>
> After upgrading to 2.6, we started getting "SSL handshake fail" exceptions.
> Curios thing is that it seems to affect only restore consumers. For mTLS, we
> use dynamic certificates that are being reloaded automatically every X
> minutes.
> We didn't have any issues with it, up until upgrading 2.6 and other stream
> processing jobs running Kafka 2.4 don't have similar problems.
> After restarting the Kafka Streams instance, issue goes away.
>
> From the stacktrace, it's visible that problem is:
> {code:java}
> Aug 07 10:36:12.478 | Caused by:
> java.security.cert.CertificateExpiredException: NotAfter: Fri Aug 07 07:45:16
> GMT 2020
> {code}
> Seems like somehow restore consumer gets stuck with old certificate and it's
> not refreshed.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KAFKA-10375) Restore consumer fails with SSL handshake fail exception
Levani Kokhreidze created KAFKA-10375:
-
Summary: Restore consumer fails with SSL handshake fail exception
Key: KAFKA-10375
URL: https://issues.apache.org/jira/browse/KAFKA-10375
Project: Kafka
Issue Type: Bug
Components: streams
Affects Versions: 2.6.0
Reporter: Levani Kokhreidze
Attachments: stacktrace.txt
After upgrading to 2.6, we started getting "SSL handshake fail" exceptions.
Curios thing is that it seems to affect only restore consumers. For mTLS, we
use dynamic certificates that are being reloaded automatically every X minutes.
We didn't have any issues with it, up until upgrading 2.6 and other stream
processing jobs running Kafka 2.4 don't have similar problems.
After restarting the Kafka Streams instance, issue goes away.
>From the stacktrace, it's visible that problem is:
{code:java}
Aug 07 10:36:12.478 | Caused by:
java.security.cert.CertificateExpiredException: NotAfter: Fri Aug 07 07:45:16
GMT 2020
{code}
Seems like somehow restore consumer gets stuck with old certificate and it's
not refreshed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9659) Kafka Streams / Consumer configured for static membership fails on "fatal exception: group.instance.id gets fenced"
[
https://issues.apache.org/jira/browse/KAFKA-9659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17173636#comment-17173636
]
Levani Kokhreidze commented on KAFKA-9659:
--
Hi [~guozhang] just wanted to let you know that after upgrading to 2.6, we no
longer see this issue.
> Kafka Streams / Consumer configured for static membership fails on "fatal
> exception: group.instance.id gets fenced"
> ---
>
> Key: KAFKA-9659
> URL: https://issues.apache.org/jira/browse/KAFKA-9659
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.5.0
>Reporter: Rohan Desai
>Assignee: Guozhang Wang
>Priority: Major
> Attachments: ksql-1.logs
>
>
> I'm running a KSQL query, which underneath is built into a Kafka Streams
> application. The application has been running without issue for a few days,
> until today, when all the streams threads exited with:
>
>
> {{[ERROR] 2020-03-05 00:57:58,776
> [_confluent-ksql-pksqlc-xm6g1query_CSAS_RATINGS_WITH_USER_AVERAGE_5-39e8046a-b6e6-44fd-8d6d-37cff78649bf-StreamThread-2]
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator handle -
> [Consumer instanceId=ksql-1-2,
> clientId=_confluent-ksql-pksqlc-xm6g1query_CSAS_RATINGS_WITH_USER_AVERAGE_5-39e8046a-b6e6-44fd-8d6d-37cff78649bf-StreamThread-2-consumer,
> groupId=_confluent-ksql-pksqlc-xm6g1query_CSAS_RATINGS_WITH_USER_AVERAGE_5]
> Received fatal exception: group.instance.id gets fenced}}
> {{[ERROR] 2020-03-05 00:57:58,776
> [_confluent-ksql-pksqlc-xm6g1query_CSAS_RATINGS_WITH_USER_AVERAGE_5-39e8046a-b6e6-44fd-8d6d-37cff78649bf-StreamThread-2]
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator onFailure -
> [Consumer instanceId=ksql-1-2,
> clientId=_confluent-ksql-pksqlc-xm6g1query_CSAS_RATINGS_WITH_USER_AVERAGE_5-39e8046a-b6e6-44fd-8d6d-37cff78649bf-StreamThread-2-consumer,
> groupId=_confluent-ksql-pksqlc-xm6g1query_CSAS_RATINGS_WITH_USER_AVERAGE_5]
> Caught fenced group.instance.id Optional[ksql-1-2] error in heartbeat thread}}
> {{[ERROR] 2020-03-05 00:57:58,776
> [_confluent-ksql-pksqlc-xm6g1query_CSAS_RATINGS_WITH_USER_AVERAGE_5-39e8046a-b6e6-44fd-8d6d-37cff78649bf-StreamThread-2]
> org.apache.kafka.streams.processor.internals.StreamThread run -
> stream-thread
> [_confluent-ksql-pksqlc-xm6g1query_CSAS_RATINGS_WITH_USER_AVERAGE_5-39e8046a-b6e6-44fd-8d6d-37cff78649bf-StreamThread-2]
> Encountered the following unexpected Kafka exception during processing, this
> usually indicate Streams internal errors:}}
> \{{ org.apache.kafka.common.errors.FencedInstanceIdException: The broker
> rejected this static consumer since another consumer with the same
> group.instance.id has registered with a different member.id.}}{{[INFO]
> 2020-03-05 00:57:58,776
> [_confluent-ksql-pksqlc-xm6g1query_CSAS_RATINGS_WITH_USER_AVERAGE_5-39e8046a-b6e6-44fd-8d6d-37cff78649bf-StreamThread-2]
> org.apache.kafka.streams.processor.internals.StreamThread setState -
> stream-thread
> [_confluent-ksql-pksqlc-xm6g1query_CSAS_RATINGS_WITH_USER_AVERAGE_5-39e8046a-b6e6-44fd-8d6d-37cff78649bf-StreamThread-2]
> State transition from RUNNING to PENDING_SHUTDOWN}}
>
> I've attached the KSQL and Kafka Streams logs to this ticket. Here's a
> summary for one of the streams threads (instance id `ksql-1-2`):
>
> Around 00:56:36 the coordinator fails over from b11 to b2:
>
> {{[INFO] 2020-03-05 00:56:36,258
> [_confluent-ksql-pksqlc-xm6g1query_CTAS_RATINGS_BY_USER_0-c1df9747-f353-47f1-82fd-30b97c20d038-StreamThread-2]
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator handle -
> [Consumer instanceId=ksql-1-2,
> clientId=_confluent-ksql-pksqlc-xm6g1query_CTAS_RATINGS_BY_USER_0-c1df9747-f353-47f1-82fd-30b97c20d038-StreamThread-2-consumer,
> groupId=_confluent-ksql-pksqlc-xm6g1query_CTAS_RATINGS_BY_USER_0] Attempt to
> heartbeat failed since coordinator
> b11-pkc-lzxjz.us-west-2.aws.devel.cpdev.cloud:9092 (id: 2147483636 rack:
> null) is either not started or not valid.}}
> {{ [INFO] 2020-03-05 00:56:36,258
> [_confluent-ksql-pksqlc-xm6g1query_CTAS_RATINGS_BY_USER_0-c1df9747-f353-47f1-82fd-30b97c20d038-StreamThread-2]
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator
> markCoordinatorUnknown - [Consumer instanceId=ksql-1-2,
> clientId=_confluent-ksql-pksqlc-xm6g1query_CTAS_RATINGS_BY_USER_0-c1df9747-f353-47f1-82fd-30b97c20d038-StreamThread-2-consumer,
> groupId=_confluent-ksql-pksqlc-xm6g1query_CTAS_RATINGS_BY_USER_0] Group
> coordinator b11-pkc-lzxjz.us-west-2.aws.devel.cpdev.cloud:9092 (id:
> 2147483636 rack: null) is unavailable or invalid, will attempt rediscovery}}
> {{ [INFO] 2020-03-05 00:56:36,270
>
[jira] [Resolved] (KAFKA-9859) kafka-streams-application-reset tool doesn't take into account topics generated by KTable foreign key join operation
[
https://issues.apache.org/jira/browse/KAFKA-9859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Levani Kokhreidze resolved KAFKA-9859.
--
Fix Version/s: 2.6.0
Resolution: Fixed
fixed with PR [https://github.com/apache/kafka/pull/8671]
> kafka-streams-application-reset tool doesn't take into account topics
> generated by KTable foreign key join operation
>
>
> Key: KAFKA-9859
> URL: https://issues.apache.org/jira/browse/KAFKA-9859
> Project: Kafka
> Issue Type: Bug
> Components: streams, tools
>Reporter: Levani Kokhreidze
>Assignee: Levani Kokhreidze
>Priority: Major
> Labels: newbie, newbie++
> Fix For: 2.6.0
>
>
> Steps to reproduce:
> * Create Kafka Streams application which uses foreign key join operation
> (without a Named parameter overload)
> * Stop Kafka streams application
> * Perform `kafka-topics-list` and verify that foreign key operation internal
> topics are generated
> * Use `kafka-streams-application-reset` to perform the cleanup of your kafka
> streams application: `kafka-streams-application-reset --application-id
> --input-topics --bootstrap-servers
> --to-datetime 2019-04-13T00:00:00.000`
> * Perform `kafka-topics-list` again, you'll see that topics generated by the
> foreign key operation are still there.
> [kafka-streams-application-reset|#L679-L680]] uses
> `-subscription-registration-topic` and `-subscription-response-topic`
> suffixes to match topics generated by the foreign key operation. While in
> reality, internal topics are generated in this format:
> {code:java}
> -KTABLE-FK-JOIN-SUBSCRIPTION-REGISTRATION- number>-topic
> -KTABLE-FK-JOIN-SUBSCRIPTION-RESPONSE- number>-topic{code}
> Please note that this problem only happens when `Named` parameter is not
> used. When named parameter is used, topics are generated with a same pattern
> as specified in StreamsResetter.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-10003) Deprecate KStream#through in favor of KStream#repartition
[ https://issues.apache.org/jira/browse/KAFKA-10003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Levani Kokhreidze updated KAFKA-10003: -- Component/s: streams > Deprecate KStream#through in favor of KStream#repartition > - > > Key: KAFKA-10003 > URL: https://issues.apache.org/jira/browse/KAFKA-10003 > Project: Kafka > Issue Type: Task > Components: streams >Reporter: Levani Kokhreidze >Priority: Major > > After introducing `KStream#repartition` in KIP-221 > ([https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+DSL+with+Connecting+Topic+Creation+and+Repartition+Hint)|https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+DSL+with+Connecting+Topic+Creation+and+Repartition+Hint], > it makes sense to deprecate `KStream#through` in favor of new operator (see > voting thread for more context: > [https://www.mail-archive.com/dev@kafka.apache.org/msg107645.html)|https://www.mail-archive.com/dev@kafka.apache.org/msg107645.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-10003) Deprecate KStream#through in favor of KStream#repartition
[ https://issues.apache.org/jira/browse/KAFKA-10003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Levani Kokhreidze updated KAFKA-10003: -- Description: After introducing `KStream#repartition` in KIP-221 ([https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+DSL+with+Connecting+Topic+Creation+and+Repartition+Hint)|https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+DSL+with+Connecting+Topic+Creation+and+Repartition+Hint], it makes sense to deprecate `KStream#through` in favor of new operator (see voting thread for more context: [https://www.mail-archive.com/dev@kafka.apache.org/msg107645.html)|https://www.mail-archive.com/dev@kafka.apache.org/msg107645.html] (was: After introducing `KStream#repartition` in [KIP-221|[https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+DSL+with+Connecting+Topic+Creation+and+Repartition+Hint]] , it makes sense to deprecate `KStream#through` in favor of new operator (see voting thread for more context: [https://www.mail-archive.com/dev@kafka.apache.org/msg107645.html)|https://www.mail-archive.com/dev@kafka.apache.org/msg107645.html]) > Deprecate KStream#through in favor of KStream#repartition > - > > Key: KAFKA-10003 > URL: https://issues.apache.org/jira/browse/KAFKA-10003 > Project: Kafka > Issue Type: Task >Reporter: Levani Kokhreidze >Priority: Major > > After introducing `KStream#repartition` in KIP-221 > ([https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+DSL+with+Connecting+Topic+Creation+and+Repartition+Hint)|https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+DSL+with+Connecting+Topic+Creation+and+Repartition+Hint], > it makes sense to deprecate `KStream#through` in favor of new operator (see > voting thread for more context: > [https://www.mail-archive.com/dev@kafka.apache.org/msg107645.html)|https://www.mail-archive.com/dev@kafka.apache.org/msg107645.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-10003) Deprecate KStream#through in favor of KStream#repartition
[ https://issues.apache.org/jira/browse/KAFKA-10003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Levani Kokhreidze updated KAFKA-10003: -- Description: After introducing `KStream#repartition` in [KIP-221|[https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+DSL+with+Connecting+Topic+Creation+and+Repartition+Hint]] , it makes sense to deprecate `KStream#through` in favor of new operator (see voting thread for more context: [https://www.mail-archive.com/dev@kafka.apache.org/msg107645.html)|https://www.mail-archive.com/dev@kafka.apache.org/msg107645.html] (was: After introducing `KStream#repartition` in [KIP-221|[https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+DSL+with+Connecting+Topic+Creation+and+Repartition+Hint]], it makes sense to deprecate `KStream#through` in favor of new operator (see voting thread for more context: [https://www.mail-archive.com/dev@kafka.apache.org/msg107645.html)|https://www.mail-archive.com/dev@kafka.apache.org/msg107645.html]) > Deprecate KStream#through in favor of KStream#repartition > - > > Key: KAFKA-10003 > URL: https://issues.apache.org/jira/browse/KAFKA-10003 > Project: Kafka > Issue Type: Task >Reporter: Levani Kokhreidze >Priority: Major > > After introducing `KStream#repartition` in > [KIP-221|[https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+DSL+with+Connecting+Topic+Creation+and+Repartition+Hint]] > , it makes sense to deprecate `KStream#through` in favor of new operator > (see voting thread for more context: > [https://www.mail-archive.com/dev@kafka.apache.org/msg107645.html)|https://www.mail-archive.com/dev@kafka.apache.org/msg107645.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-10003) Deprecate KStream#through in favor of KStream#repartition
Levani Kokhreidze created KAFKA-10003: - Summary: Deprecate KStream#through in favor of KStream#repartition Key: KAFKA-10003 URL: https://issues.apache.org/jira/browse/KAFKA-10003 Project: Kafka Issue Type: Task Reporter: Levani Kokhreidze After introducing `KStream#repartition` in [KIP-221|[https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+DSL+with+Connecting+Topic+Creation+and+Repartition+Hint]], it makes sense to deprecate `KStream#through` in favor of new operator (see voting thread for more context: [https://www.mail-archive.com/dev@kafka.apache.org/msg107645.html)|https://www.mail-archive.com/dev@kafka.apache.org/msg107645.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-9659) Kafka Streams / Consumer configured for static membership fails on "fatal exception: group.instance.id gets fenced"
[
https://issues.apache.org/jira/browse/KAFKA-9659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17106528#comment-17106528
]
Levani Kokhreidze commented on KAFKA-9659:
--
Hi [~guozhang]
Just fyi - we've experienced this error as well. Some of our Kafka streams jobs
(with static membership) sometimes die whenever we restart one of the brokers.
It affects broker version 2.3.1 and 2.4 as well.
We can reliably reproduce the issue in our "staging" environment so if some
extra debug logs can be useful from clients/brokers would be happy to help.
> Kafka Streams / Consumer configured for static membership fails on "fatal
> exception: group.instance.id gets fenced"
> ---
>
> Key: KAFKA-9659
> URL: https://issues.apache.org/jira/browse/KAFKA-9659
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.5.0
>Reporter: Rohan Desai
>Assignee: Guozhang Wang
>Priority: Major
> Attachments: ksql-1.logs
>
>
> I'm running a KSQL query, which underneath is built into a Kafka Streams
> application. The application has been running without issue for a few days,
> until today, when all the streams threads exited with:
>
>
> {{[ERROR] 2020-03-05 00:57:58,776
> [_confluent-ksql-pksqlc-xm6g1query_CSAS_RATINGS_WITH_USER_AVERAGE_5-39e8046a-b6e6-44fd-8d6d-37cff78649bf-StreamThread-2]
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator handle -
> [Consumer instanceId=ksql-1-2,
> clientId=_confluent-ksql-pksqlc-xm6g1query_CSAS_RATINGS_WITH_USER_AVERAGE_5-39e8046a-b6e6-44fd-8d6d-37cff78649bf-StreamThread-2-consumer,
> groupId=_confluent-ksql-pksqlc-xm6g1query_CSAS_RATINGS_WITH_USER_AVERAGE_5]
> Received fatal exception: group.instance.id gets fenced}}
> {{[ERROR] 2020-03-05 00:57:58,776
> [_confluent-ksql-pksqlc-xm6g1query_CSAS_RATINGS_WITH_USER_AVERAGE_5-39e8046a-b6e6-44fd-8d6d-37cff78649bf-StreamThread-2]
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator onFailure -
> [Consumer instanceId=ksql-1-2,
> clientId=_confluent-ksql-pksqlc-xm6g1query_CSAS_RATINGS_WITH_USER_AVERAGE_5-39e8046a-b6e6-44fd-8d6d-37cff78649bf-StreamThread-2-consumer,
> groupId=_confluent-ksql-pksqlc-xm6g1query_CSAS_RATINGS_WITH_USER_AVERAGE_5]
> Caught fenced group.instance.id Optional[ksql-1-2] error in heartbeat thread}}
> {{[ERROR] 2020-03-05 00:57:58,776
> [_confluent-ksql-pksqlc-xm6g1query_CSAS_RATINGS_WITH_USER_AVERAGE_5-39e8046a-b6e6-44fd-8d6d-37cff78649bf-StreamThread-2]
> org.apache.kafka.streams.processor.internals.StreamThread run -
> stream-thread
> [_confluent-ksql-pksqlc-xm6g1query_CSAS_RATINGS_WITH_USER_AVERAGE_5-39e8046a-b6e6-44fd-8d6d-37cff78649bf-StreamThread-2]
> Encountered the following unexpected Kafka exception during processing, this
> usually indicate Streams internal errors:}}
> \{{ org.apache.kafka.common.errors.FencedInstanceIdException: The broker
> rejected this static consumer since another consumer with the same
> group.instance.id has registered with a different member.id.}}{{[INFO]
> 2020-03-05 00:57:58,776
> [_confluent-ksql-pksqlc-xm6g1query_CSAS_RATINGS_WITH_USER_AVERAGE_5-39e8046a-b6e6-44fd-8d6d-37cff78649bf-StreamThread-2]
> org.apache.kafka.streams.processor.internals.StreamThread setState -
> stream-thread
> [_confluent-ksql-pksqlc-xm6g1query_CSAS_RATINGS_WITH_USER_AVERAGE_5-39e8046a-b6e6-44fd-8d6d-37cff78649bf-StreamThread-2]
> State transition from RUNNING to PENDING_SHUTDOWN}}
>
> I've attached the KSQL and Kafka Streams logs to this ticket. Here's a
> summary for one of the streams threads (instance id `ksql-1-2`):
>
> Around 00:56:36 the coordinator fails over from b11 to b2:
>
> {{[INFO] 2020-03-05 00:56:36,258
> [_confluent-ksql-pksqlc-xm6g1query_CTAS_RATINGS_BY_USER_0-c1df9747-f353-47f1-82fd-30b97c20d038-StreamThread-2]
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator handle -
> [Consumer instanceId=ksql-1-2,
> clientId=_confluent-ksql-pksqlc-xm6g1query_CTAS_RATINGS_BY_USER_0-c1df9747-f353-47f1-82fd-30b97c20d038-StreamThread-2-consumer,
> groupId=_confluent-ksql-pksqlc-xm6g1query_CTAS_RATINGS_BY_USER_0] Attempt to
> heartbeat failed since coordinator
> b11-pkc-lzxjz.us-west-2.aws.devel.cpdev.cloud:9092 (id: 2147483636 rack:
> null) is either not started or not valid.}}
> {{ [INFO] 2020-03-05 00:56:36,258
> [_confluent-ksql-pksqlc-xm6g1query_CTAS_RATINGS_BY_USER_0-c1df9747-f353-47f1-82fd-30b97c20d038-StreamThread-2]
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator
> markCoordinatorUnknown - [Consumer instanceId=ksql-1-2,
> clientId=_confluent-ksql-pksqlc-xm6g1query_CTAS_RATINGS_BY_USER_0-c1df9747-f353-47f1-82fd-30b97c20d038-StreamThread-2-consumer,
>
[jira] [Assigned] (KAFKA-9859) kafka-streams-application-reset tool doesn't take into account topics generated by KTable foreign key join operation
[
https://issues.apache.org/jira/browse/KAFKA-9859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Levani Kokhreidze reassigned KAFKA-9859:
Assignee: Levani Kokhreidze
> kafka-streams-application-reset tool doesn't take into account topics
> generated by KTable foreign key join operation
>
>
> Key: KAFKA-9859
> URL: https://issues.apache.org/jira/browse/KAFKA-9859
> Project: Kafka
> Issue Type: Bug
> Components: streams, tools
>Reporter: Levani Kokhreidze
>Assignee: Levani Kokhreidze
>Priority: Major
> Labels: newbie, newbie++
>
> Steps to reproduce:
> * Create Kafka Streams application which uses foreign key join operation
> (without a Named parameter overload)
> * Stop Kafka streams application
> * Perform `kafka-topics-list` and verify that foreign key operation internal
> topics are generated
> * Use `kafka-streams-application-reset` to perform the cleanup of your kafka
> streams application: `kafka-streams-application-reset --application-id
> --input-topics --bootstrap-servers
> --to-datetime 2019-04-13T00:00:00.000`
> * Perform `kafka-topics-list` again, you'll see that topics generated by the
> foreign key operation are still there.
> [kafka-streams-application-reset|#L679-L680]] uses
> `-subscription-registration-topic` and `-subscription-response-topic`
> suffixes to match topics generated by the foreign key operation. While in
> reality, internal topics are generated in this format:
> {code:java}
> -KTABLE-FK-JOIN-SUBSCRIPTION-REGISTRATION- number>-topic
> -KTABLE-FK-JOIN-SUBSCRIPTION-RESPONSE- number>-topic{code}
> Please note that this problem only happens when `Named` parameter is not
> used. When named parameter is used, topics are generated with a same pattern
> as specified in StreamsResetter.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9859) kafka-streams-application-reset tool doesn't take into account topics generated by KTable foreign key join operation
[
https://issues.apache.org/jira/browse/KAFKA-9859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17099196#comment-17099196
]
Levani Kokhreidze commented on KAFKA-9859:
--
Hi [~guozhang],
Was meaning to do it this week, will assign ticket to myself and send the PR.
> kafka-streams-application-reset tool doesn't take into account topics
> generated by KTable foreign key join operation
>
>
> Key: KAFKA-9859
> URL: https://issues.apache.org/jira/browse/KAFKA-9859
> Project: Kafka
> Issue Type: Bug
> Components: streams, tools
>Reporter: Levani Kokhreidze
>Priority: Major
> Labels: newbie, newbie++
>
> Steps to reproduce:
> * Create Kafka Streams application which uses foreign key join operation
> (without a Named parameter overload)
> * Stop Kafka streams application
> * Perform `kafka-topics-list` and verify that foreign key operation internal
> topics are generated
> * Use `kafka-streams-application-reset` to perform the cleanup of your kafka
> streams application: `kafka-streams-application-reset --application-id
> --input-topics --bootstrap-servers
> --to-datetime 2019-04-13T00:00:00.000`
> * Perform `kafka-topics-list` again, you'll see that topics generated by the
> foreign key operation are still there.
> [kafka-streams-application-reset|#L679-L680]] uses
> `-subscription-registration-topic` and `-subscription-response-topic`
> suffixes to match topics generated by the foreign key operation. While in
> reality, internal topics are generated in this format:
> {code:java}
> -KTABLE-FK-JOIN-SUBSCRIPTION-REGISTRATION- number>-topic
> -KTABLE-FK-JOIN-SUBSCRIPTION-RESPONSE- number>-topic{code}
> Please note that this problem only happens when `Named` parameter is not
> used. When named parameter is used, topics are generated with a same pattern
> as specified in StreamsResetter.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (KAFKA-9859) kafka-streams-application-reset tool doesn't take into account topics generated by KTable foreign key join operation
[
https://issues.apache.org/jira/browse/KAFKA-9859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091243#comment-17091243
]
Levani Kokhreidze edited comment on KAFKA-9859 at 4/24/20, 2:28 PM:
Please also note that this problem only happens when `Named` parameter is not
used. When named parameter is used, topics are generated with a same pattern as
specified in StreamsResetter.
was (Author: lkokhreidze):
Please also note that this problem only happens when `Named` parameter is not
used. When named parameter is used, topics are generated with a same patter as
specified in StreamsResetter.
> kafka-streams-application-reset tool doesn't take into account topics
> generated by KTable foreign key join operation
>
>
> Key: KAFKA-9859
> URL: https://issues.apache.org/jira/browse/KAFKA-9859
> Project: Kafka
> Issue Type: Bug
> Components: streams, tools
>Reporter: Levani Kokhreidze
>Priority: Major
>
> Steps to reproduce:
> * Create Kafka Streams application which uses foreign key join operation
> (without a Named parameter overload)
> * Stop Kafka streams application
> * Perform `kafka-topics-list` and verify that foreign key operation internal
> topics are generated
> * Use `kafka-streams-application-reset` to perform the cleanup of your kafka
> streams application: `kafka-streams-application-reset --application-id
> --input-topics --bootstrap-servers
> --to-datetime 2019-04-13T00:00:00.000`
> * Perform `kafka-topics-list` again, you'll see that topics generated by the
> foreign key operation are still there.
> [kafka-streams-application-reset|#L679-L680]] uses
> `-subscription-registration-topic` and `-subscription-response-topic`
> suffixes to match topics generated by the foreign key operation. While in
> reality, internal topics are generated in this format:
> {code:java}
> -KTABLE-FK-JOIN-SUBSCRIPTION-REGISTRATION- number>-topic
> -KTABLE-FK-JOIN-SUBSCRIPTION-RESPONSE- number>-topic{code}
> Please note that this problem only happens when `Named` parameter is not
> used. When named parameter is used, topics are generated with a same pattern
> as specified in StreamsResetter.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-9859) kafka-streams-application-reset tool doesn't take into account topics generated by KTable foreign key join operation
[
https://issues.apache.org/jira/browse/KAFKA-9859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Levani Kokhreidze updated KAFKA-9859:
-
Description:
Steps to reproduce:
* Create Kafka Streams application which uses foreign key join operation
* Stop Kafka streams application
* Perform `kafka-topics-list` and verify that foreign key operation internal
topics are generated
* Use `kafka-streams-application-reset` to perform the cleanup of your kafka
streams application: `kafka-streams-application-reset --application-id
--input-topics --bootstrap-servers
--to-datetime 2019-04-13T00:00:00.000`
* Perform `kafka-topics-list` again, you'll see that topics generated by the
foreign key operation are still there.
[kafka-streams-application-reset|#L679-L680]] uses
`-subscription-registration-topic` and `-subscription-response-topic` suffixes
to match topics generated by the foreign key operation. While in reality,
internal topics are generated in this format:
{code:java}
-KTABLE-FK-JOIN-SUBSCRIPTION-REGISTRATION--topic
-KTABLE-FK-JOIN-SUBSCRIPTION-RESPONSE--topic{code}
Please note that this problem only happens when `Named` parameter is not used.
When named parameter is used, topics are generated with a same pattern as
specified in StreamsResetter.
was:
Steps to reproduce:
* Create Kafka Streams application which uses foreign key join operation
* Stop Kafka streams application
* Perform `kafka-topics-list` and verify that foreign key operation internal
topics are generated
* Use `kafka-streams-application-reset` to perform the cleanup of your kafka
streams application: `kafka-streams-application-reset --application-id
--input-topics --bootstrap-servers
--to-datetime 2019-04-13T00:00:00.000`
* Perform `kafka-topics-list` again, you'll see that topics generated by the
foreign key operation are still there.
[kafka-streams-application-reset|#L679-L680]] uses
`-subscription-registration-topic` and `-subscription-response-topic` suffixes
to match topics generated by the foreign key operation. While in reality,
internal topics are generated in this format:
{code:java}
-KTABLE-FK-JOIN-SUBSCRIPTION-REGISTRATION--topic
-KTABLE-FK-JOIN-SUBSCRIPTION-RESPONSE--topic{code}
Please note that this problem only happens when `Named` parameter is not used.
When named parameter is used, topics are generated with a same patter as
specified in StreamsResetter.
> kafka-streams-application-reset tool doesn't take into account topics
> generated by KTable foreign key join operation
>
>
> Key: KAFKA-9859
> URL: https://issues.apache.org/jira/browse/KAFKA-9859
> Project: Kafka
> Issue Type: Bug
> Components: streams, tools
>Reporter: Levani Kokhreidze
>Priority: Major
>
> Steps to reproduce:
> * Create Kafka Streams application which uses foreign key join operation
> * Stop Kafka streams application
> * Perform `kafka-topics-list` and verify that foreign key operation internal
> topics are generated
> * Use `kafka-streams-application-reset` to perform the cleanup of your kafka
> streams application: `kafka-streams-application-reset --application-id
> --input-topics --bootstrap-servers
> --to-datetime 2019-04-13T00:00:00.000`
> * Perform `kafka-topics-list` again, you'll see that topics generated by the
> foreign key operation are still there.
> [kafka-streams-application-reset|#L679-L680]] uses
> `-subscription-registration-topic` and `-subscription-response-topic`
> suffixes to match topics generated by the foreign key operation. While in
> reality, internal topics are generated in this format:
> {code:java}
> -KTABLE-FK-JOIN-SUBSCRIPTION-REGISTRATION- number>-topic
> -KTABLE-FK-JOIN-SUBSCRIPTION-RESPONSE- number>-topic{code}
> Please note that this problem only happens when `Named` parameter is not
> used. When named parameter is used, topics are generated with a same pattern
> as specified in StreamsResetter.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-9859) kafka-streams-application-reset tool doesn't take into account topics generated by KTable foreign key join operation
[
https://issues.apache.org/jira/browse/KAFKA-9859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Levani Kokhreidze updated KAFKA-9859:
-
Description:
Steps to reproduce:
* Create Kafka Streams application which uses foreign key join operation
(without a Named parameter overload)
* Stop Kafka streams application
* Perform `kafka-topics-list` and verify that foreign key operation internal
topics are generated
* Use `kafka-streams-application-reset` to perform the cleanup of your kafka
streams application: `kafka-streams-application-reset --application-id
--input-topics --bootstrap-servers
--to-datetime 2019-04-13T00:00:00.000`
* Perform `kafka-topics-list` again, you'll see that topics generated by the
foreign key operation are still there.
[kafka-streams-application-reset|#L679-L680]] uses
`-subscription-registration-topic` and `-subscription-response-topic` suffixes
to match topics generated by the foreign key operation. While in reality,
internal topics are generated in this format:
{code:java}
-KTABLE-FK-JOIN-SUBSCRIPTION-REGISTRATION--topic
-KTABLE-FK-JOIN-SUBSCRIPTION-RESPONSE--topic{code}
Please note that this problem only happens when `Named` parameter is not used.
When named parameter is used, topics are generated with a same pattern as
specified in StreamsResetter.
was:
Steps to reproduce:
* Create Kafka Streams application which uses foreign key join operation
* Stop Kafka streams application
* Perform `kafka-topics-list` and verify that foreign key operation internal
topics are generated
* Use `kafka-streams-application-reset` to perform the cleanup of your kafka
streams application: `kafka-streams-application-reset --application-id
--input-topics --bootstrap-servers
--to-datetime 2019-04-13T00:00:00.000`
* Perform `kafka-topics-list` again, you'll see that topics generated by the
foreign key operation are still there.
[kafka-streams-application-reset|#L679-L680]] uses
`-subscription-registration-topic` and `-subscription-response-topic` suffixes
to match topics generated by the foreign key operation. While in reality,
internal topics are generated in this format:
{code:java}
-KTABLE-FK-JOIN-SUBSCRIPTION-REGISTRATION--topic
-KTABLE-FK-JOIN-SUBSCRIPTION-RESPONSE--topic{code}
Please note that this problem only happens when `Named` parameter is not used.
When named parameter is used, topics are generated with a same pattern as
specified in StreamsResetter.
> kafka-streams-application-reset tool doesn't take into account topics
> generated by KTable foreign key join operation
>
>
> Key: KAFKA-9859
> URL: https://issues.apache.org/jira/browse/KAFKA-9859
> Project: Kafka
> Issue Type: Bug
> Components: streams, tools
>Reporter: Levani Kokhreidze
>Priority: Major
>
> Steps to reproduce:
> * Create Kafka Streams application which uses foreign key join operation
> (without a Named parameter overload)
> * Stop Kafka streams application
> * Perform `kafka-topics-list` and verify that foreign key operation internal
> topics are generated
> * Use `kafka-streams-application-reset` to perform the cleanup of your kafka
> streams application: `kafka-streams-application-reset --application-id
> --input-topics --bootstrap-servers
> --to-datetime 2019-04-13T00:00:00.000`
> * Perform `kafka-topics-list` again, you'll see that topics generated by the
> foreign key operation are still there.
> [kafka-streams-application-reset|#L679-L680]] uses
> `-subscription-registration-topic` and `-subscription-response-topic`
> suffixes to match topics generated by the foreign key operation. While in
> reality, internal topics are generated in this format:
> {code:java}
> -KTABLE-FK-JOIN-SUBSCRIPTION-REGISTRATION- number>-topic
> -KTABLE-FK-JOIN-SUBSCRIPTION-RESPONSE- number>-topic{code}
> Please note that this problem only happens when `Named` parameter is not
> used. When named parameter is used, topics are generated with a same pattern
> as specified in StreamsResetter.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-9859) kafka-streams-application-reset tool doesn't take into account topics generated by KTable foreign key join operation
[
https://issues.apache.org/jira/browse/KAFKA-9859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Levani Kokhreidze updated KAFKA-9859:
-
Description:
Steps to reproduce:
* Create Kafka Streams application which uses foreign key join operation
* Stop Kafka streams application
* Perform `kafka-topics-list` and verify that foreign key operation internal
topics are generated
* Use `kafka-streams-application-reset` to perform the cleanup of your kafka
streams application: `kafka-streams-application-reset --application-id
--input-topics --bootstrap-servers
--to-datetime 2019-04-13T00:00:00.000`
* Perform `kafka-topics-list` again, you'll see that topics generated by the
foreign key operation are still there.
[kafka-streams-application-reset|#L679-L680]] uses
`-subscription-registration-topic` and `-subscription-response-topic` suffixes
to match topics generated by the foreign key operation. While in reality,
internal topics are generated in this format:
{code:java}
-KTABLE-FK-JOIN-SUBSCRIPTION-REGISTRATION--topic
-KTABLE-FK-JOIN-SUBSCRIPTION-RESPONSE--topic{code}
Please also note that this problem only happens when `Named` parameter is not
used. When named parameter is used, topics are generated with a same patter as
specified in StreamsResetter.
was:
Steps to reproduce:
* Create Kafka Streams application which uses foreign key join operation
* Stop Kafka streams application
* Perform `kafka-topics-list` and verify that foreign key operation internal
topics are generated
* Use `kafka-streams-application-reset` to perform the cleanup of your kafka
streams application: `kafka-streams-application-reset --application-id
--input-topics --bootstrap-servers
--to-datetime 2019-04-13T00:00:00.000`
* Perform `kafka-topics-list` again, you'll see that topics generated by the
foreign key operation are still there.
[kafka-streams-application-reset|#L679-L680]] uses
`-subscription-registration-topic` and `-subscription-response-topic` suffixes
to match topics generated by the foreign key operation. While in reality,
internal topics are generated in this format:
{code:java}
-KTABLE-FK-JOIN-SUBSCRIPTION-REGISTRATION--topic
-KTABLE-FK-JOIN-SUBSCRIPTION-RESPONSE--topic
{code}
> kafka-streams-application-reset tool doesn't take into account topics
> generated by KTable foreign key join operation
>
>
> Key: KAFKA-9859
> URL: https://issues.apache.org/jira/browse/KAFKA-9859
> Project: Kafka
> Issue Type: Bug
> Components: streams, tools
>Reporter: Levani Kokhreidze
>Priority: Major
>
> Steps to reproduce:
> * Create Kafka Streams application which uses foreign key join operation
> * Stop Kafka streams application
> * Perform `kafka-topics-list` and verify that foreign key operation internal
> topics are generated
> * Use `kafka-streams-application-reset` to perform the cleanup of your kafka
> streams application: `kafka-streams-application-reset --application-id
> --input-topics --bootstrap-servers
> --to-datetime 2019-04-13T00:00:00.000`
> * Perform `kafka-topics-list` again, you'll see that topics generated by the
> foreign key operation are still there.
> [kafka-streams-application-reset|#L679-L680]] uses
> `-subscription-registration-topic` and `-subscription-response-topic`
> suffixes to match topics generated by the foreign key operation. While in
> reality, internal topics are generated in this format:
> {code:java}
> -KTABLE-FK-JOIN-SUBSCRIPTION-REGISTRATION- number>-topic
> -KTABLE-FK-JOIN-SUBSCRIPTION-RESPONSE- number>-topic{code}
> Please also note that this problem only happens when `Named` parameter is not
> used. When named parameter is used, topics are generated with a same patter
> as specified in StreamsResetter.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-9859) kafka-streams-application-reset tool doesn't take into account topics generated by KTable foreign key join operation
[
https://issues.apache.org/jira/browse/KAFKA-9859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Levani Kokhreidze updated KAFKA-9859:
-
Description:
Steps to reproduce:
* Create Kafka Streams application which uses foreign key join operation
* Stop Kafka streams application
* Perform `kafka-topics-list` and verify that foreign key operation internal
topics are generated
* Use `kafka-streams-application-reset` to perform the cleanup of your kafka
streams application: `kafka-streams-application-reset --application-id
--input-topics --bootstrap-servers
--to-datetime 2019-04-13T00:00:00.000`
* Perform `kafka-topics-list` again, you'll see that topics generated by the
foreign key operation are still there.
[kafka-streams-application-reset|#L679-L680]] uses
`-subscription-registration-topic` and `-subscription-response-topic` suffixes
to match topics generated by the foreign key operation. While in reality,
internal topics are generated in this format:
{code:java}
-KTABLE-FK-JOIN-SUBSCRIPTION-REGISTRATION--topic
-KTABLE-FK-JOIN-SUBSCRIPTION-RESPONSE--topic{code}
Please note that this problem only happens when `Named` parameter is not used.
When named parameter is used, topics are generated with a same patter as
specified in StreamsResetter.
was:
Steps to reproduce:
* Create Kafka Streams application which uses foreign key join operation
* Stop Kafka streams application
* Perform `kafka-topics-list` and verify that foreign key operation internal
topics are generated
* Use `kafka-streams-application-reset` to perform the cleanup of your kafka
streams application: `kafka-streams-application-reset --application-id
--input-topics --bootstrap-servers
--to-datetime 2019-04-13T00:00:00.000`
* Perform `kafka-topics-list` again, you'll see that topics generated by the
foreign key operation are still there.
[kafka-streams-application-reset|#L679-L680]] uses
`-subscription-registration-topic` and `-subscription-response-topic` suffixes
to match topics generated by the foreign key operation. While in reality,
internal topics are generated in this format:
{code:java}
-KTABLE-FK-JOIN-SUBSCRIPTION-REGISTRATION--topic
-KTABLE-FK-JOIN-SUBSCRIPTION-RESPONSE--topic{code}
Please also note that this problem only happens when `Named` parameter is not
used. When named parameter is used, topics are generated with a same patter as
specified in StreamsResetter.
> kafka-streams-application-reset tool doesn't take into account topics
> generated by KTable foreign key join operation
>
>
> Key: KAFKA-9859
> URL: https://issues.apache.org/jira/browse/KAFKA-9859
> Project: Kafka
> Issue Type: Bug
> Components: streams, tools
>Reporter: Levani Kokhreidze
>Priority: Major
>
> Steps to reproduce:
> * Create Kafka Streams application which uses foreign key join operation
> * Stop Kafka streams application
> * Perform `kafka-topics-list` and verify that foreign key operation internal
> topics are generated
> * Use `kafka-streams-application-reset` to perform the cleanup of your kafka
> streams application: `kafka-streams-application-reset --application-id
> --input-topics --bootstrap-servers
> --to-datetime 2019-04-13T00:00:00.000`
> * Perform `kafka-topics-list` again, you'll see that topics generated by the
> foreign key operation are still there.
> [kafka-streams-application-reset|#L679-L680]] uses
> `-subscription-registration-topic` and `-subscription-response-topic`
> suffixes to match topics generated by the foreign key operation. While in
> reality, internal topics are generated in this format:
> {code:java}
> -KTABLE-FK-JOIN-SUBSCRIPTION-REGISTRATION- number>-topic
> -KTABLE-FK-JOIN-SUBSCRIPTION-RESPONSE- number>-topic{code}
> Please note that this problem only happens when `Named` parameter is not
> used. When named parameter is used, topics are generated with a same patter
> as specified in StreamsResetter.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-9859) kafka-streams-application-reset tool doesn't take into account topics generated by KTable foreign key join operation
[
https://issues.apache.org/jira/browse/KAFKA-9859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Levani Kokhreidze updated KAFKA-9859:
-
Description:
Steps to reproduce:
* Create Kafka Streams application which uses foreign key join operation
* Stop Kafka streams application
* Perform `kafka-topics-list` and verify that foreign key operation internal
topics are generated
* Use `kafka-streams-application-reset` to perform the cleanup of your kafka
streams application: `kafka-streams-application-reset --application-id
--input-topics --bootstrap-servers
--to-datetime 2019-04-13T00:00:00.000`
* Perform `kafka-topics-list` again, you'll see that topics generated by the
foreign key operation are still there.
[kafka-streams-application-reset|#L679-L680]] uses
`-subscription-registration-topic` and `-subscription-response-topic` suffixes
to match topics generated by the foreign key operation. While in reality,
internal topics are generated in this format:
{code:java}
-KTABLE-FK-JOIN-SUBSCRIPTION-REGISTRATION--topic
-KTABLE-FK-JOIN-SUBSCRIPTION-RESPONSE--topic
{code}
was:
Steps to reproduce:
* Create Kafka Streams application which uses foreign key join operation
* Stop Kafka streams application
* Perform `kafka-topics-list` and verify that foreign key operation internal
topics are generated
* Use `kafka-streams-application-reset` to perform the cleanup of your kafka
streams application: `kafka-streams-application-reset --application-id
--input-topics --bootstrap-servers
--to-datetime 2019-04-13T00:00:00.000`
* Perform `kafka-topics-list` again, you'll see that topics generated by the
foreign key operation are still there.
[kafka-streams-application-reset|[https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/StreamsResetter.java#L679-L680]]
uses `-subscription-registration-topic` and `-subscription-response-topic`
suffixes to match topics generated by the foreign key operation. While in
reality, internal topics are generated in this format:
{code:java}
-KTABLE-FK-JOIN-SUBSCRIPTION-REGISTRATION--topic
-KTABLE-FK-JOIN-SUBSCRIPTION-RESPONSE--topic
{code}
> kafka-streams-application-reset tool doesn't take into account topics
> generated by KTable foreign key join operation
>
>
> Key: KAFKA-9859
> URL: https://issues.apache.org/jira/browse/KAFKA-9859
> Project: Kafka
> Issue Type: Bug
> Components: streams, tools
>Reporter: Levani Kokhreidze
>Priority: Major
>
> Steps to reproduce:
> * Create Kafka Streams application which uses foreign key join operation
> * Stop Kafka streams application
> * Perform `kafka-topics-list` and verify that foreign key operation internal
> topics are generated
> * Use `kafka-streams-application-reset` to perform the cleanup of your kafka
> streams application: `kafka-streams-application-reset --application-id
> --input-topics --bootstrap-servers
> --to-datetime 2019-04-13T00:00:00.000`
> * Perform `kafka-topics-list` again, you'll see that topics generated by the
> foreign key operation are still there.
> [kafka-streams-application-reset|#L679-L680]] uses
> `-subscription-registration-topic` and `-subscription-response-topic`
> suffixes to match topics generated by the foreign key operation. While in
> reality, internal topics are generated in this format:
> {code:java}
> -KTABLE-FK-JOIN-SUBSCRIPTION-REGISTRATION- number>-topic
> -KTABLE-FK-JOIN-SUBSCRIPTION-RESPONSE--topic
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9859) kafka-streams-application-reset tool doesn't take into account topics generated by KTable foreign key join operation
[
https://issues.apache.org/jira/browse/KAFKA-9859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091243#comment-17091243
]
Levani Kokhreidze commented on KAFKA-9859:
--
Please also note that this problem only happens when `Named` parameter is not
used. When named parameter is used, topics are generated with a same patter as
specified in StreamsResetter.
> kafka-streams-application-reset tool doesn't take into account topics
> generated by KTable foreign key join operation
>
>
> Key: KAFKA-9859
> URL: https://issues.apache.org/jira/browse/KAFKA-9859
> Project: Kafka
> Issue Type: Bug
> Components: streams, tools
>Reporter: Levani Kokhreidze
>Priority: Major
>
> Steps to reproduce:
> * Create Kafka Streams application which uses foreign key join operation
> * Stop Kafka streams application
> * Perform `kafka-topics-list` and verify that foreign key operation internal
> topics are generated
> * Use `kafka-streams-application-reset` to perform the cleanup of your kafka
> streams application: `kafka-streams-application-reset --application-id
> --input-topics --bootstrap-servers
> --to-datetime 2019-04-13T00:00:00.000`
> * Perform `kafka-topics-list` again, you'll see that topics generated by the
> foreign key operation are still there.
> [kafka-streams-application-reset|[https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/StreamsResetter.java#L679-L680]]
> uses `-subscription-registration-topic` and `-subscription-response-topic`
> suffixes to match topics generated by the foreign key operation. While in
> reality, internal topics are generated in this format:
>
> {code:java}
> -KTABLE-FK-JOIN-SUBSCRIPTION-REGISTRATION- number>-topic
> -KTABLE-FK-JOIN-SUBSCRIPTION-RESPONSE--topic
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-9859) kafka-streams-application-reset tool doesn't take into account topics generated by KTable foreign key join operation
[
https://issues.apache.org/jira/browse/KAFKA-9859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Levani Kokhreidze updated KAFKA-9859:
-
Description:
Steps to reproduce:
* Create Kafka Streams application which uses foreign key join operation
* Stop Kafka streams application
* Perform `kafka-topics-list` and verify that foreign key operation internal
topics are generated
* Use `kafka-streams-application-reset` to perform the cleanup of your kafka
streams application: `kafka-streams-application-reset --application-id
--input-topics --bootstrap-servers
--to-datetime 2019-04-13T00:00:00.000`
* Perform `kafka-topics-list` again, you'll see that topics generated by the
foreign key operation are still there.
[kafka-streams-application-reset|[https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/StreamsResetter.java#L679-L680]]
uses `-subscription-registration-topic` and `-subscription-response-topic`
suffixes to match topics generated by the foreign key operation. While in
reality, internal topics are generated in this format:
{code:java}
-KTABLE-FK-JOIN-SUBSCRIPTION-REGISTRATION--topic
-KTABLE-FK-JOIN-SUBSCRIPTION-RESPONSE--topic
{code}
was:
Steps to reproduce:
* Create Kafka Streams application which uses foreign key join operation
* Stop Kafka streams application
* Perform `kafka-topics-list` and verify that foreign key operation internal
topics are generated
* Use `kafka-streams-application-reset` to perform the cleanup of your kafka
streams application: `kafka-streams-application-reset --application-id
--input-topics --bootstrap-servers
--to-datetime 2019-04-13T00:00:00.000`
* Perform `kafka-topics-list` again, you'll see that topics generated by the
foreign key operation are still there.
kafka-streams-application-reset uses `-subscription-registration-topic` and
`-subscription-response-topic` suffixes to match topics generated by the
foreign key operation. While in reality, internal topics are generated in this
format:
{code:java}
-KTABLE-FK-JOIN-SUBSCRIPTION-REGISTRATION--topic
-KTABLE-FK-JOIN-SUBSCRIPTION-RESPONSE--topic
{code}
> kafka-streams-application-reset tool doesn't take into account topics
> generated by KTable foreign key join operation
>
>
> Key: KAFKA-9859
> URL: https://issues.apache.org/jira/browse/KAFKA-9859
> Project: Kafka
> Issue Type: Bug
> Components: streams, tools
>Reporter: Levani Kokhreidze
>Priority: Major
>
> Steps to reproduce:
> * Create Kafka Streams application which uses foreign key join operation
> * Stop Kafka streams application
> * Perform `kafka-topics-list` and verify that foreign key operation internal
> topics are generated
> * Use `kafka-streams-application-reset` to perform the cleanup of your kafka
> streams application: `kafka-streams-application-reset --application-id
> --input-topics --bootstrap-servers
> --to-datetime 2019-04-13T00:00:00.000`
> * Perform `kafka-topics-list` again, you'll see that topics generated by the
> foreign key operation are still there.
> [kafka-streams-application-reset|[https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/StreamsResetter.java#L679-L680]]
> uses `-subscription-registration-topic` and `-subscription-response-topic`
> suffixes to match topics generated by the foreign key operation. While in
> reality, internal topics are generated in this format:
>
> {code:java}
> -KTABLE-FK-JOIN-SUBSCRIPTION-REGISTRATION- number>-topic
> -KTABLE-FK-JOIN-SUBSCRIPTION-RESPONSE--topic
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-9859) kafka-streams-application-reset tool doesn't take into account topics generated by KTable foreign key join operation
[
https://issues.apache.org/jira/browse/KAFKA-9859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Levani Kokhreidze updated KAFKA-9859:
-
Description:
Steps to reproduce:
* Create Kafka Streams application which uses foreign key join operation
* Stop Kafka streams application
* Perform `kafka-topics-list` and verify that foreign key operation internal
topics are generated
* Use `kafka-streams-application-reset` to perform the cleanup of your kafka
streams application: `kafka-streams-application-reset --application-id
--input-topics --bootstrap-servers
--to-datetime 2019-04-13T00:00:00.000`
* Perform `kafka-topics-list` again, you'll see that topics generated by the
foreign key operation are still there.
kafka-streams-application-reset uses `-subscription-registration-topic` and
`-subscription-response-topic` suffixes to match topics generated by the
foreign key operation. While in reality, internal topics are generated in this
format:
{code:java}
-KTABLE-FK-JOIN-SUBSCRIPTION-REGISTRATION--topic
-KTABLE-FK-JOIN-SUBSCRIPTION-RESPONSE--topic
{code}
was:
Steps to reproduce:
* Create Kafka Streams application which uses foreign key join operation
* Stop Kafka streams application
* Perform `kafka-topics-list` and verify that foreign key operation internal
topics are generated
* Use `kafka-streams-application-reset` to perform the cleanup of your kafka
streams application: `kafka-streams-application-reset --application-id
--input-topics --bootstrap-servers
--to-datetime 2019-04-13T00:00:00.000`
* Perform `kafka-topics-list` again, you'll see that topics generated by the
foreign key operation are still there.
`kafka-streams-application-reset` uses `-subscription-registration-topic` and
`-subscription-response-topic` suffixes to match topics generated by the
foreign key operation. While in reality, internal topics are generated in this
format:
{code:java}
-KTABLE-FK-JOIN-SUBSCRIPTION-REGISTRATION--topic
-KTABLE-FK-JOIN-SUBSCRIPTION-RESPONSE--topic
{code}
> kafka-streams-application-reset tool doesn't take into account topics
> generated by KTable foreign key join operation
>
>
> Key: KAFKA-9859
> URL: https://issues.apache.org/jira/browse/KAFKA-9859
> Project: Kafka
> Issue Type: Bug
> Components: streams, tools
>Reporter: Levani Kokhreidze
>Priority: Major
>
> Steps to reproduce:
> * Create Kafka Streams application which uses foreign key join operation
> * Stop Kafka streams application
> * Perform `kafka-topics-list` and verify that foreign key operation internal
> topics are generated
> * Use `kafka-streams-application-reset` to perform the cleanup of your kafka
> streams application: `kafka-streams-application-reset --application-id
> --input-topics --bootstrap-servers
> --to-datetime 2019-04-13T00:00:00.000`
> * Perform `kafka-topics-list` again, you'll see that topics generated by the
> foreign key operation are still there.
> kafka-streams-application-reset uses `-subscription-registration-topic` and
> `-subscription-response-topic` suffixes to match topics generated by the
> foreign key operation. While in reality, internal topics are generated in
> this format:
>
> {code:java}
> -KTABLE-FK-JOIN-SUBSCRIPTION-REGISTRATION- number>-topic
> -KTABLE-FK-JOIN-SUBSCRIPTION-RESPONSE--topic
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-9859) kafka-streams-application-reset tool doesn't take into account topics generated by KTable foreign key join operation
[
https://issues.apache.org/jira/browse/KAFKA-9859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Levani Kokhreidze updated KAFKA-9859:
-
Description:
Steps to reproduce:
* Create Kafka Streams application which uses foreign key join operation
* Stop Kafka streams application
* Perform `kafka-topics-list` and verify that foreign key operation internal
topics are generated
* Use `kafka-streams-application-reset` to perform the cleanup of your kafka
streams application: `kafka-streams-application-reset --application-id
--input-topics --bootstrap-servers
--to-datetime 2019-04-13T00:00:00.000`
* Perform `kafka-topics-list` again, you'll see that topics generated by the
foreign key operation are still there.
`kafka-streams-application-reset` uses `-subscription-registration-topic` and
`-subscription-response-topic` suffixes to match topics generated by the
foreign key operation. While in reality, internal topics are generated in this
format:
{code:java}
-KTABLE-FK-JOIN-SUBSCRIPTION-REGISTRATION--topic
-KTABLE-FK-JOIN-SUBSCRIPTION-RESPONSE--topic
{code}
was:
Steps to reproduce:
* Create Kafka Streams application which uses foreign key join operation
* Stop Kafka streams application
* Perform `kafka-topics-list` and verify that foreign key operation internal
topics are generated
* Use `kafka-streams-application-reset` to perform the cleanup of your kafka
streams application: `kafka-streams-application-reset --application-id
--input-topics --bootstrap-servers
--to-datetime 2019-04-13T00:00:00.000`
* Perform `kafka-topics-list` again, you'll see that topics generated by the
foreign key operation are still there.
`kafka-streams-application-reset` uses `-subscription-registration-topic` and
`-subscription-response-topic` suffixes to match topics generated by the
foreign key operation. While in reality, internal topics are generated in this
format:
{code:java}
-KTABLE-FK-JOIN-SUBSCRIPTION-REGISTRATION--topic
-KTABLE-FK-JOIN-SUBSCRIPTION-RESPONSE--topic
{code}
> kafka-streams-application-reset tool doesn't take into account topics
> generated by KTable foreign key join operation
>
>
> Key: KAFKA-9859
> URL: https://issues.apache.org/jira/browse/KAFKA-9859
> Project: Kafka
> Issue Type: Bug
> Components: streams, tools
>Reporter: Levani Kokhreidze
>Priority: Major
>
> Steps to reproduce:
> * Create Kafka Streams application which uses foreign key join operation
> * Stop Kafka streams application
> * Perform `kafka-topics-list` and verify that foreign key operation internal
> topics are generated
> * Use `kafka-streams-application-reset` to perform the cleanup of your kafka
> streams application: `kafka-streams-application-reset --application-id
> --input-topics --bootstrap-servers
> --to-datetime 2019-04-13T00:00:00.000`
> * Perform `kafka-topics-list` again, you'll see that topics generated by the
> foreign key operation are still there.
> `kafka-streams-application-reset` uses `-subscription-registration-topic` and
> `-subscription-response-topic` suffixes to match topics generated by the
> foreign key operation. While in reality, internal topics are generated in
> this format:
>
> {code:java}
> -KTABLE-FK-JOIN-SUBSCRIPTION-REGISTRATION- number>-topic
> -KTABLE-FK-JOIN-SUBSCRIPTION-RESPONSE--topic
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-9859) kafka-streams-application-reset tool doesn't take into account topics generated by KTable foreign key join operation
[
https://issues.apache.org/jira/browse/KAFKA-9859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Levani Kokhreidze updated KAFKA-9859:
-
Description:
Steps to reproduce:
* Create Kafka Streams application which uses foreign key join operation
* Stop Kafka streams application
* Perform `kafka-topics-list` and verify that foreign key operation internal
topics are generated
* Use `kafka-streams-application-reset` to perform the cleanup of your kafka
streams application: `kafka-streams-application-reset --application-id
--input-topics --bootstrap-servers
--to-datetime 2019-04-13T00:00:00.000`
* Perform `kafka-topics-list` again, you'll see that topics generated by the
foreign key operation are still there.
`kafka-streams-application-reset` uses `-subscription-registration-topic` and
`-subscription-response-topic` suffixes to match topics generated by the
foreign key operation. While in reality, internal topics are generated in this
format:
{code:java}
-KTABLE-FK-JOIN-SUBSCRIPTION-REGISTRATION--topic
-KTABLE-FK-JOIN-SUBSCRIPTION-RESPONSE--topic
{code}
was:
Steps to reproduce:
* Create Kafka Streams application which uses foreign key join operation
* Stop Kafka streams application
* Perform `kafka-topics-list` and verify that foreign key operation internal
topics are generated
* Use `kafka-streams-application-reset` to perform the cleanup of your kafka
streams application: `kafka-streams-application-reset --application-id
--input-topics --bootstrap-servers
--to-datetime 2019-04-13T00:00:00.000`
* Perform `kafka-topics-list` again, you'll see that topics generated by the
foreign key operation are still there.
`kafka-streams-application-reset` uses `-subscription-registration-topic` and
`-subscription-response-topic` suffixes to match topics generated by the
foreign key operation. While in reality, internal topics are generated in this
format:
{code:java}
-KTABLE-FK-JOIN-SUBSCRIPTION-REGISTRATION--topic
-KTABLE-FK-JOIN-SUBSCRIPTION-RESPONSE--topic
{code}
> kafka-streams-application-reset tool doesn't take into account topics
> generated by KTable foreign key join operation
>
>
> Key: KAFKA-9859
> URL: https://issues.apache.org/jira/browse/KAFKA-9859
> Project: Kafka
> Issue Type: Bug
> Components: streams, tools
>Reporter: Levani Kokhreidze
>Priority: Major
>
> Steps to reproduce:
> * Create Kafka Streams application which uses foreign key join operation
> * Stop Kafka streams application
> * Perform `kafka-topics-list` and verify that foreign key operation internal
> topics are generated
> * Use `kafka-streams-application-reset` to perform the cleanup of your kafka
> streams application: `kafka-streams-application-reset --application-id
> --input-topics --bootstrap-servers
> --to-datetime 2019-04-13T00:00:00.000`
> * Perform `kafka-topics-list` again, you'll see that topics generated by the
> foreign key operation are still there.
> `kafka-streams-application-reset` uses `-subscription-registration-topic` and
> `-subscription-response-topic` suffixes to match topics generated by the
> foreign key operation. While in reality, internal topics are generated in
> this format:
>
>
> {code:java}
> -KTABLE-FK-JOIN-SUBSCRIPTION-REGISTRATION- number>-topic
> -KTABLE-FK-JOIN-SUBSCRIPTION-RESPONSE--topic
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Reopened] (KAFKA-9859) kafka-streams-application-reset tool doesn't take into account topics generated by KTable foreign key join operation
[
https://issues.apache.org/jira/browse/KAFKA-9859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Levani Kokhreidze reopened KAFKA-9859:
--
> kafka-streams-application-reset tool doesn't take into account topics
> generated by KTable foreign key join operation
>
>
> Key: KAFKA-9859
> URL: https://issues.apache.org/jira/browse/KAFKA-9859
> Project: Kafka
> Issue Type: Bug
> Components: streams, tools
>Reporter: Levani Kokhreidze
>Priority: Major
>
> Steps to reproduce:
> * Create Kafka Streams application which uses foreign key join operation
> * Stop Kafka streams application
> * Perform `kafka-topics-list` and verify that foreign key operation internal
> topics are generated
> * Use `kafka-streams-application-reset` to perform the cleanup of your kafka
> streams application: `kafka-streams-application-reset --application-id
> --input-topics --bootstrap-servers
> --to-datetime 2019-04-13T00:00:00.000`
> * Perform `kafka-topics-list` again, you'll see that topics generated by the
> foreign key operation are still there.
> `kafka-streams-application-reset` uses `-subscription-registration-topic` and
> `-subscription-response-topic` suffixes to match topics generated by the
> foreign key operation. While in reality, internal topics are generated in
> this format:
>
> {code:java}
> -KTABLE-FK-JOIN-SUBSCRIPTION-REGISTRATION- number>-topic
> -KTABLE-FK-JOIN-SUBSCRIPTION-RESPONSE--topic
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9859) kafka-streams-application-reset tool doesn't take into account topics generated by KTable foreign key join operation
[
https://issues.apache.org/jira/browse/KAFKA-9859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091236#comment-17091236
]
Levani Kokhreidze commented on KAFKA-9859:
--
Thanks for spotting this, haven't paid attention to it. Re-opening the ticket
with the new findings.
> kafka-streams-application-reset tool doesn't take into account topics
> generated by KTable foreign key join operation
>
>
> Key: KAFKA-9859
> URL: https://issues.apache.org/jira/browse/KAFKA-9859
> Project: Kafka
> Issue Type: Bug
> Components: streams, tools
>Reporter: Levani Kokhreidze
>Priority: Major
>
> Steps to reproduce:
> * Create Kafka Streams application which uses foreign key join operation
> * Stop Kafka streams application
> * Perform `kafka-topics-list` and verify that foreign key operation internal
> topics are generated
> * Use `kafka-streams-application-reset` to perform the cleanup of your kafka
> streams application: `kafka-streams-application-reset --application-id
> --input-topics --bootstrap-servers
> --to-datetime 2019-04-13T00:00:00.000`
> * Perform `kafka-topics-list` again, you'll see that topics generated by the
> foreign key operation are still there.
> `kafka-streams-application-reset` uses `-subscription-registration-topic` and
> `-subscription-response-topic` suffixes to match topics generated by the
> foreign key operation. While in reality, internal topics are generated in
> this format:
>
> {code:java}
> -KTABLE-FK-JOIN-SUBSCRIPTION-REGISTRATION- number>-topic
> -KTABLE-FK-JOIN-SUBSCRIPTION-RESPONSE--topic
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-9859) kafka-streams-application-reset tool doesn't take into account topics generated by KTable foreign key join operation
[ https://issues.apache.org/jira/browse/KAFKA-9859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Levani Kokhreidze updated KAFKA-9859: - Description: Steps to reproduce: * Create Kafka Streams application which uses foreign key join operation * Stop Kafka streams application * Perform `kafka-topics-list` and verify that foreign key operation internal topics are generated * Use `kafka-streams-application-reset` to perform the cleanup of your kafka streams application: `kafka-streams-application-reset --application-id --input-topics --bootstrap-servers --to-datetime 2019-04-13T00:00:00.000` * Perform `kafka-topics-list` again, you'll see that topics generated by the foreign key operation are still there. `kafka-streams-application-reset` uses `-subscription-registration-topic` and `-subscription-response-topic` suffixes to match topics generated by the foreign key operation. While in reality, internal topics are generated in this format: was: Steps to reproduce: * Create Kafka Streams application which uses foreign key join operation * Stop Kafka streams application * Perform `kafka-topics-list` and verify that foreign key operation internal topics are generated * Use `kafka-streams-application-reset` to perform the cleanup of your kafka streams application: `kafka-streams-application-reset --application-id --input-topics --bootstrap-servers --to-datetime 2019-04-13T00:00:00.000` * Perform `kafka-topics-list` again, you'll see that topics generated by the foreign key operation are still there. `kafka-streams-application-reset` uses `repartition` and `changelog` suffixes to determine which topics needs to be deleted, as a result topics generated by the foreign key are ignored. > kafka-streams-application-reset tool doesn't take into account topics > generated by KTable foreign key join operation > > > Key: KAFKA-9859 > URL: https://issues.apache.org/jira/browse/KAFKA-9859 > Project: Kafka > Issue Type: Bug > Components: streams, tools >Reporter: Levani Kokhreidze >Priority: Major > > Steps to reproduce: > * Create Kafka Streams application which uses foreign key join operation > * Stop Kafka streams application > * Perform `kafka-topics-list` and verify that foreign key operation internal > topics are generated > * Use `kafka-streams-application-reset` to perform the cleanup of your kafka > streams application: `kafka-streams-application-reset --application-id > --input-topics --bootstrap-servers > --to-datetime 2019-04-13T00:00:00.000` > * Perform `kafka-topics-list` again, you'll see that topics generated by the > foreign key operation are still there. > `kafka-streams-application-reset` uses `-subscription-registration-topic` and > `-subscription-response-topic` suffixes to match topics generated by the > foreign key operation. While in reality, internal topics are generated in > this format: > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-9859) kafka-streams-application-reset tool doesn't take into account topics generated by KTable foreign key join operation
[
https://issues.apache.org/jira/browse/KAFKA-9859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Levani Kokhreidze updated KAFKA-9859:
-
Description:
Steps to reproduce:
* Create Kafka Streams application which uses foreign key join operation
* Stop Kafka streams application
* Perform `kafka-topics-list` and verify that foreign key operation internal
topics are generated
* Use `kafka-streams-application-reset` to perform the cleanup of your kafka
streams application: `kafka-streams-application-reset --application-id
--input-topics --bootstrap-servers
--to-datetime 2019-04-13T00:00:00.000`
* Perform `kafka-topics-list` again, you'll see that topics generated by the
foreign key operation are still there.
`kafka-streams-application-reset` uses `-subscription-registration-topic` and
`-subscription-response-topic` suffixes to match topics generated by the
foreign key operation. While in reality, internal topics are generated in this
format:
{code:java}
-KTABLE-FK-JOIN-SUBSCRIPTION-REGISTRATION--topic
-KTABLE-FK-JOIN-SUBSCRIPTION-RESPONSE--topic
{code}
was:
Steps to reproduce:
* Create Kafka Streams application which uses foreign key join operation
* Stop Kafka streams application
* Perform `kafka-topics-list` and verify that foreign key operation internal
topics are generated
* Use `kafka-streams-application-reset` to perform the cleanup of your kafka
streams application: `kafka-streams-application-reset --application-id
--input-topics --bootstrap-servers
--to-datetime 2019-04-13T00:00:00.000`
* Perform `kafka-topics-list` again, you'll see that topics generated by the
foreign key operation are still there.
`kafka-streams-application-reset` uses `-subscription-registration-topic` and
`-subscription-response-topic` suffixes to match topics generated by the
foreign key operation. While in reality, internal topics are generated in this
format:
> kafka-streams-application-reset tool doesn't take into account topics
> generated by KTable foreign key join operation
>
>
> Key: KAFKA-9859
> URL: https://issues.apache.org/jira/browse/KAFKA-9859
> Project: Kafka
> Issue Type: Bug
> Components: streams, tools
>Reporter: Levani Kokhreidze
>Priority: Major
>
> Steps to reproduce:
> * Create Kafka Streams application which uses foreign key join operation
> * Stop Kafka streams application
> * Perform `kafka-topics-list` and verify that foreign key operation internal
> topics are generated
> * Use `kafka-streams-application-reset` to perform the cleanup of your kafka
> streams application: `kafka-streams-application-reset --application-id
> --input-topics --bootstrap-servers
> --to-datetime 2019-04-13T00:00:00.000`
> * Perform `kafka-topics-list` again, you'll see that topics generated by the
> foreign key operation are still there.
> `kafka-streams-application-reset` uses `-subscription-registration-topic` and
> `-subscription-response-topic` suffixes to match topics generated by the
> foreign key operation. While in reality, internal topics are generated in
> this format:
>
> {code:java}
> -KTABLE-FK-JOIN-SUBSCRIPTION-REGISTRATION- number>-topic
> -KTABLE-FK-JOIN-SUBSCRIPTION-RESPONSE--topic
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9850) Move KStream#repartition operator validation during Topology build process
[ https://issues.apache.org/jira/browse/KAFKA-9850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17087979#comment-17087979 ] Levani Kokhreidze commented on KAFKA-9850: -- Hi [~bchen225242] Sorry for the late reply, missed this. I may pick this up, but not yet. I want to focus on some other KIP first. > Move KStream#repartition operator validation during Topology build process > --- > > Key: KAFKA-9850 > URL: https://issues.apache.org/jira/browse/KAFKA-9850 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Levani Kokhreidze >Priority: Major > Labels: help-wanted, newbie, newbie++ > > `KStream#repartition` operation performs most of its validation regarding > joining, co-partitioning, etc after starting Kafka Streams instance. Some > parts of this validation can be detected much earlier, specifically during > topology `build()`. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (KAFKA-9859) kafka-streams-application-reset tool doesn't take into account topics generated by KTable foreign key join operation
[ https://issues.apache.org/jira/browse/KAFKA-9859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17082619#comment-17082619 ] Levani Kokhreidze edited comment on KAFKA-9859 at 4/13/20, 7:55 PM: Ah sorry, I think this ticket is a bit misleading. Kafka broker I have tested this was 2.3.1 and FK join was implemented in 2.4. Seems like 2.4.0 has proper implementation around topics generated by the FK Join, see `kafka.tools.StreamsResetter#isInternalTopic`. I guess this ticket can be closed as "not a problem"? Pretty sure streams-resetter on older versions of brokers can't be backward compatible with the FK join feature. was (Author: lkokhreidze): Ah sorry, I think this ticket is a bit misleading. Kafka broker I have tested this was 2.3.1 and FK join was implemented in 2.4. Seems like 2.4.0 has proper implementation around topics generated by the FK Join, see `kafka.tools.StreamsResetter#isInternalTopic`. I guess this ticket can be closed as "not a problem"? Not sure if older versions of brokers can be backward compatible with this. > kafka-streams-application-reset tool doesn't take into account topics > generated by KTable foreign key join operation > > > Key: KAFKA-9859 > URL: https://issues.apache.org/jira/browse/KAFKA-9859 > Project: Kafka > Issue Type: Bug > Components: streams, tools >Reporter: Levani Kokhreidze >Priority: Major > > Steps to reproduce: > * Create Kafka Streams application which uses foreign key join operation > * Stop Kafka streams application > * Perform `kafka-topics-list` and verify that foreign key operation internal > topics are generated > * Use `kafka-streams-application-reset` to perform the cleanup of your kafka > streams application: `kafka-streams-application-reset --application-id > --input-topics --bootstrap-servers > --to-datetime 2019-04-13T00:00:00.000` > * Perform `kafka-topics-list` again, you'll see that topics generated by the > foreign key operation are still there. > `kafka-streams-application-reset` uses `repartition` and `changelog` suffixes > to determine which topics needs to be deleted, as a result topics generated > by the foreign key are ignored. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-9859) kafka-streams-application-reset tool doesn't take into account topics generated by KTable foreign key join operation
[ https://issues.apache.org/jira/browse/KAFKA-9859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17082619#comment-17082619 ] Levani Kokhreidze commented on KAFKA-9859: -- Ah sorry, I think this ticket is a bit misleading. Kafka broker I have tested this was 2.3.1 and FK join was implemented in 2.4. Seems like 2.4.0 has proper implementation around topics generated by the FK Join, see `kafka.tools.StreamsResetter#isInternalTopic`. I guess this ticket can be closed as "not a problem"? Not sure if older versions of brokers can be backward compatible with this. > kafka-streams-application-reset tool doesn't take into account topics > generated by KTable foreign key join operation > > > Key: KAFKA-9859 > URL: https://issues.apache.org/jira/browse/KAFKA-9859 > Project: Kafka > Issue Type: Bug > Components: streams, tools >Reporter: Levani Kokhreidze >Priority: Major > > Steps to reproduce: > * Create Kafka Streams application which uses foreign key join operation > * Stop Kafka streams application > * Perform `kafka-topics-list` and verify that foreign key operation internal > topics are generated > * Use `kafka-streams-application-reset` to perform the cleanup of your kafka > streams application: `kafka-streams-application-reset --application-id > --input-topics --bootstrap-servers > --to-datetime 2019-04-13T00:00:00.000` > * Perform `kafka-topics-list` again, you'll see that topics generated by the > foreign key operation are still there. > `kafka-streams-application-reset` uses `repartition` and `changelog` suffixes > to determine which topics needs to be deleted, as a result topics generated > by the foreign key are ignored. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-9859) kafka-streams-application-reset tool doesn't take into account topics generated by KTable foreign key join operation
[ https://issues.apache.org/jira/browse/KAFKA-9859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Levani Kokhreidze updated KAFKA-9859: - Component/s: streams > kafka-streams-application-reset tool doesn't take into account topics > generated by KTable foreign key join operation > > > Key: KAFKA-9859 > URL: https://issues.apache.org/jira/browse/KAFKA-9859 > Project: Kafka > Issue Type: Bug > Components: streams >Reporter: Levani Kokhreidze >Priority: Major > > Steps to reproduce: > * Create Kafka Streams application which uses foreign key join operation > * Stop Kafka streams application > * Perform `kafka-topics-list` and verify that foreign key operation internal > topics are generated > * Use `kafka-streams-application-reset` to perform the cleanup of your kafka > streams application: `kafka-streams-application-reset --application-id > --input-topics --bootstrap-servers > --to-datetime 2019-04-13T00:00:00.000` > * Perform `kafka-topics-list` again, you'll see that topics generated by the > foreign key operation are still there. > `kafka-streams-application-reset` uses `repartition` and `changelog` suffixes > to determine which topics needs to be deleted, as a result topics generated > by the foreign key are ignored. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-9859) kafka-streams-application-reset tool doesn't take into account topics generated by KTable foreign key join operation
Levani Kokhreidze created KAFKA-9859: Summary: kafka-streams-application-reset tool doesn't take into account topics generated by KTable foreign key join operation Key: KAFKA-9859 URL: https://issues.apache.org/jira/browse/KAFKA-9859 Project: Kafka Issue Type: Bug Reporter: Levani Kokhreidze Steps to reproduce: * Create Kafka Streams application which uses foreign key join operation * Stop Kafka streams application * Perform `kafka-topics-list` and verify that foreign key operation internal topics are generated * Use `kafka-streams-application-reset` to perform the cleanup of your kafka streams application: `kafka-streams-application-reset --application-id --input-topics --bootstrap-servers --to-datetime 2019-04-13T00:00:00.000` * Perform `kafka-topics-list` again, you'll see that topics generated by the foreign key operation are still there. `kafka-streams-application-reset` uses `repartition` and `changelog` suffixes to determine which topics needs to be deleted, as a result topics generated by the foreign key are ignored. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-9850) Move KStream#repartition operator validation during Topology build process
[ https://issues.apache.org/jira/browse/KAFKA-9850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17081314#comment-17081314 ] Levani Kokhreidze commented on KAFKA-9850: -- Related comment on the PR [https://github.com/apache/kafka/pull/7170#discussion_r401360195] > Move KStream#repartition operator validation during Topology build process > --- > > Key: KAFKA-9850 > URL: https://issues.apache.org/jira/browse/KAFKA-9850 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Levani Kokhreidze >Priority: Major > > `KStream#repartition` operation performs most of its validation regarding > joining, co-partitioning, etc after starting Kafka Streams instance. Some > parts of this validation can be detected much earlier, specifically during > topology `build()`. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-9850) Move KStream#repartition operator validation during Topology build process
Levani Kokhreidze created KAFKA-9850: Summary: Move KStream#repartition operator validation during Topology build process Key: KAFKA-9850 URL: https://issues.apache.org/jira/browse/KAFKA-9850 Project: Kafka Issue Type: Improvement Components: streams Reporter: Levani Kokhreidze `KStream#repartition` operation performs most of its validation regarding joining, co-partitioning, etc after starting Kafka Streams instance. Some parts of this validation can be detected much earlier, specifically during topology `build()`. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KAFKA-9828) Add partition to TestRecord in streams test-utils
[ https://issues.apache.org/jira/browse/KAFKA-9828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Levani Kokhreidze reassigned KAFKA-9828: Assignee: Levani Kokhreidze > Add partition to TestRecord in streams test-utils > - > > Key: KAFKA-9828 > URL: https://issues.apache.org/jira/browse/KAFKA-9828 > Project: Kafka > Issue Type: Improvement > Components: streams-test-utils >Reporter: Levani Kokhreidze >Assignee: Levani Kokhreidze >Priority: Minor > > TopologyTestDriver creates `TestRecord` for consumed events. In order to test > partitioning, when one uses custom partitioner, would be useful if > `TestRecord` had `partition` field as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-9828) Add partition to TestRecord in streams test-utils
Levani Kokhreidze created KAFKA-9828: Summary: Add partition to TestRecord in streams test-utils Key: KAFKA-9828 URL: https://issues.apache.org/jira/browse/KAFKA-9828 Project: Kafka Issue Type: Improvement Components: streams-test-utils Reporter: Levani Kokhreidze TopologyTestDriver creates `TestRecord` for consumed events. In order to test partitioning, when one uses custom partitioner, would be useful if `TestRecord` had `partition` field as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (KAFKA-9638) Do not trigger REBALANCING when specific exceptions occur in Kafka Streams
[
https://issues.apache.org/jira/browse/KAFKA-9638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17051072#comment-17051072
]
Levani Kokhreidze edited comment on KAFKA-9638 at 3/4/20, 10:10 AM:
Hi [~bchen225242]
Not sure I understand. Maybe I am missing something but whenever stream thread
encounters unrecoverable exception, all its assigned partitions with
corresponding state will be migrated to other thread (not necessarily on the
same node).
Lets have a concrete example, if we have topology similar to this with 6
threads span across 2 physical machines (3 threads on each machine). And for
simplicity, lets say that `input-topic` has also 6 partitions.
{code:java}
streamBuilder.stream("input-topic")
.selectKey((key, value) -> value.newKey())
.groupByKey()
.aggregate(0d, (key, value, aggr) -> aggr + value.invoiceValue(),
Materialized.as("my-store"));
{code}
In the aggregator function I have a bug, `value.inoiceValue()` may sometime
return null, and code above will throw NPE. As a result, stream thread will
die, and it's partition will be re-assigned (with corresponding state) to
another thread, maybe to another node altogether. Since there was an error,
offset for problematic event won't be committed and other threads are also
doomed to die with the same exception. So all the rebalancing, state migration,
etc is pretty much useless in this case.
Instead, would be great if Kafka streams would kill the thread for the
partition where problematic event is, instead of rebalancing it across
different threads. In this case, we would accumulate lag on a single partition
but all other threads would still be processing data. So instead of global
downtime on 6 partitions, we would have downtime only on a single one.
Does this make sense?
Regards,
Levani
was (Author: lkokhreidze):
Hi [~bchen225242]
Not sure I understand. Maybe I am missing something but whenever stream thread
encounters unrecoverable exception, all its assigned partitions with
corresponding state will be migrated to other thread (not necessarily on the
same node).
Lets have a concrete example, if we have topology similar to this with 6
threads span across 2 physical machines (3 threads on each machine). And for
simplicity, lets say that `input-topic` has also 6 partitions.
{code:java}
streamBuilder.stream("input-topic")
.selectKey((key, value) -> value.newKey())
.groupByKey()
.aggregate(0d, (key, value, aggr) -> aggr + value.invoiceValue(),
Materialized.as("my-store"));
{code}
In the aggregator function I have a bug, `value.inoiceValue()` may sometime
return null, and code above will throw NPE. As a result, stream thread will
die, and it's partition will be re-assigned (with corresponding state) to
another thread, maybe to another node altogether. Since there was an error,
offset for problematic event won't be committed and other threads are also
doomed to die with the same exception. So all the rebalancing, state migration,
etc is pretty much useless in this case.
Instead, would be great if Kafka streams would kill the thread for the
partition where problematic event is, instead of rebalancing it across
different threads. In this case, we would accumulate lag on a single partition
but all other threads would still be processing data. So instead of global
downtime on 6 partitions, we would have downtime only on a single one.
Does this make sense?
Regards,
- Levani
> Do not trigger REBALANCING when specific exceptions occur in Kafka Streams
> ---
>
> Key: KAFKA-9638
> URL: https://issues.apache.org/jira/browse/KAFKA-9638
> Project: Kafka
> Issue Type: New Feature
> Components: streams
>Reporter: Levani Kokhreidze
>Priority: Major
>
> As of now, when StreamThread encounters exception in Kafka Streams
> application, it will result in REBALANCING of all the tasks that were
> responsibility of the given thread. Problem with that is, if the exception
> was, lets say some logical exception, like NPE, REBALANCING is pretty much
> useless, cause all other threads will also die with the same NPE. This kind
> of mute rebalancing gives extra costs in terms of network traffic, IOPS, etc
> in case of large stateful applications.
> In addition, this behaviour causes global outage of the Kafka Streams
> application, instead of localized outage of the certain tasks. Would be great
> if Kafka Streams users could specify via some interface, exceptions that must
> not trigger rebalancing of the tasks. StreamThread may still die, but in this
> case, we would have isolated incident.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9638) Do not trigger REBALANCING when specific exceptions occur in Kafka Streams
[
https://issues.apache.org/jira/browse/KAFKA-9638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17051072#comment-17051072
]
Levani Kokhreidze commented on KAFKA-9638:
--
Hi [~bchen225242]
Not sure I understand. Maybe I am missing something but whenever stream thread
encounters unrecoverable exception, all its assigned partitions with
corresponding state will be migrated to other thread (not necessarily on the
same node).
Lets have a concrete example, if we have topology similar to this with 6
threads span across 2 physical machines (3 threads on each machine). And for
simplicity, lets say that `input-topic` has also 6 partitions.
{code:java}
streamBuilder.stream("input-topic")
.selectKey((key, value) -> value.newKey())
.groupByKey()
.aggregate(0d, (key, value, aggr) -> aggr + value.invoiceValue(),
Materialized.as("my-store"));
{code}
In the aggregator function I have a bug, `value.inoiceValue()` may sometime
return null, and code above will throw NPE. As a result, stream thread will
die, and it's partition will be re-assigned (with corresponding state) to
another thread, maybe to another node altogether. Since there was an error,
offset for problematic event won't be committed and other threads are also
doomed to die with the same exception. So all the rebalancing, state migration,
etc is pretty much useless in this case.
Instead, would be great if Kafka streams would kill the thread for the
partition where problematic event is, instead of rebalancing it across
different threads. In this case, we would accumulate lag on a single partition
but all other threads would still be processing data. So instead of global
downtime on 6 partitions, we would have downtime only on a single one.
Does this make sense?
Regards,
- Levani
> Do not trigger REBALANCING when specific exceptions occur in Kafka Streams
> ---
>
> Key: KAFKA-9638
> URL: https://issues.apache.org/jira/browse/KAFKA-9638
> Project: Kafka
> Issue Type: New Feature
> Components: streams
>Reporter: Levani Kokhreidze
>Priority: Major
>
> As of now, when StreamThread encounters exception in Kafka Streams
> application, it will result in REBALANCING of all the tasks that were
> responsibility of the given thread. Problem with that is, if the exception
> was, lets say some logical exception, like NPE, REBALANCING is pretty much
> useless, cause all other threads will also die with the same NPE. This kind
> of mute rebalancing gives extra costs in terms of network traffic, IOPS, etc
> in case of large stateful applications.
> In addition, this behaviour causes global outage of the Kafka Streams
> application, instead of localized outage of the certain tasks. Would be great
> if Kafka Streams users could specify via some interface, exceptions that must
> not trigger rebalancing of the tasks. StreamThread may still die, but in this
> case, we would have isolated incident.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-9638) Do not trigger REBALANCING when specific exceptions occur in Kafka Streams
[ https://issues.apache.org/jira/browse/KAFKA-9638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Levani Kokhreidze updated KAFKA-9638: - Description: As of now, when StreamThread encounters exception in Kafka Streams application, it will result in REBALANCING of all the tasks that were responsibility of the given thread. Problem with that is, if the exception was, lets say some logical exception, like NPE, REBALANCING is pretty much useless, cause all other threads will also die with the same NPE. This kind of mute rebalancing gives extra costs in terms of network traffic, IOPS, etc in case of large stateful applications. In addition, this behaviour causes global outage of the Kafka Streams application, instead of localized outage of the certain tasks. Would be great if Kafka Streams users could specify via some interface exceptions that must not trigger rebalancing of the tasks. StreamThread may still die, but in this case, we would have isolated incident. was: As of now, when StreamThread encounters exception in Kafka Streams application, it will result in REBALANCING of all the tasks that were responsibility of the given thread. Problem with that is, if the exception was, lets say some logical exception, like NPE, REBALANCING is pretty much useless, cause all other threads will also die with the same NPE. This kind of mute rebalancing gives extra costs in terms of network traffic, IOPS, etc in case of large stateful applications. In addition, this behaviour causes global outage of the Kafka Streams application, instead of localized outage of the certain tasks. Would be great if Kafka Streams users could specify through some interface exceptions that must not trigger rebalancing of the tasks. StreamThread may still die, but in this case, we would have isolated incident. > Do not trigger REBALANCING when specific exceptions occur in Kafka Streams > --- > > Key: KAFKA-9638 > URL: https://issues.apache.org/jira/browse/KAFKA-9638 > Project: Kafka > Issue Type: New Feature > Components: streams >Reporter: Levani Kokhreidze >Priority: Major > > As of now, when StreamThread encounters exception in Kafka Streams > application, it will result in REBALANCING of all the tasks that were > responsibility of the given thread. Problem with that is, if the exception > was, lets say some logical exception, like NPE, REBALANCING is pretty much > useless, cause all other threads will also die with the same NPE. This kind > of mute rebalancing gives extra costs in terms of network traffic, IOPS, etc > in case of large stateful applications. > In addition, this behaviour causes global outage of the Kafka Streams > application, instead of localized outage of the certain tasks. Would be great > if Kafka Streams users could specify via some interface exceptions that must > not trigger rebalancing of the tasks. StreamThread may still die, but in this > case, we would have isolated incident. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-9638) Do not trigger REBALANCING when specific exceptions occur in Kafka Streams
[ https://issues.apache.org/jira/browse/KAFKA-9638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Levani Kokhreidze updated KAFKA-9638: - Description: As of now, when StreamThread encounters exception in Kafka Streams application, it will result in REBALANCING of all the tasks that were responsibility of the given thread. Problem with that is, if the exception was, lets say some logical exception, like NPE, REBALANCING is pretty much useless, cause all other threads will also die with the same NPE. This kind of mute rebalancing gives extra costs in terms of network traffic, IOPS, etc in case of large stateful applications. In addition, this behaviour causes global outage of the Kafka Streams application, instead of localized outage of the certain tasks. Would be great if Kafka Streams users could specify via some interface, exceptions that must not trigger rebalancing of the tasks. StreamThread may still die, but in this case, we would have isolated incident. was: As of now, when StreamThread encounters exception in Kafka Streams application, it will result in REBALANCING of all the tasks that were responsibility of the given thread. Problem with that is, if the exception was, lets say some logical exception, like NPE, REBALANCING is pretty much useless, cause all other threads will also die with the same NPE. This kind of mute rebalancing gives extra costs in terms of network traffic, IOPS, etc in case of large stateful applications. In addition, this behaviour causes global outage of the Kafka Streams application, instead of localized outage of the certain tasks. Would be great if Kafka Streams users could specify via some interface exceptions that must not trigger rebalancing of the tasks. StreamThread may still die, but in this case, we would have isolated incident. > Do not trigger REBALANCING when specific exceptions occur in Kafka Streams > --- > > Key: KAFKA-9638 > URL: https://issues.apache.org/jira/browse/KAFKA-9638 > Project: Kafka > Issue Type: New Feature > Components: streams >Reporter: Levani Kokhreidze >Priority: Major > > As of now, when StreamThread encounters exception in Kafka Streams > application, it will result in REBALANCING of all the tasks that were > responsibility of the given thread. Problem with that is, if the exception > was, lets say some logical exception, like NPE, REBALANCING is pretty much > useless, cause all other threads will also die with the same NPE. This kind > of mute rebalancing gives extra costs in terms of network traffic, IOPS, etc > in case of large stateful applications. > In addition, this behaviour causes global outage of the Kafka Streams > application, instead of localized outage of the certain tasks. Would be great > if Kafka Streams users could specify via some interface, exceptions that must > not trigger rebalancing of the tasks. StreamThread may still die, but in this > case, we would have isolated incident. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-9638) Do not trigger REBALANCING when specific exceptions occur in Kafka Streams
[ https://issues.apache.org/jira/browse/KAFKA-9638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Levani Kokhreidze updated KAFKA-9638: - Component/s: streams > Do not trigger REBALANCING when specific exceptions occur in Kafka Streams > --- > > Key: KAFKA-9638 > URL: https://issues.apache.org/jira/browse/KAFKA-9638 > Project: Kafka > Issue Type: New Feature > Components: streams >Reporter: Levani Kokhreidze >Priority: Major > > As of now, when StreamThread encounters exception in Kafka Streams > application, it will result in REBALANCING of all the tasks that were > responsibility of the given thread. Problem with that is, if the exception > was, lets say some logical exception, like NPE, REBALANCING is pretty much > useless, cause all other threads will also die with the same NPE. This kind > of mute rebalancing gives extra costs in terms of network traffic, IOPS, etc > in case of large stateful applications. > In addition, this behaviour causes global outage of the Kafka Streams > application, instead of localized outage of the certain tasks. Would be great > if Kafka Streams users could specify through some interface exceptions that > must not trigger rebalancing of the tasks. StreamThread may still die, but in > this case, we would have isolated incident. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-9638) Do not trigger REBALANCING when specific exceptions occur in Kafka Streams
[ https://issues.apache.org/jira/browse/KAFKA-9638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Levani Kokhreidze updated KAFKA-9638: - Labels: (was: streams) > Do not trigger REBALANCING when specific exceptions occur in Kafka Streams > --- > > Key: KAFKA-9638 > URL: https://issues.apache.org/jira/browse/KAFKA-9638 > Project: Kafka > Issue Type: New Feature >Reporter: Levani Kokhreidze >Priority: Major > > As of now, when StreamThread encounters exception in Kafka Streams > application, it will result in REBALANCING of all the tasks that were > responsibility of the given thread. Problem with that is, if the exception > was, lets say some logical exception, like NPE, REBALANCING is pretty much > useless, cause all other threads will also die with the same NPE. This kind > of mute rebalancing gives extra costs in terms of network traffic, IOPS, etc > in case of large stateful applications. > In addition, this behaviour causes global outage of the Kafka Streams > application, instead of localized outage of the certain tasks. Would be great > if Kafka Streams users could specify through some interface exceptions that > must not trigger rebalancing of the tasks. StreamThread may still die, but in > this case, we would have isolated incident. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-9638) Do not trigger REBALANCING when specific exceptions occur in Kafka Streams
[ https://issues.apache.org/jira/browse/KAFKA-9638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Levani Kokhreidze updated KAFKA-9638: - Labels: streams (was: ) > Do not trigger REBALANCING when specific exceptions occur in Kafka Streams > --- > > Key: KAFKA-9638 > URL: https://issues.apache.org/jira/browse/KAFKA-9638 > Project: Kafka > Issue Type: New Feature >Reporter: Levani Kokhreidze >Priority: Major > Labels: streams > > As of now, when StreamThread encounters exception in Kafka Streams > application, it will result in REBALANCING of all the tasks that were > responsibility of the given thread. Problem with that is, if the exception > was, lets say some logical exception, like NPE, REBALANCING is pretty much > useless, cause all other threads will also die with the same NPE. This kind > of mute rebalancing gives extra costs in terms of network traffic, IOPS, etc > in case of large stateful applications. > In addition, this behaviour causes global outage of the Kafka Streams > application, instead of localized outage of the certain tasks. Would be great > if Kafka Streams users could specify through some interface exceptions that > must not trigger rebalancing of the tasks. StreamThread may still die, but in > this case, we would have isolated incident. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-9638) Do not trigger REBALANCING when specific exceptions occur in Kafka Streams
[ https://issues.apache.org/jira/browse/KAFKA-9638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Levani Kokhreidze updated KAFKA-9638: - Description: As of now, when StreamThread encounters exception in Kafka Streams application, it will result in REBALANCING of all the tasks that were responsibility of the given thread. Problem with that is, if the exception was, lets say some logical exception, like NPE, REBALANCING is pretty much useless, cause all other threads will also die with the same NPE. This kind of mute rebalancing gives extra costs in terms of network traffic, IOPS, etc in case of large stateful applications. In addition, this behaviour causes global outage of the Kafka Streams application, instead of localized outage of the certain tasks. Would be great if Kafka Streams users could specify through some interface exceptions that must not trigger rebalancing of the tasks. StreamThread may still die, but in this case, we would have isolated incident. was: As of now, when StreamThread encounters exception in Kafka Streams application, it will result in REBALANCING on all the tasks that were responsibility of the given thread. Problem with that is, if the exception was, lets say some logical exception, like NPE, REBALANCING is pretty much useless cause all other threads will also die with the same NPE. This kind of mute rebalancing gives extra costs in terms of network traffic, IOPS, etc in case of large stateful applications. In addition, this behaviour causes global outage of the Kafka Streams application, instead of localized outage of the certain tasks. Would be great if Kafka Streams users could specify through some interface exceptions that must not trigger rebalancing of the tasks. StreamThread may still die, but in this case, we would have isolated incident. > Do not trigger REBALANCING when specific exceptions occur in Kafka Streams > --- > > Key: KAFKA-9638 > URL: https://issues.apache.org/jira/browse/KAFKA-9638 > Project: Kafka > Issue Type: New Feature >Reporter: Levani Kokhreidze >Priority: Major > > As of now, when StreamThread encounters exception in Kafka Streams > application, it will result in REBALANCING of all the tasks that were > responsibility of the given thread. Problem with that is, if the exception > was, lets say some logical exception, like NPE, REBALANCING is pretty much > useless, cause all other threads will also die with the same NPE. This kind > of mute rebalancing gives extra costs in terms of network traffic, IOPS, etc > in case of large stateful applications. > In addition, this behaviour causes global outage of the Kafka Streams > application, instead of localized outage of the certain tasks. Would be great > if Kafka Streams users could specify through some interface exceptions that > must not trigger rebalancing of the tasks. StreamThread may still die, but in > this case, we would have isolated incident. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-9638) Do not trigger REBALANCING when specific exceptions occur in Kafka Streams
Levani Kokhreidze created KAFKA-9638: Summary: Do not trigger REBALANCING when specific exceptions occur in Kafka Streams Key: KAFKA-9638 URL: https://issues.apache.org/jira/browse/KAFKA-9638 Project: Kafka Issue Type: New Feature Reporter: Levani Kokhreidze As of now, when StreamThread encounters exception in Kafka Streams application, it will result in REBALANCING on all the tasks that were responsibility of the given thread. Problem with that is, if the exception was, lets say some logical exception, like NPE, REBALANCING is pretty much useless cause all other threads will also die with the same NPE. This kind of mute rebalancing gives extra costs in terms of network traffic, IOPS, etc in case of large stateful applications. In addition, this behaviour causes global outage of the Kafka Streams application, instead of localized outage of the certain tasks. Would be great if Kafka Streams users could specify through some interface exceptions that must not trigger rebalancing of the tasks. StreamThread may still die, but in this case, we would have isolated incident. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-4835) Avoid repartitioning when key change doesn't change partitions
[
https://issues.apache.org/jira/browse/KAFKA-4835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Levani Kokhreidze updated KAFKA-4835:
-
Fix Version/s: (was: 2.6.0)
> Avoid repartitioning when key change doesn't change partitions
> --
>
> Key: KAFKA-4835
> URL: https://issues.apache.org/jira/browse/KAFKA-4835
> Project: Kafka
> Issue Type: Sub-task
> Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Michal Borowiecki
>Priority: Major
> Labels: kip
>
> From
> https://issues.apache.org/jira/browse/KAFKA-4601?focusedCommentId=15881030=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15881030
> ...it would be good to provide users more control over the repartitioning.
> My use case is as follows (unrelated bits omitted for brevity):
> {code:java}
> KTable loggedInCustomers = builder
> .stream("customerLogins")
> .groupBy((key, activity) ->
> activity.getCustomerRef())
> .reduce((first,second) -> second, loginStore());
>
> builder
> .stream("balanceUpdates")
> .map((key, activity) -> new KeyValue<>(
> activity.getCustomerRef(),
> activity))
> .join(loggedInCustomers, (activity, session) -> ...
> .to("sessions");
> {code}
> Both "groupBy" and "map" in the underlying implementation set the
> repartitionRequired flag (since the key changes), and the aggregation/join
> that follows will create the repartitioned topic.
> However, in our case I know that both input streams are already partitioned
> by the customerRef value, which I'm mapping into the key (because it's
> required by the join operation).
> So there are 2 unnecessary intermediate topics created with their associated
> overhead, while the ultimate goal is simply to do a join on a value that we
> already use to partition the original streams anyway.
> (Note, we don't have the option to re-implement the original input streams
> to make customerRef the message key.)
> I think it would be better to allow the user to decide (from their knowledge
> of the incoming streams) whether a repartition is mandatory on aggregation
> and join operations (overloaded version of the methods with the
> repartitionRequired flag exposed maybe?)
> An alternative would be to allow users to perform a join on a value other
> than the key (a keyValueMapper parameter to join, like the one used for joins
> with global tables), but I expect that to be more involved and error-prone to
> use for people who don't understand the partitioning requirements well
> (whereas it's safe for global tables).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-4835) Avoid repartitioning when key change doesn't change partitions
[
https://issues.apache.org/jira/browse/KAFKA-4835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17049715#comment-17049715
]
Levani Kokhreidze commented on KAFKA-4835:
--
Hey [~vvcephei],
Indeed this ticket is something that shouldn't be connected to KIP-221. I'll
unlink it from the KIP and stop the progress on it.
I don't plan to add anything new to existing PR
([https://github.com/apache/kafka/pull/7170]) as it's already quite big.
Thanks for noticing this.
Regards,
Levani
> Avoid repartitioning when key change doesn't change partitions
> --
>
> Key: KAFKA-4835
> URL: https://issues.apache.org/jira/browse/KAFKA-4835
> Project: Kafka
> Issue Type: Sub-task
> Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Michal Borowiecki
>Priority: Major
> Labels: kip
> Fix For: 2.6.0
>
>
> From
> https://issues.apache.org/jira/browse/KAFKA-4601?focusedCommentId=15881030=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15881030
> ...it would be good to provide users more control over the repartitioning.
> My use case is as follows (unrelated bits omitted for brevity):
> {code:java}
> KTable loggedInCustomers = builder
> .stream("customerLogins")
> .groupBy((key, activity) ->
> activity.getCustomerRef())
> .reduce((first,second) -> second, loginStore());
>
> builder
> .stream("balanceUpdates")
> .map((key, activity) -> new KeyValue<>(
> activity.getCustomerRef(),
> activity))
> .join(loggedInCustomers, (activity, session) -> ...
> .to("sessions");
> {code}
> Both "groupBy" and "map" in the underlying implementation set the
> repartitionRequired flag (since the key changes), and the aggregation/join
> that follows will create the repartitioned topic.
> However, in our case I know that both input streams are already partitioned
> by the customerRef value, which I'm mapping into the key (because it's
> required by the join operation).
> So there are 2 unnecessary intermediate topics created with their associated
> overhead, while the ultimate goal is simply to do a join on a value that we
> already use to partition the original streams anyway.
> (Note, we don't have the option to re-implement the original input streams
> to make customerRef the message key.)
> I think it would be better to allow the user to decide (from their knowledge
> of the incoming streams) whether a repartition is mandatory on aggregation
> and join operations (overloaded version of the methods with the
> repartitionRequired flag exposed maybe?)
> An alternative would be to allow users to perform a join on a value other
> than the key (a keyValueMapper parameter to join, like the one used for joins
> with global tables), but I expect that to be more involved and error-prone to
> use for people who don't understand the partitioning requirements well
> (whereas it's safe for global tables).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (KAFKA-4835) Avoid repartitioning when key change doesn't change partitions
[
https://issues.apache.org/jira/browse/KAFKA-4835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Levani Kokhreidze reassigned KAFKA-4835:
Assignee: (was: Levani Kokhreidze)
> Avoid repartitioning when key change doesn't change partitions
> --
>
> Key: KAFKA-4835
> URL: https://issues.apache.org/jira/browse/KAFKA-4835
> Project: Kafka
> Issue Type: Sub-task
> Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Michal Borowiecki
>Priority: Major
> Labels: kip
> Fix For: 2.6.0
>
>
> From
> https://issues.apache.org/jira/browse/KAFKA-4601?focusedCommentId=15881030=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15881030
> ...it would be good to provide users more control over the repartitioning.
> My use case is as follows (unrelated bits omitted for brevity):
> {code:java}
> KTable loggedInCustomers = builder
> .stream("customerLogins")
> .groupBy((key, activity) ->
> activity.getCustomerRef())
> .reduce((first,second) -> second, loginStore());
>
> builder
> .stream("balanceUpdates")
> .map((key, activity) -> new KeyValue<>(
> activity.getCustomerRef(),
> activity))
> .join(loggedInCustomers, (activity, session) -> ...
> .to("sessions");
> {code}
> Both "groupBy" and "map" in the underlying implementation set the
> repartitionRequired flag (since the key changes), and the aggregation/join
> that follows will create the repartitioned topic.
> However, in our case I know that both input streams are already partitioned
> by the customerRef value, which I'm mapping into the key (because it's
> required by the join operation).
> So there are 2 unnecessary intermediate topics created with their associated
> overhead, while the ultimate goal is simply to do a join on a value that we
> already use to partition the original streams anyway.
> (Note, we don't have the option to re-implement the original input streams
> to make customerRef the message key.)
> I think it would be better to allow the user to decide (from their knowledge
> of the incoming streams) whether a repartition is mandatory on aggregation
> and join operations (overloaded version of the methods with the
> repartitionRequired flag exposed maybe?)
> An alternative would be to allow users to perform a join on a value other
> than the key (a keyValueMapper parameter to join, like the one used for joins
> with global tables), but I expect that to be more involved and error-prone to
> use for people who don't understand the partitioning requirements well
> (whereas it's safe for global tables).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-6037) Make Sub-topology Parallelism Tunable
[ https://issues.apache.org/jira/browse/KAFKA-6037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Levani Kokhreidze updated KAFKA-6037: - Labels: kip (was: needs-kip) > Make Sub-topology Parallelism Tunable > - > > Key: KAFKA-6037 > URL: https://issues.apache.org/jira/browse/KAFKA-6037 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Guozhang Wang >Assignee: Levani Kokhreidze >Priority: Major > Labels: kip > > Today the downstream sub-topology's parallelism (aka the number of tasks) are > purely dependent on the upstream sub-topology's parallelism, which ultimately > depends on the source topic's num. partitions. However this does not work > perfectly with dynamic scaling scenarios. > Imagine if your have a simple aggregation application, it would have two > sub-topologies cut by the repartition topic, the first sub-topology would be > very light as it reads from input topics and write to repartition topic based > on the agg-key; the second sub-topology would do the actual work with the agg > state store, etc, hence is heavy computational. Right now the first and > second topology will always have the same number of tasks as the repartition > topic num.partitions is defined to be the same as the source topic > num.partitions, so to scale up we have to increase the number of input topic > partitions. > One way to improve on that, is to use a default large number for repartition > topics and also allow users to override it (either through DSL code, or > through config). Doing this different sub-topologies would have different > number of tasks, i.e. parallelism units. In addition, users may also use this > config to "hint" the DSL translator to NOT create the repartition topics > (i.e. to not break up the sub-topologies) when she has the knowledge of the > data format. -- This message was sent by Atlassian Jira (v8.3.4#803005)
