[jira] [Comment Edited] (KAFKA-15417) JoinWindow does not seem to work properly with a KStream - KStream - LeftJoin()
[
https://issues.apache.org/jira/browse/KAFKA-15417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17768550#comment-17768550
]
Victor van den Hoven edited comment on KAFKA-15417 at 9/25/23 2:22 PM:
---
What to do with "Flaky tests"?
I do not think my PR has anything to do with it.
Just commit and try again and pray?
!Afbeelding 1.png|width=133,height=156!
!Afbeelding 1-1.png|width=483,height=99!
was (Author: victorvandenhoven):
What to do with "Flaky tests"?
I do not think my PR has anything to do with it.
Just commit and try again and pray?
!Afbeelding 1.png|width=133,height=156!
> JoinWindow does not seem to work properly with a KStream - KStream -
> LeftJoin()
>
>
> Key: KAFKA-15417
> URL: https://issues.apache.org/jira/browse/KAFKA-15417
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 3.4.0
>Reporter: Victor van den Hoven
>Assignee: Victor van den Hoven
>Priority: Major
> Attachments: Afbeelding 1-1.png, Afbeelding 1.png,
> SimpleStreamTopology.java, SimpleStreamTopologyTest.java
>
>
> In Kafka-streams 3.4.0 :
> According to the javadoc of the Joinwindow:
> _There are three different window configuration supported:_
> * _before = after = time-difference_
> * _before = 0 and after = time-difference_
> * _*before = time-difference and after = 0*_
>
> However if I use a joinWindow with *before = time-difference and after = 0*
> on a kstream-kstream-leftjoin the *after=0* part does not seem to work.
> When using _stream1.leftjoin(stream2, joinWindow)_ with
> {_}joinWindow{_}.{_}after=0 and joinWindow.before=30s{_}, any new message on
> stream 1 that can not be joined with any messages on stream2 should be joined
> with a null-record after the _joinWindow.after_ has ended and a new message
> has arrived on stream1.
> It does not.
> Only if the new message arrives after the value of _joinWindow.before_ the
> previous message will be joined with a null-record.
>
> Attached you can find two files with a TopologyTestDriver Unit test to
> reproduce.
> topology: stream1.leftjoin( stream2, joiner, joinwindow)
> joinWindow has before=5000ms and after=0
> message1(key1) -> stream1
> after 4000ms message2(key2) -> stream1 -> NO null-record join was made, but
> the after period was expired.
> after 4900ms message2(key2) -> stream1 -> NO null-record join was made, but
> the after period was expired.
> after 5000ms message2(key2) -> stream1 -> A null-record join was made,
> before period was expired.
> after 6000ms message2(key2) -> stream1 -> A null-record join was made,
> before period was expired.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-15417) JoinWindow does not seem to work properly with a KStream - KStream - LeftJoin()
[
https://issues.apache.org/jira/browse/KAFKA-15417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victor van den Hoven updated KAFKA-15417:
-
Attachment: Afbeelding 1-1.png
> JoinWindow does not seem to work properly with a KStream - KStream -
> LeftJoin()
>
>
> Key: KAFKA-15417
> URL: https://issues.apache.org/jira/browse/KAFKA-15417
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 3.4.0
>Reporter: Victor van den Hoven
>Assignee: Victor van den Hoven
>Priority: Major
> Attachments: Afbeelding 1-1.png, Afbeelding 1.png,
> SimpleStreamTopology.java, SimpleStreamTopologyTest.java
>
>
> In Kafka-streams 3.4.0 :
> According to the javadoc of the Joinwindow:
> _There are three different window configuration supported:_
> * _before = after = time-difference_
> * _before = 0 and after = time-difference_
> * _*before = time-difference and after = 0*_
>
> However if I use a joinWindow with *before = time-difference and after = 0*
> on a kstream-kstream-leftjoin the *after=0* part does not seem to work.
> When using _stream1.leftjoin(stream2, joinWindow)_ with
> {_}joinWindow{_}.{_}after=0 and joinWindow.before=30s{_}, any new message on
> stream 1 that can not be joined with any messages on stream2 should be joined
> with a null-record after the _joinWindow.after_ has ended and a new message
> has arrived on stream1.
> It does not.
> Only if the new message arrives after the value of _joinWindow.before_ the
> previous message will be joined with a null-record.
>
> Attached you can find two files with a TopologyTestDriver Unit test to
> reproduce.
> topology: stream1.leftjoin( stream2, joiner, joinwindow)
> joinWindow has before=5000ms and after=0
> message1(key1) -> stream1
> after 4000ms message2(key2) -> stream1 -> NO null-record join was made, but
> the after period was expired.
> after 4900ms message2(key2) -> stream1 -> NO null-record join was made, but
> the after period was expired.
> after 5000ms message2(key2) -> stream1 -> A null-record join was made,
> before period was expired.
> after 6000ms message2(key2) -> stream1 -> A null-record join was made,
> before period was expired.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (KAFKA-15417) JoinWindow does not seem to work properly with a KStream - KStream - LeftJoin()
[
https://issues.apache.org/jira/browse/KAFKA-15417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17768550#comment-17768550
]
Victor van den Hoven edited comment on KAFKA-15417 at 9/25/23 1:22 PM:
---
What to do with "Flaky tests"?
I do not think my PR has anything to do with it.
Just commit and try again and pray?
!Afbeelding 1.png|width=133,height=156!
was (Author: victorvandenhoven):
What to do with "Flaky tests"?
I do not think my PR has anything to do with it.
Just commit and try again and pray?
> JoinWindow does not seem to work properly with a KStream - KStream -
> LeftJoin()
>
>
> Key: KAFKA-15417
> URL: https://issues.apache.org/jira/browse/KAFKA-15417
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 3.4.0
>Reporter: Victor van den Hoven
>Assignee: Victor van den Hoven
>Priority: Major
> Attachments: Afbeelding 1.png, SimpleStreamTopology.java,
> SimpleStreamTopologyTest.java
>
>
> In Kafka-streams 3.4.0 :
> According to the javadoc of the Joinwindow:
> _There are three different window configuration supported:_
> * _before = after = time-difference_
> * _before = 0 and after = time-difference_
> * _*before = time-difference and after = 0*_
>
> However if I use a joinWindow with *before = time-difference and after = 0*
> on a kstream-kstream-leftjoin the *after=0* part does not seem to work.
> When using _stream1.leftjoin(stream2, joinWindow)_ with
> {_}joinWindow{_}.{_}after=0 and joinWindow.before=30s{_}, any new message on
> stream 1 that can not be joined with any messages on stream2 should be joined
> with a null-record after the _joinWindow.after_ has ended and a new message
> has arrived on stream1.
> It does not.
> Only if the new message arrives after the value of _joinWindow.before_ the
> previous message will be joined with a null-record.
>
> Attached you can find two files with a TopologyTestDriver Unit test to
> reproduce.
> topology: stream1.leftjoin( stream2, joiner, joinwindow)
> joinWindow has before=5000ms and after=0
> message1(key1) -> stream1
> after 4000ms message2(key2) -> stream1 -> NO null-record join was made, but
> the after period was expired.
> after 4900ms message2(key2) -> stream1 -> NO null-record join was made, but
> the after period was expired.
> after 5000ms message2(key2) -> stream1 -> A null-record join was made,
> before period was expired.
> after 6000ms message2(key2) -> stream1 -> A null-record join was made,
> before period was expired.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-15417) JoinWindow does not seem to work properly with a KStream - KStream - LeftJoin()
[
https://issues.apache.org/jira/browse/KAFKA-15417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victor van den Hoven updated KAFKA-15417:
-
Attachment: Afbeelding 1.png
> JoinWindow does not seem to work properly with a KStream - KStream -
> LeftJoin()
>
>
> Key: KAFKA-15417
> URL: https://issues.apache.org/jira/browse/KAFKA-15417
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 3.4.0
>Reporter: Victor van den Hoven
>Assignee: Victor van den Hoven
>Priority: Major
> Attachments: Afbeelding 1.png, SimpleStreamTopology.java,
> SimpleStreamTopologyTest.java
>
>
> In Kafka-streams 3.4.0 :
> According to the javadoc of the Joinwindow:
> _There are three different window configuration supported:_
> * _before = after = time-difference_
> * _before = 0 and after = time-difference_
> * _*before = time-difference and after = 0*_
>
> However if I use a joinWindow with *before = time-difference and after = 0*
> on a kstream-kstream-leftjoin the *after=0* part does not seem to work.
> When using _stream1.leftjoin(stream2, joinWindow)_ with
> {_}joinWindow{_}.{_}after=0 and joinWindow.before=30s{_}, any new message on
> stream 1 that can not be joined with any messages on stream2 should be joined
> with a null-record after the _joinWindow.after_ has ended and a new message
> has arrived on stream1.
> It does not.
> Only if the new message arrives after the value of _joinWindow.before_ the
> previous message will be joined with a null-record.
>
> Attached you can find two files with a TopologyTestDriver Unit test to
> reproduce.
> topology: stream1.leftjoin( stream2, joiner, joinwindow)
> joinWindow has before=5000ms and after=0
> message1(key1) -> stream1
> after 4000ms message2(key2) -> stream1 -> NO null-record join was made, but
> the after period was expired.
> after 4900ms message2(key2) -> stream1 -> NO null-record join was made, but
> the after period was expired.
> after 5000ms message2(key2) -> stream1 -> A null-record join was made,
> before period was expired.
> after 6000ms message2(key2) -> stream1 -> A null-record join was made,
> before period was expired.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (KAFKA-15417) JoinWindow does not seem to work properly with a KStream - KStream - LeftJoin()
[
https://issues.apache.org/jira/browse/KAFKA-15417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17768550#comment-17768550
]
Victor van den Hoven edited comment on KAFKA-15417 at 9/25/23 12:26 PM:
What to do with "Flaky tests"?
I do not think my PR has anything to do with it.
Just commit and try again and pray?
was (Author: victorvandenhoven):
Not sure what I can do about this:
> Task :streams:test
> org.apache.kafka.streams.integration.EOSUncleanShutdownIntegrationTest.shouldWorkWithUncleanShutdownWipeOutStateStore[exactly_once]
> failed, log available in
> /home/jenkins/jenkins-agent/workspace/Kafka_kafka-pr_PR-14426@2/streams/build/reports/testOutput/org.apache.kafka.streams.integration.EOSUncleanShutdownIntegrationTest.shouldWorkWithUncleanShutdownWipeOutStateStore[exactly_once].test.stdout
> Gradle Test Run :streams:test > Gradle Test Executor 85 >
> EOSUncleanShutdownIntegrationTest > [exactly_once] >
> shouldWorkWithUncleanShutdownWipeOutStateStore[exactly_once] FAILED
:(
> JoinWindow does not seem to work properly with a KStream - KStream -
> LeftJoin()
>
>
> Key: KAFKA-15417
> URL: https://issues.apache.org/jira/browse/KAFKA-15417
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 3.4.0
>Reporter: Victor van den Hoven
>Assignee: Victor van den Hoven
>Priority: Major
> Attachments: SimpleStreamTopology.java, SimpleStreamTopologyTest.java
>
>
> In Kafka-streams 3.4.0 :
> According to the javadoc of the Joinwindow:
> _There are three different window configuration supported:_
> * _before = after = time-difference_
> * _before = 0 and after = time-difference_
> * _*before = time-difference and after = 0*_
>
> However if I use a joinWindow with *before = time-difference and after = 0*
> on a kstream-kstream-leftjoin the *after=0* part does not seem to work.
> When using _stream1.leftjoin(stream2, joinWindow)_ with
> {_}joinWindow{_}.{_}after=0 and joinWindow.before=30s{_}, any new message on
> stream 1 that can not be joined with any messages on stream2 should be joined
> with a null-record after the _joinWindow.after_ has ended and a new message
> has arrived on stream1.
> It does not.
> Only if the new message arrives after the value of _joinWindow.before_ the
> previous message will be joined with a null-record.
>
> Attached you can find two files with a TopologyTestDriver Unit test to
> reproduce.
> topology: stream1.leftjoin( stream2, joiner, joinwindow)
> joinWindow has before=5000ms and after=0
> message1(key1) -> stream1
> after 4000ms message2(key2) -> stream1 -> NO null-record join was made, but
> the after period was expired.
> after 4900ms message2(key2) -> stream1 -> NO null-record join was made, but
> the after period was expired.
> after 5000ms message2(key2) -> stream1 -> A null-record join was made,
> before period was expired.
> after 6000ms message2(key2) -> stream1 -> A null-record join was made,
> before period was expired.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-15417) JoinWindow does not seem to work properly with a KStream - KStream - LeftJoin()
[
https://issues.apache.org/jira/browse/KAFKA-15417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17768550#comment-17768550
]
Victor van den Hoven commented on KAFKA-15417:
--
Not sure what I can do about this:
> Task :streams:test
> org.apache.kafka.streams.integration.EOSUncleanShutdownIntegrationTest.shouldWorkWithUncleanShutdownWipeOutStateStore[exactly_once]
> failed, log available in
> /home/jenkins/jenkins-agent/workspace/Kafka_kafka-pr_PR-14426@2/streams/build/reports/testOutput/org.apache.kafka.streams.integration.EOSUncleanShutdownIntegrationTest.shouldWorkWithUncleanShutdownWipeOutStateStore[exactly_once].test.stdout
> Gradle Test Run :streams:test > Gradle Test Executor 85 >
> EOSUncleanShutdownIntegrationTest > [exactly_once] >
> shouldWorkWithUncleanShutdownWipeOutStateStore[exactly_once] FAILED
:(
> JoinWindow does not seem to work properly with a KStream - KStream -
> LeftJoin()
>
>
> Key: KAFKA-15417
> URL: https://issues.apache.org/jira/browse/KAFKA-15417
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 3.4.0
>Reporter: Victor van den Hoven
>Assignee: Victor van den Hoven
>Priority: Major
> Attachments: SimpleStreamTopology.java, SimpleStreamTopologyTest.java
>
>
> In Kafka-streams 3.4.0 :
> According to the javadoc of the Joinwindow:
> _There are three different window configuration supported:_
> * _before = after = time-difference_
> * _before = 0 and after = time-difference_
> * _*before = time-difference and after = 0*_
>
> However if I use a joinWindow with *before = time-difference and after = 0*
> on a kstream-kstream-leftjoin the *after=0* part does not seem to work.
> When using _stream1.leftjoin(stream2, joinWindow)_ with
> {_}joinWindow{_}.{_}after=0 and joinWindow.before=30s{_}, any new message on
> stream 1 that can not be joined with any messages on stream2 should be joined
> with a null-record after the _joinWindow.after_ has ended and a new message
> has arrived on stream1.
> It does not.
> Only if the new message arrives after the value of _joinWindow.before_ the
> previous message will be joined with a null-record.
>
> Attached you can find two files with a TopologyTestDriver Unit test to
> reproduce.
> topology: stream1.leftjoin( stream2, joiner, joinwindow)
> joinWindow has before=5000ms and after=0
> message1(key1) -> stream1
> after 4000ms message2(key2) -> stream1 -> NO null-record join was made, but
> the after period was expired.
> after 4900ms message2(key2) -> stream1 -> NO null-record join was made, but
> the after period was expired.
> after 5000ms message2(key2) -> stream1 -> A null-record join was made,
> before period was expired.
> after 6000ms message2(key2) -> stream1 -> A null-record join was made,
> before period was expired.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (KAFKA-15417) JoinWindow does not seem to work properly with a KStream - KStream - LeftJoin()
[
https://issues.apache.org/jira/browse/KAFKA-15417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17767979#comment-17767979
]
Victor van den Hoven edited comment on KAFKA-15417 at 9/22/23 12:18 PM:
There is a PR submitted for this: [https://github.com/apache/kafka/pull/14426]
Not only the joinSpuriousLookBackTimeMs was flipped.
Also I think that the emit-logic should come after the join-logic.
Because only then you can determine whether you need to emit non-joined items
or not.
Otherwise you will always emit non-joined items, even if there is found a
joined item.
;)
was (Author: victorvandenhoven):
There is a PR submitted for this: [https://github.com/apache/kafka/pull/14426]
;)
> JoinWindow does not seem to work properly with a KStream - KStream -
> LeftJoin()
>
>
> Key: KAFKA-15417
> URL: https://issues.apache.org/jira/browse/KAFKA-15417
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 3.4.0
>Reporter: Victor van den Hoven
>Assignee: Victor van den Hoven
>Priority: Major
> Attachments: SimpleStreamTopology.java, SimpleStreamTopologyTest.java
>
>
> In Kafka-streams 3.4.0 :
> According to the javadoc of the Joinwindow:
> _There are three different window configuration supported:_
> * _before = after = time-difference_
> * _before = 0 and after = time-difference_
> * _*before = time-difference and after = 0*_
>
> However if I use a joinWindow with *before = time-difference and after = 0*
> on a kstream-kstream-leftjoin the *after=0* part does not seem to work.
> When using _stream1.leftjoin(stream2, joinWindow)_ with
> {_}joinWindow{_}.{_}after=0 and joinWindow.before=30s{_}, any new message on
> stream 1 that can not be joined with any messages on stream2 should be joined
> with a null-record after the _joinWindow.after_ has ended and a new message
> has arrived on stream1.
> It does not.
> Only if the new message arrives after the value of _joinWindow.before_ the
> previous message will be joined with a null-record.
>
> Attached you can find two files with a TopologyTestDriver Unit test to
> reproduce.
> topology: stream1.leftjoin( stream2, joiner, joinwindow)
> joinWindow has before=5000ms and after=0
> message1(key1) -> stream1
> after 4000ms message2(key2) -> stream1 -> NO null-record join was made, but
> the after period was expired.
> after 4900ms message2(key2) -> stream1 -> NO null-record join was made, but
> the after period was expired.
> after 5000ms message2(key2) -> stream1 -> A null-record join was made,
> before period was expired.
> after 6000ms message2(key2) -> stream1 -> A null-record join was made,
> before period was expired.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-15417) JoinWindow does not seem to work properly with a KStream - KStream - LeftJoin()
[
https://issues.apache.org/jira/browse/KAFKA-15417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17767979#comment-17767979
]
Victor van den Hoven commented on KAFKA-15417:
--
There is a PR submitted for this: [https://github.com/apache/kafka/pull/14426]
;)
> JoinWindow does not seem to work properly with a KStream - KStream -
> LeftJoin()
>
>
> Key: KAFKA-15417
> URL: https://issues.apache.org/jira/browse/KAFKA-15417
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 3.4.0
>Reporter: Victor van den Hoven
>Assignee: Victor van den Hoven
>Priority: Major
> Attachments: SimpleStreamTopology.java, SimpleStreamTopologyTest.java
>
>
> In Kafka-streams 3.4.0 :
> According to the javadoc of the Joinwindow:
> _There are three different window configuration supported:_
> * _before = after = time-difference_
> * _before = 0 and after = time-difference_
> * _*before = time-difference and after = 0*_
>
> However if I use a joinWindow with *before = time-difference and after = 0*
> on a kstream-kstream-leftjoin the *after=0* part does not seem to work.
> When using _stream1.leftjoin(stream2, joinWindow)_ with
> {_}joinWindow{_}.{_}after=0 and joinWindow.before=30s{_}, any new message on
> stream 1 that can not be joined with any messages on stream2 should be joined
> with a null-record after the _joinWindow.after_ has ended and a new message
> has arrived on stream1.
> It does not.
> Only if the new message arrives after the value of _joinWindow.before_ the
> previous message will be joined with a null-record.
>
> Attached you can find two files with a TopologyTestDriver Unit test to
> reproduce.
> topology: stream1.leftjoin( stream2, joiner, joinwindow)
> joinWindow has before=5000ms and after=0
> message1(key1) -> stream1
> after 4000ms message2(key2) -> stream1 -> NO null-record join was made, but
> the after period was expired.
> after 4900ms message2(key2) -> stream1 -> NO null-record join was made, but
> the after period was expired.
> after 5000ms message2(key2) -> stream1 -> A null-record join was made,
> before period was expired.
> after 6000ms message2(key2) -> stream1 -> A null-record join was made,
> before period was expired.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] (KAFKA-15417) JoinWindow does not seem to work properly with a KStream - KStream - LeftJoin()
[ https://issues.apache.org/jira/browse/KAFKA-15417 ]
Victor van den Hoven deleted comment on KAFKA-15417:
--
was (Author: victorvandenhoven):
The before/after for the joinSpuriousLookBackTimeMs in the left/right join node
were flipped indeed.
After setting them correctly all unit tests still continued to pass, because
there were no unit tests available for leftJoin scenario's.
So I added two leftJoin tests: one for leftJoin with joined items and one for
left-join with null-joined items.
All tests seem to work successfully and also my own application works as
expected.
Shall I create the PR?
> JoinWindow does not seem to work properly with a KStream - KStream -
> LeftJoin()
>
>
> Key: KAFKA-15417
> URL: https://issues.apache.org/jira/browse/KAFKA-15417
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 3.4.0
>Reporter: Victor van den Hoven
>Assignee: Victor van den Hoven
>Priority: Major
> Attachments: SimpleStreamTopology.java, SimpleStreamTopologyTest.java
>
>
> In Kafka-streams 3.4.0 :
> According to the javadoc of the Joinwindow:
> _There are three different window configuration supported:_
> * _before = after = time-difference_
> * _before = 0 and after = time-difference_
> * _*before = time-difference and after = 0*_
>
> However if I use a joinWindow with *before = time-difference and after = 0*
> on a kstream-kstream-leftjoin the *after=0* part does not seem to work.
> When using _stream1.leftjoin(stream2, joinWindow)_ with
> {_}joinWindow{_}.{_}after=0 and joinWindow.before=30s{_}, any new message on
> stream 1 that can not be joined with any messages on stream2 should be joined
> with a null-record after the _joinWindow.after_ has ended and a new message
> has arrived on stream1.
> It does not.
> Only if the new message arrives after the value of _joinWindow.before_ the
> previous message will be joined with a null-record.
>
> Attached you can find two files with a TopologyTestDriver Unit test to
> reproduce.
> topology: stream1.leftjoin( stream2, joiner, joinwindow)
> joinWindow has before=5000ms and after=0
> message1(key1) -> stream1
> after 4000ms message2(key2) -> stream1 -> NO null-record join was made, but
> the after period was expired.
> after 4900ms message2(key2) -> stream1 -> NO null-record join was made, but
> the after period was expired.
> after 5000ms message2(key2) -> stream1 -> A null-record join was made,
> before period was expired.
> after 6000ms message2(key2) -> stream1 -> A null-record join was made,
> before period was expired.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-15417) JoinWindow does not seem to work properly with a KStream - KStream - LeftJoin()
[
https://issues.apache.org/jira/browse/KAFKA-15417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17766843#comment-17766843
]
Victor van den Hoven commented on KAFKA-15417:
--
The before/after for the joinSpuriousLookBackTimeMs in the left/right join node
were flipped indeed.
After setting them correctly all unit tests still continued to pass, because
there were no unit tests available for leftJoin scenario's.
So I added two leftJoin tests: one for leftJoin with joined items and one for
left-join with null-joined items.
All tests seem to work successfully and also my own application works as
expected.
Shall I create the PR?
> JoinWindow does not seem to work properly with a KStream - KStream -
> LeftJoin()
>
>
> Key: KAFKA-15417
> URL: https://issues.apache.org/jira/browse/KAFKA-15417
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 3.4.0
>Reporter: Victor van den Hoven
>Assignee: Victor van den Hoven
>Priority: Major
> Attachments: SimpleStreamTopology.java, SimpleStreamTopologyTest.java
>
>
> In Kafka-streams 3.4.0 :
> According to the javadoc of the Joinwindow:
> _There are three different window configuration supported:_
> * _before = after = time-difference_
> * _before = 0 and after = time-difference_
> * _*before = time-difference and after = 0*_
>
> However if I use a joinWindow with *before = time-difference and after = 0*
> on a kstream-kstream-leftjoin the *after=0* part does not seem to work.
> When using _stream1.leftjoin(stream2, joinWindow)_ with
> {_}joinWindow{_}.{_}after=0 and joinWindow.before=30s{_}, any new message on
> stream 1 that can not be joined with any messages on stream2 should be joined
> with a null-record after the _joinWindow.after_ has ended and a new message
> has arrived on stream1.
> It does not.
> Only if the new message arrives after the value of _joinWindow.before_ the
> previous message will be joined with a null-record.
>
> Attached you can find two files with a TopologyTestDriver Unit test to
> reproduce.
> topology: stream1.leftjoin( stream2, joiner, joinwindow)
> joinWindow has before=5000ms and after=0
> message1(key1) -> stream1
> after 4000ms message2(key2) -> stream1 -> NO null-record join was made, but
> the after period was expired.
> after 4900ms message2(key2) -> stream1 -> NO null-record join was made, but
> the after period was expired.
> after 5000ms message2(key2) -> stream1 -> A null-record join was made,
> before period was expired.
> after 6000ms message2(key2) -> stream1 -> A null-record join was made,
> before period was expired.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (KAFKA-15417) JoinWindow does not seem to work properly with a KStream - KStream - LeftJoin()
[
https://issues.apache.org/jira/browse/KAFKA-15417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17762676#comment-17762676
]
Victor van den Hoven edited comment on KAFKA-15417 at 9/7/23 4:13 PM:
--
Yes, I think it was incorrect and flipped.
Sure, I can give it a try to work on a PR.
was (Author: victorvandenhoven):
Yes, I think it was incorrectly and flipped.
Sure, I can give it a try to work on a PR.
> JoinWindow does not seem to work properly with a KStream - KStream -
> LeftJoin()
>
>
> Key: KAFKA-15417
> URL: https://issues.apache.org/jira/browse/KAFKA-15417
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 3.4.0
>Reporter: Victor van den Hoven
>Priority: Major
> Attachments: SimpleStreamTopology.java, SimpleStreamTopologyTest.java
>
>
> In Kafka-streams 3.4.0 :
> According to the javadoc of the Joinwindow:
> _There are three different window configuration supported:_
> * _before = after = time-difference_
> * _before = 0 and after = time-difference_
> * _*before = time-difference and after = 0*_
>
> However if I use a joinWindow with *before = time-difference and after = 0*
> on a kstream-kstream-leftjoin the *after=0* part does not seem to work.
> When using _stream1.leftjoin(stream2, joinWindow)_ with
> {_}joinWindow{_}.{_}after=0 and joinWindow.before=30s{_}, any new message on
> stream 1 that can not be joined with any messages on stream2 should be joined
> with a null-record after the _joinWindow.after_ has ended and a new message
> has arrived on stream1.
> It does not.
> Only if the new message arrives after the value of _joinWindow.before_ the
> previous message will be joined with a null-record.
>
> Attached you can find two files with a TopologyTestDriver Unit test to
> reproduce.
> topology: stream1.leftjoin( stream2, joiner, joinwindow)
> joinWindow has before=5000ms and after=0
> message1(key1) -> stream1
> after 4000ms message2(key2) -> stream1 -> NO null-record join was made, but
> the after period was expired.
> after 4900ms message2(key2) -> stream1 -> NO null-record join was made, but
> the after period was expired.
> after 5000ms message2(key2) -> stream1 -> A null-record join was made,
> before period was expired.
> after 6000ms message2(key2) -> stream1 -> A null-record join was made,
> before period was expired.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (KAFKA-15417) JoinWindow does not seem to work properly with a KStream - KStream - LeftJoin()
[
https://issues.apache.org/jira/browse/KAFKA-15417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17762676#comment-17762676
]
Victor van den Hoven edited comment on KAFKA-15417 at 9/7/23 9:46 AM:
--
Yes, I think it was incorrectly and flipped.
Sure, I can give it a try to work on a PR.
was (Author: victorvandenhoven):
Yes, I think it was incorrectly and flipped.
Sure, I can try to work on a PR.
> JoinWindow does not seem to work properly with a KStream - KStream -
> LeftJoin()
>
>
> Key: KAFKA-15417
> URL: https://issues.apache.org/jira/browse/KAFKA-15417
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 3.4.0
>Reporter: Victor van den Hoven
>Priority: Major
> Attachments: SimpleStreamTopology.java, SimpleStreamTopologyTest.java
>
>
> In Kafka-streams 3.4.0 :
> According to the javadoc of the Joinwindow:
> _There are three different window configuration supported:_
> * _before = after = time-difference_
> * _before = 0 and after = time-difference_
> * _*before = time-difference and after = 0*_
>
> However if I use a joinWindow with *before = time-difference and after = 0*
> on a kstream-kstream-leftjoin the *after=0* part does not seem to work.
> When using _stream1.leftjoin(stream2, joinWindow)_ with
> {_}joinWindow{_}.{_}after=0 and joinWindow.before=30s{_}, any new message on
> stream 1 that can not be joined with any messages on stream2 should be joined
> with a null-record after the _joinWindow.after_ has ended and a new message
> has arrived on stream1.
> It does not.
> Only if the new message arrives after the value of _joinWindow.before_ the
> previous message will be joined with a null-record.
>
> Attached you can find two files with a TopologyTestDriver Unit test to
> reproduce.
> topology: stream1.leftjoin( stream2, joiner, joinwindow)
> joinWindow has before=5000ms and after=0
> message1(key1) -> stream1
> after 4000ms message2(key2) -> stream1 -> NO null-record join was made, but
> the after period was expired.
> after 4900ms message2(key2) -> stream1 -> NO null-record join was made, but
> the after period was expired.
> after 5000ms message2(key2) -> stream1 -> A null-record join was made,
> before period was expired.
> after 6000ms message2(key2) -> stream1 -> A null-record join was made,
> before period was expired.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-15417) JoinWindow does not seem to work properly with a KStream - KStream - LeftJoin()
[
https://issues.apache.org/jira/browse/KAFKA-15417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17762676#comment-17762676
]
Victor van den Hoven commented on KAFKA-15417:
--
Yes, I think it was incorrectly and flipped.
Sure, I can try to work on a PR.
> JoinWindow does not seem to work properly with a KStream - KStream -
> LeftJoin()
>
>
> Key: KAFKA-15417
> URL: https://issues.apache.org/jira/browse/KAFKA-15417
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 3.4.0
>Reporter: Victor van den Hoven
>Priority: Major
> Attachments: SimpleStreamTopology.java, SimpleStreamTopologyTest.java
>
>
> In Kafka-streams 3.4.0 :
> According to the javadoc of the Joinwindow:
> _There are three different window configuration supported:_
> * _before = after = time-difference_
> * _before = 0 and after = time-difference_
> * _*before = time-difference and after = 0*_
>
> However if I use a joinWindow with *before = time-difference and after = 0*
> on a kstream-kstream-leftjoin the *after=0* part does not seem to work.
> When using _stream1.leftjoin(stream2, joinWindow)_ with
> {_}joinWindow{_}.{_}after=0 and joinWindow.before=30s{_}, any new message on
> stream 1 that can not be joined with any messages on stream2 should be joined
> with a null-record after the _joinWindow.after_ has ended and a new message
> has arrived on stream1.
> It does not.
> Only if the new message arrives after the value of _joinWindow.before_ the
> previous message will be joined with a null-record.
>
> Attached you can find two files with a TopologyTestDriver Unit test to
> reproduce.
> topology: stream1.leftjoin( stream2, joiner, joinwindow)
> joinWindow has before=5000ms and after=0
> message1(key1) -> stream1
> after 4000ms message2(key2) -> stream1 -> NO null-record join was made, but
> the after period was expired.
> after 4900ms message2(key2) -> stream1 -> NO null-record join was made, but
> the after period was expired.
> after 5000ms message2(key2) -> stream1 -> A null-record join was made,
> before period was expired.
> after 6000ms message2(key2) -> stream1 -> A null-record join was made,
> before period was expired.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-15417) JoinWindow does not seem to work properly with a KStream - KStream - LeftJoin()
[
https://issues.apache.org/jira/browse/KAFKA-15417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17761996#comment-17761996
]
Victor van den Hoven commented on KAFKA-15417:
--
Thank you for your comment.
There is unfortunately no improvement.
I tested it with the following:
_{color:#6a3e3e}props{color}{color:#00}.put(StreamsConfig.InternalConfig.{color}{color:#c0}EMIT_INTERVAL_MS_KSTREAMS_OUTER_JOIN_SPURIOUS_RESULTS_FIX{color}{color:#00},
0);{color}_
At startup I get the following warning:
{quote}_WARN 25120 --- [ main] o.a.k.clients.consumer.ConsumerConfig
: These configurations '[specific.avro.reader, schema.registry.url,
__emit.interval.ms.kstreams.outer.join.spurious.results.fix__]' were supplied
but are not used yet._
{quote}
I use the following joinwindow:
### JoinWindow size: 2ms
### JoinWindow before: 2ms
### JoinWindow after: 0ms
Started with empty topics:
Test1:
streamA.leftjoin( streamB);
put message on streamA (key1, a1)
wait 5000ms
put message on streamA (key1, a2)
expected join-record (key1, null) but there is not.
Started with empty topics:
Test2:
streamA.leftjoin( streamB);
put message on streamA (key1, a1)
wait 25000ms
put message on streamA (key1, a2)
there is a join-record (key1, null) .
> JoinWindow does not seem to work properly with a KStream - KStream -
> LeftJoin()
>
>
> Key: KAFKA-15417
> URL: https://issues.apache.org/jira/browse/KAFKA-15417
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 3.4.0
>Reporter: Victor van den Hoven
>Priority: Major
> Attachments: SimpleStreamTopology.java, SimpleStreamTopologyTest.java
>
>
> In Kafka-streams 3.4.0 :
> According to the javadoc of the Joinwindow:
> _There are three different window configuration supported:_
> * _before = after = time-difference_
> * _before = 0 and after = time-difference_
> * _*before = time-difference and after = 0*_
>
> However if I use a joinWindow with *before = time-difference and after = 0*
> on a kstream-kstream-leftjoin the *after=0* part does not seem to work.
> When using _stream1.leftjoin(stream2, joinWindow)_ with
> {_}joinWindow{_}.{_}after=0 and joinWindow.before=30s{_}, any new message on
> stream 1 that can not be joined with any messages on stream2 should be joined
> with a null-record after the _joinWindow.after_ has ended and a new message
> has arrived on stream1.
> It does not.
> Only if the new message arrives after the value of _joinWindow.before_ the
> previous message will be joined with a null-record.
>
> Attached you can find two files with a TopologyTestDriver Unit test to
> reproduce.
> topology: stream1.leftjoin( stream2, joiner, joinwindow)
> joinWindow has before=5000ms and after=0
> message1(key1) -> stream1
> after 4000ms message2(key2) -> stream1 -> NO null-record join was made, but
> the after period was expired.
> after 4900ms message2(key2) -> stream1 -> NO null-record join was made, but
> the after period was expired.
> after 5000ms message2(key2) -> stream1 -> A null-record join was made,
> before period was expired.
> after 6000ms message2(key2) -> stream1 -> A null-record join was made,
> before period was expired.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-15417) JoinWindow does not seem to work properly with a KStream - KStream - LeftJoin()
Victor van den Hoven created KAFKA-15417:
Summary: JoinWindow does not seem to work properly with a KStream
- KStream - LeftJoin()
Key: KAFKA-15417
URL: https://issues.apache.org/jira/browse/KAFKA-15417
Project: Kafka
Issue Type: Bug
Components: streams
Affects Versions: 3.4.0
Reporter: Victor van den Hoven
Attachments: SimpleStreamTopology.java, SimpleStreamTopologyTest.java
In Kafka-streams 3.4.0 :
According to the javadoc of the Joinwindow:
_There are three different window configuration supported:_
* _before = after = time-difference_
* _before = 0 and after = time-difference_
* _*before = time-difference and after = 0*_
However if I use a joinWindow with *before = time-difference and after = 0*
on a kstream-kstream-leftjoin the *after=0* part does not seem to work.
When using _stream1.leftjoin(stream2, joinWindow)_ with
{_}joinWindow{_}.{_}after=0 and joinWindow.before=30s{_}, any new message on
stream 1 that can not be joined with any messages on stream2 should be joined
with a null-record after the _joinWindow.after_ has ended and a new message has
arrived on stream1.
It does not.
Only if the new message arrives after the value of _joinWindow.before_ the
previous message will be joined with a null-record.
Attached you can find two files with a TopologyTestDriver Unit test to
reproduce.
topology: stream1.leftjoin( stream2, joiner, joinwindow)
joinWindow has before=5000ms and after=0
message1(key1) -> stream1
after 4000ms message2(key2) -> stream1 -> NO null-record join was made, but
the after period was expired.
after 4900ms message2(key2) -> stream1 -> NO null-record join was made, but
the after period was expired.
after 5000ms message2(key2) -> stream1 -> A null-record join was made,
before period was expired.
after 6000ms message2(key2) -> stream1 -> A null-record join was made,
before period was expired.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KAFKA-14400) KStream - KStream - LeftJoin() does not call ValueJoiner with null value
[
https://issues.apache.org/jira/browse/KAFKA-14400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victor van den Hoven resolved KAFKA-14400.
--
Resolution: Not A Bug
It behaves differently because stream-stram-leftjoin semantics have changed.
> KStream - KStream - LeftJoin() does not call ValueJoiner with null value
> -
>
> Key: KAFKA-14400
> URL: https://issues.apache.org/jira/browse/KAFKA-14400
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 3.1.1, 3.3.1
> Environment: Windows PC
>Reporter: Victor van den Hoven
>Priority: Major
> Attachments: Afbeelding 2.png, SimpleStreamTopology.java,
> SimpleStreamTopologyTest.java
>
>
> In Kafka-streams 3.1.1 :
> When using +JoinWindows.ofTimeDifferenceWithNoGrace(Duration.ofMillis(1))+
> the KStream {*}leftJoin{*}(KStream otherStream, ValueJoiner {_}joiner{_},
> JoinWindows windows) does not seem to call the _joiner_ with null value when
> join predicate is not satisfied (not expected).
>
> When using deprecated +JoinWindows.of(Duration.ofMillis(1));+
> the KStream {*}leftJoin{*}(KStream otherStream, ValueJoiner {_}joiner{_},
> JoinWindows windows) does
> call the _joiner_ with null value when join predicate is not satisfied (as
> expected and documented).
>
> Attached you can find two files with TopologyTestDriver Unit test to
> reproduce.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-14400) KStream - KStream - LeftJoin() does not call ValueJoiner with null value
[
https://issues.apache.org/jira/browse/KAFKA-14400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17642366#comment-17642366
]
Victor van den Hoven commented on KAFKA-14400:
--
Thank you for answering my questions.
:)
> KStream - KStream - LeftJoin() does not call ValueJoiner with null value
> -
>
> Key: KAFKA-14400
> URL: https://issues.apache.org/jira/browse/KAFKA-14400
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 3.1.1, 3.3.1
> Environment: Windows PC
>Reporter: Victor van den Hoven
>Priority: Major
> Attachments: Afbeelding 2.png, SimpleStreamTopology.java,
> SimpleStreamTopologyTest.java
>
>
> In Kafka-streams 3.1.1 :
> When using +JoinWindows.ofTimeDifferenceWithNoGrace(Duration.ofMillis(1))+
> the KStream {*}leftJoin{*}(KStream otherStream, ValueJoiner {_}joiner{_},
> JoinWindows windows) does not seem to call the _joiner_ with null value when
> join predicate is not satisfied (not expected).
>
> When using deprecated +JoinWindows.of(Duration.ofMillis(1));+
> the KStream {*}leftJoin{*}(KStream otherStream, ValueJoiner {_}joiner{_},
> JoinWindows windows) does
> call the _joiner_ with null value when join predicate is not satisfied (as
> expected and documented).
>
> Attached you can find two files with TopologyTestDriver Unit test to
> reproduce.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (KAFKA-14400) KStream - KStream - LeftJoin() does not call ValueJoiner with null value
[
https://issues.apache.org/jira/browse/KAFKA-14400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17637240#comment-17637240
]
Victor van den Hoven edited comment on KAFKA-14400 at 11/23/22 4:09 PM:
Am I correct that when one +LAST+ input record is written to the left-input
stream not within any window and when no more input records are written to any
of the input streams, then this LAST input record will never be emitted?
The question remains though
In the kstream-kstream-join table of the "Improved Left/Outer Stream-Stream
Join" on the wiki page that was referred to: Why is "d,ts=14" not also joined
with "A,ts=3" as the window size is 15? And why is "D,ts=15" not also joined
with "A,ts=3"?
Thank you for taking the time to explain the new API-semantics to me.
It is all clear to me now. I wasn't aware of the new semantics.
Apologies for the inconvenience
So it is not a bug indeed and the issue can be closed.
was (Author: victorvandenhoven):
Thank you for taking the time to explain the new API-semantics to me.
It is all clear to me now. I wasn't aware of the new semantics.
Apologies for the inconvenience
So it is not a bug indeed and the issue can be closed.
Am I correct that when one +LAST+ input record is written to the left-input
stream not within any window and when no more input records are written to any
of the input streams, then this LAST input record will never be emitted?
The question remains though
In the kstream-kstream-join table of the "Improved Left/Outer Stream-Stream
Join" on the wiki page that was referred to: Why is "d,ts=14" not also joined
with "A,ts=3" as the window size is 15? And why is "D,ts=15" not also joined
with "A,ts=3"?
> KStream - KStream - LeftJoin() does not call ValueJoiner with null value
> -
>
> Key: KAFKA-14400
> URL: https://issues.apache.org/jira/browse/KAFKA-14400
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 3.1.1, 3.3.1
> Environment: Windows PC
>Reporter: Victor van den Hoven
>Priority: Major
> Attachments: Afbeelding 2.png, SimpleStreamTopology.java,
> SimpleStreamTopologyTest.java
>
>
> In Kafka-streams 3.1.1 :
> When using +JoinWindows.ofTimeDifferenceWithNoGrace(Duration.ofMillis(1))+
> the KStream {*}leftJoin{*}(KStream otherStream, ValueJoiner {_}joiner{_},
> JoinWindows windows) does not seem to call the _joiner_ with null value when
> join predicate is not satisfied (not expected).
>
> When using deprecated +JoinWindows.of(Duration.ofMillis(1));+
> the KStream {*}leftJoin{*}(KStream otherStream, ValueJoiner {_}joiner{_},
> JoinWindows windows) does
> call the _joiner_ with null value when join predicate is not satisfied (as
> expected and documented).
>
> Attached you can find two files with TopologyTestDriver Unit test to
> reproduce.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-14400) KStream - KStream - LeftJoin() does not call ValueJoiner with null value
[
https://issues.apache.org/jira/browse/KAFKA-14400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17637240#comment-17637240
]
Victor van den Hoven commented on KAFKA-14400:
--
Thank you for taking the time to explain the new API-semantics to me.
It is all clear to me now. I wasn't aware of the new semantics.
Apologies for the inconvenience
So it is not a bug indeed and the issue can be closed.
Am I correct that when one +LAST+ input record is written to the left-input
stream not within any window and when no more input records are written to any
of the input streams, then this LAST input record will never be emitted?
The question remains though
In the kstream-kstream-join table of the "Improved Left/Outer Stream-Stream
Join" on the wiki page that was referred to: Why is "d,ts=14" not also joined
with "A,ts=3" as the window size is 15? And why is "D,ts=15" not also joined
with "A,ts=3"?
> KStream - KStream - LeftJoin() does not call ValueJoiner with null value
> -
>
> Key: KAFKA-14400
> URL: https://issues.apache.org/jira/browse/KAFKA-14400
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 3.1.1, 3.3.1
> Environment: Windows PC
>Reporter: Victor van den Hoven
>Priority: Major
> Attachments: Afbeelding 2.png, SimpleStreamTopology.java,
> SimpleStreamTopologyTest.java
>
>
> In Kafka-streams 3.1.1 :
> When using +JoinWindows.ofTimeDifferenceWithNoGrace(Duration.ofMillis(1))+
> the KStream {*}leftJoin{*}(KStream otherStream, ValueJoiner {_}joiner{_},
> JoinWindows windows) does not seem to call the _joiner_ with null value when
> join predicate is not satisfied (not expected).
>
> When using deprecated +JoinWindows.of(Duration.ofMillis(1));+
> the KStream {*}leftJoin{*}(KStream otherStream, ValueJoiner {_}joiner{_},
> JoinWindows windows) does
> call the _joiner_ with null value when join predicate is not satisfied (as
> expected and documented).
>
> Attached you can find two files with TopologyTestDriver Unit test to
> reproduce.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (KAFKA-14400) KStream - KStream - LeftJoin() does not call ValueJoiner with null value
[
https://issues.apache.org/jira/browse/KAFKA-14400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17636509#comment-17636509
]
Victor van den Hoven edited comment on KAFKA-14400 at 11/21/22 8:07 AM:
I agree, but actually it is also happening with the ofTimeDifferenceAndGrace()
method.
As far as I can tell it has nothing to do with the grace period.
It just is not working correctly. Also Join() does seem to work, leftJoin()
isn't.
Maybe you can reproduce with the unit-test that I provided?.
was (Author: victorvandenhoven):
I agree, but actually it is also happening with the ofTimeDifferenceAndGrace()
method.
As far as I can tell it has nothing to do with the grace period.
It just is not working correctly. Also Join() does seem to work, leftJoin()
isn't.
Maybe you can reproduce with.
> KStream - KStream - LeftJoin() does not call ValueJoiner with null value
> -
>
> Key: KAFKA-14400
> URL: https://issues.apache.org/jira/browse/KAFKA-14400
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 3.1.1, 3.3.1
> Environment: Windows PC
>Reporter: Victor van den Hoven
>Priority: Major
> Attachments: Afbeelding 2.png, SimpleStreamTopology.java,
> SimpleStreamTopologyTest.java
>
>
> In Kafka-streams 3.1.1 :
> When using +JoinWindows.ofTimeDifferenceWithNoGrace(Duration.ofMillis(1))+
> the KStream {*}leftJoin{*}(KStream otherStream, ValueJoiner {_}joiner{_},
> JoinWindows windows) does not seem to call the _joiner_ with null value when
> join predicate is not satisfied (not expected).
>
> When using deprecated +JoinWindows.of(Duration.ofMillis(1));+
> the KStream {*}leftJoin{*}(KStream otherStream, ValueJoiner {_}joiner{_},
> JoinWindows windows) does
> call the _joiner_ with null value when join predicate is not satisfied (as
> expected and documented).
>
> Attached you can find two files with TopologyTestDriver Unit test to
> reproduce.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (KAFKA-14400) KStream - KStream - LeftJoin() does not call ValueJoiner with null value
[
https://issues.apache.org/jira/browse/KAFKA-14400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17636512#comment-17636512
]
Victor van den Hoven edited comment on KAFKA-14400 at 11/21/22 8:06 AM:
Also with the deprecated joinWindows (using ofTimeDifferenceAndGrace()) a
different topology is being produced:
In line 13 and line 16 you will find a new
{*}KSTREAM-OUTERSHARED-04-store{*}.
Where is that coming from?
!Afbeelding 2.png!
was (Author: victorvandenhoven):
Also with the deprecated joinWindows a diferent topology is being produced:
In line 13 and line 16 you will find a new KSTREAM-OUTERSHARED-04-store.
Where is that coming from?
!Afbeelding 2.png!
> KStream - KStream - LeftJoin() does not call ValueJoiner with null value
> -
>
> Key: KAFKA-14400
> URL: https://issues.apache.org/jira/browse/KAFKA-14400
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 3.1.1, 3.3.1
> Environment: Windows PC
>Reporter: Victor van den Hoven
>Priority: Major
> Attachments: Afbeelding 2.png, SimpleStreamTopology.java,
> SimpleStreamTopologyTest.java
>
>
> In Kafka-streams 3.1.1 :
> When using +JoinWindows.ofTimeDifferenceWithNoGrace(Duration.ofMillis(1))+
> the KStream {*}leftJoin{*}(KStream otherStream, ValueJoiner {_}joiner{_},
> JoinWindows windows) does not seem to call the _joiner_ with null value when
> join predicate is not satisfied (not expected).
>
> When using deprecated +JoinWindows.of(Duration.ofMillis(1));+
> the KStream {*}leftJoin{*}(KStream otherStream, ValueJoiner {_}joiner{_},
> JoinWindows windows) does
> call the _joiner_ with null value when join predicate is not satisfied (as
> expected and documented).
>
> Attached you can find two files with TopologyTestDriver Unit test to
> reproduce.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (KAFKA-14400) KStream - KStream - LeftJoin() does not call ValueJoiner with null value
[
https://issues.apache.org/jira/browse/KAFKA-14400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17636512#comment-17636512
]
Victor van den Hoven edited comment on KAFKA-14400 at 11/21/22 8:05 AM:
Also with the deprecated joinWindows a diferent topology is being produced:
In line 13 and line 16 you will find a new KSTREAM-OUTERSHARED-04-store.
Where is that coming from?
!Afbeelding 2.png!
was (Author: victorvandenhoven):
!Afbeelding 2.png!
> KStream - KStream - LeftJoin() does not call ValueJoiner with null value
> -
>
> Key: KAFKA-14400
> URL: https://issues.apache.org/jira/browse/KAFKA-14400
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 3.1.1, 3.3.1
> Environment: Windows PC
>Reporter: Victor van den Hoven
>Priority: Major
> Attachments: Afbeelding 2.png, SimpleStreamTopology.java,
> SimpleStreamTopologyTest.java
>
>
> In Kafka-streams 3.1.1 :
> When using +JoinWindows.ofTimeDifferenceWithNoGrace(Duration.ofMillis(1))+
> the KStream {*}leftJoin{*}(KStream otherStream, ValueJoiner {_}joiner{_},
> JoinWindows windows) does not seem to call the _joiner_ with null value when
> join predicate is not satisfied (not expected).
>
> When using deprecated +JoinWindows.of(Duration.ofMillis(1));+
> the KStream {*}leftJoin{*}(KStream otherStream, ValueJoiner {_}joiner{_},
> JoinWindows windows) does
> call the _joiner_ with null value when join predicate is not satisfied (as
> expected and documented).
>
> Attached you can find two files with TopologyTestDriver Unit test to
> reproduce.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-14400) KStream - KStream - LeftJoin() does not call ValueJoiner with null value
[
https://issues.apache.org/jira/browse/KAFKA-14400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victor van den Hoven updated KAFKA-14400:
-
Attachment: Afbeelding 2.png
> KStream - KStream - LeftJoin() does not call ValueJoiner with null value
> -
>
> Key: KAFKA-14400
> URL: https://issues.apache.org/jira/browse/KAFKA-14400
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 3.1.1, 3.3.1
> Environment: Windows PC
>Reporter: Victor van den Hoven
>Priority: Major
> Attachments: Afbeelding 2.png, SimpleStreamTopology.java,
> SimpleStreamTopologyTest.java
>
>
> In Kafka-streams 3.1.1 :
> When using +JoinWindows.ofTimeDifferenceWithNoGrace(Duration.ofMillis(1))+
> the KStream {*}leftJoin{*}(KStream otherStream, ValueJoiner {_}joiner{_},
> JoinWindows windows) does not seem to call the _joiner_ with null value when
> join predicate is not satisfied (not expected).
>
> When using deprecated +JoinWindows.of(Duration.ofMillis(1));+
> the KStream {*}leftJoin{*}(KStream otherStream, ValueJoiner {_}joiner{_},
> JoinWindows windows) does
> call the _joiner_ with null value when join predicate is not satisfied (as
> expected and documented).
>
> Attached you can find two files with TopologyTestDriver Unit test to
> reproduce.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-14400) KStream - KStream - LeftJoin() does not call ValueJoiner with null value
[
https://issues.apache.org/jira/browse/KAFKA-14400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17636512#comment-17636512
]
Victor van den Hoven commented on KAFKA-14400:
--
!Afbeelding 2.png!
> KStream - KStream - LeftJoin() does not call ValueJoiner with null value
> -
>
> Key: KAFKA-14400
> URL: https://issues.apache.org/jira/browse/KAFKA-14400
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 3.1.1, 3.3.1
> Environment: Windows PC
>Reporter: Victor van den Hoven
>Priority: Major
> Attachments: Afbeelding 2.png, SimpleStreamTopology.java,
> SimpleStreamTopologyTest.java
>
>
> In Kafka-streams 3.1.1 :
> When using +JoinWindows.ofTimeDifferenceWithNoGrace(Duration.ofMillis(1))+
> the KStream {*}leftJoin{*}(KStream otherStream, ValueJoiner {_}joiner{_},
> JoinWindows windows) does not seem to call the _joiner_ with null value when
> join predicate is not satisfied (not expected).
>
> When using deprecated +JoinWindows.of(Duration.ofMillis(1));+
> the KStream {*}leftJoin{*}(KStream otherStream, ValueJoiner {_}joiner{_},
> JoinWindows windows) does
> call the _joiner_ with null value when join predicate is not satisfied (as
> expected and documented).
>
> Attached you can find two files with TopologyTestDriver Unit test to
> reproduce.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-14400) KStream - KStream - LeftJoin() does not call ValueJoiner with null value
[
https://issues.apache.org/jira/browse/KAFKA-14400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17636509#comment-17636509
]
Victor van den Hoven commented on KAFKA-14400:
--
I agree, but actually it is also happening with the ofTimeDifferenceAndGrace()
method.
As far as I can tell it has nothing to do with the grace period.
It just is not working correctly. Also Join() does seem to work, leftJoin()
isn't.
Maybe you can reproduce with.
> KStream - KStream - LeftJoin() does not call ValueJoiner with null value
> -
>
> Key: KAFKA-14400
> URL: https://issues.apache.org/jira/browse/KAFKA-14400
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 3.1.1, 3.3.1
> Environment: Windows PC
>Reporter: Victor van den Hoven
>Priority: Major
> Attachments: SimpleStreamTopology.java, SimpleStreamTopologyTest.java
>
>
> In Kafka-streams 3.1.1 :
> When using +JoinWindows.ofTimeDifferenceWithNoGrace(Duration.ofMillis(1))+
> the KStream {*}leftJoin{*}(KStream otherStream, ValueJoiner {_}joiner{_},
> JoinWindows windows) does not seem to call the _joiner_ with null value when
> join predicate is not satisfied (not expected).
>
> When using deprecated +JoinWindows.of(Duration.ofMillis(1));+
> the KStream {*}leftJoin{*}(KStream otherStream, ValueJoiner {_}joiner{_},
> JoinWindows windows) does
> call the _joiner_ with null value when join predicate is not satisfied (as
> expected and documented).
>
> Attached you can find two files with TopologyTestDriver Unit test to
> reproduce.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-14400) KStream - KStream - LeftJoin() does not call ValueJoiner with null value
[
https://issues.apache.org/jira/browse/KAFKA-14400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victor van den Hoven updated KAFKA-14400:
-
Description:
In Kafka-streams 3.1.1 :
When using +JoinWindows.ofTimeDifferenceWithNoGrace(Duration.ofMillis(1))+
the KStream {*}leftJoin{*}(KStream otherStream, ValueJoiner {_}joiner{_},
JoinWindows windows) does not seem to call the _joiner_ with null value when
join predicate is not satisfied (not expected).
When using deprecated +JoinWindows.of(Duration.ofMillis(1));+
the KStream {*}leftJoin{*}(KStream otherStream, ValueJoiner {_}joiner{_},
JoinWindows windows) does
call the _joiner_ with null value when join predicate is not satisfied (as
expected and documented).
Attached you can find two files with TopologyTestDriver Unit test to reproduce.
was:
In Kafka-streams 3.1.1 :
When using +JoinWindows.ofTimeDifferenceWithNoGrace(Duration.ofMillis(1))+
the KStream {*}leftJoin{*}(KStream otherStream, ValueJoiner {_}joiner{_},
JoinWindows windows) does not seem to call the _joiner_ with null value when
join predicate is not satisfied.
When using deprecated +JoinWindows.of(Duration.ofMillis(1));+
the KStream {*}leftJoin{*}(KStream otherStream, ValueJoiner {_}joiner{_},
JoinWindows windows) does
all the _joiner_ with null value when join predicate is not satisfied.
Attached you can find two files with TopologyTestDriver Unit test to reproduce.
> KStream - KStream - LeftJoin() does not call ValueJoiner with null value
> -
>
> Key: KAFKA-14400
> URL: https://issues.apache.org/jira/browse/KAFKA-14400
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 3.1.1, 3.3.1
> Environment: Windows PC
>Reporter: Victor van den Hoven
>Priority: Major
> Attachments: SimpleStreamTopology.java, SimpleStreamTopologyTest.java
>
>
> In Kafka-streams 3.1.1 :
> When using +JoinWindows.ofTimeDifferenceWithNoGrace(Duration.ofMillis(1))+
> the KStream {*}leftJoin{*}(KStream otherStream, ValueJoiner {_}joiner{_},
> JoinWindows windows) does not seem to call the _joiner_ with null value when
> join predicate is not satisfied (not expected).
>
> When using deprecated +JoinWindows.of(Duration.ofMillis(1));+
> the KStream {*}leftJoin{*}(KStream otherStream, ValueJoiner {_}joiner{_},
> JoinWindows windows) does
> call the _joiner_ with null value when join predicate is not satisfied (as
> expected and documented).
>
> Attached you can find two files with TopologyTestDriver Unit test to
> reproduce.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-14400) KStream - KStream - LeftJoin() does not call ValueJoiner with null value
Victor van den Hoven created KAFKA-14400:
Summary: KStream - KStream - LeftJoin() does not call ValueJoiner
with null value
Key: KAFKA-14400
URL: https://issues.apache.org/jira/browse/KAFKA-14400
Project: Kafka
Issue Type: Bug
Components: streams
Affects Versions: 3.3.1, 3.1.1
Environment: Windows PC
Reporter: Victor van den Hoven
Attachments: SimpleStreamTopology.java, SimpleStreamTopologyTest.java
In Kafka-streams 3.1.1 :
When using +JoinWindows.ofTimeDifferenceWithNoGrace(Duration.ofMillis(1))+
the KStream {*}leftJoin{*}(KStream otherStream, ValueJoiner {_}joiner{_},
JoinWindows windows) does not seem to call the _joiner_ with null value when
join predicate is not satisfied.
When using deprecated +JoinWindows.of(Duration.ofMillis(1));+
the KStream {*}leftJoin{*}(KStream otherStream, ValueJoiner {_}joiner{_},
JoinWindows windows) does
all the _joiner_ with null value when join predicate is not satisfied.
Attached you can find two files with TopologyTestDriver Unit test to reproduce.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-13009) Metrics recorder is re-initialised with different task
[
https://issues.apache.org/jira/browse/KAFKA-13009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victor van den Hoven updated KAFKA-13009:
-
Description:
When starting my Kafka Stream application, I get in the Logs:
[SmartMeterActionService-e0d0f403-87c7-4502-b1be-875d544899e2-StreamThread-1]
State transition from STARTING to
PARTITIONS_ASSIGNED[SmartMeterActionService-e0d0f403-87c7-4502-b1be-875d544899e2-StreamThread-1]
State transition from STARTING to PARTITIONS_ASSIGNED2021-06-29 07:35:58.258
ERROR 1 — [-StreamThread-1] o.a.k.s.p.internals.StreamThread :
stream-thread
[SmartMeterActionService-e0d0f403-87c7-4502-b1be-875d544899e2-StreamThread-1]
Encountered the following exception during processing and the thread is going
to shut down:
java.lang.IllegalStateException: Metrics recorder is re-initialised with
different task: previous task is -1_-1 whereas current task is 0_1.
*{color:#ff}This is a bug in Kafka Streams. Please open a bug report under
[https://issues.apache.org/jira/projects/KAFKA/issues] at{color}*
org.apache.kafka.streams.state.internals.metrics.RocksDBMetricsRecorder.init(RocksDBMetricsRecorder.java:137)
~[kafka-streams-2.7.0.jar!/:na] at
org.apache.kafka.streams.state.internals.RocksDBStore.init(RocksDBStore.java:252)
~[kafka-streams-2.7.0.jar!/:na] at
org.apache.kafka.streams.state.internals.WrappedStateStore.init(WrappedStateStore.java:55)
~[kafka-streams-2.7.0.jar!/:na] at
org.apache.kafka.streams.state.internals.CachingKeyValueStore.init(CachingKeyValueStore.java:74)
~[kafka-streams-2.7.0.jar!/:na] at
org.apache.kafka.streams.state.internals.WrappedStateStore.init(WrappedStateStore.java:55)
~[kafka-streams-2.7.0.jar!/:na] at
org.apache.kafka.streams.state.internals.MeteredKeyValueStore.lambda$init$1(MeteredKeyValueStore.java:120)
~[kafka-streams-2.7.0.jar!/:na] at
org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl.maybeMeasureLatency(StreamsMetricsImpl.java:883)
~[kafka-streams-2.7.0.jar!/:na] at
org.apache.kafka.streams.state.internals.MeteredKeyValueStore.init(MeteredKeyValueStore.java:120)
~[kafka-streams-2.7.0.jar!/:na] at
org.apache.kafka.streams.processor.internals.ProcessorStateManager.registerStateStores(ProcessorStateManager.java:201)
~[kafka-streams-2.7.0.jar!/:na] at
org.apache.kafka.streams.processor.internals.StateManagerUtil.registerStateStores(StateManagerUtil.java:103)
~[kafka-streams-2.7.0.jar!/:na] at
org.apache.kafka.streams.processor.internals.StandbyTask.initializeIfNeeded(StandbyTask.java:93)
~[kafka-streams-2.7.0.jar!/:na] at
org.apache.kafka.streams.processor.internals.TaskManager.tryToCompleteRestoration(TaskManager.java:473)
~[kafka-streams-2.7.0.jar!/:na] at
org.apache.kafka.streams.processor.internals.StreamThread.initializeAndRestorePhase(StreamThread.java:728)
~[kafka-streams-2.7.0.jar!/:na] at
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:625)
~[kafka-streams-2.7.0.jar!/:na] at
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:553)
~[kafka-streams-2.7.0.jar!/:na] at
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:512)
~[kafka-streams-2.7.0.jar!/:na]
2021-06-29 07:35:58.259 INFO 1 — [-StreamThread-1]
o.a.k.s.p.internals.StreamThread : stream-thread
[SmartMeterActionService-e0d0f403-87c7-4502-b1be-875d544899e2-StreamThread-1]
State transition from PARTITIONS_ASSIGNED to PENDING_SHUTDOWN2021-06-29
07:35:58.259 INFO 1 — [-StreamThread-1] o.a.k.s.p.internals.StreamThread
: stream-thread
[SmartMeterActionService-e0d0f403-87c7-4502-b1be-875d544899e2-StreamThread-1]
Shutting down
After this the application shuts down!
After removing the internal change-log-topic the application could start again
without the issue.
was:
When starting my Kafka Stream application, I get in the Logs:
[SmartMeterActionService-e0d0f403-87c7-4502-b1be-875d544899e2-StreamThread-1]
State transition from STARTING to
PARTITIONS_ASSIGNED[SmartMeterActionService-e0d0f403-87c7-4502-b1be-875d544899e2-StreamThread-1]
State transition from STARTING to PARTITIONS_ASSIGNED2021-06-29 07:35:58.258
ERROR 1 — [-StreamThread-1] o.a.k.s.p.internals.StreamThread :
stream-thread
[SmartMeterActionService-e0d0f403-87c7-4502-b1be-875d544899e2-StreamThread-1]
Encountered the following exception during processing and the thread is going
to shut down:
java.lang.IllegalStateException: Metrics recorder is re-initialised with
different task: previous task is -1_-1 whereas current task is 0_1.
*{color:#FF}This is a bug in Kafka Streams. Please open a bug report under
[https://issues.apache.org/jira/projects/KAFKA/issues] at{color}*
org.apache.kafka.streams.state.internals.metrics.RocksDBMetricsRecorder.init(RocksDBMetricsRecorder.java:137)
~[kafka-streams-2.7.0.jar!/:na]
[jira] [Updated] (KAFKA-13009) Metrics recorder is re-initialised with different task
[
https://issues.apache.org/jira/browse/KAFKA-13009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victor van den Hoven updated KAFKA-13009:
-
Description:
When starting my Kafka Stream application, I get in the Logs:
[SmartMeterActionService-e0d0f403-87c7-4502-b1be-875d544899e2-StreamThread-1]
State transition from STARTING to
PARTITIONS_ASSIGNED[SmartMeterActionService-e0d0f403-87c7-4502-b1be-875d544899e2-StreamThread-1]
State transition from STARTING to PARTITIONS_ASSIGNED2021-06-29 07:35:58.258
ERROR 1 — [-StreamThread-1] o.a.k.s.p.internals.StreamThread :
stream-thread
[SmartMeterActionService-e0d0f403-87c7-4502-b1be-875d544899e2-StreamThread-1]
Encountered the following exception during processing and the thread is going
to shut down:
java.lang.IllegalStateException: Metrics recorder is re-initialised with
different task: previous task is -1_-1 whereas current task is 0_1.
*{color:#FF}This is a bug in Kafka Streams. Please open a bug report under
[https://issues.apache.org/jira/projects/KAFKA/issues] at{color}*
org.apache.kafka.streams.state.internals.metrics.RocksDBMetricsRecorder.init(RocksDBMetricsRecorder.java:137)
~[kafka-streams-2.7.0.jar!/:na] at
org.apache.kafka.streams.state.internals.RocksDBStore.init(RocksDBStore.java:252)
~[kafka-streams-2.7.0.jar!/:na] at
org.apache.kafka.streams.state.internals.WrappedStateStore.init(WrappedStateStore.java:55)
~[kafka-streams-2.7.0.jar!/:na] at
org.apache.kafka.streams.state.internals.CachingKeyValueStore.init(CachingKeyValueStore.java:74)
~[kafka-streams-2.7.0.jar!/:na] at
org.apache.kafka.streams.state.internals.WrappedStateStore.init(WrappedStateStore.java:55)
~[kafka-streams-2.7.0.jar!/:na] at
org.apache.kafka.streams.state.internals.MeteredKeyValueStore.lambda$init$1(MeteredKeyValueStore.java:120)
~[kafka-streams-2.7.0.jar!/:na] at
org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl.maybeMeasureLatency(StreamsMetricsImpl.java:883)
~[kafka-streams-2.7.0.jar!/:na] at
org.apache.kafka.streams.state.internals.MeteredKeyValueStore.init(MeteredKeyValueStore.java:120)
~[kafka-streams-2.7.0.jar!/:na] at
org.apache.kafka.streams.processor.internals.ProcessorStateManager.registerStateStores(ProcessorStateManager.java:201)
~[kafka-streams-2.7.0.jar!/:na] at
org.apache.kafka.streams.processor.internals.StateManagerUtil.registerStateStores(StateManagerUtil.java:103)
~[kafka-streams-2.7.0.jar!/:na] at
org.apache.kafka.streams.processor.internals.StandbyTask.initializeIfNeeded(StandbyTask.java:93)
~[kafka-streams-2.7.0.jar!/:na] at
org.apache.kafka.streams.processor.internals.TaskManager.tryToCompleteRestoration(TaskManager.java:473)
~[kafka-streams-2.7.0.jar!/:na] at
org.apache.kafka.streams.processor.internals.StreamThread.initializeAndRestorePhase(StreamThread.java:728)
~[kafka-streams-2.7.0.jar!/:na] at
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:625)
~[kafka-streams-2.7.0.jar!/:na] at
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:553)
~[kafka-streams-2.7.0.jar!/:na] at
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:512)
~[kafka-streams-2.7.0.jar!/:na]
2021-06-29 07:35:58.259 INFO 1 — [-StreamThread-1]
o.a.k.s.p.internals.StreamThread : stream-thread
[SmartMeterActionService-e0d0f403-87c7-4502-b1be-875d544899e2-StreamThread-1]
State transition from PARTITIONS_ASSIGNED to PENDING_SHUTDOWN2021-06-29
07:35:58.259 INFO 1 — [-StreamThread-1] o.a.k.s.p.internals.StreamThread
: stream-thread
[SmartMeterActionService-e0d0f403-87c7-4502-b1be-875d544899e2-StreamThread-1]
Shutting down
After this the application shuts down!
was:
When starting my Kafka Stream application, I get in the Logs:
[SmartMeterActionService-e0d0f403-87c7-4502-b1be-875d544899e2-StreamThread-1]
State transition from STARTING to
PARTITIONS_ASSIGNED[SmartMeterActionService-e0d0f403-87c7-4502-b1be-875d544899e2-StreamThread-1]
State transition from STARTING to PARTITIONS_ASSIGNED2021-06-29 07:35:58.258
ERROR 1 --- [-StreamThread-1] o.a.k.s.p.internals.StreamThread :
stream-thread
[SmartMeterActionService-e0d0f403-87c7-4502-b1be-875d544899e2-StreamThread-1]
Encountered the following exception during processing and the thread is going
to shut down:
java.lang.IllegalStateException: Metrics recorder is re-initialised with
different task: previous task is -1_-1 whereas current task is 0_1. This is a
bug in Kafka Streams. Please open a bug report under
https://issues.apache.org/jira/projects/KAFKA/issues at
org.apache.kafka.streams.state.internals.metrics.RocksDBMetricsRecorder.init(RocksDBMetricsRecorder.java:137)
~[kafka-streams-2.7.0.jar!/:na] at
org.apache.kafka.streams.state.internals.RocksDBStore.init(RocksDBStore.java:252)
~[kafka-streams-2.7.0.jar!/:na] at
[jira] [Created] (KAFKA-13009) Metrics recorder is re-initialised with different task
Victor van den Hoven created KAFKA-13009: Summary: Metrics recorder is re-initialised with different task Key: KAFKA-13009 URL: https://issues.apache.org/jira/browse/KAFKA-13009 Project: Kafka Issue Type: Bug Components: streams Affects Versions: 2.7.0 Environment: Docker container Reporter: Victor van den Hoven When starting my Kafka Stream application, I get in the Logs: [SmartMeterActionService-e0d0f403-87c7-4502-b1be-875d544899e2-StreamThread-1] State transition from STARTING to PARTITIONS_ASSIGNED[SmartMeterActionService-e0d0f403-87c7-4502-b1be-875d544899e2-StreamThread-1] State transition from STARTING to PARTITIONS_ASSIGNED2021-06-29 07:35:58.258 ERROR 1 --- [-StreamThread-1] o.a.k.s.p.internals.StreamThread : stream-thread [SmartMeterActionService-e0d0f403-87c7-4502-b1be-875d544899e2-StreamThread-1] Encountered the following exception during processing and the thread is going to shut down: java.lang.IllegalStateException: Metrics recorder is re-initialised with different task: previous task is -1_-1 whereas current task is 0_1. This is a bug in Kafka Streams. Please open a bug report under https://issues.apache.org/jira/projects/KAFKA/issues at org.apache.kafka.streams.state.internals.metrics.RocksDBMetricsRecorder.init(RocksDBMetricsRecorder.java:137) ~[kafka-streams-2.7.0.jar!/:na] at org.apache.kafka.streams.state.internals.RocksDBStore.init(RocksDBStore.java:252) ~[kafka-streams-2.7.0.jar!/:na] at org.apache.kafka.streams.state.internals.WrappedStateStore.init(WrappedStateStore.java:55) ~[kafka-streams-2.7.0.jar!/:na] at org.apache.kafka.streams.state.internals.CachingKeyValueStore.init(CachingKeyValueStore.java:74) ~[kafka-streams-2.7.0.jar!/:na] at org.apache.kafka.streams.state.internals.WrappedStateStore.init(WrappedStateStore.java:55) ~[kafka-streams-2.7.0.jar!/:na] at org.apache.kafka.streams.state.internals.MeteredKeyValueStore.lambda$init$1(MeteredKeyValueStore.java:120) ~[kafka-streams-2.7.0.jar!/:na] at org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl.maybeMeasureLatency(StreamsMetricsImpl.java:883) ~[kafka-streams-2.7.0.jar!/:na] at org.apache.kafka.streams.state.internals.MeteredKeyValueStore.init(MeteredKeyValueStore.java:120) ~[kafka-streams-2.7.0.jar!/:na] at org.apache.kafka.streams.processor.internals.ProcessorStateManager.registerStateStores(ProcessorStateManager.java:201) ~[kafka-streams-2.7.0.jar!/:na] at org.apache.kafka.streams.processor.internals.StateManagerUtil.registerStateStores(StateManagerUtil.java:103) ~[kafka-streams-2.7.0.jar!/:na] at org.apache.kafka.streams.processor.internals.StandbyTask.initializeIfNeeded(StandbyTask.java:93) ~[kafka-streams-2.7.0.jar!/:na] at org.apache.kafka.streams.processor.internals.TaskManager.tryToCompleteRestoration(TaskManager.java:473) ~[kafka-streams-2.7.0.jar!/:na] at org.apache.kafka.streams.processor.internals.StreamThread.initializeAndRestorePhase(StreamThread.java:728) ~[kafka-streams-2.7.0.jar!/:na] at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:625) ~[kafka-streams-2.7.0.jar!/:na] at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:553) ~[kafka-streams-2.7.0.jar!/:na] at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:512) ~[kafka-streams-2.7.0.jar!/:na] 2021-06-29 07:35:58.259 INFO 1 --- [-StreamThread-1] o.a.k.s.p.internals.StreamThread : stream-thread [SmartMeterActionService-e0d0f403-87c7-4502-b1be-875d544899e2-StreamThread-1] State transition from PARTITIONS_ASSIGNED to PENDING_SHUTDOWN2021-06-29 07:35:58.259 INFO 1 --- [-StreamThread-1] o.a.k.s.p.internals.StreamThread : stream-thread [SmartMeterActionService-e0d0f403-87c7-4502-b1be-875d544899e2-StreamThread-1] Shutting down After this the application shuts down! -- This message was sent by Atlassian Jira (v8.3.4#803005)
