from:"Matthias J. Sax"



[ 
https://issues.apache.org/jira/browse/KAFKA-14839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17716493#comment-17716493
 ] 

Matthias J. Sax commented on KAFKA-14839:
-

Sure!

> Exclude protected variable from JavaDocs
> 
>
> Key: KAFKA-14839
> URL: https://issues.apache.org/jira/browse/KAFKA-14839
> Project: Kafka
>  Issue Type: Bug
>  Components: documentation, streams
>            Reporter: Matthias J. Sax
>Priority: Major
>
> Cf 
> [https://kafka.apache.org/31/javadoc/org/apache/kafka/streams/kstream/JoinWindows.html#enableSpuriousResultFix]
> The variable `enableSpuriousResultFix` is protected, and it's not public API, 
> and thus should not show up in the JavaDocs.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (KAFKA-14936) Add Grace Period To Stream Table Join



 [ 
https://issues.apache.org/jira/browse/KAFKA-14936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax updated KAFKA-14936:

Labels: kip streams  (was: streams)

> Add Grace Period To Stream Table Join
> -
>
> Key: KAFKA-14936
> URL: https://issues.apache.org/jira/browse/KAFKA-14936
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Walker Carlson
>Assignee: Walker Carlson
>Priority: Major
>  Labels: kip, streams
>
> Include the grace period for stream table joins as described in kip 923.
> Also add a rocksDB time based queueing implementation of 
> `TimeOrderedKeyValueBuffer`



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (KAFKA-14172) bug: State stores lose state when tasks are reassigned under EOS wit…



 [ 
https://issues.apache.org/jira/browse/KAFKA-14172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax updated KAFKA-14172:

Fix Version/s: 3.4.1

> bug: State stores lose state when tasks are reassigned under EOS wit…
> -
>
> Key: KAFKA-14172
> URL: https://issues.apache.org/jira/browse/KAFKA-14172
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 3.1.1
>Reporter: Martin Hørslev
>Assignee: Guozhang Wang
>Priority: Critical
> Fix For: 3.5.0, 3.4.1
>
>
> h1. State stores lose state when tasks are reassigned under EOS with standby 
> replicas and default acceptable lag.
> I have observed that state stores used in a transform step under a Exactly 
> Once semantics ends up losing state after a rebalancing event that includes 
> reassignment of tasks to previous standby task within the acceptable standby 
> lag.
>  
> The problem is reproduceable and an integration test have been created to 
> showcase the [issue|https://github.com/apache/kafka/pull/12540]. 
> A detailed description of the observed issue is provided 
> [here|https://github.com/apache/kafka/pull/12540/files?short_path=3ca480e#diff-3ca480ef093a1faa18912e1ebc679be492b341147b96d7a85bda59911228ef45]
> Similar issues have been observed and reported to StackOverflow for example 
> [here|https://stackoverflow.com/questions/69038181/kafka-streams-aggregation-data-loss-between-instance-restarts-and-rebalances].
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (KAFKA-14922) kafka-streams-application-reset deletes topics not belonging to specified application-id



 [ 
https://issues.apache.org/jira/browse/KAFKA-14922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax updated KAFKA-14922:

Labels: beginner needs-kip newbie  (was: )

> kafka-streams-application-reset deletes topics not belonging to specified 
> application-id
> 
>
> Key: KAFKA-14922
> URL: https://issues.apache.org/jira/browse/KAFKA-14922
> Project: Kafka
>  Issue Type: Bug
>  Components: streams, tools
>Affects Versions: 3.4.0
>Reporter: Jørgen
>Priority: Major
>  Labels: beginner, needs-kip, newbie
>
> Slack-thread: 
> [https://confluentcommunity.slack.com/archives/C48AHTCUQ/p1681908267206849]
> When running the command _kafka-streams-application-reset --bootstrap-servers 
> $BOOTSTRAP --application-id foo_ all internal topics that _starts with_ foo 
> is deleted. This happens even if there's no application-id named foo.
> Example:
> {code:java}
> Application IDs:
> foo-v1
> foo-v2
> Internal topics:
> foo-v1-repartition-topic-repartition
> foo-v2-repartition-topic-repartition 
> Application reset:
> kafka-streams-application-reset --bootstrap-servers $BOOTSTRAP 
> --application-id foo
> > No input or intermediate topics specified. Skipping seek.
> Deleting inferred internal topics [foo-v2-repartition-topic-repartition, 
> foo-v1-repartition-topic-repartition]
> Done.{code}
> Expected behaviour is that the command fails as there are no application-id's 
> with the name foo instead of deleting all foo* topics. 
> This is critical on typos or if application-ids starts with the same name as 
> others (for example if we had foo-v21 and wanted to reset foo-v2)
> The bug should be located here: 
> [https://github.com/apache/kafka/blob/c14f56b48461f01743146d58987bc8661ba0d459/tools/src/main/java/org/apache/kafka/tools/StreamsResetter.java#L693]
> Should check that the topics matches the application-id exactly instead of 
> checking that it starts with the application-id.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (KAFKA-14862) Outer stream-stream join does not output all results with multiple input partitions

2023-04-24 Thread Matthias J. Sax (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-14862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax updated KAFKA-14862:

Affects Version/s: 3.1.0

> Outer stream-stream join does not output all results with multiple input 
> partitions
> ---
>
> Key: KAFKA-14862
> URL: https://issues.apache.org/jira/browse/KAFKA-14862
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 3.1.0
>Reporter: Bruno Cadonna
>    Assignee: Matthias J. Sax
>Priority: Major
> Fix For: 3.5.0, 3.4.1
>
>
> If I execute the following Streams app once with two input topics each with 1 
> partition and then with input topics each with two partitions, I get 
> different results.
>   
> {code:java}
> final KStream leftSide = builder.stream(leftSideTopic);
> final KStream rightSide = builder.stream(rightSideTopic);
> final KStream leftAndRight = leftSide.outerJoin(
> rightSide,
> (leftValue, rightValue) ->
> (rightValue == null) ? leftValue + "/NOTPRESENT": leftValue + "/" + 
> rightValue,
> JoinWindows.ofTimeDifferenceAndGrace(
> Duration.ofSeconds(20), 
> Duration.ofSeconds(10)),
> StreamJoined.with(
> Serdes.String(), /* key */
> Serdes.String(), /* left value */
> Serdes.String()  /* right value */
> ));
> leftAndRight.print(Printed.toSysOut());
> {code}
> To reproduce, produce twice the following batch of records with an interval 
> greater than window + grace period (i.e. > 30 seconds) in between the two 
> batches:
> {code}
> (0, 0)
> (1, 1)
> (2, 2)
> (3, 3)
> (4, 4)
> (5, 5)
> (6, 6)
> (7, 7)
> (8, 8)
> (9, 9)
> {code}
> With input topics with 1 partition I get:
> {code}
> [KSTREAM-PROCESSVALUES-08]: 0, 0/NOTPRESENT
> [KSTREAM-PROCESSVALUES-08]: 1, 1/NOTPRESENT
> [KSTREAM-PROCESSVALUES-08]: 2, 2/NOTPRESENT
> [KSTREAM-PROCESSVALUES-08]: 3, 3/NOTPRESENT
> [KSTREAM-PROCESSVALUES-08]: 4, 4/NOTPRESENT
> [KSTREAM-PROCESSVALUES-08]: 5, 5/NOTPRESENT
> [KSTREAM-PROCESSVALUES-08]: 6, 6/NOTPRESENT
> [KSTREAM-PROCESSVALUES-08]: 7, 7/NOTPRESENT
> [KSTREAM-PROCESSVALUES-08]: 8, 8/NOTPRESENT
> [KSTREAM-PROCESSVALUES-08]: 9, 9/NOTPRESENT
> {code}
> With input topics with 2 partitions I get:
> {code}
> [KSTREAM-PROCESSVALUES-08]: 1, 1/NOTPRESENT
> [KSTREAM-PROCESSVALUES-08]: 3, 3/NOTPRESENT
> [KSTREAM-PROCESSVALUES-08]: 4, 4/NOTPRESENT
> [KSTREAM-PROCESSVALUES-08]: 7, 7/NOTPRESENT
> [KSTREAM-PROCESSVALUES-08]: 8, 8/NOTPRESENT
> [KSTREAM-PROCESSVALUES-08]: 9, 9/NOTPRESENT
> {code}
> I would expect to get the same set of records, maybe in a different order due 
> to the partitioning.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] New PMC chair: Mickael Maison

2023-04-21 Thread Matthias J. Sax


Congrats Mickael!

And thanks a lot for taking on this additional task! Glad to have you!


-Matthias

On 4/21/23 9:40 AM, Viktor Somogyi-Vass wrote:

Jun, thank you for all your hard work! Also, congrats Mickael, it is very
well deserved :)

Best,
Viktor

On Fri, Apr 21, 2023, 18:15 Adam Bellemare  wrote:


Thank you for all your hard work Jun - that's a decade-long legacy!
And congratulations to you Mickael!

On Fri, Apr 21, 2023 at 11:20 AM Josep Prat 
wrote:


Thanks Jun for your work as Chair all these years!
Congratulations Mickael!

Best,

———
Josep Prat

Aiven Deutschland GmbH

Alexanderufer 3-7, 10117 Berlin

Amtsgericht Charlottenburg, HRB 209739 B

Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen

m: +491715557497

w: aiven.io

e: josep.p...@aiven.io

On Fri, Apr 21, 2023, 17:10 Jun Rao  wrote:


Hi, everyone,

After more than 10 years, I am stepping down as the PMC chair of Apache
Kafka. We now have a new chair Mickael Maison, who has been a PMC

member

since 2020. I plan to continue to contribute to Apache Kafka myself.

Congratulations, Mickael!

Jun

Re: [ANNOUNCE] New PMC chair: Mickael Maison

2023-04-21 Thread Matthias J. Sax


Congrats Mickael!

And thanks a lot for taking on this additional task! Glad to have you!


-Matthias

On 4/21/23 9:40 AM, Viktor Somogyi-Vass wrote:

Jun, thank you for all your hard work! Also, congrats Mickael, it is very
well deserved :)

Best,
Viktor

On Fri, Apr 21, 2023, 18:15 Adam Bellemare  wrote:


Thank you for all your hard work Jun - that's a decade-long legacy!
And congratulations to you Mickael!

On Fri, Apr 21, 2023 at 11:20 AM Josep Prat 
wrote:


Thanks Jun for your work as Chair all these years!
Congratulations Mickael!

Best,

———
Josep Prat

Aiven Deutschland GmbH

Alexanderufer 3-7, 10117 Berlin

Amtsgericht Charlottenburg, HRB 209739 B

Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen

m: +491715557497

w: aiven.io

e: josep.p...@aiven.io

On Fri, Apr 21, 2023, 17:10 Jun Rao  wrote:


Hi, everyone,

After more than 10 years, I am stepping down as the PMC chair of Apache
Kafka. We now have a new chair Mickael Maison, who has been a PMC

member

since 2020. I plan to continue to contribute to Apache Kafka myself.

Congratulations, Mickael!

Jun

[jira] [Commented] (KAFKA-14722) Make BooleanSerde public

2023-04-21 Thread Matthias J. Sax (Jira)



[ 
https://issues.apache.org/jira/browse/KAFKA-14722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17715157#comment-17715157
 ] 

Matthias J. Sax commented on KAFKA-14722:
-

I did a PR: [https://github.com/apache/kafka/pull/13577] – Just merged it.

> Make BooleanSerde public
> 
>
> Key: KAFKA-14722
> URL: https://issues.apache.org/jira/browse/KAFKA-14722
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>            Reporter: Matthias J. Sax
>Assignee: Spacrocket
>Priority: Minor
>  Labels: beginner, kip, newbie
> Fix For: 3.5.0
>
>
> KIP-907: 
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-907%3A+Add+Boolean+Serde+to+public+interface]
>  
> We introduce a "BooleanSerde" via 
> [https://github.com/apache/kafka/pull/13249] as internal class. We could make 
> it public.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (KAFKA-14922) kafka-streams-application-reset deletes topics not belonging to specified application-id

2023-04-20 Thread Matthias J. Sax (Jira)



[ 
https://issues.apache.org/jira/browse/KAFKA-14922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17714770#comment-17714770
 ] 

Matthias J. Sax commented on KAFKA-14922:
-

{quote}We could add a warning with the list of internal topics found to delete 
and ask for a confirmation to make it harder to inadvertently delete internal 
topics of other application IDs.
{quote}

There is already a `--dry-run` option, but we could of course also try adding a 
`--execute` one and flip it around... Would need a KIP of course.
 
{quote}An _improvement_ would be to only return topics that exactly contains 
the applicationId provided.{quote}
{color:#172b4d}I don't believe it's possible to implement this.{color}
 
{quote}Would not cover the case where other applicationIds starts with 
applicationId provided (foo-v1 would delete foo-v1-2 topics, etc)
{quote}
 
Both seem to be the same issue? Note: we already do 
`topicName.startsWith(options.valueOf(applicationIdOption) + "-")`, ie, we add 
the expected `-` – if your app.id uses a dash like `myApp-v1` there is nothing 
we can do about it. It provides a protection for `appV1` vs `appV2` and if you 
pass in `app` it won't match either of them, but if `-` is use inside app.id, 
it seems there is nothing we can do about it.

> kafka-streams-application-reset deletes topics not belonging to specified 
> application-id
> 
>
> Key: KAFKA-14922
> URL: https://issues.apache.org/jira/browse/KAFKA-14922
> Project: Kafka
>  Issue Type: Bug
>  Components: streams, tools
>Affects Versions: 3.4.0
>Reporter: Jørgen
>Priority: Major
>
> Slack-thread: 
> [https://confluentcommunity.slack.com/archives/C48AHTCUQ/p1681908267206849]
> When running the command _kafka-streams-application-reset --bootstrap-servers 
> $BOOTSTRAP --application-id foo_ all internal topics that _starts with_ foo 
> is deleted. This happens even if there's no application-id named foo.
> Example:
> {code:java}
> Application IDs:
> foo-v1
> foo-v2
> Internal topics:
> foo-v1-repartition-topic-repartition
> foo-v2-repartition-topic-repartition 
> Application reset:
> kafka-streams-application-reset --bootstrap-servers $BOOTSTRAP 
> --application-id foo
> > No input or intermediate topics specified. Skipping seek.
> Deleting inferred internal topics [foo-v2-repartition-topic-repartition, 
> foo-v1-repartition-topic-repartition]
> Done.{code}
> Expected behaviour is that the command fails as there are no application-id's 
> with the name foo instead of deleting all foo* topics. 
> This is critical on typos or if application-ids starts with the same name as 
> others (for example if we had foo-v21 and wanted to reset foo-v2)
> The bug should be located here: 
> [https://github.com/apache/kafka/blob/c14f56b48461f01743146d58987bc8661ba0d459/tools/src/main/java/org/apache/kafka/tools/StreamsResetter.java#L693]
> Should check that the topics matches the application-id exactly instead of 
> checking that it starts with the application-id.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (KAFKA-14922) kafka-streams-application-reset deletes topics not belonging to specified application-id



 [ 
https://issues.apache.org/jira/browse/KAFKA-14922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax updated KAFKA-14922:

Component/s: streams
 tools

> kafka-streams-application-reset deletes topics not belonging to specified 
> application-id
> 
>
> Key: KAFKA-14922
> URL: https://issues.apache.org/jira/browse/KAFKA-14922
> Project: Kafka
>  Issue Type: Bug
>  Components: streams, tools
>Affects Versions: 3.4.0
>Reporter: Jørgen
>Priority: Major
>
> Slack-thread: 
> [https://confluentcommunity.slack.com/archives/C48AHTCUQ/p1681908267206849]
> When running the command _kafka-streams-application-reset --bootstrap-servers 
> $BOOTSTRAP --application-id foo_ all internal topics that _starts with_ foo 
> is deleted. This happens even if there's no application-id named foo.
> Example:
> {code:java}
> Application IDs:
> foo-v1
> foo-v2
> Internal topics:
> foo-v1-repartition-topic-repartition
> foo-v2-repartition-topic-repartition 
> Application reset:
> kafka-streams-application-reset --bootstrap-servers $BOOTSTRAP 
> --application-id foo
> > No input or intermediate topics specified. Skipping seek.
> Deleting inferred internal topics [foo-v2-repartition-topic-repartition, 
> foo-v1-repartition-topic-repartition]
> Done.{code}
> Expected behaviour is that the command fails as there are no application-id's 
> with the name foo instead of deleting all foo* topics. 
> This is critical on typos or if application-ids starts with the same name as 
> others (for example if we had foo-v21 and wanted to reset foo-v2)
> The bug should be located here: 
> [https://github.com/apache/kafka/blob/c14f56b48461f01743146d58987bc8661ba0d459/tools/src/main/java/org/apache/kafka/tools/StreamsResetter.java#L693]
> Should check that the topics matches the application-id exactly instead of 
> checking that it starts with the application-id.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Re-visit end of life policy

2023-04-19 Thread Matthias J. Sax


While I understand the desire, I tend to agree with Ismael.

In general, it's a significant amount of work not just to do the actual 
releases, but also the cherry-pick bug-fixed to older branches. Code 
diverges very quickly, and a clean cherry-pick is usually only possible 
for one or two branches. And it's not just simple conflicts that are 
easy to resolve, but it often even implies to do a full new fix, if the 
corresponding code was refactored, what is more often the case than one 
might think.


If there is no very strong ask from the community, I would rather let 
committer spent their time reviewing PRs instead and help contributors 
to get the work merged.


Just my 2ct.

-Matthias


On 4/13/23 2:52 PM, Ismael Juma wrote:

Clarification below.

I did not understand your point about maintenance expense to ensure

compatibility. I am confused because, IMO, irrespective of our bug fix
support duration for minor versions, we should ensure that all prior minor
versions are compatible. Hence, increasing the support duration to 24
months will not add more expense than today to ensure compatibility.



No, I am not saying that. I am saying that there is no reason not to
upgrade from one minor release to another since we provide full
compatibility between minor releases. The expensive part is that we release
3 times a year, so you have to support 6 releases at any given point in
time. More importantly, you have to validate all these releases, handle any
additional bugs and so on. When it comes to the CVE stuff, you also have to
deal with cases where a project you depend on forces an upgrade to a
release with compatibility impact and so on. Having seen this first hand,
it's a significant amount of work.

Ismael

[jira] [Commented] (KAFKA-14922) kafka-streams-application-reset deletes topics not belonging to specified application-id



[ 
https://issues.apache.org/jira/browse/KAFKA-14922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17714354#comment-17714354
 ] 

Matthias J. Sax commented on KAFKA-14922:
-

Thanks for creating this ticket. It's a know issue but it's unclear how it 
could be fixed.

The problem is, that topic name have the patter 
`--` – I am not sure how we could look for 
an _exact_ match (we don't know the full topic name)? If there is a way, please 
let us know. But I think we need to close this as "won't fix" unfortunately. 

> kafka-streams-application-reset deletes topics not belonging to specified 
> application-id
> 
>
> Key: KAFKA-14922
> URL: https://issues.apache.org/jira/browse/KAFKA-14922
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 3.4.0
>Reporter: Jørgen
>Priority: Major
>
> Slack-thread: 
> [https://confluentcommunity.slack.com/archives/C48AHTCUQ/p1681908267206849]
> When running the command _kafka-streams-application-reset --bootstrap-servers 
> $BOOTSTRAP --application-id foo_ all internal topics that _starts with_ foo 
> is deleted. This happens even if there's no application-id named foo.
> Example:
> {code:java}
> Application IDs:
> foo-v1
> foo-v2
> Internal topics:
> foo-v1-repartition-topic-repartition
> foo-v2-repartition-topic-repartition 
> Application reset:
> kafka-streams-application-reset --bootstrap-servers $BOOTSTRAP 
> --application-id foo
> > No input or intermediate topics specified. Skipping seek.
> Deleting inferred internal topics [foo-v2-repartition-topic-repartition, 
> foo-v1-repartition-topic-repartition]
> Done.{code}
> Expected behaviour is that the command fails as there are no application-id's 
> with the name foo instead of deleting all foo* topics. 
> This is critical on typos or if application-ids starts with the same name as 
> others (for example if we had foo-v21 and wanted to reset foo-v2)
> The bug should be located here: 
> [https://github.com/apache/kafka/blob/c14f56b48461f01743146d58987bc8661ba0d459/tools/src/main/java/org/apache/kafka/tools/StreamsResetter.java#L693]
> Should check that the topics matches the application-id exactly instead of 
> checking that it starts with the application-id.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-4327) Move Reset Tool from core to streams



 [ 
https://issues.apache.org/jira/browse/KAFKA-4327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-4327.

Fix Version/s: (was: 4.0.0)
   Resolution: Fixed

This was resolved via https://issues.apache.org/jira/browse/KAFKA-14586.

> Move Reset Tool from core to streams
> 
>
> Key: KAFKA-4327
> URL: https://issues.apache.org/jira/browse/KAFKA-4327
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>            Reporter: Matthias J. Sax
>Priority: Blocker
>  Labels: kip
>
> This is a follow up on https://issues.apache.org/jira/browse/KAFKA-4008
> Currently, Kafka Streams Application Reset Tool is part of {{core}} module 
> due to ZK dependency. After KIP-4 got merged, this dependency can be dropped 
> and the Reset Tool can be moved to {{streams}} module.
> This should also update {{InternalTopicManager#filterExistingTopics}} that 
> revers to ResetTool in an exception message:
>  {{"Use 'kafka.tools.StreamsResetter' tool"}}
>  -> {{"Use '" + kafka.tools.StreamsResetter.getClass().getName() + "' tool"}}
> Doing this JIRA also requires to update the docs with regard to broker 
> backward compatibility – not all broker support "topic delete request" and 
> thus, the reset tool will not be backward compatible to all broker versions.
> KIP-756: 
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-756%3A+Move+StreamsResetter+tool+outside+of+core]
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-4327) Move Reset Tool from core to streams



 [ 
https://issues.apache.org/jira/browse/KAFKA-4327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-4327.

Fix Version/s: (was: 4.0.0)
   Resolution: Fixed

This was resolved via https://issues.apache.org/jira/browse/KAFKA-14586.

> Move Reset Tool from core to streams
> 
>
> Key: KAFKA-4327
> URL: https://issues.apache.org/jira/browse/KAFKA-4327
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>            Reporter: Matthias J. Sax
>Priority: Blocker
>  Labels: kip
>
> This is a follow up on https://issues.apache.org/jira/browse/KAFKA-4008
> Currently, Kafka Streams Application Reset Tool is part of {{core}} module 
> due to ZK dependency. After KIP-4 got merged, this dependency can be dropped 
> and the Reset Tool can be moved to {{streams}} module.
> This should also update {{InternalTopicManager#filterExistingTopics}} that 
> revers to ResetTool in an exception message:
>  {{"Use 'kafka.tools.StreamsResetter' tool"}}
>  -> {{"Use '" + kafka.tools.StreamsResetter.getClass().getName() + "' tool"}}
> Doing this JIRA also requires to update the docs with regard to broker 
> backward compatibility – not all broker support "topic delete request" and 
> thus, the reset tool will not be backward compatible to all broker versions.
> KIP-756: 
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-756%3A+Move+StreamsResetter+tool+outside+of+core]
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Assigned] (KAFKA-4327) Move Reset Tool from core to streams



 [ 
https://issues.apache.org/jira/browse/KAFKA-4327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax reassigned KAFKA-4327:
--

Assignee: (was: Jorge Esteban Quilcate Otoya)

> Move Reset Tool from core to streams
> 
>
> Key: KAFKA-4327
> URL: https://issues.apache.org/jira/browse/KAFKA-4327
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>            Reporter: Matthias J. Sax
>Priority: Blocker
>  Labels: kip
> Fix For: 4.0.0
>
>
> This is a follow up on https://issues.apache.org/jira/browse/KAFKA-4008
> Currently, Kafka Streams Application Reset Tool is part of {{core}} module 
> due to ZK dependency. After KIP-4 got merged, this dependency can be dropped 
> and the Reset Tool can be moved to {{streams}} module.
> This should also update {{InternalTopicManager#filterExistingTopics}} that 
> revers to ResetTool in an exception message:
>  {{"Use 'kafka.tools.StreamsResetter' tool"}}
>  -> {{"Use '" + kafka.tools.StreamsResetter.getClass().getName() + "' tool"}}
> Doing this JIRA also requires to update the docs with regard to broker 
> backward compatibility – not all broker support "topic delete request" and 
> thus, the reset tool will not be backward compatible to all broker versions.
> KIP-756: 
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-756%3A+Move+StreamsResetter+tool+outside+of+core]
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (KAFKA-14586) Move StreamsResetter to tools

2023-04-18 Thread Matthias J. Sax (Jira)



[ 
https://issues.apache.org/jira/browse/KAFKA-14586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713836#comment-17713836
 ] 

Matthias J. Sax commented on KAFKA-14586:
-

Thanks for providing context, and no worries about not knowing about the other 
KIP (there is too many things going on, and I also just realized the overlap).

Yes, `StreamsResetter` might be used programmatically, so we should add a 
redirection. Who will do this? Guess we should get it in before code freeze to 
not delay the release.

I am not worried about moving the test because it's not user facing.

Overall, it seem we can close out the other KIP and ticket as "subsumed" by 
this ticket/KIP. I can do the cleanup for it.

Just let me know if there is anything I can help with, or if the matter is 
resolved after we got the missing redirection merged.

 

> Move StreamsResetter to tools
> -
>
> Key: KAFKA-14586
> URL: https://issues.apache.org/jira/browse/KAFKA-14586
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Mickael Maison
>Assignee: Sagar Rao
>Priority: Major
> Fix For: 3.5.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Assigned] (KAFKA-14862) Outer stream-stream join does not output all results with multiple input partitions

2023-04-17 Thread Matthias J. Sax (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-14862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax reassigned KAFKA-14862:
---

Assignee: Matthias J. Sax

> Outer stream-stream join does not output all results with multiple input 
> partitions
> ---
>
> Key: KAFKA-14862
> URL: https://issues.apache.org/jira/browse/KAFKA-14862
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Bruno Cadonna
>    Assignee: Matthias J. Sax
>Priority: Major
>
> If I execute the following Streams app once with two input topics each with 1 
> partition and then with input topics each with two partitions, I get 
> different results.
>   
> {code:java}
> final KStream leftSide = builder.stream(leftSideTopic);
> final KStream rightSide = builder.stream(rightSideTopic);
> final KStream leftAndRight = leftSide.outerJoin(
> rightSide,
> (leftValue, rightValue) ->
> (rightValue == null) ? leftValue + "/NOTPRESENT": leftValue + "/" + 
> rightValue,
> JoinWindows.ofTimeDifferenceAndGrace(
> Duration.ofSeconds(20), 
> Duration.ofSeconds(10)),
> StreamJoined.with(
> Serdes.String(), /* key */
> Serdes.String(), /* left value */
> Serdes.String()  /* right value */
> ));
> leftAndRight.print(Printed.toSysOut());
> {code}
> To reproduce, produce twice the following batch of records with an interval 
> greater than window + grace period (i.e. > 30 seconds) in between the two 
> batches:
> {code}
> (0, 0)
> (1, 1)
> (2, 2)
> (3, 3)
> (4, 4)
> (5, 5)
> (6, 6)
> (7, 7)
> (8, 8)
> (9, 9)
> {code}
> With input topics with 1 partition I get:
> {code}
> [KSTREAM-PROCESSVALUES-08]: 0, 0/NOTPRESENT
> [KSTREAM-PROCESSVALUES-08]: 1, 1/NOTPRESENT
> [KSTREAM-PROCESSVALUES-08]: 2, 2/NOTPRESENT
> [KSTREAM-PROCESSVALUES-08]: 3, 3/NOTPRESENT
> [KSTREAM-PROCESSVALUES-08]: 4, 4/NOTPRESENT
> [KSTREAM-PROCESSVALUES-08]: 5, 5/NOTPRESENT
> [KSTREAM-PROCESSVALUES-08]: 6, 6/NOTPRESENT
> [KSTREAM-PROCESSVALUES-08]: 7, 7/NOTPRESENT
> [KSTREAM-PROCESSVALUES-08]: 8, 8/NOTPRESENT
> [KSTREAM-PROCESSVALUES-08]: 9, 9/NOTPRESENT
> {code}
> With input topics with 2 partitions I get:
> {code}
> [KSTREAM-PROCESSVALUES-08]: 1, 1/NOTPRESENT
> [KSTREAM-PROCESSVALUES-08]: 3, 3/NOTPRESENT
> [KSTREAM-PROCESSVALUES-08]: 4, 4/NOTPRESENT
> [KSTREAM-PROCESSVALUES-08]: 7, 7/NOTPRESENT
> [KSTREAM-PROCESSVALUES-08]: 8, 8/NOTPRESENT
> [KSTREAM-PROCESSVALUES-08]: 9, 9/NOTPRESENT
> {code}
> I would expect to get the same set of records, maybe in a different order due 
> to the partitioning.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (KAFKA-14586) Move StreamsResetter to tools

2023-04-17 Thread Matthias J. Sax (Jira)



[ 
https://issues.apache.org/jira/browse/KAFKA-14586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713233#comment-17713233
 ] 

Matthias J. Sax commented on KAFKA-14586:
-

[~mimaison] [~sagarrao] – I am just realizing that we did this as part of 3.5 – 
We actually had https://issues.apache.org/jira/browse/KAFKA-4327 that we did 
not do in the past, because we thought we should only do it in major release, 
as it seems to be a breaking change (we also had 
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-756%3A+Move+StreamsResetter+tool+outside+of+core]
 for it)

Seems you solve the issue about introducing a breaking change with some 
"redirection" according to KIP-906. So we can we close K4327 and the K-756?

> Move StreamsResetter to tools
> -
>
> Key: KAFKA-14586
> URL: https://issues.apache.org/jira/browse/KAFKA-14586
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Mickael Maison
>Assignee: Sagar Rao
>Priority: Major
> Fix For: 3.5.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (KAFKA-14911) Add system tests for rolling upgrade path of KIP-904

2023-04-17 Thread Matthias J. Sax (Jira)



[ 
https://issues.apache.org/jira/browse/KAFKA-14911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713230#comment-17713230
 ] 

Matthias J. Sax commented on KAFKA-14911:
-

[~fqpublic] – are you planning to pickup this ticket?

> Add system tests for rolling upgrade path of KIP-904
> 
>
> Key: KAFKA-14911
> URL: https://issues.apache.org/jira/browse/KAFKA-14911
> Project: Kafka
>  Issue Type: Test
>Reporter: Farooq Qaiser
>Priority: Major
> Fix For: 3.5.0
>
>
> As per [~mjsax] comment 
> [here|https://github.com/apache/kafka/pull/10747#pullrequestreview-1376539752],
>  we should add a system test to test the rolling upgrade path for 
> [KIP-904|https://cwiki.apache.org/confluence/x/P5VbDg] which introduces a new 
> serialization format for groupBy internal repartition topics and was 
> implemented as part of https://issues.apache.org/jira/browse/KAFKA-12446 
> There is `StreamsUpgradeTest.java` and `streams_upgrade_test.py` (cf 
> `test_rolling_upgrade_with_2_bounces`) as a starting point.
> Might be best to do a similar thing as for FK-joins, and add a new test 
> variation. 
> The tricky thing about the test would be, to ensure that the repartition 
> topic is not empty when we do the bounce, so the test should be setup 
> accordingly.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Apache Kafka 3.5.0 release

2023-04-14 Thread Matthias J. Sax


Thanks a lot!

On 4/14/23 5:32 AM, Mickael Maison wrote:

Hi Matthias,

I merged the PR before cutting the 3.5 branch.

Thanks,
Mickael

On Fri, Apr 14, 2023 at 2:31 PM Mickael Maison  wrote:


Hi David,

I've created the 3.5 branch. Feel free to cherry pick these 2 commits
when they are ready.

Thanks,
Mickael

On Fri, Apr 14, 2023 at 11:23 AM Satish Duggana
 wrote:


Thanks Luke for helping with the reviews and adding a few tests in a
couple of PRs.

Hi Mickael,
I raised 3 PRs recently for tiered storage, one is merged. The other 2
PRs are in the critical path of non-tiered storage changes also.
Especially, in consumer fetch and retention cleanup paths. These need
to be thoroughly reviewed and avoid any regressions in that area. We
should merge them to the trunk as soon as possible to make it easier
to work on follow-up PRs. IMO, we can avoid merging these PRs in 3.5
just before the release without baking for a longer duration. We can
take a call on this later after the reviews are done.

Many of the individual functionalities related to tiered storage like
default topic based RLMM implementation, enhanced follower fetch
protocol implementation for tiered storage, copying remote log
segments are merged.
There are 2 PRs for consumer fetch for remote record reads, remote
retention cleanup and topic deletion functionality are under review.

I do not think it can be considered as an early access review even
with the 2 PRs in review. Luke and I synced up and agreed on the same.
Most of the recent functionality is added with a few unit tests. We
plan to have follow-up PRs on the immediate pending items and also
raise PRs in the next few weeks on unit tests, integration test
framework and several integration tests for many of these
functionalities tying together.

Thanks,
Satish.


On Fri, 14 Apr 2023 at 12:52, Matthias J. Sax  wrote:


Hey Mickael,

we have one open PR for KIP-914 left. Would be great if you could merge
it before cutting the 3.5 branch. If you don't want to merge it and
prefer that I cherry-pick it to 3.5 branch later, also works for me.

I did close the ticket already as resolved. It's just a minor change the
KIP.

https://github.com/apache/kafka/pull/13565


Thanks a lot!
-Matthias


On 4/13/23 4:32 AM, David Jacot wrote:

Hi Mickael,

Thanks for the heads up. As raised by Jeff earlier in this thread, we would
like to get the two small patches [1][2] for KIP-915 in 3.5. The PRs are in
review and I should be able to merge them in the next few days. I will
cherry-pick them to the release branch in your create it in the meantime.

[1] https://github.com/apache/kafka/pull/13511
[2] https://github.com/apache/kafka/pull/13526

Best,
David

On Thu, Apr 13, 2023 at 12:55 PM Mickael Maison 
wrote:


Hi Luke,

Thanks for the heads up. This would be great to get Tiered Storage in
Early Access. Let me know if you can't get everything done this week.

Mickael

On Thu, Apr 13, 2023 at 12:54 PM Mickael Maison
 wrote:


Hi,

We've now reached feature freeze for 3.5.0. From now on, only bug
fixes and changes related to stabilizing the release should be merged.

I plan to create the release branch tomorrow (Friday 14). After this
point, you'll have to cherry pick changes to the release branch when
merging a PR. I'll send another message once the branch has been
created.

I've updated the release plan and started moving KIPs that are not
complete to the postponed section. For now I've kept a few KIPs that
are still in progress. If they are not fully merged when I create a
branch, I'll mark them as postponed too.

The next milestone is code freeze on April 26.

Thanks,
Mickael

On Wed, Apr 12, 2023 at 12:24 PM Luke Chen  wrote:


Hi Mickael,

I'd like to ask for some more days for KIP-405 tiered storage PRs to
include in v3.5.
Currently, we have 1 PR under reviewing (
https://github.com/apache/kafka/pull/13535), and 1 PR soon will be

opened

for review.
After these 2 PRs merged, we can have an "Early Access" for tiered

storage

feature, that allow users to use in non-production environments.
Does that work for you?

Thank you.
Luke

On Thu, Apr 6, 2023 at 2:49 AM Jeff Kim 


wrote:


Hi Mickael,

Thank you.

Best,
Jeff

On Wed, Apr 5, 2023 at 1:28 PM Mickael Maison <

mickael.mai...@gmail.com>

wrote:


Hi Jeff,

Ok, I've added KIP-915 to the release plan.

Thanks,
Mickael

On Wed, Apr 5, 2023 at 6:48 PM Jeff Kim



wrote:


Hi Mickael,

I would like to bring up that KIP-915 proposes to patch 3.5
although it missed the KIP freeze date. If the patch is done

before the

feature freeze date, 4/13, would this be acceptable? If so,

should this

be added to the 3.5.0 Release Plan wiki?

Best,
Jeff

On Mon, Mar 27, 2023 at 1:02 PM Greg Harris



wrote:


Mickael,

Just wanted to let you know that I will not be including

KIP-898 in

the

3.5.0 release.
I think the change needed is not reviewable before the feature

freeze

deadline, and would take resources away from other more

necessary

c

Re: [DISCUSS] Apache Kafka 3.5.0 release

2023-04-14 Thread Matthias J. Sax


Hey Mickael,

we have one open PR for KIP-914 left. Would be great if you could merge 
it before cutting the 3.5 branch. If you don't want to merge it and 
prefer that I cherry-pick it to 3.5 branch later, also works for me.


I did close the ticket already as resolved. It's just a minor change the 
KIP.


https://github.com/apache/kafka/pull/13565


Thanks a lot!
  -Matthias


On 4/13/23 4:32 AM, David Jacot wrote:

Hi Mickael,

Thanks for the heads up. As raised by Jeff earlier in this thread, we would
like to get the two small patches [1][2] for KIP-915 in 3.5. The PRs are in
review and I should be able to merge them in the next few days. I will
cherry-pick them to the release branch in your create it in the meantime.

[1] https://github.com/apache/kafka/pull/13511
[2] https://github.com/apache/kafka/pull/13526

Best,
David

On Thu, Apr 13, 2023 at 12:55 PM Mickael Maison 
wrote:


Hi Luke,

Thanks for the heads up. This would be great to get Tiered Storage in
Early Access. Let me know if you can't get everything done this week.

Mickael

On Thu, Apr 13, 2023 at 12:54 PM Mickael Maison
 wrote:


Hi,

We've now reached feature freeze for 3.5.0. From now on, only bug
fixes and changes related to stabilizing the release should be merged.

I plan to create the release branch tomorrow (Friday 14). After this
point, you'll have to cherry pick changes to the release branch when
merging a PR. I'll send another message once the branch has been
created.

I've updated the release plan and started moving KIPs that are not
complete to the postponed section. For now I've kept a few KIPs that
are still in progress. If they are not fully merged when I create a
branch, I'll mark them as postponed too.

The next milestone is code freeze on April 26.

Thanks,
Mickael

On Wed, Apr 12, 2023 at 12:24 PM Luke Chen  wrote:


Hi Mickael,

I'd like to ask for some more days for KIP-405 tiered storage PRs to
include in v3.5.
Currently, we have 1 PR under reviewing (
https://github.com/apache/kafka/pull/13535), and 1 PR soon will be

opened

for review.
After these 2 PRs merged, we can have an "Early Access" for tiered

storage

feature, that allow users to use in non-production environments.
Does that work for you?

Thank you.
Luke

On Thu, Apr 6, 2023 at 2:49 AM Jeff Kim 


wrote:


Hi Mickael,

Thank you.

Best,
Jeff

On Wed, Apr 5, 2023 at 1:28 PM Mickael Maison <

mickael.mai...@gmail.com>

wrote:


Hi Jeff,

Ok, I've added KIP-915 to the release plan.

Thanks,
Mickael

On Wed, Apr 5, 2023 at 6:48 PM Jeff Kim



wrote:


Hi Mickael,

I would like to bring up that KIP-915 proposes to patch 3.5
although it missed the KIP freeze date. If the patch is done

before the

feature freeze date, 4/13, would this be acceptable? If so,

should this

be added to the 3.5.0 Release Plan wiki?

Best,
Jeff

On Mon, Mar 27, 2023 at 1:02 PM Greg Harris



wrote:


Mickael,

Just wanted to let you know that I will not be including

KIP-898 in

the

3.5.0 release.
I think the change needed is not reviewable before the feature

freeze

deadline, and would take resources away from other more

necessary

changes.


Thanks!
Greg

On Thu, Mar 23, 2023 at 9:01 AM Chia-Ping Tsai <

chia7...@gmail.com>

wrote:



If you have a KIP that is accepted, make sure it is listed

in





https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.5.0

and that it's status is accurate.


Thanks for the reminder. Have added KIP-641 to the list.

Thanks,
Chia-Ping


Mickael Maison  於 2023年3月23日

下午11:51

寫道：


Hi all,

KIP Freeze was yesterday. The next milestone is feature

freeze on

April

12.

If you have a KIP that is accepted, make sure it is listed

in





https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.5.0

and that it's status is accurate.

Thanks,
Mickael

On Fri, Mar 17, 2023 at 6:22 PM Christo Lolov <

christolo...@gmail.com>

wrote:


Hello!

What would you suggest as the best way to get more eyes on

KIP-902 as

I

would like it to be included it in 3.5.0?


Best,
Christo


On 16 Mar 2023, at 10:33, Mickael Maison <

mickael.mai...@gmail.com>

wrote:


Hi,

This is a reminder that KIP freeze is less than a week

away (22

Mar).

For a KIP to be considered for this release, it must be

voted

and

accepted by that date.

Feature freeze will be 3 weeks after this, so if you

want KIPs

or

other significant changes in the release, please get

them ready

soon.


Thanks,
Mickael


On Tue, Feb 14, 2023 at 10:44 PM Ismael Juma <

ism...@juma.me.uk



wrote:


Thanks!

Ismael

On Tue, Feb 14, 2023 at 1:07 PM Mickael Maison <

mickael.mai...@gmail.com>

wrote:


Hi Ismael,

Good call. I shifted all dates by 2 weeks and moved

them to

Wednesdays.


Thanks,
Mickael

On Tue, Feb 14, 2023 at 6:01 PM Ismael Juma <

ism...@juma.me.uk



wrote:


Thanks Mickael. A couple of notes:

1. We typically choose a Wednesday for the various

freeze

dates -

there

are

often 1-2 day slips and it's better if that doesn't

require

people

working

[jira] [Resolved] (KAFKA-7499) Extend ProductionExceptionHandler to cover serialization exceptions



 [ 
https://issues.apache.org/jira/browse/KAFKA-7499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-7499.

Fix Version/s: 3.5.0
   Resolution: Fixed

> Extend ProductionExceptionHandler to cover serialization exceptions
> ---
>
> Key: KAFKA-7499
> URL: https://issues.apache.org/jira/browse/KAFKA-7499
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>            Reporter: Matthias J. Sax
>Assignee: Philip Nee
>Priority: Major
>  Labels: beginner, kip, newbie, newbie++
> Fix For: 3.5.0
>
>
> In 
> [KIP-210|https://cwiki.apache.org/confluence/display/KAFKA/KIP-210+-+Provide+for+custom+error+handling++when+Kafka+Streams+fails+to+produce],
>  an exception handler for the write path was introduced. This exception 
> handler covers exception that are raised in the producer callback.
> However, serialization happens before the data is handed to the producer with 
> Kafka Streams itself and the producer uses `byte[]/byte[]` key-value-pair 
> types.
> Thus, we might want to extend the ProductionExceptionHandler to cover 
> serialization exception, too, to skip over corrupted output messages. An 
> example could be a "String" message that contains invalid JSON and should be 
> serialized as JSON.
> KIP-399 (not voted yet; feel free to pick it up): 
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-399%3A+Extend+ProductionExceptionHandler+to+cover+serialization+exceptions]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-7499) Extend ProductionExceptionHandler to cover serialization exceptions



 [ 
https://issues.apache.org/jira/browse/KAFKA-7499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-7499.

Fix Version/s: 3.5.0
   Resolution: Fixed

> Extend ProductionExceptionHandler to cover serialization exceptions
> ---
>
> Key: KAFKA-7499
> URL: https://issues.apache.org/jira/browse/KAFKA-7499
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>            Reporter: Matthias J. Sax
>Assignee: Philip Nee
>Priority: Major
>  Labels: beginner, kip, newbie, newbie++
> Fix For: 3.5.0
>
>
> In 
> [KIP-210|https://cwiki.apache.org/confluence/display/KAFKA/KIP-210+-+Provide+for+custom+error+handling++when+Kafka+Streams+fails+to+produce],
>  an exception handler for the write path was introduced. This exception 
> handler covers exception that are raised in the producer callback.
> However, serialization happens before the data is handed to the producer with 
> Kafka Streams itself and the producer uses `byte[]/byte[]` key-value-pair 
> types.
> Thus, we might want to extend the ProductionExceptionHandler to cover 
> serialization exception, too, to skip over corrupted output messages. An 
> example could be a "String" message that contains invalid JSON and should be 
> serialized as JSON.
> KIP-399 (not voted yet; feel free to pick it up): 
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-399%3A+Extend+ProductionExceptionHandler+to+cover+serialization+exceptions]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-14834) Improved processor semantics for versioned stores



 [ 
https://issues.apache.org/jira/browse/KAFKA-14834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-14834.
-
Fix Version/s: 3.5.0
   Resolution: Fixed

> Improved processor semantics for versioned stores
> -
>
> Key: KAFKA-14834
> URL: https://issues.apache.org/jira/browse/KAFKA-14834
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Victoria Xia
>Assignee: Victoria Xia
>Priority: Major
>  Labels: kip, streams
> Fix For: 3.5.0
>
>
> With the introduction of versioned state stores in 
> [KIP-889|https://cwiki.apache.org/confluence/display/KAFKA/KIP-889%3A+Versioned+State+Stores],
>  we should leverage them to provide improved join semantics. 
> As described in 
> [KIP-914|https://cwiki.apache.org/confluence/display/KAFKA/KIP-914%3A+DSL+Processor+Semantics+for+Versioned+Stores],
>  we will make the following four improvements:
>  * stream-table joins will perform a timestamped lookup (using the 
> stream-side record timestamp) if the table is versioned
>  * table-table joins, including foreign key joins, will not produce new join 
> results on out-of-order records (by key) from versioned tables
>  * table filters will disable the existing optimization to not send duplicate 
> tombstones when applied to a versioned table
>  * table aggregations will ignore out-of-order records when aggregating a 
> versioned table



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-14834) Improved processor semantics for versioned stores



 [ 
https://issues.apache.org/jira/browse/KAFKA-14834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-14834.
-
Fix Version/s: 3.5.0
   Resolution: Fixed

> Improved processor semantics for versioned stores
> -
>
> Key: KAFKA-14834
> URL: https://issues.apache.org/jira/browse/KAFKA-14834
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Victoria Xia
>Assignee: Victoria Xia
>Priority: Major
>  Labels: kip, streams
> Fix For: 3.5.0
>
>
> With the introduction of versioned state stores in 
> [KIP-889|https://cwiki.apache.org/confluence/display/KAFKA/KIP-889%3A+Versioned+State+Stores],
>  we should leverage them to provide improved join semantics. 
> As described in 
> [KIP-914|https://cwiki.apache.org/confluence/display/KAFKA/KIP-914%3A+DSL+Processor+Semantics+for+Versioned+Stores],
>  we will make the following four improvements:
>  * stream-table joins will perform a timestamped lookup (using the 
> stream-side record timestamp) if the table is versioned
>  * table-table joins, including foreign key joins, will not produce new join 
> results on out-of-order records (by key) from versioned tables
>  * table filters will disable the existing optimization to not send duplicate 
> tombstones when applied to a versioned table
>  * table aggregations will ignore out-of-order records when aggregating a 
> versioned table



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (KAFKA-14209) Optimize stream stream self join to use single state store

2023-04-12 Thread Matthias J. Sax (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-14209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax updated KAFKA-14209:

Description: 
KIP-862: 
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-862%3A+Self-join+optimization+for+stream-stream+joins]
 

For stream-stream joins that join the same source, we can omit one state store 
since they contain the same data.

  was:For stream-stream joins that join the same source, we can omit one state 
store since they contain the same data.


> Optimize stream stream self join to use single state store
> --
>
> Key: KAFKA-14209
> URL: https://issues.apache.org/jira/browse/KAFKA-14209
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Vicky Papavasileiou
>Assignee: Vicky Papavasileiou
>Priority: Major
>  Labels: kip
> Fix For: 3.4.0
>
>
> KIP-862: 
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-862%3A+Self-join+optimization+for+stream-stream+joins]
>  
> For stream-stream joins that join the same source, we can omit one state 
> store since they contain the same data.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (KAFKA-14209) Optimize stream stream self join to use single state store

2023-04-12 Thread Matthias J. Sax (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-14209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax updated KAFKA-14209:

Labels: kip  (was: )

> Optimize stream stream self join to use single state store
> --
>
> Key: KAFKA-14209
> URL: https://issues.apache.org/jira/browse/KAFKA-14209
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Vicky Papavasileiou
>Assignee: Vicky Papavasileiou
>Priority: Major
>  Labels: kip
> Fix For: 3.4.0
>
>
> For stream-stream joins that join the same source, we can omit one state 
> store since they contain the same data.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Fwd: [VOTE] KIP-914 Join Processor Semantics for Versioned Stores

2023-04-11 Thread Matthias J. Sax

If we send old and new value as two messages, this should work I guess? 
Victory could confirm. -- But not if we send old/new as a single message 
in case the new-key does not change?


-Matthias

On 4/11/23 5:25 AM, Lucas Brutschy wrote:

Hi,

No concerns at all, just a clarifying question from my side: for
detecting out-of-order records, I need both new and old timestamp, I
suppose I get it for the new record via timestamp extractor, can I not
get it the same way from the old record that is passed down to the
aggregation after KIP-904?

Thanks,
Lucas

On Tue, Apr 11, 2023 at 5:35 AM Matthias J. Sax  wrote:


Thanks.

One question: for the repartition topic format change, do we want to
re-use flag=2, or should we introduce flag=3, and determine when
compiling the DSL into the Topology if we want/need to include the
timestamp, and if not, use format version=2 to avoid unnecessary overhead?


-Matthias

On 4/10/23 5:47 PM, Victoria Xia wrote:

Hi everyone,

While wrapping up the implementation for KIP-914, I have discovered that
two more DSL processors require semantic updates in the presence of
versioned tables:

 - The table filter processor has an optimization to drop nulls if the
 previous filtered value is also null. When the upstream table is versioned,
 this optimization should be disabled in order to preserve proper version
 history in the presence of out-of-order data.
 - When performing an aggregation over a versioned table, only the latest
 value by timestamp (per key) should be included in the final aggregate
 value. This is not happening today in the presence of out-of-order data,
 due to the way that TableSourceNodes call `get(key)` in order to determine
 the "old value" which is to be removed from the aggregate as part of
 applying an update. To fix this, aggregations should ignore out-of-order
 records when aggregating versioned tables.
- In order to implement this change, table aggregate processors need
a way to determine whether a record is out-of-order or not. This
cannot be
done by querying the source table value getter as that store belongs to 
a
different subtopology (because a repartition occurs before
aggregation). As
such, an additional timestamp must be included in the repartition topic.
The 3.5 release already includes an update to the repartition
topic format
(with upgrade implications properly handled) via KIP-904

<https://cwiki.apache.org/confluence/display/KAFKA/KIP-904%3A+Kafka+Streams+-+Guarantee+subtractor+is+called+before+adder+if+key+has+not+changed>,
so making an additional change to the repartition topic format to add a
timestamp comes at no additional cost to users.


I have updated the KIP
<https://cwiki.apache.org/confluence/display/KAFKA/KIP-914%3A+DSL+Processor+Semantics+for+Versioned+Stores>
itself with more detail about each of these changes. Please let me know if
there are any concerns. In the absence of dissent, I'd like to include
these changes along with the rest of KIP-914 in the 3.5 release.

Apologies for not noticing these additional semantics implications earlier,
Victoria

-- Forwarded message -
From: Victoria Xia 
Date: Wed, Mar 22, 2023 at 10:08 AM
Subject: Re: [VOTE] KIP-914 Join Processor Semantics for Versioned Stores
To: 


Thanks for voting, everyone! We have three binding yes votes with no
objections during four full days of voting. I will close the vote and mark
the KIP as accepted, right in time for the 3.5 release.

Thanks,
Victoria

On Wed, Mar 22, 2023 at 7:11 AM Bruno Cadonna  wrote:


+1 (binding)

Thanks Victoria!

Best,
Bruno

On 20.03.23 17:13, Matthias J. Sax wrote:

+1 (binding)

On 3/20/23 9:05 AM, Guozhang Wang wrote:

+1, thank you Victoria!

On Sat, Mar 18, 2023 at 8:27 AM Victoria Xia
 wrote:


Hi all,

I'd like to start a vote on KIP-914 for updating the Kafka Streams join
processors to use proper timestamp-based semantics in applications with
versioned stores:


https://cwiki.apache.org/confluence/display/KAFKA/KIP-914%3A+Join+Processor+Semantics+for+Versioned+Stores


To avoid compatibility concerns, I'd like to include the changes from
this
KIP together with KIP-889
<

https://cwiki.apache.org/confluence/display/KAFKA/KIP-889%3A+Versioned+State+Stores



(for introducing versioned stores) in the upcoming 3.5 release. I will
close the vote on the 3.5 KIP deadline, March 22, if there are no
objections before then.

Thanks,
Victoria

Re: Fwd: [VOTE] KIP-914 Join Processor Semantics for Versioned Stores

2023-04-10 Thread Matthias J. Sax


Thanks.

One question: for the repartition topic format change, do we want to 
re-use flag=2, or should we introduce flag=3, and determine when 
compiling the DSL into the Topology if we want/need to include the 
timestamp, and if not, use format version=2 to avoid unnecessary overhead?



-Matthias

On 4/10/23 5:47 PM, Victoria Xia wrote:

Hi everyone,

While wrapping up the implementation for KIP-914, I have discovered that
two more DSL processors require semantic updates in the presence of
versioned tables:

- The table filter processor has an optimization to drop nulls if the
previous filtered value is also null. When the upstream table is versioned,
this optimization should be disabled in order to preserve proper version
history in the presence of out-of-order data.
- When performing an aggregation over a versioned table, only the latest
value by timestamp (per key) should be included in the final aggregate
value. This is not happening today in the presence of out-of-order data,
due to the way that TableSourceNodes call `get(key)` in order to determine
the "old value" which is to be removed from the aggregate as part of
applying an update. To fix this, aggregations should ignore out-of-order
records when aggregating versioned tables.
   - In order to implement this change, table aggregate processors need
   a way to determine whether a record is out-of-order or not. This
cannot be
   done by querying the source table value getter as that store belongs to a
   different subtopology (because a repartition occurs before
aggregation). As
   such, an additional timestamp must be included in the repartition topic.
   The 3.5 release already includes an update to the repartition
topic format
   (with upgrade implications properly handled) via KIP-904
   
<https://cwiki.apache.org/confluence/display/KAFKA/KIP-904%3A+Kafka+Streams+-+Guarantee+subtractor+is+called+before+adder+if+key+has+not+changed>,
   so making an additional change to the repartition topic format to add a
   timestamp comes at no additional cost to users.


I have updated the KIP
<https://cwiki.apache.org/confluence/display/KAFKA/KIP-914%3A+DSL+Processor+Semantics+for+Versioned+Stores>
itself with more detail about each of these changes. Please let me know if
there are any concerns. In the absence of dissent, I'd like to include
these changes along with the rest of KIP-914 in the 3.5 release.

Apologies for not noticing these additional semantics implications earlier,
Victoria

-- Forwarded message -
From: Victoria Xia 
Date: Wed, Mar 22, 2023 at 10:08 AM
Subject: Re: [VOTE] KIP-914 Join Processor Semantics for Versioned Stores
To: 


Thanks for voting, everyone! We have three binding yes votes with no
objections during four full days of voting. I will close the vote and mark
the KIP as accepted, right in time for the 3.5 release.

Thanks,
Victoria

On Wed, Mar 22, 2023 at 7:11 AM Bruno Cadonna  wrote:


+1 (binding)

Thanks Victoria!

Best,
Bruno

On 20.03.23 17:13, Matthias J. Sax wrote:

+1 (binding)

On 3/20/23 9:05 AM, Guozhang Wang wrote:

+1, thank you Victoria!

On Sat, Mar 18, 2023 at 8:27 AM Victoria Xia
 wrote:


Hi all,

I'd like to start a vote on KIP-914 for updating the Kafka Streams join
processors to use proper timestamp-based semantics in applications with
versioned stores:


https://cwiki.apache.org/confluence/display/KAFKA/KIP-914%3A+Join+Processor+Semantics+for+Versioned+Stores


To avoid compatibility concerns, I'd like to include the changes from
this
KIP together with KIP-889
<

https://cwiki.apache.org/confluence/display/KAFKA/KIP-889%3A+Versioned+State+Stores



(for introducing versioned stores) in the upcoming 3.5 release. I will
close the vote on the 3.5 KIP deadline, March 22, if there are no
objections before then.

Thanks,
Victoria

[jira] [Assigned] (KAFKA-14054) Unexpected client shutdown as TimeoutException is thrown as IllegalStateException

2023-04-10 Thread Matthias J. Sax (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-14054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax reassigned KAFKA-14054:
---

Assignee: Matthias J. Sax

> Unexpected client shutdown as TimeoutException is thrown as 
> IllegalStateException
> -
>
> Key: KAFKA-14054
> URL: https://issues.apache.org/jira/browse/KAFKA-14054
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 3.1.0, 3.2.0, 3.1.1
>Reporter: Donald
>    Assignee: Matthias J. Sax
>Priority: Major
>
>  Re: 
> https://forum.confluent.io/t/bug-timeoutexception-is-thrown-as-illegalstateexception-causing-client-shutdown/5460/2
> 1) TimeoutException is thrown as IllegalStateException in 
> {_}org.apache.kafka.streams.processor.internals.StreamTask#commitNeeded{_}. 
> Which causes the client to shutdown in 
> {_}org.apache.kafka.streams.KafkaStreams#getActionForThrowable{_}.
> 2) Should Timeout be a recoverable error which is expected to be handled by 
> User?
> 3) This issue is exposed by change KAFKA-12887 which was introduced in 
> kafka-streams ver 3.1.0
> *code referenced*
> {code:java|title=org.apache.kafka.streams.processor.internals.StreamTask#commitNeeded}
> public boolean commitNeeded() {
> if (commitNeeded) {
> return true;
> } else {
> for (final Map.Entry entry : 
> consumedOffsets.entrySet()) {
> final TopicPartition partition = entry.getKey();
> try {
> final long offset = mainConsumer.position(partition);
> if (offset > entry.getValue() + 1) {
> commitNeeded = true;
> entry.setValue(offset - 1);
> }
> } catch (final TimeoutException error) {
> // the `consumer.position()` call should never block, 
> because we know that we did process data
> // for the requested partition and thus the consumer 
> should have a valid local position
> // that it can return immediately
> // hence, a `TimeoutException` indicates a bug and thus 
> we rethrow it as fatal `IllegalStateException`
> throw new IllegalStateException(error);
> } catch (final KafkaException fatal) {
> throw new StreamsException(fatal);
> }
> }
> return commitNeeded;
> }
> }
> {code}
> {code:java|title=org.apache.kafka.streams.KafkaStreams#getActionForThrowable}
> private StreamsUncaughtExceptionHandler.StreamThreadExceptionResponse 
> getActionForThrowable(final Throwable throwable,
>   
>   final StreamsUncaughtExceptionHandler 
> streamsUncaughtExceptionHandler) {
> final StreamsUncaughtExceptionHandler.StreamThreadExceptionResponse 
> action;
> if (wrappedExceptionIsIn(throwable, 
> EXCEPTIONS_NOT_TO_BE_HANDLED_BY_USERS)) {
> action = SHUTDOWN_CLIENT;
> } else {
> action = streamsUncaughtExceptionHandler.handle(throwable);
> }
> return action;
> }
> private void handleStreamsUncaughtException(final Throwable throwable,
> final 
> StreamsUncaughtExceptionHandler streamsUncaughtExceptionHandler,
> final boolean 
> skipThreadReplacement) {
> final StreamsUncaughtExceptionHandler.StreamThreadExceptionResponse 
> action = getActionForThrowable(throwable, streamsUncaughtExceptionHandler);
> if (oldHandler) {
> log.warn("Stream's new uncaught exception handler is set as well 
> as the deprecated old handler." +
> "The old handler will be ignored as long as a new handler 
> is set.");
> }
> switch (action) {
> case REPLACE_THREAD:
> if (!skipThreadReplacement) {
> log.error("Replacing thread in the streams uncaught 
> exception handler", throwable);
> replaceStreamThread(throwable);
> } else {
> log.debug("Skipping thread replacement for recoverable 
> error");
> }
> break;
> case SHUTDOWN_CLIENT:
> log.error("En

[jira] [Reopened] (KAFKA-14318) KIP-878: Autoscaling for Statically Partitioned Streams



 [ 
https://issues.apache.org/jira/browse/KAFKA-14318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax reopened KAFKA-14318:
-

> KIP-878: Autoscaling for Statically Partitioned Streams
> ---
>
> Key: KAFKA-14318
> URL: https://issues.apache.org/jira/browse/KAFKA-14318
> Project: Kafka
>  Issue Type: New Feature
>  Components: streams
>Reporter: A. Sophie Blee-Goldman
>Priority: Major
>  Labels: kip
>
> [KIP-878: Autoscaling for Statically Partitioned 
> Streams|https://cwiki.apache.org/confluence/display/KAFKA/KIP-878%3A+Autoscaling+for+Statically+Partitioned+Streams]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-14318) KIP-878: Autoscaling for Statically Partitioned Streams



 [ 
https://issues.apache.org/jira/browse/KAFKA-14318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-14318.
-
Resolution: Fixed

> KIP-878: Autoscaling for Statically Partitioned Streams
> ---
>
> Key: KAFKA-14318
> URL: https://issues.apache.org/jira/browse/KAFKA-14318
> Project: Kafka
>  Issue Type: New Feature
>  Components: streams
>Reporter: A. Sophie Blee-Goldman
>Priority: Major
>  Labels: kip
>
> [KIP-878: Autoscaling for Statically Partitioned 
> Streams|https://cwiki.apache.org/confluence/display/KAFKA/KIP-878%3A+Autoscaling+for+Statically+Partitioned+Streams]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-14318) KIP-878: Autoscaling for Statically Partitioned Streams



 [ 
https://issues.apache.org/jira/browse/KAFKA-14318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-14318.
-
Resolution: Fixed

> KIP-878: Autoscaling for Statically Partitioned Streams
> ---
>
> Key: KAFKA-14318
> URL: https://issues.apache.org/jira/browse/KAFKA-14318
> Project: Kafka
>  Issue Type: New Feature
>  Components: streams
>Reporter: A. Sophie Blee-Goldman
>Priority: Major
>  Labels: kip
>
> [KIP-878: Autoscaling for Statically Partitioned 
> Streams|https://cwiki.apache.org/confluence/display/KAFKA/KIP-878%3A+Autoscaling+for+Statically+Partitioned+Streams]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Reopened] (KAFKA-14318) KIP-878: Autoscaling for Statically Partitioned Streams



 [ 
https://issues.apache.org/jira/browse/KAFKA-14318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax reopened KAFKA-14318:
-

> KIP-878: Autoscaling for Statically Partitioned Streams
> ---
>
> Key: KAFKA-14318
> URL: https://issues.apache.org/jira/browse/KAFKA-14318
> Project: Kafka
>  Issue Type: New Feature
>  Components: streams
>Reporter: A. Sophie Blee-Goldman
>Priority: Major
>  Labels: kip
>
> [KIP-878: Autoscaling for Statically Partitioned 
> Streams|https://cwiki.apache.org/confluence/display/KAFKA/KIP-878%3A+Autoscaling+for+Statically+Partitioned+Streams]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Assigned] (KAFKA-14318) KIP-878: Autoscaling for Statically Partitioned Streams



 [ 
https://issues.apache.org/jira/browse/KAFKA-14318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax reassigned KAFKA-14318:
---

Assignee: (was: A. Sophie Blee-Goldman)

> KIP-878: Autoscaling for Statically Partitioned Streams
> ---
>
> Key: KAFKA-14318
> URL: https://issues.apache.org/jira/browse/KAFKA-14318
> Project: Kafka
>  Issue Type: New Feature
>  Components: streams
>Reporter: A. Sophie Blee-Goldman
>Priority: Major
>  Labels: kip
>
> [KIP-878: Autoscaling for Statically Partitioned 
> Streams|https://cwiki.apache.org/confluence/display/KAFKA/KIP-878%3A+Autoscaling+for+Statically+Partitioned+Streams]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (KAFKA-14318) KIP-878: Autoscaling for Statically Partitioned Streams



 [ 
https://issues.apache.org/jira/browse/KAFKA-14318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax updated KAFKA-14318:

Fix Version/s: (was: 3.5.0)

> KIP-878: Autoscaling for Statically Partitioned Streams
> ---
>
> Key: KAFKA-14318
> URL: https://issues.apache.org/jira/browse/KAFKA-14318
> Project: Kafka
>  Issue Type: New Feature
>  Components: streams
>Reporter: A. Sophie Blee-Goldman
>Assignee: A. Sophie Blee-Goldman
>Priority: Major
>  Labels: kip
>
> [KIP-878: Autoscaling for Statically Partitioned 
> Streams|https://cwiki.apache.org/confluence/display/KAFKA/KIP-878%3A+Autoscaling+for+Statically+Partitioned+Streams]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-14491) Introduce Versioned Key-Value Stores to Kafka Streams

2023-04-05 Thread Matthias J. Sax (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-14491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-14491.
-
Fix Version/s: 3.5.0
   Resolution: Fixed

> Introduce Versioned Key-Value Stores to Kafka Streams
> -
>
> Key: KAFKA-14491
> URL: https://issues.apache.org/jira/browse/KAFKA-14491
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Victoria Xia
>Assignee: Victoria Xia
>Priority: Major
>  Labels: kip, streams
> Fix For: 3.5.0
>
>
> The key-value state stores used by Kafka Streams today maintain only the 
> latest value associated with each key. In order to support applications which 
> require access to older record versions, Kafka Streams should have versioned 
> state stores. Versioned state stores are similar to key-value stores except 
> they can store multiple record versions for a single key. An example use case 
> for versioned key-value stores is in providing proper temporal join semantics 
> for stream-tables joins with regards to out-of-order data.
> See KIP for more: 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-889%3A+Versioned+State+Stores



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-14491) Introduce Versioned Key-Value Stores to Kafka Streams

2023-04-05 Thread Matthias J. Sax (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-14491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-14491.
-
Fix Version/s: 3.5.0
   Resolution: Fixed

> Introduce Versioned Key-Value Stores to Kafka Streams
> -
>
> Key: KAFKA-14491
> URL: https://issues.apache.org/jira/browse/KAFKA-14491
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Victoria Xia
>Assignee: Victoria Xia
>Priority: Major
>  Labels: kip, streams
> Fix For: 3.5.0
>
>
> The key-value state stores used by Kafka Streams today maintain only the 
> latest value associated with each key. In order to support applications which 
> require access to older record versions, Kafka Streams should have versioned 
> state stores. Versioned state stores are similar to key-value stores except 
> they can store multiple record versions for a single key. An example use case 
> for versioned key-value stores is in providing proper temporal join semantics 
> for stream-tables joins with regards to out-of-order data.
> See KIP for more: 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-889%3A+Versioned+State+Stores



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-14864) Memory leak in KStreamWindowAggregate with ON_WINDOW_CLOSE emit strategy



 [ 
https://issues.apache.org/jira/browse/KAFKA-14864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-14864.
-
Fix Version/s: 3.4.1
   3.3.3
   Resolution: Fixed

> Memory leak in KStreamWindowAggregate with ON_WINDOW_CLOSE emit strategy
> 
>
> Key: KAFKA-14864
> URL: https://issues.apache.org/jira/browse/KAFKA-14864
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 3.4.0, 3.3.2
>Reporter: Victoria Xia
>Assignee: Victoria Xia
>Priority: Major
> Fix For: 3.5.0, 3.4.1, 3.3.3
>
>
> The Streams DSL processor implementation for the ON_WINDOW_CLOSE emit 
> strategy during KStream windowed aggregations opens a key-value iterator but 
> does not call `close()` on it 
> ([link|https://github.com/apache/kafka/blob/5afedd9ac37c4d740f47867cfd31eaed15dc542f/streams/src/main/java/org/apache/kafka/streams/kstream/internals/AbstractKStreamTimeWindowAggregateProcessor.java#L203]),
>  despite the Javadocs for the iterator making clear that users must do so in 
> order to release resources 
> ([link|https://github.com/apache/kafka/blob/5afedd9ac37c4d740f47867cfd31eaed15dc542f/streams/src/main/java/org/apache/kafka/streams/state/KeyValueIterator.java#L27]).
>   
> I discovered this bug while running load testing benchmarks and noticed that 
> some runs were sporadically hitting OOMs, so it is definitely possible to hit 
> this in practice.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-14864) Memory leak in KStreamWindowAggregate with ON_WINDOW_CLOSE emit strategy



 [ 
https://issues.apache.org/jira/browse/KAFKA-14864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-14864.
-
Fix Version/s: 3.4.1
   3.3.3
   Resolution: Fixed

> Memory leak in KStreamWindowAggregate with ON_WINDOW_CLOSE emit strategy
> 
>
> Key: KAFKA-14864
> URL: https://issues.apache.org/jira/browse/KAFKA-14864
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 3.4.0, 3.3.2
>Reporter: Victoria Xia
>Assignee: Victoria Xia
>Priority: Major
> Fix For: 3.5.0, 3.4.1, 3.3.3
>
>
> The Streams DSL processor implementation for the ON_WINDOW_CLOSE emit 
> strategy during KStream windowed aggregations opens a key-value iterator but 
> does not call `close()` on it 
> ([link|https://github.com/apache/kafka/blob/5afedd9ac37c4d740f47867cfd31eaed15dc542f/streams/src/main/java/org/apache/kafka/streams/kstream/internals/AbstractKStreamTimeWindowAggregateProcessor.java#L203]),
>  despite the Javadocs for the iterator making clear that users must do so in 
> order to release resources 
> ([link|https://github.com/apache/kafka/blob/5afedd9ac37c4d740f47867cfd31eaed15dc542f/streams/src/main/java/org/apache/kafka/streams/state/KeyValueIterator.java#L27]).
>   
> I discovered this bug while running load testing benchmarks and noticed that 
> some runs were sporadically hitting OOMs, so it is definitely possible to hit 
> this in practice.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (KAFKA-14722) Make BooleanSerde public



[ 
https://issues.apache.org/jira/browse/KAFKA-14722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17708211#comment-17708211
 ] 

Matthias J. Sax commented on KAFKA-14722:
-

The docs PRs was not merged yet – thus the work is not yet completed. We keep 
Jiras open as reminders about this. Docs are as important as the feature itself.

> Make BooleanSerde public
> 
>
> Key: KAFKA-14722
> URL: https://issues.apache.org/jira/browse/KAFKA-14722
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>            Reporter: Matthias J. Sax
>Assignee: Spacrocket
>Priority: Minor
>  Labels: beginner, kip, newbie
>
> KIP-907: 
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-907%3A+Add+Boolean+Serde+to+public+interface]
>  
> We introduce a "BooleanSerde" via 
> [https://github.com/apache/kafka/pull/13249] as internal class. We could make 
> it public.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (KAFKA-14722) Make BooleanSerde public



 [ 
https://issues.apache.org/jira/browse/KAFKA-14722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax updated KAFKA-14722:

Fix Version/s: 3.5.0

> Make BooleanSerde public
> 
>
> Key: KAFKA-14722
> URL: https://issues.apache.org/jira/browse/KAFKA-14722
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>            Reporter: Matthias J. Sax
>Assignee: Spacrocket
>Priority: Minor
>  Labels: beginner, kip, newbie
> Fix For: 3.5.0
>
>
> KIP-907: 
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-907%3A+Add+Boolean+Serde+to+public+interface]
>  
> We introduce a "BooleanSerde" via 
> [https://github.com/apache/kafka/pull/13249] as internal class. We could make 
> it public.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (KAFKA-14864) Memory leak in KStreamWindowAggregate with ON_WINDOW_CLOSE emit strategy



 [ 
https://issues.apache.org/jira/browse/KAFKA-14864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax updated KAFKA-14864:

Affects Version/s: 3.3.2
   3.4.0

> Memory leak in KStreamWindowAggregate with ON_WINDOW_CLOSE emit strategy
> 
>
> Key: KAFKA-14864
> URL: https://issues.apache.org/jira/browse/KAFKA-14864
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 3.4.0, 3.3.2
>Reporter: Victoria Xia
>Assignee: Victoria Xia
>Priority: Major
> Fix For: 3.5.0
>
>
> The Streams DSL processor implementation for the ON_WINDOW_CLOSE emit 
> strategy during KStream windowed aggregations opens a key-value iterator but 
> does not call `close()` on it 
> ([link|https://github.com/apache/kafka/blob/5afedd9ac37c4d740f47867cfd31eaed15dc542f/streams/src/main/java/org/apache/kafka/streams/kstream/internals/AbstractKStreamTimeWindowAggregateProcessor.java#L203]),
>  despite the Javadocs for the iterator making clear that users must do so in 
> order to release resources 
> ([link|https://github.com/apache/kafka/blob/5afedd9ac37c4d740f47867cfd31eaed15dc542f/streams/src/main/java/org/apache/kafka/streams/state/KeyValueIterator.java#L27]).
>   
> I discovered this bug while running load testing benchmarks and noticed that 
> some runs were sporadically hitting OOMs, so it is definitely possible to hit 
> this in practice.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (KAFKA-14847) Separate the callers of commitAllTasks v.s. commitTasks for EOS(-v2) and ALOS

2023-03-25 Thread Matthias J. Sax (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-14847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax updated KAFKA-14847:

Description: 
Today, EOS-v2/v1 and ALOS shares the same internal callpath inside 
TaskManager/TaskExecutor for committing tasks from various scenarios, the call 
path {{commitTasksAndMaybeUpdateCommitableOffsets}} -> 
{{commitOffsetsOrTransaction}} takes in a list of tasks as its input, which can 
be a subset of the tasks that thread / task manager owns. For EOS-v1 / ALOS, 
this is fine to commit just a subset of the tasks; however for EOS-v2, since 
all tasks participate in the same txn it could lead to dangerous violations, 
and today we are relying on all the callers of the commit function to make sure 
that the list of tasks they passed in, under EOS-v1, would still not violate 
the semantics. As summarized today (thanks to Matthias), today that callee 
could be triggered in the following cases:

1) Inside handleRevocation() – this is a clean path, an we add all non-revoked 
tasks with commitNeeded() flag set to the commit – so this seems to be fine.
2) tryCloseCleanAllActiveTasks() – here we only call it, if 
tasksToCloseDirty.isEmpty() – so it seems fine, too.
3) commit() with a list of task handed in – we call commit() inside the TM 
three time
3.a) inside commitAll() as commit(tasks.values()) (passing in all tasks)
3.b) inside maybeCommitActiveTasksPerUserRequested as 
commit(activeTaskIterable()); (passing in all tasks)
3.c) inside handleCorruption() – here, we only consider RUNNING and RESTORING 
tasks, which are not corrupted – note we only throw a TaskCorruptedException 
during restore state initialization, thus, corrupted tasks did not process 
anything yet, and all other tasks should be clean to be committed.
3.d) commitSuccessfullyProcessedTasks() – under ALOS/EOS-v1, as we just commit 
a subset of tasks' source offsets while at the same time we still commit those 
unsuccessful task's outgoing records if there are any. (For EOS-v2 the list of 
tasks should be empty.)

Just going through this list of callers itself, as demonstrated above, is 
already pretty complex, and very vulnerable to bugs. It's better to not rely on 
the callers, but the callees to make sure that's the case. More concretely, I 
think we can introduce a new function called {{commitAllTasks}} such that under 
EOS-v2, the caller always call {{commitAllTasks}} instead, and if there are 
some tasks that should not be committed because we know they have not processed 
any data, the {{commitAllTasks}} callee itself would do some clever filtering 
internally.

Given its scope, I think it's better to do this refactoring after EOS-v1 is 
removed.

  was:
Today, EOS-v2/v1 and ALOS shares the same internal callpath inside 
TaskManager/TaskExecutor for committing tasks from various scenarios, the call 
path {{commitTasksAndMaybeUpdateCommitableOffsets}} -> 
{{commitOffsetsOrTransaction}} takes in a list of tasks as its input, which can 
be a subset of the tasks that thread / task manager owns. For EOS-v1 / ALOS, 
this is fine to commit just a subset of the tasks; however for EOS-v2, since 
all tasks participate in the same txn it could lead to dangerous violations, 
and today we are relying on all the callers of the commit function to make sure 
that the list of tasks they passed in, under EOS-v1, would still not violate 
the semantics. As summarized today (thanks to Matthias), today that callee 
could be triggered in the following cases:

1) Inside handleRevocation() – this is a clean path, an we add all non-revoked 
tasks with commitNeeded() flag set to the commit – so this seems to be fine.
2) tryCloseCleanAllActiveTasks() – here we only call it, if 
tasksToCloseDirty.isEmpty() – so it seems fine, too.
3) commit() with a list of task handed in – we call commit() inside the TM 
three time
3.a) inside commitAll() as commit(tasks.values()) (passing in all tasks)
3.b) inside maybeCommitActiveTasksPerUserRequested as 
commit(activeTaskIterable()); (passing in all tasks)
3.c) inside handleCorruption() – here, we only consider RUNNING and RESTORING 
tasks, which are not corrupted – note we only throw a TaskCorruptedException 
during restore state initialization, thus, corrupted tasks did not process 
anything yet, and all other tasks should be clean to be committed.
3.d) commitSuccessfullyProcessedTasks() – under ALOS/EOS-v1, as we just commit 
a subset of tasks' source offsets while at the same time we still commit those 
unsuccessful task's outgoing records if there are any.

Just going through this list of callers itself, as demonstrated above, is 
already pretty complex, and very vulnerable to bugs. It's better to not rely on 
the callers, but the callees to make sure that's the case. More concretely, I 
think we can introduce a new function called {{commitAllTasks}} such that under 
EOS-v2, the caller alway

[jira] [Updated] (KAFKA-14847) Separate the callers of commitAllTasks v.s. commitTasks for EOS(-v2) and ALOS

2023-03-25 Thread Matthias J. Sax (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-14847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax updated KAFKA-14847:

Description: 
Today, EOS-v2/v1 and ALOS shares the same internal callpath inside 
TaskManager/TaskExecutor for committing tasks from various scenarios, the call 
path {{commitTasksAndMaybeUpdateCommitableOffsets}} -> 
{{commitOffsetsOrTransaction}} takes in a list of tasks as its input, which can 
be a subset of the tasks that thread / task manager owns. For EOS-v1 / ALOS, 
this is fine to commit just a subset of the tasks; however for EOS-v2, since 
all tasks participate in the same txn it could lead to dangerous violations, 
and today we are relying on all the callers of the commit function to make sure 
that the list of tasks they passed in, under EOS-v1, would still not violate 
the semantics. As summarized today (thanks to Matthias), today that callee 
could be triggered in the following cases:

1) Inside handleRevocation() – this is a clean path, an we add all non-revoked 
tasks with commitNeeded() flag set to the commit – so this seems to be fine.
2) tryCloseCleanAllActiveTasks() – here we only call it, if 
tasksToCloseDirty.isEmpty() – so it seems fine, too.
3) commit() with a list of task handed in – we call commit() inside the TM 
three time
3.a) inside commitAll() as commit(tasks.values()) (passing in all tasks)
3.b) inside maybeCommitActiveTasksPerUserRequested as 
commit(activeTaskIterable()); (passing in all tasks)
3.c) inside handleCorruption() – here, we only consider RUNNING and RESTORING 
tasks, which are not corrupted – note we only throw a TaskCorruptedException 
during restore state initialization, thus, corrupted tasks did not process 
anything yet, and all other tasks should be clean to be committed.
3.d) commitSuccessfullyProcessedTasks() – under ALOS/EOS-v1, as we just commit 
a subset of tasks' source offsets while at the same time we still commit those 
unsuccessful task's outgoing records if there are any.

Just going through this list of callers itself, as demonstrated above, is 
already pretty complex, and very vulnerable to bugs. It's better to not rely on 
the callers, but the callees to make sure that's the case. More concretely, I 
think we can introduce a new function called {{commitAllTasks}} such that under 
EOS-v2, the caller always call {{commitAllTasks}} instead, and if there are 
some tasks that should not be committed because we know they have not processed 
any data, the {{commitAllTasks}} callee itself would do some clever filtering 
internally.

Given its scope, I think it's better to do this refactoring after EOS-v1 is 
removed.

  was:
Today, EOS-v2/v1 and ALOS shares the same internal callpath inside 
TaskManager/TaskExecutor for committing tasks from various scenarios, the call 
path {{commitTasksAndMaybeUpdateCommitableOffsets}} -> 
{{commitOffsetsOrTransaction}} takes in a list of tasks as its input, which can 
be a subset of the tasks that thread / task manager owns. For EOS-v1 / ALOS, 
this is fine to commit just a subset of the tasks; however for EOS-v2, since 
all tasks participate in the same txn it could lead to dangerous violations, 
and today we are relying on all the callers of the commit function to make sure 
that the list of tasks they passed in, under EOS-v1, would still not violate 
the semantics. As summarized today (thanks to Matthias), today that callee 
could be triggered in the following cases:

1) Inside handleRevocation() – this is a clean path, an we add all non-revoked 
tasks with commitNeeded() flag set to the commit – so this seems to be fine.
2) tryCloseCleanAllActiveTasks() – here we only call it, if 
tasksToCloseDirty.isEmpty() – so it seems fine, too.
3) commit() with a list of task handed in – we call commit() inside the TM 
three time
3.a) inside commitAll() as commit(tasks.values()) (passing in all tasks)
3.b) inside maybeCommitActiveTasksPerUserRequested as 
commit(activeTaskIterable()); (passing in all tasks)
3.c) inside handleCorruption() – here, we only consider RUNNING and RESTORING 
tasks, which are not corrupted – note we only throw a TaskCorruptedException 
during restore state initialization, thus, corrupted tasks did not process 
anything yet, and all other tasks should be clean to be committed.
3.d) commitSuccessfullyProcessedTasks() – under EOS-v1, as we just commit a 
subset of tasks' source offsets while at the same time we still commit those 
unsuccessful task's outgoing records if there are any.

Just going through this list of callers itself, as demonstrated above, is 
already pretty complex, and very vulnerable to bugs. It's better to not rely on 
the callers, but the callees to make sure that's the case. More concretely, I 
think we can introduce a new function called {{commitAllTasks}} such that under 
EOS-v2, the caller always call {{commitAllTasks}} instead, and if there are 
some tasks that

[jira] [Updated] (KAFKA-14847) Separate the callers of commitAllTasks v.s. commitTasks for EOS(-v2) and ALOS

2023-03-25 Thread Matthias J. Sax (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-14847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax updated KAFKA-14847:

Description: 
Today, EOS-v2/v1 and ALOS shares the same internal callpath inside 
TaskManager/TaskExecutor for committing tasks from various scenarios, the call 
path {{commitTasksAndMaybeUpdateCommitableOffsets}} -> 
{{commitOffsetsOrTransaction}} takes in a list of tasks as its input, which can 
be a subset of the tasks that thread / task manager owns. For EOS-v1 / ALOS, 
this is fine to commit just a subset of the tasks; however for EOS-v2, since 
all tasks participate in the same txn it could lead to dangerous violations, 
and today we are relying on all the callers of the commit function to make sure 
that the list of tasks they passed in, under EOS-v1, would still not violate 
the semantics. As summarized today (thanks to Matthias), today that callee 
could be triggered in the following cases:

1) Inside handleRevocation() – this is a clean path, an we add all non-revoked 
tasks with commitNeeded() flag set to the commit – so this seems to be fine.
2) tryCloseCleanAllActiveTasks() – here we only call it, if 
tasksToCloseDirty.isEmpty() – so it seems fine, too.
3) commit() with a list of task handed in – we call commit() inside the TM 
three time
3.a) inside commitAll() as commit(tasks.values()) (passing in all tasks)
3.b) inside maybeCommitActiveTasksPerUserRequested as 
commit(activeTaskIterable()); (passing in all tasks)
3.c) inside handleCorruption() – here, we only consider RUNNING and RESTORING 
tasks, which are not corrupted – note we only throw a TaskCorruptedException 
during restore state initialization, thus, corrupted tasks did not process 
anything yet, and all other tasks should be clean to be committed.
3.d) commitSuccessfullyProcessedTasks() – under EOS-v1, as we just commit a 
subset of tasks' source offsets while at the same time we still commit those 
unsuccessful task's outgoing records if there are any.

Just going through this list of callers itself, as demonstrated above, is 
already pretty complex, and very vulnerable to bugs. It's better to not rely on 
the callers, but the callees to make sure that's the case. More concretely, I 
think we can introduce a new function called {{commitAllTasks}} such that under 
EOS-v2, the caller always call {{commitAllTasks}} instead, and if there are 
some tasks that should not be committed because we know they have not processed 
any data, the {{commitAllTasks}} callee itself would do some clever filtering 
internally.

Given its scope, I think it's better to do this refactoring after EOS-v1 is 
removed.

  was:
Today, EOS-v2/v1 and ALOS shares the same internal callpath inside 
TaskManager/TaskExecutor for committing tasks from various scenarios, the call 
path {{commitTasksAndMaybeUpdateCommitableOffsets}} -> 
{{commitOffsetsOrTransaction}} takes in a list of tasks as its input, which can 
be a subset of the tasks that thread / task manager owns. For EOS-v1 / ALOS, 
this is fine to commit just a subset of the tasks; however for EOS-v1, since 
all tasks participate in the same txn it could lead to dangerous violations, 
and today we are relying on all the callers of the commit function to make sure 
that the list of tasks they passed in, under EOS-v2, would still not violate 
the semantics. As summarized today (thanks to Matthias), today that callee 
could be triggered in the following cases:

1) Inside handleRevocation() -- this is a clean path, an we add all non-revoked 
tasks with commitNeeded() flag set to the commit -- so this seems to be fine.
2) tryCloseCleanAllActiveTasks() -- here we only call it, if 
tasksToCloseDirty.isEmpty() -- so it seems fine, too.
3) commit() with a list of task handed in -- we call commit() inside the TM 
three time
3.a) inside commitAll() as commit(tasks.values()) (passing in all tasks)
3.b) inside maybeCommitActiveTasksPerUserRequested as 
commit(activeTaskIterable()); (passing in all tasks)
3.c) inside handleCorruption() -- here, we only consider RUNNING and RESTORING 
tasks, which are not corrupted -- note we only throw a TaskCorruptedException 
during restore state initialization, thus, corrupted tasks did not process 
anything yet, and all other tasks should be clean to be committed.
3.d) commitSuccessfullyProcessedTasks() -- under EOS-v2, as we just commit a 
subset of tasks' source offsets while at the same time we still commit those 
unsuccessful task's outgoing records if there are any.

Just going through this list of callers itself, as demonstrated above, is 
already pretty complex, and very vulnerable to bugs. It's better to not rely on 
the callers, but the callees to make sure that's the case. More concretely, I 
think we can introduce a new function called {{commitAllTasks}} such that under 
EOS-v2, the caller always call {{commitAllTasks}} instead, and if there are 
some

[jira] [Created] (KAFKA-14839) Exclude protected variable from JavaDocs

2023-03-23 Thread Matthias J. Sax (Jira)

Matthias J. Sax created KAFKA-14839:
---

 Summary: Exclude protected variable from JavaDocs
 Key: KAFKA-14839
 URL: https://issues.apache.org/jira/browse/KAFKA-14839
 Project: Kafka
  Issue Type: Bug
  Components: documentation, streams
Reporter: Matthias J. Sax


Cf 
[https://kafka.apache.org/31/javadoc/org/apache/kafka/streams/kstream/JoinWindows.html#enableSpuriousResultFix]

The variable `enableSpuriousResultFix` is protected, and it's not public API, 
and thus should not show up in the JavaDocs.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14839) Exclude protected variable from JavaDocs

2023-03-23 Thread Matthias J. Sax (Jira)

Matthias J. Sax created KAFKA-14839:
---

 Summary: Exclude protected variable from JavaDocs
 Key: KAFKA-14839
 URL: https://issues.apache.org/jira/browse/KAFKA-14839
 Project: Kafka
  Issue Type: Bug
  Components: documentation, streams
Reporter: Matthias J. Sax


Cf 
[https://kafka.apache.org/31/javadoc/org/apache/kafka/streams/kstream/JoinWindows.html#enableSpuriousResultFix]

The variable `enableSpuriousResultFix` is protected, and it's not public API, 
and thus should not show up in the JavaDocs.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Request small bug or minor issue

2023-03-21 Thread Matthias J. Sax


Thanks for your interest.

Please checkout https://kafka.apache.org/contributing to get started.

You can look for tickets labeled "newbie" or "beginner": 
https://issues.apache.org/jira/browse/KAFKA-8971?jql=project%20%3D%20KAFKA%20AND%20resolution%20%3D%20Unresolved%20AND%20labels%20in%20(newbie%2C%20beginner)%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC



-Matthias

On 3/20/23 3:32 PM, Sina Kashipazha wrote:

Hey there,

I'm Sina, just another software engineer :-)

I've been utilizing Kafka for some time now and am interested in 
contributing to the project. Do you happen to have any minor issues or 
bugs that need attention.



Kind regard,
Sina

Re: can Kafka streams support ordering across 2 different topics when consuming from multiple source topics?

2023-03-21 Thread Matthias J. Sax

In general there is no ordering guarantee between topics. So it might 
depend a lot ofnthe details of your use case.


For example, if you know that it will be always two event, you could 
buffer the first one in a state-store, and wait for the second one to 
arrive and decide in which order to forward both events downstream for 
actual processing.



HTH, Matthias


On 3/20/23 11:57 PM, Pushkar Deole wrote:

Hi All,

We have a kafka streams application that consumes from 2 different topics
say topic A and topic B. The application uses data of telephone call on
those topics and each call has a call id which is used as key to send
events to those 2 topics. e.g. for a telephone call, the 1st event related
to that call is sent to A with call id however subsequent event for that
same call might go to topic B again with call id as key.

*At times, we need to process those 2 events in an order, which is not
possible with the current topology that we are using*. *Can someone suggest
if this is possible to achieve with streams?*
The topology is as below:

Topic A has 6 partitions
Topics B has 6 partitions
Call id used as key on both topics
Kafka streams application has 3 instances that consumes from both of the
topics as source topics.
Each streams application instance has 2 stream threads thus total 6 stream
threads across 3 instances of streams application cater to 6 partitions of
inputs topics.

[jira] [Commented] (KAFKA-7224) KIP-328: Add spill-to-disk for Suppression

2023-03-20 Thread Matthias J. Sax (Jira)



[ 
https://issues.apache.org/jira/browse/KAFKA-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17702980#comment-17702980
 ] 

Matthias J. Sax commented on KAFKA-7224:


With 
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-825%3A+introduce+a+new+API+to+control+when+aggregated+results+are+produced]
 added to 3.3, so we still want/need this one?

> KIP-328: Add spill-to-disk for Suppression
> --
>
> Key: KAFKA-7224
> URL: https://issues.apache.org/jira/browse/KAFKA-7224
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: John Roesler
>Priority: Major
>
> As described in 
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-328%3A+Ability+to+suppress+updates+for+KTables]
> Following on KAFKA-7223, implement the spill-to-disk buffering strategy.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-914 Join Processor Semantics for Versioned Stores

2023-03-20 Thread Matthias J. Sax


+1 (binding)

On 3/20/23 9:05 AM, Guozhang Wang wrote:

+1, thank you Victoria!

On Sat, Mar 18, 2023 at 8:27 AM Victoria Xia
 wrote:


Hi all,

I'd like to start a vote on KIP-914 for updating the Kafka Streams join
processors to use proper timestamp-based semantics in applications with
versioned stores:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-914%3A+Join+Processor+Semantics+for+Versioned+Stores

To avoid compatibility concerns, I'd like to include the changes from this
KIP together with KIP-889

(for introducing versioned stores) in the upcoming 3.5 release. I will
close the vote on the 3.5 KIP deadline, March 22, if there are no
objections before then.

Thanks,
Victoria

Re: [DISCUSS] KIP-914 Join Processor Semantics for Versioned Stores

2023-03-15 Thread Matthias J. Sax

Thanks for the KIP! Great to see a first step towards using the new
versioned stores!

I think the described tradeoffs make sense and I like make a pragmatic
step into the right direction, and avoid boiling the ocean. Thus, I
agree to the proposed solution.

One minor thing, that I believe just need clarification in the KIP (does
not seem to be a change to the KIP itself):

For stream-table joins, I think we need to elaborate that a `get(k, ts)`
call now might return `null` if the history retention of the store is
too short. For inner-joins it would result in no output record (ie,
stream input record is dropped). Would be good to have it mentioned in
the KIP explicitly.

We should also discuss how left-joins should work for this case. I think
it's ok (better) to include the stream record in the result if the
lookup returns `null` -- either because no key exist in the exiting
history for the provided timestamp, or (the actual case in question)
because we query older than available history. If you agree, can we add
this to the KIP?

For left-table-table joins, there seems to be no special impact, but it
should be called out, too. The lookup itself does not go into the
history of the table so no change here (as we don't have the "query
older than history case") -- and for out-of-order records, we just
"drop" them anyway, so no change for left-joins either I believe.

-Matthias

On 3/15/23 2:00 PM, Guozhang Wang wrote:

Sounds good to me. Thanks!

On Wed, Mar 15, 2023 at 12:07 PM Victoria Xia
wrote:

Thanks for kicking off the discussion, John and Guozhang!

Just one thing that might be out of scope: if users want to enable the

versioned table feature across the topology, should we allow them to do it
via a single config rather than changing the materialized object at each
place?

Yes, I think this would be a great usability improvement and am in favor of
introducing such a config. As long as the config defaults to using
unversioned stores (which makes sense anyway), there will be no
compatibility concerns with introducing the config in a future release.
It's out of scope for this particular KIP as a result, but can hopefully be
introduced as part of the next release after 3.5.

Best,
Victoria

On Wed, Mar 15, 2023 at 10:49 AM Guozhang Wang
wrote:

Thanks Victoria for the great writeup, with a thorough analysis and
trade-offs. I do not have any major questions about the proposal.

Just one thing that might be out of scope: if users want to enable the
versioned table feature across the topology, should we allow them to
do it via a single config rather than changing the materialized object
at each place? Maybe we can defer that for future discussions, but
just want to hear your thoughts.

Anyways, I think this proposal is great just as-is even if we agree to
do the configuration improvement later.

Guozhang

On Thu, Mar 9, 2023 at 7:52 PM John Roesler wrote:

Thanks for the KIP, Victoria!

I had some questions/concerns, but you addressed them in the Rejected

Alternatives section. Thanks for the thorough proposal!

-John

On Thu, Mar 9, 2023, at 18:59, Victoria Xia wrote:

Hi everyone,

I have a proposal for updating Kafka Streams's stream-table join and
table-table join semantics for the new versioned key-value state stores
introduced in KIP-889
<

https://cwiki.apache.org/confluence/display/KAFKA/KIP-889%3A+Versioned+State+Stores

Would love to hear your thoughts and suggestions.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-914%3A+Join+Processor+Semantics+for+Versioned+Stores

Thanks,
Victoria

[jira] [Updated] (KAFKA-14385) Flaky Test QueryableStateIntegrationTest.shouldNotMakeStoreAvailableUntilAllStoresAvailable



 [ 
https://issues.apache.org/jira/browse/KAFKA-14385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax updated KAFKA-14385:

Priority: Critical  (was: Major)

> Flaky Test 
> QueryableStateIntegrationTest.shouldNotMakeStoreAvailableUntilAllStoresAvailable
> ---
>
> Key: KAFKA-14385
> URL: https://issues.apache.org/jira/browse/KAFKA-14385
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: A. Sophie Blee-Goldman
>Priority: Critical
>  Labels: flaky-test
>
> Failed twice on the same build (Java 8 & 11)
> h3. Stacktrace
> java.lang.AssertionError: KafkaStreams did not transit to RUNNING state 
> within 15000 milli seconds. Expected:  but: was  at 
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20) at 
> org.apache.kafka.test.StreamsTestUtils.startKafkaStreamsAndWaitForRunningState(StreamsTestUtils.java:134)
>  at 
> org.apache.kafka.test.StreamsTestUtils.startKafkaStreamsAndWaitForRunningState(StreamsTestUtils.java:121)
>  at 
> org.apache.kafka.streams.integration.QueryableStateIntegrationTest.shouldNotMakeStoreAvailableUntilAllStoresAvailable(QueryableStateIntegrationTest.java:1038)
>  
> https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-12836/3/testReport/org.apache.kafka.streams.integration/QueryableStateIntegrationTest/Build___JDK_11_and_Scala_2_13___shouldNotMakeStoreAvailableUntilAllStoresAvailable/



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (KAFKA-14384) Flaky Test SelfJoinUpgradeIntegrationTest.shouldUpgradeWithTopologyOptimizationOff



 [ 
https://issues.apache.org/jira/browse/KAFKA-14384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax updated KAFKA-14384:

Priority: Critical  (was: Major)

> Flaky Test 
> SelfJoinUpgradeIntegrationTest.shouldUpgradeWithTopologyOptimizationOff
> --
>
> Key: KAFKA-14384
> URL: https://issues.apache.org/jira/browse/KAFKA-14384
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: A. Sophie Blee-Goldman
>Priority: Critical
>  Labels: flaky-test
>
> h3. Stacktrace
> java.lang.AssertionError: Did not receive all 5 records from topic 
> selfjoin-outputSelfJoinUpgradeIntegrationTestshouldUpgradeWithTopologyOptimizationOff
>  within 6 ms Expected: is a value equal to or greater than <5> but: <0> 
> was less than <5> at 
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20) at 
> org.apache.kafka.streams.integration.utils.IntegrationTestUtils.lambda$waitUntilMinKeyValueWithTimestampRecordsReceived$2(IntegrationTestUtils.java:763)
>  at 
> org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:382)
>  at 
> org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:350)
>  at 
> org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueWithTimestampRecordsReceived(IntegrationTestUtils.java:759)
>  at 
> org.apache.kafka.streams.integration.SelfJoinUpgradeIntegrationTest.processKeyValueAndVerifyCount(SelfJoinUpgradeIntegrationTest.java:244)
>  at 
> org.apache.kafka.streams.integration.SelfJoinUpgradeIntegrationTest.shouldUpgradeWithTopologyOptimizationOff(SelfJoinUpgradeIntegrationTest.java:155)
>  
> https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-12835/4/testReport/org.apache.kafka.streams.integration/SelfJoinUpgradeIntegrationTest/Build___JDK_11_and_Scala_2_13___shouldUpgradeWithTopologyOptimizationOff/



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (KAFKA-10184) Flaky HighAvailabilityTaskAssignorIntegrationTest.shouldScaleOutWithWarmupTasksAndPersistentStores



 [ 
https://issues.apache.org/jira/browse/KAFKA-10184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax updated KAFKA-10184:

Priority: Critical  (was: Minor)

> Flaky 
> HighAvailabilityTaskAssignorIntegrationTest.shouldScaleOutWithWarmupTasksAndPersistentStores
> --
>
> Key: KAFKA-10184
> URL: https://issues.apache.org/jira/browse/KAFKA-10184
> Project: Kafka
>  Issue Type: Test
>  Components: streams, unit tests
>Reporter: Guozhang Wang
>Assignee: John Roesler
>Priority: Critical
>
> {code}
> Stacktrace
> java.lang.AssertionError: Condition not met within timeout 12. Input 
> records haven't all been written to the changelog: 442
>   at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:26)
>   at 
> org.apache.kafka.test.TestUtils.lambda$waitForCondition$6(TestUtils.java:401)
>   at 
> org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:449)
>   at 
> org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:417)
>   at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:398)
>   at 
> org.apache.kafka.streams.integration.HighAvailabilityTaskAssignorIntegrationTest.shouldScaleOutWithWarmupTasks(HighAvailabilityTaskAssignorIntegrationTest.java:149)
>   at 
> org.apache.kafka.streams.integration.HighAvailabilityTaskAssignorIntegrationTest.shouldScaleOutWithWarmupTasksAndPersistentStores(HighAvailabilityTaskAssignorIntegrationTest.java:91)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
>   at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
>   at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
>   at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
>   at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
>   at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>   at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
>   at 
> org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
>   at 
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
>   at 
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch

[jira] [Updated] (KAFKA-8691) Flakey test ProcessorContextTest#shouldNotAllowToScheduleZeroMillisecondPunctuation



 [ 
https://issues.apache.org/jira/browse/KAFKA-8691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax updated KAFKA-8691:
---
Priority: Critical  (was: Major)

> Flakey test  
> ProcessorContextTest#shouldNotAllowToScheduleZeroMillisecondPunctuation
> 
>
> Key: KAFKA-8691
> URL: https://issues.apache.org/jira/browse/KAFKA-8691
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Boyang Chen
>Priority: Critical
>
> [https://builds.apache.org/job/kafka-pr-jdk11-scala2.12/6384/consoleFull]
> org.apache.kafka.streams.processor.internals.ProcessorContextTest > 
> shouldNotAllowToScheduleZeroMillisecondPunctuation PASSED*23:37:09* ERROR: 
> Failed to write output for test null.Gradle Test Executor 5*23:37:09* 
> java.lang.NullPointerException: Cannot invoke method write() on null 
> object*23:37:09*at 
> org.codehaus.groovy.runtime.NullObject.invokeMethod(NullObject.java:91)*23:37:09*
> at 
> org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:47)*23:37:09*
>  at 
> org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)*23:37:09*
>   at 
> org.codehaus.groovy.runtime.callsite.NullCallSite.call(NullCallSite.java:34)*23:37:09*
>at 
> org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)*23:37:09*
>   at java_io_FileOutputStream$write.call(Unknown Source)*23:37:09*
> at 
> build_5nv3fyjgqff9aim9wbxfnad9z$_run_closure5$_closure75$_closure108.doCall(/home/jenkins/jenkins-slave/workspace/kafka-pr-jdk11-scala2.12/build.gradle:244)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (KAFKA-14533) Flaky Test SmokeTestDriverIntegrationTest.shouldWorkWithRebalance



 [ 
https://issues.apache.org/jira/browse/KAFKA-14533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax updated KAFKA-14533:

Priority: Critical  (was: Major)

> Flaky Test SmokeTestDriverIntegrationTest.shouldWorkWithRebalance
> -
>
> Key: KAFKA-14533
> URL: https://issues.apache.org/jira/browse/KAFKA-14533
> Project: Kafka
>  Issue Type: Test
>  Components: streams, unit tests
>Reporter: Greg Harris
>Assignee: Guozhang Wang
>Priority: Critical
>  Labels: flaky-test
>
> The SmokeTestDriverIntegrationTest appears to be flakey failing in recent 
> runs:
> ```
>     
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka/detail/trunk/1444/tests/
>         java.util.concurrent.TimeoutException: 
> shouldWorkWithRebalance(boolean) timed out after 600 seconds
>     
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka/detail/trunk/1443/tests/
>         java.util.concurrent.TimeoutException: 
> shouldWorkWithRebalance(boolean) timed out after 600 seconds
>     
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka/detail/trunk/1441/tests/
>         java.util.concurrent.TimeoutException: 
> shouldWorkWithRebalance(boolean) timed out after 600 seconds
>     
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka/detail/trunk/1440/tests/
>         java.util.concurrent.TimeoutException: 
> shouldWorkWithRebalance(boolean) timed out after 600 seconds
>     
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka/detail/trunk/1438/tests/
>         java.util.concurrent.TimeoutException: 
> shouldWorkWithRebalance(boolean) timed out after 600 seconds
>     
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka/detail/trunk/1434/tests/
>         java.util.concurrent.TimeoutException: 
> shouldWorkWithRebalance(boolean) timed out after 600 seconds
> ```
> The stacktrace appears to be:
> ```
> java.util.concurrent.TimeoutException: shouldWorkWithRebalance(boolean) timed 
> out after 600 seconds
>  at 
> org.junit.jupiter.engine.extension.TimeoutExceptionFactory.create(TimeoutExceptionFactory.java:29)
>  at 
> org.junit.jupiter.engine.extension.SameThreadTimeoutInvocation.proceed(SameThreadTimeoutInvocation.java:58)
>  at 
> org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
> ...
>  Suppressed: java.lang.InterruptedException: sleep interrupted
>  at java.lang.Thread.sleep(Native Method)
>  at 
> org.apache.kafka.streams.integration.SmokeTestDriverIntegrationTest.shouldWorkWithRebalance(SmokeTestDriverIntegrationTest.java:151)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>  at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>  at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:498)
>  at 
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)
>  at 
> org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
>  at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
>  at 
> org.junit.jupiter.engine.extension.SameThreadTimeoutInvocation.proceed(SameThreadTimeoutInvocation.java:45)
>  ... 134 more
> ```
> The test appears to be timing out waiting for the SmokeTestClient to complete 
> its asynchronous close, and taking significantly longer to do so (600s 
> instead of 60s) than a typical local test execution time.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Assigned] (KAFKA-10688) Handle accidental truncation of repartition topics as exceptional failure

2023-03-13 Thread Matthias J. Sax (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-10688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax reassigned KAFKA-10688:
---

Assignee: (was: Guozhang Wang)

> Handle accidental truncation of repartition topics as exceptional failure
> -
>
> Key: KAFKA-10688
> URL: https://issues.apache.org/jira/browse/KAFKA-10688
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Guozhang Wang
>Priority: Major
>  Labels: new-streams-runtime-should-fix
>
> Today we always handle InvalidOffsetException from the main consumer by the 
> resetting policy assuming they are for source topics. But repartition topics 
> are also source topics and should never be truncated and hence cause 
> InvalidOffsetException.
> We should differentiate these repartition topics from external source topics 
> and treat the InvalidOffsetException from repartition topics as fatal and 
> close the whole application.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] New Kafka PMC Member: Chris Egerton

2023-03-09 Thread Matthias J. Sax


Congrats!

On 3/9/23 2:59 PM, José Armando García Sancio wrote:

Congrats Chris.

On Thu, Mar 9, 2023 at 2:01 PM Kowshik Prakasam  wrote:


Congrats Chris!

On Thu, Mar 9, 2023 at 1:33 PM Divij Vaidya  wrote:


Congratulations Chris! I am in awe with the amount of effort you put in
code reviews and helping out the community members. Very well deserved.

--
Divij Vaidya



On Thu, Mar 9, 2023 at 9:49 PM Jorge Esteban Quilcate Otoya <
quilcate.jo...@gmail.com> wrote:


So well deserved! Congratulations Chris!!!

On Thu, 9 Mar 2023 at 22:09, Lucas Brutschy 
wrote:


Congratulations!

On Thu, Mar 9, 2023 at 8:48 PM Roman Schmitz 
wrote:


Congratulations Chris!

Am Do., 9. März 2023 um 20:33 Uhr schrieb Chia-Ping Tsai <

chia7...@gmail.com

:



Congratulations Chris!


Mickael Maison  於 2023年3月10日 上午2:21

寫道：


Congratulations Chris!


On Thu, Mar 9, 2023 at 7:17 PM Bill Bejeck 

wrote:


Congratulations Chris!


On Thu, Mar 9, 2023 at 1:12 PM Jun Rao




wrote:


Hi, Everyone,

Chris Egerton has been a Kafka committer since July 2022. He

has

been

very

instrumental to the community since becoming a committer. It's

my

pleasure

to announce that Chris is now a member of Kafka PMC.

Congratulations Chris!

Jun
on behalf of Apache Kafka PMC

Re: [ANNOUNCE] New Kafka PMC Member: Chris Egerton

2023-03-09 Thread Matthias J. Sax


Congrats!

On 3/9/23 2:59 PM, José Armando García Sancio wrote:

Congrats Chris.

On Thu, Mar 9, 2023 at 2:01 PM Kowshik Prakasam  wrote:


Congrats Chris!

On Thu, Mar 9, 2023 at 1:33 PM Divij Vaidya  wrote:


Congratulations Chris! I am in awe with the amount of effort you put in
code reviews and helping out the community members. Very well deserved.

--
Divij Vaidya



On Thu, Mar 9, 2023 at 9:49 PM Jorge Esteban Quilcate Otoya <
quilcate.jo...@gmail.com> wrote:


So well deserved! Congratulations Chris!!!

On Thu, 9 Mar 2023 at 22:09, Lucas Brutschy 
wrote:


Congratulations!

On Thu, Mar 9, 2023 at 8:48 PM Roman Schmitz 
wrote:


Congratulations Chris!

Am Do., 9. März 2023 um 20:33 Uhr schrieb Chia-Ping Tsai <

chia7...@gmail.com

:



Congratulations Chris!


Mickael Maison  於 2023年3月10日 上午2:21

寫道：


Congratulations Chris!


On Thu, Mar 9, 2023 at 7:17 PM Bill Bejeck 

wrote:


Congratulations Chris!


On Thu, Mar 9, 2023 at 1:12 PM Jun Rao




wrote:


Hi, Everyone,

Chris Egerton has been a Kafka committer since July 2022. He

has

been

very

instrumental to the community since becoming a committer. It's

my

pleasure

to announce that Chris is now a member of Kafka PMC.

Congratulations Chris!

Jun
on behalf of Apache Kafka PMC

Re: [ANNOUNCE] New Kafka PMC Member: David Arthur

2023-03-09 Thread Matthias J. Sax


Congrats!

On 3/9/23 2:59 PM, José Armando García Sancio wrote:

Congrats David!

On Thu, Mar 9, 2023 at 2:00 PM Kowshik Prakasam  wrote:


Congrats David!

On Thu, Mar 9, 2023 at 12:09 PM Lucas Brutschy
 wrote:


Congratulations!

On Thu, Mar 9, 2023 at 8:37 PM Manikumar 
wrote:


Congrats David!


On Fri, Mar 10, 2023 at 12:24 AM Josep Prat 

Congrats David!

———
Josep Prat

Aiven Deutschland GmbH

Alexanderufer 3-7, 10117 Berlin

Amtsgericht Charlottenburg, HRB 209739 B

Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen

m: +491715557497

w: aiven.io

e: josep.p...@aiven.io

On Thu, Mar 9, 2023, 19:22 Mickael Maison 

wrote:



Congratulations David!

On Thu, Mar 9, 2023 at 7:20 PM Chris Egerton 


wrote:


Congrats David!

On Thu, Mar 9, 2023 at 1:17 PM Bill Bejeck 

wrote:



Congratulations David!

On Thu, Mar 9, 2023 at 1:12 PM Jun Rao 


wrote:



Hi, Everyone,

David Arthur has been a Kafka committer since 2013. He has been

very

instrumental to the community since becoming a committer. It's

my

pleasure

to announce that David is now a member of Kafka PMC.

Congratulations David!

Jun
on behalf of Apache Kafka PMC

Re: [ANNOUNCE] New Kafka PMC Member: David Arthur

2023-03-09 Thread Matthias J. Sax


Congrats!

On 3/9/23 2:59 PM, José Armando García Sancio wrote:

Congrats David!

On Thu, Mar 9, 2023 at 2:00 PM Kowshik Prakasam  wrote:


Congrats David!

On Thu, Mar 9, 2023 at 12:09 PM Lucas Brutschy
 wrote:


Congratulations!

On Thu, Mar 9, 2023 at 8:37 PM Manikumar 
wrote:


Congrats David!


On Fri, Mar 10, 2023 at 12:24 AM Josep Prat 

Congrats David!

———
Josep Prat

Aiven Deutschland GmbH

Alexanderufer 3-7, 10117 Berlin

Amtsgericht Charlottenburg, HRB 209739 B

Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen

m: +491715557497

w: aiven.io

e: josep.p...@aiven.io

On Thu, Mar 9, 2023, 19:22 Mickael Maison 

wrote:



Congratulations David!

On Thu, Mar 9, 2023 at 7:20 PM Chris Egerton 


wrote:


Congrats David!

On Thu, Mar 9, 2023 at 1:17 PM Bill Bejeck 

wrote:



Congratulations David!

On Thu, Mar 9, 2023 at 1:12 PM Jun Rao 


wrote:



Hi, Everyone,

David Arthur has been a Kafka committer since 2013. He has been

very

instrumental to the community since becoming a committer. It's

my

pleasure

to announce that David is now a member of Kafka PMC.

Congratulations David!

Jun
on behalf of Apache Kafka PMC

[jira] [Assigned] (KAFKA-4969) State-store workload-aware StreamsPartitionAssignor

2023-03-07 Thread Matthias J. Sax (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-4969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax reassigned KAFKA-4969:
--

Assignee: Bill Bejeck

> State-store workload-aware StreamsPartitionAssignor
> ---
>
> Key: KAFKA-4969
> URL: https://issues.apache.org/jira/browse/KAFKA-4969
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams
>            Reporter: Matthias J. Sax
>Assignee: Bill Bejeck
>Priority: Major
> Fix For: 2.6.0
>
>
> Currently, {{StreamPartitionsAssigner}} does not distinguish different 
> "types" of tasks. For example, task can be stateless of have one or multiple 
> stores.
> This can lead to an suboptimal task placement: assume there are 2 stateless 
> and 2 stateful tasks and the app is running with 2 instances. To share the 
> "store load" it would be good to place one stateless and one stateful task 
> per instance. Right now, there is no guarantee about this, and it can happen, 
> that one instance processed both stateless tasks while the other processes 
> both stateful tasks.
> We should improve {{StreamPartitionAssignor}} and introduce "task types" 
> including a cost model for task placement. We should consider the following 
> parameters:
>  - number of stores
>  - number of sources/sinks
>  - number of processors
>  - regular task vs standby task
>  - in the case of standby tasks, which tasks have progressed the most with 
> respect to restoration
> This improvement should be backed by a design document in the project wiki 
> (no KIP required though) as it's a fairly complex change.
>  
> There have been some additional discussions around task assignment on a 
> related PR https://github.com/apache/kafka/pull/5390



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (KAFKA-14778) Kafka Streams 2.7.1 to 3.3.1 rolling upgrade with static membership triggers a rebalance



[ 
https://issues.apache.org/jira/browse/KAFKA-14778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17697181#comment-17697181
 ] 

Matthias J. Sax commented on KAFKA-14778:
-

Thanks for reporting this. – I believe the issue is as follows:
 * on restart, the consumer sends join group request (with laters subscription 
version)
 * broker returns buffered assignment without a rebalance (as it should)
 * KS inspects assignment and observes lower subscription – now things go 
"wrong"
 ** KS thinks that the assignment is invalid (because the leader should have 
sent an empty assignment on version probing only encoding it's version number)
 ** KS triggers a new rebalance sending it's subscription with lower 
subscription-version (as it assumes the leader could not decode the first 
subscription it sent)

It seem the fix might be to check if static membership is enabled and if the 
received assignment is empty of not. If static membership is enabled and if the 
assignment is not empty, it's not necessary to trigger a new rebalance.

One drawback is that all KS instances would stay on the old assignment, thus it 
might actually be desirable to have a single rebalance that allows all 
instances to switch to the new subscription/assignment version. This single 
rebalance should only happen _after_ all instances got updated. However, it's 
unclear how we could trigger such an rebalance, and it's also an open question 
if this rebalance is really desired or not?

> Kafka Streams 2.7.1 to 3.3.1 rolling upgrade with static membership triggers 
> a rebalance
> 
>
> Key: KAFKA-14778
> URL: https://issues.apache.org/jira/browse/KAFKA-14778
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 2.7.1, 3.3.1
>Reporter: Vinoth Rengarajan
>Priority: Major
>
> Trying to upgrade Kaka Streams application from 2.7.1 to 3.3.1 with static 
> membership but it triggers a rebalance
> Brokers are running on Kafka 2.7.1. Enabled the static membership in the 
> application. Below are the configs {*}(Stream Config & Consumer Config){*}.
> Followed below steps to upgrade
>  * Brokers are running on Kafka 2.7.1(tried with 3.3.1 version then also 
> rebalance happens).
>  * Application is running with 2.7.1 Kafka streams libraries.
>  * Deployed the latest version of the application with 3.3.1 Kafka streams 
> libraries, and configured the *upgrade.from* property to 2.7 (based on the 
> upgrade documentation available here 
> [https://kafka.apache.org/33/documentation/streams/upgrade-guide]).
>  * Doing a rolling bounce with the latest changes, rebalance is being 
> triggered on other instances in the cluster.
> Below are logs on the instance which is being bounced, forcing a rebalance on 
> others. 
> *Logs:*
>  
> {code:java}
> INFO  2023-02-27 09:52:16.805 | streams.KafkaStreams stream-client 
> [kafka_upgrade.Kafka_Upgrade_Test] State transition from CREATED to 
> REBALANCING
> INFO  2023-02-27 09:52:16.946 | internals.ConsumerCoordinator [Consumer 
> instanceId=kafka_upgrade.Kafka_Upgrade_Test-4, 
> clientId=kafka_upgrade.Kafka_Upgrade_Test-StreamThread-4-consumer, 
> groupId=kafka_upgrade.Kafka_Upgrade_Test] Notifying assignor about the new 
> Assignment(partitions=[kafka_upgrade.Kafka_Upgrade_Test-version-updates-11, 
> kafka_upgrade.Kafka_Upgrade_Test-version-updates-23], userDataSize=56)
> INFO  2023-02-27 09:52:16.947 | internals.StreamsPartitionAssignor 
> stream-thread [kafka_upgrade.Kafka_Upgrade_Test-StreamThread-3-consumer] Sent 
> a version 11 subscription and got version 8 assignment back (successful 
> version probing). Downgrade subscription metadata to commonly supported 
> version 8 and trigger new rebalance.
> INFO  2023-02-27 09:52:16.947 | internals.StreamsPartitionAssignor 
> stream-thread [kafka_upgrade.Kafka_Upgrade_Test-StreamThread-2-consumer] Sent 
> a version 11 subscription and got version 8 assignment back (successful 
> version probing). Downgrade subscription metadata to commonly supported 
> version 8 and trigger new rebalance.
> INFO  2023-02-27 09:52:16.947 | internals.StreamsPartitionAssignor 
> stream-thread [kafka_upgrade.Kafka_Upgrade_Test-StreamThread-4-consumer] Sent 
> a version 11 subscription and got version 8 assignment back (successful 
> version probing). Downgrade subscription metadata to commonly supported 
> version 8 and trigger new rebalance.
> INFO  2023-02-27 09:52:16.947 | internals.StreamsPartitionAssignor 
> stream-thread [kafka_upgrade.Kafka_Upgrade_Test-StreamThread-1-consumer] Sent 
> a version 11 subscription a

[jira] [Updated] (KAFKA-14722) Make BooleanSerde public



 [ 
https://issues.apache.org/jira/browse/KAFKA-14722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax updated KAFKA-14722:

Description: 
KIP-907: 
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-907%3A+Add+Boolean+Serde+to+public+interface]

 

We introduce a "BooleanSerde" via [https://github.com/apache/kafka/pull/13249] 
as internal class. We could make it public.

  was:We introduce a "BooleanSerde" via 
[https://github.com/apache/kafka/pull/13249] as internal class. We could make 
it public.


> Make BooleanSerde public
> 
>
> Key: KAFKA-14722
> URL: https://issues.apache.org/jira/browse/KAFKA-14722
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Matthias J. Sax
>Assignee: Spacrocket
>Priority: Minor
>  Labels: beginner, kip, newbie
>
> KIP-907: 
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-907%3A+Add+Boolean+Serde+to+public+interface]
>  
> We introduce a "BooleanSerde" via 
> [https://github.com/apache/kafka/pull/13249] as internal class. We could make 
> it public.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (KAFKA-14722) Make BooleanSerde public



 [ 
https://issues.apache.org/jira/browse/KAFKA-14722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax updated KAFKA-14722:

Labels: beginner kip newbie  (was: beginner need-kip newbie)

> Make BooleanSerde public
> 
>
> Key: KAFKA-14722
> URL: https://issues.apache.org/jira/browse/KAFKA-14722
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>            Reporter: Matthias J. Sax
>Assignee: Spacrocket
>Priority: Minor
>  Labels: beginner, kip, newbie
>
> We introduce a "BooleanSerde" via 
> [https://github.com/apache/kafka/pull/13249] as internal class. We could make 
> it public.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (KAFKA-12446) Define KGroupedTable#aggregate subtractor + adder order of execution



 [ 
https://issues.apache.org/jira/browse/KAFKA-12446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax updated KAFKA-12446:

Description: 
KIP-904: 
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-904%3A+Kafka+Streams+-+Guarantee+subtractor+is+called+before+adder+if+key+has+not+changed]
 

 

Currently, when an update is processed by KGroupedTable#aggregate, the 
subtractor is called first, then the adder. But per the docs the order of 
execution is not defined (ie. could change in future releases).

[https://kafka.apache.org/26/documentation/streams/developer-guide/dsl-api.html#streams-developer-guide-dsl-aggregating]
{quote}When subsequent non-null values are received for a key (e.g., UPDATE), 
then (1) the subtractor is called with the old value as stored in the table and 
(2) the adder is called with the new value of the input record that was just 
received. The order of execution for the subtractor and adder is not defined.
{quote}
This ticket proposes making the current order of execution part of the public 
contract.

That would allow Kafka Streams DSL users the freedom to use aggregates such as: 
{code:java}
aggregate(
  HashMap::new,
  (aggKey, newValue, aggValue) -> { aggValue.put(newValue.getKey(), 
newValue.getValue() }, // adder
  (aggKey, oldValue, aggValue) -> { aggValue.remove(newValue.getKey() } // 
subtractor
){code}
and handle updates where key remains the same but value changes.

The Kafka Music Example at

[https://github.com/confluentinc/kafka-streams-examples/blob/6.0.1-post/src/main/java/io/confluent/examples/streams/interactivequeries/kafkamusic/KafkaMusicExample.java#L345]

relies on the subtractor being called first.

 

See discussion at 
[https://github.com/confluentinc/kafka-streams-examples/issues/380]

See also the more general point made at 
[https://stackoverflow.com/questions/65888756/clarify-the-order-of-execution-for-the-subtractor-and-adder-is-not-defined]
 
{quote}If the adder and subtractor are non-commutative operations and the order 
in which they are executed can vary, you can end up with different results 
depending on the order of execution of adder and subtractor. An example of a 
useful non-commutative operation would be something like if we’re aggregating 
records into a Set:{color:#172b4d} {color}
{quote}
{code:java}
.aggregate[Set[Animal]](Set.empty)(
 adder = (zooKey, animalValue, setOfAnimals) => setOfAnimals + animalValue,
 subtractor = (zooKey, animalValue, setOfAnimals) => setOfAnimals - animalValue
)
{code}
{quote}In this example, for duplicated events, if the adder is called before 
the subtractor you would end up removing the value entirely from the set (which 
would be problematic for most use-cases I imagine).
{quote}
As [~mjsax] notes on 
[https://github.com/confluentinc/kafka-streams-examples/issues/380]

 
{quote}the implementation used the same order since 0.10.0 release and it was 
never changed
{quote}
so making this behavior part of the standard amounts to making official what 
has already been stable for a long time.

Cost:
 *  Limits your options for the future. If you ever needed Kafka Streams to 
change the order of execution (or make that order indeterminate instead of its 
current hard coded order), you would have to make that a breaking change.

Benefit:
 * Encourages wider use of the KGroupedTable#aggregate method (current lack of 
a defined order prevents using aggregate with non-commutative adder/subtractor 
functions)
 * Simplifies reasoning about how to use KGroupedTable#aggregate (knowing that 
a given order can be relied upon makes the method itself easier to understand)

 

 

 

  was:
Currently, when an update is processed by KGroupedTable#aggregate, the 
subtractor is called first, then the adder. But per the docs the order of 
execution is not defined (ie. could change in future releases).

[https://kafka.apache.org/26/documentation/streams/developer-guide/dsl-api.html#streams-developer-guide-dsl-aggregating]
{quote}When subsequent non-null values are received for a key (e.g., UPDATE), 
then (1) the subtractor is called with the old value as stored in the table and 
(2) the adder is called with the new value of the input record that was just 
received. The order of execution for the subtractor and adder is not defined.
{quote}
This ticket proposes making the current order of execution part of the public 
contract.

That would allow Kafka Streams DSL users the freedom to use aggregates such as: 
{code:java}
aggregate(
  HashMap::new,
  (aggKey, newValue, aggValue) -> { aggValue.put(newValue.getKey(), 
newValue.getValue() }, // adder
  (aggKey, oldValue, aggValue) -> { aggValue.remove(newValue.getKey() } // 
subtractor
){code}
and handle updates where key remains the same but value changes.

The Kafka Music Example at

[https://github.com/confluentinc/kafka-streams-examples/blob/6.0.1-post/src/main/java/io/

[jira] [Updated] (KAFKA-12446) Define KGroupedTable#aggregate subtractor + adder order of execution



 [ 
https://issues.apache.org/jira/browse/KAFKA-12446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax updated KAFKA-12446:

Labels: kip  (was: )

> Define KGroupedTable#aggregate subtractor + adder order of execution
> 
>
> Key: KAFKA-12446
> URL: https://issues.apache.org/jira/browse/KAFKA-12446
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Ben Ellis
>Assignee: Ben Ellis
>Priority: Minor
>  Labels: kip
>
> Currently, when an update is processed by KGroupedTable#aggregate, the 
> subtractor is called first, then the adder. But per the docs the order of 
> execution is not defined (ie. could change in future releases).
> [https://kafka.apache.org/26/documentation/streams/developer-guide/dsl-api.html#streams-developer-guide-dsl-aggregating]
> {quote}When subsequent non-null values are received for a key (e.g., UPDATE), 
> then (1) the subtractor is called with the old value as stored in the table 
> and (2) the adder is called with the new value of the input record that was 
> just received. The order of execution for the subtractor and adder is not 
> defined.
> {quote}
> This ticket proposes making the current order of execution part of the public 
> contract.
> That would allow Kafka Streams DSL users the freedom to use aggregates such 
> as: 
> {code:java}
> aggregate(
>   HashMap::new,
>   (aggKey, newValue, aggValue) -> { aggValue.put(newValue.getKey(), 
> newValue.getValue() }, // adder
>   (aggKey, oldValue, aggValue) -> { aggValue.remove(newValue.getKey() } // 
> subtractor
> ){code}
> and handle updates where key remains the same but value changes.
> The Kafka Music Example at
> [https://github.com/confluentinc/kafka-streams-examples/blob/6.0.1-post/src/main/java/io/confluent/examples/streams/interactivequeries/kafkamusic/KafkaMusicExample.java#L345]
> relies on the subtractor being called first.
>  
> See discussion at 
> [https://github.com/confluentinc/kafka-streams-examples/issues/380]
> See also the more general point made at 
> [https://stackoverflow.com/questions/65888756/clarify-the-order-of-execution-for-the-subtractor-and-adder-is-not-defined]
>  
> {quote}If the adder and subtractor are non-commutative operations and the 
> order in which they are executed can vary, you can end up with different 
> results depending on the order of execution of adder and subtractor. An 
> example of a useful non-commutative operation would be something like if 
> we’re aggregating records into a Set:{color:#172b4d} {color}
> {quote}
> {code:java}
> .aggregate[Set[Animal]](Set.empty)(
>  adder = (zooKey, animalValue, setOfAnimals) => setOfAnimals + animalValue,
>  subtractor = (zooKey, animalValue, setOfAnimals) => setOfAnimals - 
> animalValue
> )
> {code}
> {quote}In this example, for duplicated events, if the adder is called before 
> the subtractor you would end up removing the value entirely from the set 
> (which would be problematic for most use-cases I imagine).
> {quote}
> As [~mjsax] notes on 
> [https://github.com/confluentinc/kafka-streams-examples/issues/380]
>  
> {quote}the implementation used the same order since 0.10.0 release and it was 
> never changed
> {quote}
> so making this behavior part of the standard amounts to making official what 
> has already been stable for a long time.
> Cost:
>  *  Limits your options for the future. If you ever needed Kafka Streams to 
> change the order of execution (or make that order indeterminate instead of 
> its current hard coded order), you would have to make that a breaking change.
> Benefit:
>  * Encourages wider use of the KGroupedTable#aggregate method (current lack 
> of a defined order prevents using aggregate with non-commutative 
> adder/subtractor functions)
>  * Simplifies reasoning about how to use KGroupedTable#aggregate (knowing 
> that a given order can be relied upon makes the method itself easier to 
> understand)
>  
>  
> 
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (KAFKA-14748) Relax non-null FK left-join requirement



[ 
https://issues.apache.org/jira/browse/KAFKA-14748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17696313#comment-17696313
 ] 

Matthias J. Sax commented on KAFKA-14748:
-

Yes, the behavior would change. But make this change is the goal, isn't it? – 
That is also why I brought up the KIP question – if we apply a change in 
behavior, we might need a KIP.

The other question was: if we apply with change of behavior (even if we do a 
KIP), and there are users who want to old behavior can they still get it. And I 
think the answer is yes, via upstream filtering.

> Relax non-null FK left-join requirement
> ---
>
> Key: KAFKA-14748
> URL: https://issues.apache.org/jira/browse/KAFKA-14748
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>            Reporter: Matthias J. Sax
>Priority: Major
>
> Kafka Streams enforces a strict non-null-key policy in the DSL across all 
> key-dependent operations (like aggregations and joins).
> This also applies to FK-joins, in particular to the ForeignKeyExtractor. If 
> it returns `null`, it's treated as invalid. For left-joins, it might make 
> sense to still accept a `null`, and add the left-hand record with an empty 
> right-hand-side to the result.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (KAFKA-9234) Consider using @Nullable and @Nonnull annotations



[ 
https://issues.apache.org/jira/browse/KAFKA-9234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17696306#comment-17696306
 ] 

Matthias J. Sax commented on KAFKA-9234:


Thanks for the pointer – I am not familiar with JSpecify – let me take a look.

Overall, this ticket has broader impact, and while we don't need a KIP, we 
should make a broader decision as it affects Kafka holistically. \cc [~ijuma] 
[~guozhang] [~hachikuji] [~ChrisEgerton] 

Should we maybe have a discussion on the dev mailing list about it?

> Consider using @Nullable and @Nonnull annotations
> -
>
> Key: KAFKA-9234
> URL: https://issues.apache.org/jira/browse/KAFKA-9234
> Project: Kafka
>  Issue Type: Improvement
>  Components: admin, clients, consumer, KafkaConnect, producer , 
> streams, streams-test-utils
>        Reporter: Matthias J. Sax
>Assignee: Ganesh Sahu
>Priority: Minor
>  Labels: beginner, newbie
>
> Java7 was dropped some time ago, and we might want to consider usein Java8 
> `@Nullable` and `@Nonnull` annotations for all public facing APIs instead of 
> documenting it in JavaDocs only.
> This tickets should be broken down in a series of smaller PRs to keep the 
> scope of each PR contained, allowing for more effective reviews.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (KAFKA-9234) Consider using @Nullable and @Nonnull annotations



[ 
https://issues.apache.org/jira/browse/KAFKA-9234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17696249#comment-17696249
 ] 

Matthias J. Sax commented on KAFKA-9234:


{quote}I just joined the community this week.
{quote}
Welcome!
{quote} I noticed there hasn't been any activity on this, so thought of taking 
it
{quote}
Thanks!
{quote}Can you please advise on how I can reassign the task to myself? Do I 
need to obtain contributor access?
{quote}
FIxed. Just added you as a "contributor" to Jira and assigned the ticket to 
you. You can now also self-assign tickets.

> Consider using @Nullable and @Nonnull annotations
> -
>
> Key: KAFKA-9234
> URL: https://issues.apache.org/jira/browse/KAFKA-9234
> Project: Kafka
>  Issue Type: Improvement
>  Components: admin, clients, consumer, KafkaConnect, producer , 
> streams, streams-test-utils
>Reporter: Matthias J. Sax
>Assignee: Ganesh Sahu
>Priority: Minor
>  Labels: beginner, newbie
>
> Java7 was dropped some time ago, and we might want to consider usein Java8 
> `@Nullable` and `@Nonnull` annotations for all public facing APIs instead of 
> documenting it in JavaDocs only.
> This tickets should be broken down in a series of smaller PRs to keep the 
> scope of each PR contained, allowing for more effective reviews.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Assigned] (KAFKA-9234) Consider using @Nullable and @Nonnull annotations



 [ 
https://issues.apache.org/jira/browse/KAFKA-9234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax reassigned KAFKA-9234:
--

Assignee: Ganesh Sahu  (was: Manasvi Gupta)

> Consider using @Nullable and @Nonnull annotations
> -
>
> Key: KAFKA-9234
> URL: https://issues.apache.org/jira/browse/KAFKA-9234
> Project: Kafka
>  Issue Type: Improvement
>  Components: admin, clients, consumer, KafkaConnect, producer , 
> streams, streams-test-utils
>        Reporter: Matthias J. Sax
>Assignee: Ganesh Sahu
>Priority: Minor
>  Labels: beginner, newbie
>
> Java7 was dropped some time ago, and we might want to consider usein Java8 
> `@Nullable` and `@Nonnull` annotations for all public facing APIs instead of 
> documenting it in JavaDocs only.
> This tickets should be broken down in a series of smaller PRs to keep the 
> scope of each PR contained, allowing for more effective reviews.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (KAFKA-14748) Relax non-null FK left-join requirement

2023-03-02 Thread Matthias J. Sax (Jira)



[ 
https://issues.apache.org/jira/browse/KAFKA-14748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17695945#comment-17695945
 ] 

Matthias J. Sax commented on KAFKA-14748:
-

{quote}But for table-table FK-joins, today the former case would emit while the 
latter case would not?
{quote}
Cannot follow. What two cases do you mean?

> Relax non-null FK left-join requirement
> ---
>
> Key: KAFKA-14748
> URL: https://issues.apache.org/jira/browse/KAFKA-14748
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>            Reporter: Matthias J. Sax
>Priority: Major
>
> Kafka Streams enforces a strict non-null-key policy in the DSL across all 
> key-dependent operations (like aggregations and joins).
> This also applies to FK-joins, in particular to the ForeignKeyExtractor. If 
> it returns `null`, it's treated as invalid. For left-joins, it might make 
> sense to still accept a `null`, and add the left-hand record with an empty 
> right-hand-side to the result.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (KAFKA-14747) FK join should record discarded subscription responses

2023-03-02 Thread Matthias J. Sax (Jira)



[ 
https://issues.apache.org/jira/browse/KAFKA-14747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17695848#comment-17695848
 ] 

Matthias J. Sax commented on KAFKA-14747:
-

You would create a branch from `trunk` – for more info, read the "Developer 
Info" on the Kafka web page: [https://kafka.apache.org/project]

If course, if you have more question, just let us know.

> FK join should record discarded subscription responses
> --
>
> Key: KAFKA-14747
> URL: https://issues.apache.org/jira/browse/KAFKA-14747
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Matthias J. Sax
>Assignee: Koma Zhang
>Priority: Minor
>  Labels: beginner, newbie
>
> FK-joins are subject to a race condition: If the left-hand side record is 
> updated, a subscription is sent to the right-hand side (including a hash 
> value of the left-hand side record), and the right-hand side might send back 
> join responses (also including the original hash). The left-hand side only 
> processed the responses if the returned hash matches to current hash of the 
> left-hand side record, because a different hash implies that the lef- hand 
> side record was updated in the mean time (including sending a new 
> subscription to the right hand side), and thus the data is stale and the 
> response should not be processed (joining the response to the new record 
> could lead to incorrect results).
> A similar thing can happen on a right-hand side update that triggers a 
> response, that might be dropped if the left-hand side record was updated in 
> parallel.
> While the behavior is correct, we don't record if this happens. We should 
> consider to record this using the existing "dropped record" sensor or maybe 
> add a new sensor.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (KAFKA-9234) Consider using @Nullable and @Nonnull annotations

2023-03-02 Thread Matthias J. Sax (Jira)



[ 
https://issues.apache.org/jira/browse/KAFKA-9234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17695846#comment-17695846
 ] 

Matthias J. Sax commented on KAFKA-9234:


 [~rndgstn] did look into the PRs - maybe he knows best? – In general, yes, 
contributions are welcome to address this.

> Consider using @Nullable and @Nonnull annotations
> -
>
> Key: KAFKA-9234
> URL: https://issues.apache.org/jira/browse/KAFKA-9234
> Project: Kafka
>  Issue Type: Improvement
>  Components: admin, clients, consumer, KafkaConnect, producer , 
> streams, streams-test-utils
>        Reporter: Matthias J. Sax
>Assignee: Manasvi Gupta
>Priority: Minor
>  Labels: beginner, newbie
>
> Java7 was dropped some time ago, and we might want to consider usein Java8 
> `@Nullable` and `@Nonnull` annotations for all public facing APIs instead of 
> documenting it in JavaDocs only.
> This tickets should be broken down in a series of smaller PRs to keep the 
> scope of each PR contained, allowing for more effective reviews.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-904: Kafka Streams - Guarantee subtractor is called before adder if key has not changed

2023-03-01 Thread Matthias J. Sax


+1 (binding)

Thanks for the KIP!

On 3/1/23 10:58 AM, Walker Carlson wrote:

+1 Binding

On Mon, Feb 27, 2023 at 12:48 PM Guozhang Wang 
wrote:


+1.

On Sun, Feb 26, 2023 at 4:27 PM Fq Public  wrote:


Hi everyone,

I'd like to start the vote on KIP-904: Kafka Streams - Guarantee

subtractor

is called before adder if key has not changed.
The KIP is available here: https://cwiki.apache.org/confluence/x/P5VbDg
The easiest way to view the entire discussion thread is via this search
link: https://lists.apache.org/list?dev@kafka.apache.org:lte=1M:KIP-904
Please take a look and vote.

Thank you,
Farooq

Re: [VOTE] KIP-907: Add Boolean Serde to public interface

2023-03-01 Thread Matthias J. Sax


+1 (binding)

Thanks for the KIP!

On 3/1/23 10:59 AM, Walker Carlson wrote:

+1 Binding

On Mon, Feb 27, 2023 at 1:46 PM Chia-Ping Tsai  wrote:


+1 (binding)

[jira] [Assigned] (KAFKA-14533) Flaky Test SmokeTestDriverIntegrationTest.shouldWorkWithRebalance

2023-02-28 Thread Matthias J. Sax (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-14533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax reassigned KAFKA-14533:
---

Assignee: Guozhang Wang

> Flaky Test SmokeTestDriverIntegrationTest.shouldWorkWithRebalance
> -
>
> Key: KAFKA-14533
> URL: https://issues.apache.org/jira/browse/KAFKA-14533
> Project: Kafka
>  Issue Type: Test
>  Components: streams, unit tests
>Reporter: Greg Harris
>Assignee: Guozhang Wang
>Priority: Major
>  Labels: flaky-test
>
> The SmokeTestDriverIntegrationTest appears to be flakey failing in recent 
> runs:
> ```
>     
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka/detail/trunk/1444/tests/
>         java.util.concurrent.TimeoutException: 
> shouldWorkWithRebalance(boolean) timed out after 600 seconds
>     
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka/detail/trunk/1443/tests/
>         java.util.concurrent.TimeoutException: 
> shouldWorkWithRebalance(boolean) timed out after 600 seconds
>     
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka/detail/trunk/1441/tests/
>         java.util.concurrent.TimeoutException: 
> shouldWorkWithRebalance(boolean) timed out after 600 seconds
>     
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka/detail/trunk/1440/tests/
>         java.util.concurrent.TimeoutException: 
> shouldWorkWithRebalance(boolean) timed out after 600 seconds
>     
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka/detail/trunk/1438/tests/
>         java.util.concurrent.TimeoutException: 
> shouldWorkWithRebalance(boolean) timed out after 600 seconds
>     
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka/detail/trunk/1434/tests/
>         java.util.concurrent.TimeoutException: 
> shouldWorkWithRebalance(boolean) timed out after 600 seconds
> ```
> The stacktrace appears to be:
> ```
> java.util.concurrent.TimeoutException: shouldWorkWithRebalance(boolean) timed 
> out after 600 seconds
>  at 
> org.junit.jupiter.engine.extension.TimeoutExceptionFactory.create(TimeoutExceptionFactory.java:29)
>  at 
> org.junit.jupiter.engine.extension.SameThreadTimeoutInvocation.proceed(SameThreadTimeoutInvocation.java:58)
>  at 
> org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
> ...
>  Suppressed: java.lang.InterruptedException: sleep interrupted
>  at java.lang.Thread.sleep(Native Method)
>  at 
> org.apache.kafka.streams.integration.SmokeTestDriverIntegrationTest.shouldWorkWithRebalance(SmokeTestDriverIntegrationTest.java:151)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>  at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>  at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:498)
>  at 
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)
>  at 
> org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
>  at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
>  at 
> org.junit.jupiter.engine.extension.SameThreadTimeoutInvocation.proceed(SameThreadTimeoutInvocation.java:45)
>  ... 134 more
> ```
> The test appears to be timing out waiting for the SmokeTestClient to complete 
> its asynchronous close, and taking significantly longer to do so (600s 
> instead of 60s) than a typical local test execution time.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Kafka Streams 2.7.1 to 3.3.1 rolling upgrade

2023-02-27 Thread Matthias J. Sax


Hmmm... that's interesting...

It seems that Kafka Streams "version probing" does not play well static 
group membership...


Sounds like a "bug" to me -- well, more like a missing integration. Not 
sure right now, if/how we could fix it.


Can you file a ticket?

For now, I don't think you can do anything about it. Sorry. :(


-Matthias



On 2/27/23 6:50 AM, Vinoth Rengarajan wrote:

Hi Team,

I am trying to upgrade my Kaka Streams application from 2.7.1 to 3.3.1.
Brokers are running on Kafka 2.7.1. The plan is to upgrade the clients
first and then then brokers

I have already enabled the static membership in our application so that we
I am not expecting a rebalance. Below are the configs *(Stream Config &
Consumer Config)*.

As mentioned earlier, the application is running on Kafka 2.7.1. I deployed
the latest version of the app with 3.3.1 streams libraries, and configured
the '*upgrade.from' *property to 2.7 (based on the upgrade documentation
available here
https://kafka.apache.org/33/documentation/streams/upgrade-guide). When I
do a rolling bounce with the latest changes, I can see a rebalance being
triggered on other instances in the cluster.

I can see the below logs on the instance which is being bounced, forcing a
rebalance on others. Am I missing something? How can I avoid other
instances in the cluster from rebalancing?


*Logs:*
INFO  2023-02-27 09:52:16.805 | streams.KafkaStreams stream-client
[kafka_upgrade.Kafka_Upgrade_Test] State transition from CREATED to
REBALANCING
INFO  2023-02-27 09:52:16.946 | internals.ConsumerCoordinator [Consumer
instanceId=kafka_upgrade.Kafka_Upgrade_Test-4,
clientId=kafka_upgrade.Kafka_Upgrade_Test-StreamThread-4-consumer,
groupId=kafka_upgrade.Kafka_Upgrade_Test] Notifying assignor about the new
Assignment(partitions=[kafka_upgrade.Kafka_Upgrade_Test-version-updates-11,
kafka_upgrade.Kafka_Upgrade_Test-version-updates-23], userDataSize=56)
INFO  2023-02-27 09:52:16.947 | internals.StreamsPartitionAssignor
stream-thread [kafka_upgrade.Kafka_Upgrade_Test-StreamThread-3-consumer]
Sent a version 11 subscription and got version 8 assignment back
(successful version probing). Downgrade subscription metadata to commonly
supported version 8 and trigger new rebalance.
INFO  2023-02-27 09:52:16.947 | internals.StreamsPartitionAssignor
stream-thread [kafka_upgrade.Kafka_Upgrade_Test-StreamThread-2-consumer]
Sent a version 11 subscription and got version 8 assignment back
(successful version probing). Downgrade subscription metadata to commonly
supported version 8 and trigger new rebalance.
INFO  2023-02-27 09:52:16.947 | internals.StreamsPartitionAssignor
stream-thread [kafka_upgrade.Kafka_Upgrade_Test-StreamThread-4-consumer]
Sent a version 11 subscription and got version 8 assignment back
(successful version probing). Downgrade subscription metadata to commonly
supported version 8 and trigger new rebalance.
INFO  2023-02-27 09:52:16.947 | internals.StreamsPartitionAssignor
stream-thread [kafka_upgrade.Kafka_Upgrade_Test-StreamThread-1-consumer]
Sent a version 11 subscription and got version 8 assignment back
(successful version probing). Downgrade subscription metadata to commonly
supported version 8 and trigger new rebalance.
INFO  2023-02-27 09:52:16.947 | internals.StreamsPartitionAssignor
stream-thread [kafka_upgrade.Kafka_Upgrade_Test-StreamThread-2-consumer]
Requested to schedule immediate rebalance due to version probing.
INFO  2023-02-27 09:52:16.948 | internals.StreamsPartitionAssignor
stream-thread [kafka_upgrade.Kafka_Upgrade_Test-StreamThread-1-consumer]
Requested to schedule immediate rebalance due to version probing.
INFO  2023-02-27 09:52:16.948 | internals.StreamsPartitionAssignor
stream-thread [kafka_upgrade.Kafka_Upgrade_Test-StreamThread-4-consumer]
Requested to schedule immediate rebalance due to version probing.
INFO  2023-02-27 09:52:16.948 | internals.StreamsPartitionAssignor
stream-thread [kafka_upgrade.Kafka_Upgrade_Test-StreamThread-3-consumer]
Requested to schedule immediate rebalance due to version probing.

*Streams Config:*

acceptable.recovery.lag = 1
application.id = Kafka_Upgrade_Test
application.server =
bootstrap.servers = [broker1, broker2, broker3]
buffered.records.per.partition = 1000
built.in.metrics.version = latest
cache.max.bytes.buffering = 10485760
client.id = kafka_upgrade.Kafka_Upgrade_Test
commit.interval.ms = 3
connections.max.idle.ms = 54
default.deserialization.exception.handler = class
org.apache.kafka.streams.errors.LogAndFailExceptionHandler
default.dsl.store = rocksDB
default.key.serde = null
default.list.key.serde.inner = null
default.list.key.serde.type = null
default.list.value.serde.inner = null
default.list.value.serde.type = null
default.production.exception.handler = class
org.apache.kafka.streams.errors.DefaultProductionExceptionHandler
default.timestamp.extractor = class
org.apache.kafka.streams.processor.FailOnInvalidTimestamp
default.value.serde = null
max.task.idle.ms = 0

[jira] [Commented] (KAFKA-14748) Relax non-null FK left-join requirement

2023-02-27 Thread Matthias J. Sax (Jira)



[ 
https://issues.apache.org/jira/browse/KAFKA-14748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17694229#comment-17694229
 ] 

Matthias J. Sax commented on KAFKA-14748:
-

{quote}Hence, adding a `filter` operator after the `join` operator alone for 
`` cannot preserve the old behavior if a developer really 
wants that..
{quote}
Well, user could have a filter before the join, that leverages the 
key-extractor and drop the record if the key-extractor returns `null`?

About the question, "do we need to distinguish both cases": I think in the 
past, we wanted to distinguish both because we had the "eager emit" 
implementation for stream-stream joins, which could lead to "spurious 
duplicates" that one might want to filter downstream. Given, the new 
implementation of "emit-on-window-close", this issue goes away, and thus, I 
don't think we need to be able to distinguish both any longer?

I actually also believe, that "left join because key-extractor returns null" 
and "left join because no right hand side value found" is actually the same 
anyway?

> Relax non-null FK left-join requirement
> ---
>
> Key: KAFKA-14748
> URL: https://issues.apache.org/jira/browse/KAFKA-14748
> Project: Kafka
>      Issue Type: Improvement
>  Components: streams
>Reporter: Matthias J. Sax
>Priority: Major
>
> Kafka Streams enforces a strict non-null-key policy in the DSL across all 
> key-dependent operations (like aggregations and joins).
> This also applies to FK-joins, in particular to the ForeignKeyExtractor. If 
> it returns `null`, it's treated as invalid. For left-joins, it might make 
> sense to still accept a `null`, and add the left-hand record with an empty 
> right-hand-side to the result.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Reopened] (KAFKA-4106) Consumer / add configure method to PartitionAssignor interface



 [ 
https://issues.apache.org/jira/browse/KAFKA-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax reopened KAFKA-4106:


> Consumer / add configure method to PartitionAssignor interface
> --
>
> Key: KAFKA-4106
> URL: https://issues.apache.org/jira/browse/KAFKA-4106
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients, consumer
>Affects Versions: 0.10.0.1
>Reporter: Florian Hussonnois
>Assignee: Jason Gustafson
>Priority: Minor
>
> Currently, we can implement a custom PartitionAssignor which will forward 
> user data that will be used during the assignments protocol. For example, 
> data can be used to implement a rack-aware assignor
> However, currently we cannot dynamically configure a PartitionAssignor 
> instance.
> It would be nice to add a method configure(Map PartitionAssignor interface. Then, this method will be invoked by the 
> KafkaConsumer  on each assignor, as this is do for deserializers.
> The code modifications are pretty straight-forward but involve modifying the 
> public interface PartitionAssignor. Does that mean this JIRA needs a KIP ?
> I can contribute to that improvement.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-4106) Consumer / add configure method to PartitionAssignor interface



 [ 
https://issues.apache.org/jira/browse/KAFKA-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-4106.

Resolution: Fixed

> Consumer / add configure method to PartitionAssignor interface
> --
>
> Key: KAFKA-4106
> URL: https://issues.apache.org/jira/browse/KAFKA-4106
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients, consumer
>Affects Versions: 0.10.0.1
>Reporter: Florian Hussonnois
>Assignee: Jason Gustafson
>Priority: Minor
>
> Currently, we can implement a custom PartitionAssignor which will forward 
> user data that will be used during the assignments protocol. For example, 
> data can be used to implement a rack-aware assignor
> However, currently we cannot dynamically configure a PartitionAssignor 
> instance.
> It would be nice to add a method configure(Map PartitionAssignor interface. Then, this method will be invoked by the 
> KafkaConsumer  on each assignor, as this is do for deserializers.
> The code modifications are pretty straight-forward but involve modifying the 
> public interface PartitionAssignor. Does that mean this JIRA needs a KIP ?
> I can contribute to that improvement.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Reopened] (KAFKA-3117) Fail test at: PlaintextConsumerTest. testAutoCommitOnRebalance



 [ 
https://issues.apache.org/jira/browse/KAFKA-3117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax reopened KAFKA-3117:


> Fail test at: PlaintextConsumerTest. testAutoCommitOnRebalance 
> ---
>
> Key: KAFKA-3117
> URL: https://issues.apache.org/jira/browse/KAFKA-3117
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Affects Versions: 0.9.0.0
> Environment: oracle java764bit
> ubuntu 13.10 
>Reporter: edwardt
>Assignee: Jason Gustafson
>Priority: Major
>  Labels: newbie, test, transient-unit-test-failure
>
> java.lang.AssertionError: Expected partitions [topic-0, topic-1, topic2-0, 
> topic2-1] but actually got [topic-0, topic-1]
>   at org.junit.Assert.fail(Assert.java:88)
>   at kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:730)
>   at 
> kafka.api.BaseConsumerTest.testAutoCommitOnRebalance(BaseConsumerTest.scala:125)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:22



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-8177) Allow for separate connect instances to have sink connectors with the same name



 [ 
https://issues.apache.org/jira/browse/KAFKA-8177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-8177.

Resolution: Fixed

> Allow for separate connect instances to have sink connectors with the same 
> name
> ---
>
> Key: KAFKA-8177
> URL: https://issues.apache.org/jira/browse/KAFKA-8177
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Reporter: Paul Whalen
>Priority: Minor
>  Labels: connect
>
> If you have multiple Connect instances (either a single standalone or 
> distributed group of workers) running against the same Kafka cluster, the 
> connect instances cannot each have a sink connector with the same name and 
> still operate independently. This is because the consumer group ID used 
> internally for reading from the source topic(s) is entirely derived from the 
> connector's name: 
> [https://github.com/apache/kafka/blob/d0e436c471ba4122ddcc0f7a1624546f97c4a517/connect/runtime/src/main/java/org/apache/kafka/connect/util/SinkUtils.java#L24]
> The documentation of Connect implies to me that it supports "multi-tenancy," 
> that is, as long as...
>  * In standalone mode, the {{offset.storage.file.filename}} is not shared 
> between instances
>  * In distributed mode, {{group.id}} and {{config.storage.topic}}, 
> {{offset.storage.topic}}, and {{status.storage.topic}} are not the same 
> between instances
> ... then the connect instances can operate completely independently without 
> fear of conflict.  But the sink connector consumer group naming policy makes 
> this untrue. Obviously this can be achieved by uniquely naming connectors 
> across instances, but in some environments that could be a bit of a nuisance, 
> or a challenging policy to enforce. For instance, imagine a large group of 
> developers or data analysts all running their own standalone Connect to load 
> into a SQL database for their own analysis, or replicating to mirroring to 
> their own local cluster for testing.
> The obvious solution is allow supplying config that gives a Connect instance 
> some notion of identity, and to use that when creating the sink task consumer 
> group. Distributed mode already has this obviously ({{group.id}}), but it 
> would need to be added for standalone mode. Maybe {{instance.id}}? Given that 
> solution it seems like this would need a small KIP.
> I could also imagine this solving this problem through better documentation 
> ("ensure your connector names are unique!"), but having that subtlety doesn't 
> seem worth it to me. (Optionally) assigning identity to every Connect 
> instance seems strictly more clear, without any downside.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Reopened] (KAFKA-8177) Allow for separate connect instances to have sink connectors with the same name



 [ 
https://issues.apache.org/jira/browse/KAFKA-8177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax reopened KAFKA-8177:


> Allow for separate connect instances to have sink connectors with the same 
> name
> ---
>
> Key: KAFKA-8177
> URL: https://issues.apache.org/jira/browse/KAFKA-8177
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Reporter: Paul Whalen
>Priority: Minor
>  Labels: connect
>
> If you have multiple Connect instances (either a single standalone or 
> distributed group of workers) running against the same Kafka cluster, the 
> connect instances cannot each have a sink connector with the same name and 
> still operate independently. This is because the consumer group ID used 
> internally for reading from the source topic(s) is entirely derived from the 
> connector's name: 
> [https://github.com/apache/kafka/blob/d0e436c471ba4122ddcc0f7a1624546f97c4a517/connect/runtime/src/main/java/org/apache/kafka/connect/util/SinkUtils.java#L24]
> The documentation of Connect implies to me that it supports "multi-tenancy," 
> that is, as long as...
>  * In standalone mode, the {{offset.storage.file.filename}} is not shared 
> between instances
>  * In distributed mode, {{group.id}} and {{config.storage.topic}}, 
> {{offset.storage.topic}}, and {{status.storage.topic}} are not the same 
> between instances
> ... then the connect instances can operate completely independently without 
> fear of conflict.  But the sink connector consumer group naming policy makes 
> this untrue. Obviously this can be achieved by uniquely naming connectors 
> across instances, but in some environments that could be a bit of a nuisance, 
> or a challenging policy to enforce. For instance, imagine a large group of 
> developers or data analysts all running their own standalone Connect to load 
> into a SQL database for their own analysis, or replicating to mirroring to 
> their own local cluster for testing.
> The obvious solution is allow supplying config that gives a Connect instance 
> some notion of identity, and to use that when creating the sink task consumer 
> group. Distributed mode already has this obviously ({{group.id}}), but it 
> would need to be added for standalone mode. Maybe {{instance.id}}? Given that 
> solution it seems like this would need a small KIP.
> I could also imagine this solving this problem through better documentation 
> ("ensure your connector names are unique!"), but having that subtlety doesn't 
> seem worth it to me. (Optionally) assigning identity to every Connect 
> instance seems strictly more clear, without any downside.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-5452) Aggressive log compaction ratio appears to have no negative effect on log-compacted topics



 [ 
https://issues.apache.org/jira/browse/KAFKA-5452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-5452.

Resolution: Fixed

> Aggressive log compaction ratio appears to have no negative effect on 
> log-compacted topics
> --
>
> Key: KAFKA-5452
> URL: https://issues.apache.org/jira/browse/KAFKA-5452
> Project: Kafka
>  Issue Type: Improvement
>  Components: config, core, log
>Affects Versions: 0.10.2.0, 0.10.2.1
> Environment: Ubuntu Trusty (14.04.5), Oracle JDK 8
>Reporter: Jeff Chao
>Priority: Major
>  Labels: performance
> Attachments: 200mbs-dirty0-dirty-1-dirty05.png, 
> flame-graph-200mbs-dirty0.png, flame-graph-200mbs-dirty0.svg
>
>
> Some of our users are seeing unintuitive/unexpected behavior with 
> log-compacted topics where they receive multiple records for the same key 
> when consuming. This is a result of low throughput on log-compacted topics 
> such that conditions ({{min.cleanable.dirty.ratio = 0.5}}, default) aren't 
> met for compaction to kick in.
> This prompted us to test and tune {{min.cleanable.dirty.ratio}} in our 
> clusters. It appears that having more aggressive log compaction ratios don't 
> have negative effects on CPU and memory utilization. If this is truly the 
> case, we should consider changing the default from {{0.5}} to something more 
> aggressive.
> Setup:
> # 8 brokers
> # 5 zk nodes
> # 32 partitions on a topic
> # replication factor 3
> # log roll 3 hours
> # log segment bytes 1 GB
> # log retention 24 hours
> # all messages to a single key
> # all messages to a unique key
> # all messages to a bounded key range [0, 999]
> # {{min.cleanable.dirty.ratio}} per topic = {{0}}, {{0.5}}, and {{1}}
> # 200 MB/s sustained, produce and consume traffic
> Observations:
> We were able to verify log cleaner threads were performing work by checking 
> the logs and verifying the {{cleaner-offset-checkpoint}} file for all topics. 
> We also observed the log cleaner's {{time-since-last-run-ms}} metric was 
> normal, never going above the default of 15 seconds.
> Under-replicated partitions stayed steady, same for replication lag.
> Here's an example test run where we try out {{min.cleanable.dirty.ratio = 
> 0}}, {{min.cleanable.dirty.ratio = 1}}, and {{min.cleanable.dirty.ratio = 
> 0.5}}. Troughs in between the peaks represent zero traffic and reconfiguring 
> of topics.
> (200mbs-dirty-0-dirty1-dirty05.png attached)
> !200mbs-dirty0-dirty-1-dirty05.png|thumbnail!
> Memory utilization is fine, but more interestingly, CPU doesn't appear to 
> have much difference.
> To get more detail, here is a flame graph (raw svg attached) of the run for 
> {{min.cleanable.dirty.ratio = 0}}. The conservative and default ratio flame 
> graphs are equivalent.
> (flame-graph-200mbs-dirty0.png attached)
> !flame-graph-200mbs-dirty0.png|thumbnail!
> Notice that the majority of CPU is coming from:
> # SSL operations (on reads/writes)
> # KafkaApis::handleFetchRequest (ReplicaManager::fetchMessages)
> # KafkaApis::handleOffsetFetchRequest
> We also have examples from small scale test runs which show similar behavior 
> but with scaled down CPU usage.
> It seems counterintuitive that there's no apparent difference in CPU whether 
> it be aggressive or conservative compaction ratios, so we'd like to get some 
> thoughts from the community.
> We're looking for feedback on whether or not anyone else has experienced this 
> behavior before as well or, if CPU isn't affected, has anyone seen something 
> related instead.
> If this is true, then we'd be happy to discuss further and provide a patch.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-5452) Aggressive log compaction ratio appears to have no negative effect on log-compacted topics



 [ 
https://issues.apache.org/jira/browse/KAFKA-5452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-5452.

Resolution: Fixed

> Aggressive log compaction ratio appears to have no negative effect on 
> log-compacted topics
> --
>
> Key: KAFKA-5452
> URL: https://issues.apache.org/jira/browse/KAFKA-5452
> Project: Kafka
>  Issue Type: Improvement
>  Components: config, core, log
>Affects Versions: 0.10.2.0, 0.10.2.1
> Environment: Ubuntu Trusty (14.04.5), Oracle JDK 8
>Reporter: Jeff Chao
>Priority: Major
>  Labels: performance
> Attachments: 200mbs-dirty0-dirty-1-dirty05.png, 
> flame-graph-200mbs-dirty0.png, flame-graph-200mbs-dirty0.svg
>
>
> Some of our users are seeing unintuitive/unexpected behavior with 
> log-compacted topics where they receive multiple records for the same key 
> when consuming. This is a result of low throughput on log-compacted topics 
> such that conditions ({{min.cleanable.dirty.ratio = 0.5}}, default) aren't 
> met for compaction to kick in.
> This prompted us to test and tune {{min.cleanable.dirty.ratio}} in our 
> clusters. It appears that having more aggressive log compaction ratios don't 
> have negative effects on CPU and memory utilization. If this is truly the 
> case, we should consider changing the default from {{0.5}} to something more 
> aggressive.
> Setup:
> # 8 brokers
> # 5 zk nodes
> # 32 partitions on a topic
> # replication factor 3
> # log roll 3 hours
> # log segment bytes 1 GB
> # log retention 24 hours
> # all messages to a single key
> # all messages to a unique key
> # all messages to a bounded key range [0, 999]
> # {{min.cleanable.dirty.ratio}} per topic = {{0}}, {{0.5}}, and {{1}}
> # 200 MB/s sustained, produce and consume traffic
> Observations:
> We were able to verify log cleaner threads were performing work by checking 
> the logs and verifying the {{cleaner-offset-checkpoint}} file for all topics. 
> We also observed the log cleaner's {{time-since-last-run-ms}} metric was 
> normal, never going above the default of 15 seconds.
> Under-replicated partitions stayed steady, same for replication lag.
> Here's an example test run where we try out {{min.cleanable.dirty.ratio = 
> 0}}, {{min.cleanable.dirty.ratio = 1}}, and {{min.cleanable.dirty.ratio = 
> 0.5}}. Troughs in between the peaks represent zero traffic and reconfiguring 
> of topics.
> (200mbs-dirty-0-dirty1-dirty05.png attached)
> !200mbs-dirty0-dirty-1-dirty05.png|thumbnail!
> Memory utilization is fine, but more interestingly, CPU doesn't appear to 
> have much difference.
> To get more detail, here is a flame graph (raw svg attached) of the run for 
> {{min.cleanable.dirty.ratio = 0}}. The conservative and default ratio flame 
> graphs are equivalent.
> (flame-graph-200mbs-dirty0.png attached)
> !flame-graph-200mbs-dirty0.png|thumbnail!
> Notice that the majority of CPU is coming from:
> # SSL operations (on reads/writes)
> # KafkaApis::handleFetchRequest (ReplicaManager::fetchMessages)
> # KafkaApis::handleOffsetFetchRequest
> We also have examples from small scale test runs which show similar behavior 
> but with scaled down CPU usage.
> It seems counterintuitive that there's no apparent difference in CPU whether 
> it be aggressive or conservative compaction ratios, so we'd like to get some 
> thoughts from the community.
> We're looking for feedback on whether or not anyone else has experienced this 
> behavior before as well or, if CPU isn't affected, has anyone seen something 
> related instead.
> If this is true, then we'd be happy to discuss further and provide a patch.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-4106) Consumer / add configure method to PartitionAssignor interface



 [ 
https://issues.apache.org/jira/browse/KAFKA-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-4106.

Resolution: Fixed

> Consumer / add configure method to PartitionAssignor interface
> --
>
> Key: KAFKA-4106
> URL: https://issues.apache.org/jira/browse/KAFKA-4106
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients, consumer
>Affects Versions: 0.10.0.1
>Reporter: Florian Hussonnois
>Assignee: Jason Gustafson
>Priority: Minor
>
> Currently, we can implement a custom PartitionAssignor which will forward 
> user data that will be used during the assignments protocol. For example, 
> data can be used to implement a rack-aware assignor
> However, currently we cannot dynamically configure a PartitionAssignor 
> instance.
> It would be nice to add a method configure(Map PartitionAssignor interface. Then, this method will be invoked by the 
> KafkaConsumer  on each assignor, as this is do for deserializers.
> The code modifications are pretty straight-forward but involve modifying the 
> public interface PartitionAssignor. Does that mean this JIRA needs a KIP ?
> I can contribute to that improvement.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Reopened] (KAFKA-4106) Consumer / add configure method to PartitionAssignor interface



 [ 
https://issues.apache.org/jira/browse/KAFKA-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax reopened KAFKA-4106:


> Consumer / add configure method to PartitionAssignor interface
> --
>
> Key: KAFKA-4106
> URL: https://issues.apache.org/jira/browse/KAFKA-4106
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients, consumer
>Affects Versions: 0.10.0.1
>Reporter: Florian Hussonnois
>Assignee: Jason Gustafson
>Priority: Minor
>
> Currently, we can implement a custom PartitionAssignor which will forward 
> user data that will be used during the assignments protocol. For example, 
> data can be used to implement a rack-aware assignor
> However, currently we cannot dynamically configure a PartitionAssignor 
> instance.
> It would be nice to add a method configure(Map PartitionAssignor interface. Then, this method will be invoked by the 
> KafkaConsumer  on each assignor, as this is do for deserializers.
> The code modifications are pretty straight-forward but involve modifying the 
> public interface PartitionAssignor. Does that mean this JIRA needs a KIP ?
> I can contribute to that improvement.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Reopened] (KAFKA-3117) Fail test at: PlaintextConsumerTest. testAutoCommitOnRebalance



 [ 
https://issues.apache.org/jira/browse/KAFKA-3117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax reopened KAFKA-3117:


> Fail test at: PlaintextConsumerTest. testAutoCommitOnRebalance 
> ---
>
> Key: KAFKA-3117
> URL: https://issues.apache.org/jira/browse/KAFKA-3117
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Affects Versions: 0.9.0.0
> Environment: oracle java764bit
> ubuntu 13.10 
>Reporter: edwardt
>Assignee: Jason Gustafson
>Priority: Major
>  Labels: newbie, test, transient-unit-test-failure
>
> java.lang.AssertionError: Expected partitions [topic-0, topic-1, topic2-0, 
> topic2-1] but actually got [topic-0, topic-1]
>   at org.junit.Assert.fail(Assert.java:88)
>   at kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:730)
>   at 
> kafka.api.BaseConsumerTest.testAutoCommitOnRebalance(BaseConsumerTest.scala:125)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:22



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-6014) new consumer mirror maker halts after committing offsets to a deleted topic



 [ 
https://issues.apache.org/jira/browse/KAFKA-6014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-6014.

Resolution: Fixed

> new consumer mirror maker halts after committing offsets to a deleted topic
> ---
>
> Key: KAFKA-6014
> URL: https://issues.apache.org/jira/browse/KAFKA-6014
> Project: Kafka
>  Issue Type: Bug
>Reporter: Onur Karaman
>Assignee: Jason Gustafson
>Priority: Major
>
> New consumer throws an unexpected KafkaException when trying to commit to a 
> topic that has been deleted. MirrorMaker.commitOffsets doesn't attempt to 
> catch the KafkaException and just kills the process. We didn't see this in 
> the old consumer because old consumer just silently drops failed offset 
> commits.
> I ran a quick experiment locally to prove the behavior. The experiment:
> 1. start up a single broker
> 2. create a single-partition topic t
> 3. create a new consumer that consumes topic t
> 4. make the consumer commit every few seconds
> 5. delete topic t
> 6. expect: KafkaException that kills the process.
> Here's my script:
> {code}
> package org.apache.kafka.clients.consumer;
> import org.apache.kafka.common.TopicPartition;
> import java.util.Collections;
> import java.util.List;
> import java.util.Properties;
> public class OffsetCommitTopicDeletionTest {
> public static void main(String[] args) throws InterruptedException {
> Properties props = new Properties();
> props.setProperty(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, 
> "localhost:9090");
> props.setProperty(ConsumerConfig.GROUP_ID_CONFIG, "g");
> props.setProperty(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, 
> "org.apache.kafka.common.serialization.ByteArrayDeserializer");
> props.setProperty(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, 
> "org.apache.kafka.common.serialization.ByteArrayDeserializer");
> props.setProperty(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
> KafkaConsumer kafkaConsumer = new 
> KafkaConsumer<>(props);
> TopicPartition partition = new TopicPartition("t", 0);
> List partitions = 
> Collections.singletonList(partition);
> kafkaConsumer.assign(partitions);
> while (true) {
> kafkaConsumer.commitSync(Collections.singletonMap(partition, new 
> OffsetAndMetadata(0, "")));
> Thread.sleep(1000);
> }
> }
> }
> {code}
> Here are the other commands:
> {code}
> > rm -rf /tmp/zookeeper/ /tmp/kafka-logs* logs*
> > ./gradlew clean jar
> > ./bin/zookeeper-server-start.sh config/zookeeper.properties
> > export LOG_DIR=logs0 && ./bin/kafka-server-start.sh 
> > config/server0.properties
> > ./bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic t 
> > --partitions 1 --replication-factor 1
> > ./bin/kafka-run-class.sh 
> > org.apache.kafka.clients.consumer.OffsetCommitTopicDeletionTest
> > ./bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic t
> {code}
> Here is the output:
> {code}
> [2017-10-04 20:00:14,451] ERROR [Consumer clientId=consumer-1, groupId=g] 
> Offset commit failed on partition t-0 at offset 0: This server does not host 
> this topic-partition. 
> (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
> Exception in thread "main" org.apache.kafka.common.KafkaException: Partition 
> t-0 may not exist or user may not have Describe access to topic
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle(ConsumerCoordinator.java:789)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle(ConsumerCoordinator.java:734)
>   at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:808)
>   at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:788)
>   at 
> org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:204)
>   at 
> org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:167)
>   at 
> org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:127)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestF

[jira] [Resolved] (KAFKA-8177) Allow for separate connect instances to have sink connectors with the same name



 [ 
https://issues.apache.org/jira/browse/KAFKA-8177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-8177.

Resolution: Fixed

> Allow for separate connect instances to have sink connectors with the same 
> name
> ---
>
> Key: KAFKA-8177
> URL: https://issues.apache.org/jira/browse/KAFKA-8177
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Reporter: Paul Whalen
>Priority: Minor
>  Labels: connect
>
> If you have multiple Connect instances (either a single standalone or 
> distributed group of workers) running against the same Kafka cluster, the 
> connect instances cannot each have a sink connector with the same name and 
> still operate independently. This is because the consumer group ID used 
> internally for reading from the source topic(s) is entirely derived from the 
> connector's name: 
> [https://github.com/apache/kafka/blob/d0e436c471ba4122ddcc0f7a1624546f97c4a517/connect/runtime/src/main/java/org/apache/kafka/connect/util/SinkUtils.java#L24]
> The documentation of Connect implies to me that it supports "multi-tenancy," 
> that is, as long as...
>  * In standalone mode, the {{offset.storage.file.filename}} is not shared 
> between instances
>  * In distributed mode, {{group.id}} and {{config.storage.topic}}, 
> {{offset.storage.topic}}, and {{status.storage.topic}} are not the same 
> between instances
> ... then the connect instances can operate completely independently without 
> fear of conflict.  But the sink connector consumer group naming policy makes 
> this untrue. Obviously this can be achieved by uniquely naming connectors 
> across instances, but in some environments that could be a bit of a nuisance, 
> or a challenging policy to enforce. For instance, imagine a large group of 
> developers or data analysts all running their own standalone Connect to load 
> into a SQL database for their own analysis, or replicating to mirroring to 
> their own local cluster for testing.
> The obvious solution is allow supplying config that gives a Connect instance 
> some notion of identity, and to use that when creating the sink task consumer 
> group. Distributed mode already has this obviously ({{group.id}}), but it 
> would need to be added for standalone mode. Maybe {{instance.id}}? Given that 
> solution it seems like this would need a small KIP.
> I could also imagine this solving this problem through better documentation 
> ("ensure your connector names are unique!"), but having that subtlety doesn't 
> seem worth it to me. (Optionally) assigning identity to every Connect 
> instance seems strictly more clear, without any downside.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Reopened] (KAFKA-4187) Adding a flag to prefix topics with mirror maker



 [ 
https://issues.apache.org/jira/browse/KAFKA-4187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax reopened KAFKA-4187:


> Adding a flag to prefix topics with mirror maker
> 
>
> Key: KAFKA-4187
> URL: https://issues.apache.org/jira/browse/KAFKA-4187
> Project: Kafka
>  Issue Type: Improvement
>  Components: tools
>Affects Versions: 0.8.2.1, 0.9.0.1, 0.10.0.0, 0.10.0.1
>Reporter: Vincent Rischmann
>Priority: Minor
>
> So I have a setup where I need to mirror our production cluster to our 
> preproduction cluster, but can't use the original topic names.
> I've patched mirror maker to allow me to define a prefix for each topic and I 
> basically prefix everything with mirror_. I'm wondering if there's interest 
> for this feature upstream ?
> I have a patch available for Kafka 0.9.0.1 (what I'm using) and from what 
> I've seen it should apply well to Kafka 0.10.0.X too.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-4187) Adding a flag to prefix topics with mirror maker



 [ 
https://issues.apache.org/jira/browse/KAFKA-4187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-4187.

Resolution: Fixed

> Adding a flag to prefix topics with mirror maker
> 
>
> Key: KAFKA-4187
> URL: https://issues.apache.org/jira/browse/KAFKA-4187
> Project: Kafka
>  Issue Type: Improvement
>  Components: tools
>Affects Versions: 0.8.2.1, 0.9.0.1, 0.10.0.0, 0.10.0.1
>Reporter: Vincent Rischmann
>Priority: Minor
>
> So I have a setup where I need to mirror our production cluster to our 
> preproduction cluster, but can't use the original topic names.
> I've patched mirror maker to allow me to define a prefix for each topic and I 
> basically prefix everything with mirror_. I'm wondering if there's interest 
> for this feature upstream ?
> I have a patch available for Kafka 0.9.0.1 (what I'm using) and from what 
> I've seen it should apply well to Kafka 0.10.0.X too.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Reopened] (KAFKA-8177) Allow for separate connect instances to have sink connectors with the same name



 [ 
https://issues.apache.org/jira/browse/KAFKA-8177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax reopened KAFKA-8177:


> Allow for separate connect instances to have sink connectors with the same 
> name
> ---
>
> Key: KAFKA-8177
> URL: https://issues.apache.org/jira/browse/KAFKA-8177
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Reporter: Paul Whalen
>Priority: Minor
>  Labels: connect
>
> If you have multiple Connect instances (either a single standalone or 
> distributed group of workers) running against the same Kafka cluster, the 
> connect instances cannot each have a sink connector with the same name and 
> still operate independently. This is because the consumer group ID used 
> internally for reading from the source topic(s) is entirely derived from the 
> connector's name: 
> [https://github.com/apache/kafka/blob/d0e436c471ba4122ddcc0f7a1624546f97c4a517/connect/runtime/src/main/java/org/apache/kafka/connect/util/SinkUtils.java#L24]
> The documentation of Connect implies to me that it supports "multi-tenancy," 
> that is, as long as...
>  * In standalone mode, the {{offset.storage.file.filename}} is not shared 
> between instances
>  * In distributed mode, {{group.id}} and {{config.storage.topic}}, 
> {{offset.storage.topic}}, and {{status.storage.topic}} are not the same 
> between instances
> ... then the connect instances can operate completely independently without 
> fear of conflict.  But the sink connector consumer group naming policy makes 
> this untrue. Obviously this can be achieved by uniquely naming connectors 
> across instances, but in some environments that could be a bit of a nuisance, 
> or a challenging policy to enforce. For instance, imagine a large group of 
> developers or data analysts all running their own standalone Connect to load 
> into a SQL database for their own analysis, or replicating to mirroring to 
> their own local cluster for testing.
> The obvious solution is allow supplying config that gives a Connect instance 
> some notion of identity, and to use that when creating the sink task consumer 
> group. Distributed mode already has this obviously ({{group.id}}), but it 
> would need to be added for standalone mode. Maybe {{instance.id}}? Given that 
> solution it seems like this would need a small KIP.
> I could also imagine this solving this problem through better documentation 
> ("ensure your connector names are unique!"), but having that subtlety doesn't 
> seem worth it to me. (Optionally) assigning identity to every Connect 
> instance seems strictly more clear, without any downside.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-3117) Fail test at: PlaintextConsumerTest. testAutoCommitOnRebalance



 [ 
https://issues.apache.org/jira/browse/KAFKA-3117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-3117.

Resolution: Fixed

> Fail test at: PlaintextConsumerTest. testAutoCommitOnRebalance 
> ---
>
> Key: KAFKA-3117
> URL: https://issues.apache.org/jira/browse/KAFKA-3117
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Affects Versions: 0.9.0.0
> Environment: oracle java764bit
> ubuntu 13.10 
>Reporter: edwardt
>Assignee: Jason Gustafson
>Priority: Major
>  Labels: newbie, test, transient-unit-test-failure
>
> java.lang.AssertionError: Expected partitions [topic-0, topic-1, topic2-0, 
> topic2-1] but actually got [topic-0, topic-1]
>   at org.junit.Assert.fail(Assert.java:88)
>   at kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:730)
>   at 
> kafka.api.BaseConsumerTest.testAutoCommitOnRebalance(BaseConsumerTest.scala:125)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:22



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Reopened] (KAFKA-4187) Adding a flag to prefix topics with mirror maker



 [ 
https://issues.apache.org/jira/browse/KAFKA-4187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax reopened KAFKA-4187:


> Adding a flag to prefix topics with mirror maker
> 
>
> Key: KAFKA-4187
> URL: https://issues.apache.org/jira/browse/KAFKA-4187
> Project: Kafka
>  Issue Type: Improvement
>  Components: tools
>Affects Versions: 0.8.2.1, 0.9.0.1, 0.10.0.0, 0.10.0.1
>Reporter: Vincent Rischmann
>Priority: Minor
>
> So I have a setup where I need to mirror our production cluster to our 
> preproduction cluster, but can't use the original topic names.
> I've patched mirror maker to allow me to define a prefix for each topic and I 
> basically prefix everything with mirror_. I'm wondering if there's interest 
> for this feature upstream ?
> I have a patch available for Kafka 0.9.0.1 (what I'm using) and from what 
> I've seen it should apply well to Kafka 0.10.0.X too.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-4187) Adding a flag to prefix topics with mirror maker



 [ 
https://issues.apache.org/jira/browse/KAFKA-4187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-4187.

Resolution: Fixed

> Adding a flag to prefix topics with mirror maker
> 
>
> Key: KAFKA-4187
> URL: https://issues.apache.org/jira/browse/KAFKA-4187
> Project: Kafka
>  Issue Type: Improvement
>  Components: tools
>Affects Versions: 0.8.2.1, 0.9.0.1, 0.10.0.0, 0.10.0.1
>Reporter: Vincent Rischmann
>Priority: Minor
>
> So I have a setup where I need to mirror our production cluster to our 
> preproduction cluster, but can't use the original topic names.
> I've patched mirror maker to allow me to define a prefix for each topic and I 
> basically prefix everything with mirror_. I'm wondering if there's interest 
> for this feature upstream ?
> I have a patch available for Kafka 0.9.0.1 (what I'm using) and from what 
> I've seen it should apply well to Kafka 0.10.0.X too.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-3117) Fail test at: PlaintextConsumerTest. testAutoCommitOnRebalance