Re: [VOTE] KIP-953: partition method to be overloaded to accept headers as well.

[Chris Egerton]

Hi Jack,

+1 (binding)

Some friendly, non-blocking suggestions:

- IMO it's still worth specifying that the headers will be read-only; this
clarifies the intended API contract both for reviewers of the KIP who
haven't read the GitHub PR yet, and for developers who may leverage this
new method
- May be worth mentioning in the compatibility section that any
partitioners that only implement the new interface will be incompatible
with older Kafka clients versions (this is less likely to be a serious
problem in the clients world, but it's a much hairier problem with Connect,
where cross-compatibility between newer/older versions of connectors and
the Kafka Connect runtime is a serious concern)

Again, these are not blockers and I'm in favor of the KIP with or without
them since I believe both can be addressed at least partially during PR
review and don't have to be tackled at this stage.

Cheers,

Chris

On Sat, Aug 12, 2023 at 12:43 AM Sagar  wrote:

> Hey jack ,
>
> +1 (non binding)
>
> Sagar.
>
> On Sat, 12 Aug 2023 at 8:04 AM, Jack Tomy  wrote:
>
> > Hey everyone,
> >
> > Please consider this as a gentle reminder.
> >
> > On Mon, Aug 7, 2023 at 5:55 PM Jack Tomy  wrote:
> >
> > > Hey everyone.
> > >
> > > I would like to call for a vote on KIP-953: partition method to be
> > > overloaded to accept headers as well.
> > >
> > > KIP :
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263424937
> > > Discussion thread :
> > > https://lists.apache.org/thread/0f20kvfqkmhdqrwcb8vqgqn80szcrcdd
> > >
> > > Thanks
> > > --
> > > Best Regards
> > > *Jack*
> > >
> >
> >
> > --
> > Best Regards
> > *Jack*
> >
>

Re: [VOTE] KIP-953: partition method to be overloaded to accept headers as well.

[Sagar]

Hey jack ,

+1 (non binding)

Sagar.

On Sat, 12 Aug 2023 at 8:04 AM, Jack Tomy  wrote:

> Hey everyone,
>
> Please consider this as a gentle reminder.
>
> On Mon, Aug 7, 2023 at 5:55 PM Jack Tomy  wrote:
>
> > Hey everyone.
> >
> > I would like to call for a vote on KIP-953: partition method to be
> > overloaded to accept headers as well.
> >
> > KIP :
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263424937
> > Discussion thread :
> > https://lists.apache.org/thread/0f20kvfqkmhdqrwcb8vqgqn80szcrcdd
> >
> > Thanks
> > --
> > Best Regards
> > *Jack*
> >
>
>
> --
> Best Regards
> *Jack*
>

Re: [VOTE] KIP-953: partition method to be overloaded to accept headers as well.

[Jack Tomy]

Hey everyone,

Please consider this as a gentle reminder.

On Mon, Aug 7, 2023 at 5:55 PM Jack Tomy  wrote:

> Hey everyone.
>
> I would like to call for a vote on KIP-953: partition method to be
> overloaded to accept headers as well.
>
> KIP :
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263424937
> Discussion thread :
> https://lists.apache.org/thread/0f20kvfqkmhdqrwcb8vqgqn80szcrcdd
>
> Thanks
> --
> Best Regards
> *Jack*
>


-- 
Best Regards
*Jack*

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #2093

[Apache Jenkins Server]

See 

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.5 #58

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 285233 lines...]
Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
VersionedKeyValueStoreIntegrationTest > shouldSetChangelogTopicProperties PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
VersionedKeyValueStoreIntegrationTest > shouldRestore STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
VersionedKeyValueStoreIntegrationTest > shouldRestore PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
VersionedKeyValueStoreIntegrationTest > shouldPutGetAndDelete STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
VersionedKeyValueStoreIntegrationTest > shouldPutGetAndDelete PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
VersionedKeyValueStoreIntegrationTest > 
shouldManualUpgradeFromNonVersionedTimestampedToVersioned STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
VersionedKeyValueStoreIntegrationTest > 
shouldManualUpgradeFromNonVersionedTimestampedToVersioned PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
HandlingSourceTopicDeletionIntegrationTest > 
shouldThrowErrorAfterSourceTopicDeleted STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
HandlingSourceTopicDeletionIntegrationTest > 
shouldThrowErrorAfterSourceTopicDeleted PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
StreamsAssignmentScaleTest > testHighAvailabilityTaskAssignorLargeNumConsumers 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
StreamsAssignmentScaleTest > testHighAvailabilityTaskAssignorLargeNumConsumers 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorLargePartitionCount STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorLargePartitionCount PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorManyThreadsPerClient STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorManyThreadsPerClient PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
StreamsAssignmentScaleTest > testStickyTaskAssignorManyStandbys STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
StreamsAssignmentScaleTest > testStickyTaskAssignorManyStandbys PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
StreamsAssignmentScaleTest > testStickyTaskAssignorManyThreadsPerClient STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
StreamsAssignmentScaleTest > testStickyTaskAssignorManyThreadsPerClient PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
StreamsAssignmentScaleTest > testFallbackPriorTaskAssignorManyThreadsPerClient 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
StreamsAssignmentScaleTest > testFallbackPriorTaskAssignorManyThreadsPerClient 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
StreamsAssignmentScaleTest > testFallbackPriorTaskAssignorLargePartitionCount 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
StreamsAssignmentScaleTest > testFallbackPriorTaskAssignorLargePartitionCount 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
StreamsAssignmentScaleTest > testStickyTaskAssignorLargePartitionCount STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
StreamsAssignmentScaleTest > testStickyTaskAssignorLargePartitionCount PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
StreamsAssignmentScaleTest > testFallbackPriorTaskAssignorManyStandbys STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
StreamsAssignmentScaleTest > testFallbackPriorTaskAssignorManyStandbys PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
StreamsAssignmentScaleTest > testHighAvailabilityTaskAssignorManyStandbys 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
StreamsAssignmentScaleTest > testHighAvailabilityTaskAssignorManyStandbys PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
StreamsAssignmentScaleTest > testFallbackPriorTaskAssignorLargeNumConsumers 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
StreamsAssignmentScaleTest > testFallbackPriorTaskAssignorLargeNumConsumers 
PASSED

Gradle Test Run :strea

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2092

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 401822 lines...]
[INFO] --- compiler:3.1:compile (default-compile) @ streams.examples ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 3 source files to 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk_2/streams/quickstart/test-streams-archetype/streams.examples/target/classes
[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time:  2.087 s
[INFO] Finished at: 2023-08-11T21:23:30Z
[INFO] 
[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // stage
[Pipeline] }

Gradle Test Run :streams:integrationTest > Gradle Test Executor 179 > 
KStreamAggregationDedupIntegrationTest > shouldReduce(TestInfo) PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 179 > 
KStreamAggregationDedupIntegrationTest > shouldGroupByKey(TestInfo) STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 179 > 
KStreamAggregationDedupIntegrationTest > shouldGroupByKey(TestInfo) PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 179 > 
KStreamAggregationDedupIntegrationTest > shouldReduceWindowed(TestInfo) STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 179 > 
KStreamAggregationDedupIntegrationTest > shouldReduceWindowed(TestInfo) PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 179 > 
KStreamKStreamIntegrationTest > shouldOuterJoin() STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 179 > 
KStreamKStreamIntegrationTest > shouldOuterJoin() PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 179 > 
KTableKTableForeignKeyInnerJoinMultiIntegrationTest > 
shouldInnerJoinMultiPartitionQueryable() STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 179 > 
KTableKTableForeignKeyInnerJoinMultiIntegrationTest > 
shouldInnerJoinMultiPartitionQueryable() PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 179 > 
KTableSourceTopicRestartIntegrationTest > 
shouldRestoreAndProgressWhenTopicNotWrittenToDuringRestoration() STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 179 > 
KTableSourceTopicRestartIntegrationTest > 
shouldRestoreAndProgressWhenTopicNotWrittenToDuringRestoration() PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 179 > 
KTableSourceTopicRestartIntegrationTest > 
shouldRestoreAndProgressWhenTopicWrittenToDuringRestorationWithEosAlphaEnabled()
 STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 179 > 
KTableSourceTopicRestartIntegrationTest > 
shouldRestoreAndProgressWhenTopicWrittenToDuringRestorationWithEosAlphaEnabled()
 PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 179 > 
KTableSourceTopicRestartIntegrationTest > 
shouldRestoreAndProgressWhenTopicWrittenToDuringRestorationWithEosDisabled() 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 179 > 
KTableSourceTopicRestartIntegrationTest > 
shouldRestoreAndProgressWhenTopicWrittenToDuringRestorationWithEosDisabled() 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 179 > 
KTableSourceTopicRestartIntegrationTest > 
shouldRestoreAndProgressWhenTopicWrittenToDuringRestorationWithEosV2Enabled() 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 179 > 
KTableSourceTopicRestartIntegrationTest > 
shouldRestoreAndProgressWhenTopicWrittenToDuringRestorationWithEosV2Enabled() 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 179 > 
LagFetchIntegrationTest > shouldFetchLagsDuringRebalancingWithOptimization() 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 179 > 
LagFetchIntegrationTest > shouldFetchLagsDuringRebalancingWithOptimization() 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 179 > 
LagFetchIntegrationTest > shouldFetchLagsDuringRestoration() STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 179 > 
LagFetchIntegrationTest > shouldFetchLagsDuringRestoration() PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 179 > 
LagFetchIntegrationTest > shouldFetchLagsDuringRebalancingWithNoOptimization() 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 179 > 
LagFetchIntegrationTest > shouldFetchLagsDuringRebalancingWithNoOptimization() 
PASSED

[GitHub] [kafka-site] jolshan merged pull request #537: MINOR: Add Lucas Bradstreet to committers

[via GitHub]

jolshan merged PR #537:
URL: https://github.com/apache/kafka-site/pull/537


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] jolshan commented on pull request #537: MINOR: Add Lucas Bradstreet to committers

[via GitHub]

jolshan commented on PR #537:
URL: https://github.com/apache/kafka-site/pull/537#issuecomment-1675403860

   Hmm - you should be able to merge this too. If you don't you may want to 
check your permissions. 😅  (See I merged my own -- 
https://github.com/apache/kafka-site/pull/473)
   
   But I can go ahead and merge.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

RE: [DISCUSS] KIP-942: Add Power(ppc64le) support

[Vaibhav Nazare]

Hi Divij,

1.No the different versions of JVM don't work differently for power 
architecture. It will be sufficient if we just run it with the latest supported 
JDK (20) + latest supported scala (2.13)
2. The plan has been updated accordingly for rejected alternatives

-Original Message-
From: Divij Vaidya  
Sent: Thursday, August 3, 2023 2:33 PM
To: dev@kafka.apache.org
Subject: [EXTERNAL] Re: [DISCUSS] KIP-942: Add Power(ppc64le) support

Hey Vaibhav

1. KIP says "Enable CI for power architecture and run tests with Java 8, 11 and 
17 and Scala 2.13". Do different versions of JVM work differently for power 
architecture? Would it be sufficient if we just run it with the latest 
supported JDK (20) + latest supported scala
(2.13) ?

2. Can you also please add that we plan to run this only on branch builder and 
not on every PR. Note that we have two CI runs configured today, one is "branch 
builder" which runs when a commit is merged to trunk or preceding versions and 
another is "PR builder" which runs on every commit on every PR. From our 
earlier discussion on this thread, we discussed to only add it for "branch 
builder". Also, please add option of adding test to "PR builder" in the 
rejected alternative section.


--
Divij Vaidya

On Thu, Aug 3, 2023 at 8:40 AM Vaibhav Nazare  
wrote:
>
> Hi Divij
>
> Thanks for the response. Agree with you, also I have updated the KIP 
> accordingly.
>

[jira] [Resolved] (KAFKA-13197) KStream-GlobalKTable join semantics don't match documentation

[Matthias J. Sax (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-13197.
-
Fix Version/s: 3.6.0
   3.5.2
   Resolution: Fixed

> KStream-GlobalKTable join semantics don't match documentation
> -
>
> Key: KAFKA-13197
> URL: https://issues.apache.org/jira/browse/KAFKA-13197
> Project: Kafka
>  Issue Type: Bug
>  Components: documentation, streams
>Affects Versions: 2.7.0
>Reporter: Tommy Becker
>Assignee: Florin Akermann
>Priority: Major
> Fix For: 3.6.0, 3.5.2
>
>
> As part of KAFKA-10277, the behavior of KStream-GlobalKTable joins was 
> changed. It appears the change was intended to merely relax a requirement but 
> it actually broke backwards compatibility. Although it does allow {{null}} 
> keys and values in the KStream to be joined, it now excludes {{null}} results 
> of the {{KeyValueMapper}}. We have an application which can return {{null}} 
> from the {{KeyValueMapper}} for non-null keys in the KStream, and relies on 
> these nulls being passed to the {{ValueJoiner}}. Indeed the javadoc still 
> explicitly says this is done:
> {quote}If a KStream input record key or value is null the record will not be 
> included in the join operation and thus no output record will be added to the 
> resulting KStream.
>  If keyValueMapper returns null implying no match exists, a null value will 
> be provided to ValueJoiner.
> {quote}
> Both these statements are incorrect.
> I think the new behavior is worse than the previous/documented behavior. It 
> feels more reasonable to have a non-null stream record map to a null join key 
> (our use-case is event-enhancement where the incoming record doesn't have the 
> join field), than the reverse.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15054) Add configs and logic to decide if rack aware assignment should be enabled

[Hao Li (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hao Li resolved KAFKA-15054.

Fix Version/s: 3.6.0
   Resolution: Fixed

> Add configs and logic to decide if rack aware assignment should be enabled
> --
>
> Key: KAFKA-15054
> URL: https://issues.apache.org/jira/browse/KAFKA-15054
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Hao Li
>Assignee: Hao Li
>Priority: Major
> Fix For: 3.6.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15025) Implement min-cost flow without balancing tasks for same subtopology

[Hao Li (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hao Li resolved KAFKA-15025.

Fix Version/s: 3.6.0
   Resolution: Fixed

> Implement min-cost flow without balancing tasks for same subtopology
> 
>
> Key: KAFKA-15025
> URL: https://issues.apache.org/jira/browse/KAFKA-15025
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Hao Li
>Assignee: Hao Li
>Priority: Major
> Fix For: 3.6.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15027) Implement rack aware assignment for standby tasks

[Hao Li (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hao Li resolved KAFKA-15027.

Fix Version/s: 3.6.0
   Resolution: Fixed

> Implement rack aware assignment for standby tasks
> -
>
> Key: KAFKA-15027
> URL: https://issues.apache.org/jira/browse/KAFKA-15027
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Hao Li
>Assignee: Hao Li
>Priority: Major
> Fix For: 3.6.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15024) Add cost function for task/client

[Hao Li (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hao Li resolved KAFKA-15024.

Fix Version/s: 3.6.0
   Resolution: Fixed

> Add cost function for task/client
> -
>
> Key: KAFKA-15024
> URL: https://issues.apache.org/jira/browse/KAFKA-15024
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Hao Li
>Priority: Major
> Fix For: 3.6.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15023) Get rack information for source topic partitions for a task

[Hao Li (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hao Li resolved KAFKA-15023.

Fix Version/s: 3.6.0
   Resolution: Fixed

> Get rack information for source topic partitions for a task
> ---
>
> Key: KAFKA-15023
> URL: https://issues.apache.org/jira/browse/KAFKA-15023
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Hao Li
>Assignee: Hao Li
>Priority: Major
> Fix For: 3.6.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-955: Add stream-table join on foreign key

[Igor Fomenko]

Matthias,

I think that I clouded this discussion a bit with the possible 'fat'
message requirement for the one specific use case that I worked on.
Therefore I would like to take a step back and to focus just on the actual
KIP-955 that only proposes to create a stream-table join on foreign key.
This is regardless if any aggregation (like 'fat' message) is required
afterwards  or not.

Also I do not think that emitting multiple records from the stream-table FK
join is a 'weird' behaviour because this is exactly how the standard SQL
behaves and many stream processing tools try to mimic SQL at the DSL layer
- for example Spark Structured Streaming or Hadoop.  The normal behaviour
of SQL is to select records with the join as the recordset first and only
then as the next step to aggregate or otherwise transform results if it is
required. This sequence is much more flexible and efficient compared to
aggregating everything just in case some records will be selected by the
join.
We can imagine many use cases for joining on FK without subsequent
aggregation or with an aggregation on something other than stream message
key. For example we could stream all orderItems for completedOrders as
separate messages (result of FK join) and then could count them based on
time window and on orderItemId to update the current inventory of these
items in the warehouse.

Consequently I think that for Streams DSL to be consistent it should
provide the capability to join on FK first and then to aggregate optionally
with the separate operation only if required.
Creating a 'fat' message is an edge use case and I think it can not be done
in a separate subsequent operation as you rightfully noticed. So maybe for
that one we can have a separate aggregationJoin as you are proposing with
FK for this use case?
I however think that we should not call 'normal' FK join the 'flatMapJoin'
join, because flat map means we are splitting something when in fact we
just join without an aggregation.

To summarize I would compare two potential solutions for joining
stream-table on FK as follows:

   - Solution proposed in KIP-955: Join stream to table on FK without any
   aggregation. Aggregate as the next step if required (this would be a
   separate KIP - like aggregateJoin on FK)
  - Pros:
 - better performance because no aggregation with subsequent split
 will be performed
 - Logical separation of join and aggregate. This KIP only proposes
 to add FK join. Aggregating could be added in a separate KIP.
 - There are could be different types of aggregation or other data
 transformations added to the pipeline after FK join - not
just creating
 "fat" massage
  - Cons:
 - Solution requires new DSL API
  - Alternative solution discussed in email below: Use existing DSL to
   rekey the right table first to the new table with FK as the PK, and
   aggregate the records with the same PK into arrays. Join stream to table on
   PK (common message key)
  - Pros:
 - Use existing DSL, no new DSL is required to be implemented for
 the cases when 'fat' message is required at the end and
scalability is not
 an important consideration.
  - Cons:
 - This solution does not solve the more general problem of
 stream-table on FK join when no aggregation is required or
different type
 of aggregation is required (for example on child table key)
 - Solution will not scale well in a situation when there is a high
 number of updates on the right table. In my example of the
real life use
 case in email below there are several hundreds updates to
each left hand
 table of the join for each event in the right hand stream.
Overall number
 of updates to left hand tables in Kafka is several thousands
per sec at
 peak hours. For the reason above even if "fat" messages are
required the
 performance will be better with aggregateJoin based on
KIP-955 design when
 messages are aggregated per stream event and not per table update.

Disclaimer: I did not test both solutions side by side for performance. For
now I am just using design observations for performance/scalability
projections.

Any additions to pros/cons? Any other solution alternatives?

Regards,

Igor



On Thu, Aug 10, 2023 at 7:58 PM Matthias J. Sax  wrote:

> Thanks. Seems we are on the same page now what the requirement are?
> That's good progress!
>
>
> > This solution was considered when in KIP-213 for the existing
> > table-table FK join. There is a discussion on disadvantages of using this
> > approach in the article related to KIP-213 and I think the same
> > disadvantages will apply to this KIP.
>
> I am not sure. The difference (to me) is, that for KIP-213, if we
> aggregate the right input table, we would need to split the "fat result"
> record to flatten the individual result record we want to have. But for
> your case it seem

[jira] [Created] (KAFKA-15336) Connect plugin Javadocs should mention serviceloader manifests

[Greg Harris (Jira)]

Greg Harris created KAFKA-15336:
---

 Summary: Connect plugin Javadocs should mention serviceloader 
manifests
 Key: KAFKA-15336
 URL: https://issues.apache.org/jira/browse/KAFKA-15336
 Project: Kafka
  Issue Type: Improvement
  Components: KafkaConnect
Affects Versions: 3.6.0
Reporter: Greg Harris
Assignee: Greg Harris


Similar to the ConfigProvider, the Javadocs for the Connect plugin classes 
should mention that plugin implementations should have ServiceLoader manifests.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

[Jeff Kim]

Hi Calvin,

Thanks for the KIP! I'm still digesting it but I have two questions:

> In the scenario raised in the motivation section, the server may receive
ack=1 messages during T1 and advance High Watermark when the leader
is the only one in ISR.

To confirm, the current protocol allows advancing the HWM if all brokers in
the ISR append to their logs (in this case only the leader). And we're
proposing
to advance the HWM only when  brokers
replicate. Is this correct?

> Then, if we elect broker 1 as the leader at T4, though we can guarantee
the safety of ack=all messages, the High Watermark may move backward
which causes further impacts on the consumers.

How can broker 1 become the leader if it was ineligible in T3? Or are
you referring to broker 2?

Thanks,
Jeff

On Thu, Aug 10, 2023 at 6:48 PM Calvin Liu 
wrote:

> Hi everyone,
> I'd like to discuss a series of enhancement to the replication protocol.
>
> A partition replica can experience local data loss in unclean shutdown
> scenarios where unflushed data in the OS page cache is lost - such as an
> availability zone power outage or a server error. The Kafka replication
> protocol is designed to handle these situations by removing such replicas
> from the ISR and only re-adding them once they have caught up and therefore
> recovered any lost data. This prevents replicas that lost an arbitrary log
> suffix, which included committed data, from being elected leader.
> However, there is a "last replica standing" state which when combined with
> a data loss unclean shutdown event can turn a local data loss scenario into
> a global data loss scenario, i.e., committed data can be removed from all
> replicas. When the last replica in the ISR experiences an unclean shutdown
> and loses committed data, it will be reelected leader after starting up
> again, causing rejoining followers to truncate their logs and thereby
> removing the last copies of the committed records which the leader lost
> initially.
>
> The new KIP will maximize the protection and provides MinISR-1 tolerance to
> data loss unclean shutdown events.
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-966%3A+Eligible+Leader+Replicas
>

Re: [DISCUSS] KIP-714: Client metrics and observability

[Tom Bentley]

Hi Andrew,

Thanks for picking this KIP up. I know you've started a vote, so these are
unhelpfully late... sorry about that, but hopefully they're still useful.

1. "The Kafka Java client provides an API for the application to read the
generated client instance id to assist in mapping and identification of the
client based on the collected metrics." In the multi-client, single-process
model perhaps it would be desirable to have the option of including this in
log messages emitted by the client library.

2. "Mapping the client instance id to an actual application instance
running on a (virtual) machine can be done by inspecting the metrics
resource labels, such as the client source address and source port, or
security principal, all of which are added by the receiving broker." The
specific example of correlation given here (source IP address) is
problematic in environments where there may be network proxies (e.g.
Kubernetes ingress) on the path between client and broker: The broker sees
the IP address of the proxy. This is a rough edge which could be smoothed
out if Kafka supported the PROXY protocol[1] which seems to have become
something of a defacto standard. I'm not suggesting this need to be part of
the KIP, but perhaps it could be added to Future Work?
[1]: http://www.haproxy.org/download/2.9/doc/proxy-protocol.txt

3. Compression... just an idle idea, but I wonder if a useful further
improvement in compression ratio could be achieve using zstd's support for
dictionary compression[2]. I.e. a client could initially use training mode
when sending metrics, but eventually include a dictionary to be used for
subsequent metric sends. It's probably not worth the effort (at least for
the initial implementation), but since you've gone to the effort of
providing some numbers anyway, maybe it's not much additional effort to at
least find out whether this makes a useful difference.
[2]: http://facebook.github.io/zstd/#small-data

4. Maybe I didn't spot it, but I assume the initial
GetTelemetrySubscriptionsRequest
happens after authentication?

5. Rogue clients -- There are some interesting things to consider if we're
trying to defend against a genuinely adversarial client.

a) Client sends profiling information to all brokers at the maximum rate.
Each broker forwards to the time series DB. Obviously this scales linearly
with number of brokers, but it's clear that the load on the tsdb could be
many times larger than users might expect.
b) Client sends crafted compressed data which decompresses to require more
memory that the broker can allocate.

6. Shadowing the OLTP and protobuf jars -- to be clear by this you mean
both bundling _and_ relocating?

7. "In case the cluster load induced from metrics requests becomes
unmanageable the remedy is to temporarily remove or limit configured
metrics subscriptions.  " How would a user know that the observed load was
due to handling metrics requests?

8. If I understand correctly, when the configured metrics reporter does not
implement the new interface the client would still follow the described
protocol only to have nowhere to send the metrics. Am I overlooking
something?

Thanks again,

Tom

On Fri, 11 Aug 2023 at 07:52, Andrew Schofield <
andrew_schofield_j...@outlook.com> wrote:

> Hi Doguscan,
> Thanks for your question.
>
> If the target broker is unreachable, the client can send the metrics to
> another
> broker. It can select any of the other brokers for this purpose. What I
> expect in
> practice is that it loses connection to the broker it’s being using for
> metrics,
> chooses or establishes a connection to another broker, and then selects
> that
> broker for subsequent metrics pushes.
>
> Thanks,
> Andrew
>
> > On 8 Aug 2023, at 08:34, Doğuşcan Namal 
> wrote:
> >
> > Thanks for your answers Andrew. I share your pain that it took a while
> for
> > you to get this KIP approved and you want to reduce the scope of it, will
> > be happy to help you with the implementation :)
> >
> > Could you help me walk through what happens if the target broker is
> > unreachable? Is the client going to drop these metrics or is it going to
> > send it to the other brokers it is connected to? This information is
> > crucial to understand the client side impact on leadership failovers.
> > Moreover, in case of partial outages, such as only the network between
> the
> > client and the broker is partitioned whereas the network within the
> cluster
> > is healthy, practically there is no other way than the client side
> metrics
> > to identify this problem.
> >
> > Doguscan
> >
> > On Fri, 4 Aug 2023 at 15:33, Andrew Schofield <
> > andrew_schofield_j...@outlook.com> wrote:
> >
> >> Hi Doguscan,
> >> Thanks for your comments. I’m glad to hear you’re interested in this
> KIP.
> >>
> >> 1) It is preferred that a client sends its metrics to the same broker
> >> connection
> >> but actually it is able to send them to any broker. As a result, if a
> >> broker becomes
> >> unhealthy, the cli

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2091

[Apache Jenkins Server]

See 

Re: [VOTE] KIP-962 Relax non-null key requirement in Kafka Streams

[Bill Bejeck]

+1(binding)

On Fri, Aug 11, 2023 at 7:33 AM Lucas Brutschy
 wrote:

> +1 (non-binding)
>
> On Fri, Aug 11, 2023 at 1:08 AM Matthias J. Sax  wrote:
> >
> > +1 (binding)
> >
> > On 8/10/23 12:31 PM, Florin Akermann wrote:
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-962%3A+Relax+non-null+key+requirement+in+Kafka+Streams
> > >
>

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2090

[Apache Jenkins Server]

See 

Re: [VOTE] KIP-962 Relax non-null key requirement in Kafka Streams

[Lucas Brutschy]

+1 (non-binding)

On Fri, Aug 11, 2023 at 1:08 AM Matthias J. Sax  wrote:
>
> +1 (binding)
>
> On 8/10/23 12:31 PM, Florin Akermann wrote:
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-962%3A+Relax+non-null+key+requirement+in+Kafka+Streams
> >

[jira] [Created] (KAFKA-15335) Support custom SSL configuration for Kafka Connect RestServer

[Taras Ledkov (Jira)]

Taras Ledkov created KAFKA-15335:


 Summary: Support custom SSL configuration for Kafka Connect 
RestServer
 Key: KAFKA-15335
 URL: https://issues.apache.org/jira/browse/KAFKA-15335
 Project: Kafka
  Issue Type: New Feature
  Components: connect
Reporter: Taras Ledkov


Root to track KIP-967



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: What are the biggest issues with Apache Kafka?

[Divij Vaidya]

Hey Liam

Thanks for asking this question. I have been meaning to write a post to the
community for a long time about potential open areas where newcomers can
contribute but it never made it to priority in my to-do list.

In addition to what others mentioned above, here's a couple of options to
pick from. It's not an exhaustive list and I would be able to help more if
you tell me what you folks are interested in working on (e.g. on server,
client side, streams etc.) and what is the current familiarity with Kafka
code base. I can personally provide rapid reviews for option 1 and option
3, since those are the ones I feel most passionate about, but can't promise
time commitment from my side for other options.

*Option 1: KIP-405 (Tiered Storage) related work*

We are targeting an early access [1] release for KIP-405 [2] (tiered
storage in Kafka) for the upcoming version in 3.6. There is loads of work
left to polish this feature and make it production ready. If you like, you
can help over there. You can pick up any "unassigned" ticket from
https://issues.apache.org/jira/browse/KAFKA-7739 OR pick up a ticket where
the assigned person hasn't provided an update in the last 1 month.

*Option 2: Metrics related work*

We currently use two different ways of capturing metrics on the
broker/server. Historically we started with Yammer, moved to using
KafkaMetrics starting on clients but more recently we started using
KafkaMetrics on broker too. Currently the majority of broker metrics use
Yammer (which has it's own set of problems such as we are using a 10 year
old library) but the alternative KafkaMetrics has a slow histogram [2].
Here's a recent discussion about this:
https://lists.apache.org/thread/jww851jcyjtsq010bbt81b5dgwzqrgwx and
https://lists.apache.org/thread/f5wknqhmoo5lml99np7ksocz7fyk3m0r. You will
find that on the broker, KafkaRaftMetrics uses KafkaMetrics but
QuorumControllerMetrics uses Yammer metrics.We need someone in the
community pick up unifying this so that we can start using only one
methodology moving ahead. My recommendation would be to upgrade the library
of Yammer to use the latest drop wizard library as proposed in
https://cwiki.apache.org/confluence/display/KAFKA/KIP-510%3A+Metrics+library+upgrade
but there are backwarrd compatibility problems associated with it. My
colleague Christo has done some digging in the past on this and found that
the major problem of completing KIP-510 comes from the usage of
https://github.com/xvrl/kafka/blob/01208fd218286d2cd318a891f2cb5883422283b1/core/src/main/java/kafka/metrics/FilteringJmxReporter.java
introduced in KIP-544. This functionality is no longer directly available
in Dropwizard 4.2.0.
Can you dig more into this and see if there is a way to upgrade without
impacting backward compatibility?

To summarise option 2, we have the following problems:
1. We use 10 year old version of a library for capturing yammer metrics
2. Histogram calculation in metrics is very expensive. See:
https://issues.apache.org/jira/browse/KAFKA-15192?focusedCommentId=17744169&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17744169

3. KafkaMetrics library and Yammer metrics both have downsides as captured
in https://issues.apache.org/jira/browse/KAFKA-15058,
https://issues.apache.org/jira/browse/KAFKA-15154 and

*Option 3: Zero copy over SSL*

This is more of a personal project which I am not getting time to finish
up. Today zero copy doesn't have SSL enabled in Kafka. However, there is a
path forward on newer linux kernels by using kTLS. My idea is to have Kafka
use dynamically bound openssl (>=3.0) via netty-tcnative. Openssl 3.0 and
above can be compiled with the ability to enable kTLS. Hence, it should be
possible to use Kafka + netty-tcnative + openSSL compiled with ktls flag on
the OS to enable zero-copy even for SSL workloads. I can fill you in if
this is something that you are interested in pursuing.

*Option 4: Getting rid of easy mock & power mock dependencies from Kafka*

We have been making slow and steady progress towards achieving this goal
and it is being tracked in https://issues.apache.org/jira/browse/KAFKA-7438.
But it has been slow moving either because of code reviewer bandwidth or
because of lack of folks implementing the tests. We can use your help in
bringing it across the finish line.


[1]
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Tiered+Storage+Early+Access+Release+Notes
[2]
https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage

--
Divij Vaidya

Divij Vaidya



On Fri, Aug 11, 2023 at 4:55 AM ziming deng 
wrote:

> Hi Liam,
>
> The Apache Kafka project has several modules, I think you should firstly
> select a module you are interested in.
>
> For example, we are currently working on KIP-500 related features, which
> includes
> 1. KIP-856: KRaft Disk Failure Recovery,
> 2. KIP-642: Dynamic quorum reassignment,
> 3. kafka-metadata-shell.sh,
> 4. KIP-866: ZooKeeper to KRaft Migr