Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1259

[Apache Jenkins Server]

See 

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #1258

[Apache Jenkins Server]

See 

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1257

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 507023 lines...]
[2022-09-26T23:29:02.737Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 166 > TimeWindowedKStreamIntegrationTest > [ON_WINDOW_CLOSE_true] 
> 
org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest.shouldAggregateWindowedWithGrace[ON_WINDOW_CLOSE_true]
 STARTED
[2022-09-26T23:29:02.737Z] 
[2022-09-26T23:29:02.737Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 166 > TimeWindowedKStreamIntegrationTest > [ON_WINDOW_CLOSE_true] 
> 
org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest.shouldAggregateWindowedWithGrace[ON_WINDOW_CLOSE_true]
 PASSED
[2022-09-26T23:29:02.737Z] 
[2022-09-26T23:29:02.737Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 166 > TimeWindowedKStreamIntegrationTest > [ON_WINDOW_CLOSE_true] 
> 
org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest.shouldRestoreAfterJoinRestart[ON_WINDOW_CLOSE_true]
 STARTED
[2022-09-26T23:29:51.384Z] 
[2022-09-26T23:29:51.384Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 166 > TimeWindowedKStreamIntegrationTest > [ON_WINDOW_CLOSE_true] 
> 
org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest.shouldRestoreAfterJoinRestart[ON_WINDOW_CLOSE_true]
 PASSED
[2022-09-26T23:29:51.384Z] 
[2022-09-26T23:29:51.384Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 166 > TimeWindowedKStreamIntegrationTest > 
[ON_WINDOW_CLOSE_false] > 
org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest.shouldThrowUnlimitedWindows[ON_WINDOW_CLOSE_false]
 STARTED
[2022-09-26T23:29:51.384Z] 
[2022-09-26T23:29:51.384Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 166 > TimeWindowedKStreamIntegrationTest > 
[ON_WINDOW_CLOSE_false] > 
org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest.shouldThrowUnlimitedWindows[ON_WINDOW_CLOSE_false]
 PASSED
[2022-09-26T23:29:51.384Z] 
[2022-09-26T23:29:51.384Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 166 > TimeWindowedKStreamIntegrationTest > 
[ON_WINDOW_CLOSE_false] > 
org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest.shouldAggregateWindowedWithNoGrace[ON_WINDOW_CLOSE_false]
 STARTED
[2022-09-26T23:29:51.384Z] 
[2022-09-26T23:29:51.384Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 166 > TimeWindowedKStreamIntegrationTest > 
[ON_WINDOW_CLOSE_false] > 
org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest.shouldAggregateWindowedWithNoGrace[ON_WINDOW_CLOSE_false]
 PASSED
[2022-09-26T23:29:51.384Z] 
[2022-09-26T23:29:51.384Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 166 > TimeWindowedKStreamIntegrationTest > 
[ON_WINDOW_CLOSE_false] > 
org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest.shouldAggregateWindowedWithGrace[ON_WINDOW_CLOSE_false]
 STARTED
[2022-09-26T23:29:51.384Z] 
[2022-09-26T23:29:51.384Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 166 > TimeWindowedKStreamIntegrationTest > 
[ON_WINDOW_CLOSE_false] > 
org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest.shouldAggregateWindowedWithGrace[ON_WINDOW_CLOSE_false]
 PASSED
[2022-09-26T23:29:51.384Z] 
[2022-09-26T23:29:51.384Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 166 > TimeWindowedKStreamIntegrationTest > 
[ON_WINDOW_CLOSE_false] > 
org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest.shouldRestoreAfterJoinRestart[ON_WINDOW_CLOSE_false]
 STARTED
[2022-09-26T23:30:40.028Z] 
[2022-09-26T23:30:40.028Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 166 > TimeWindowedKStreamIntegrationTest > 
[ON_WINDOW_CLOSE_false] > 
org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest.shouldRestoreAfterJoinRestart[ON_WINDOW_CLOSE_false]
 PASSED
[2022-09-26T23:30:41.796Z] 
[2022-09-26T23:30:41.796Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 166 > HandlingSourceTopicDeletionIntegrationTest > 
shouldThrowErrorAfterSourceTopicDeleted STARTED
[2022-09-26T23:30:48.852Z] 
[2022-09-26T23:30:48.852Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 166 > HandlingSourceTopicDeletionIntegrationTest > 
shouldThrowErrorAfterSourceTopicDeleted PASSED
[2022-09-26T23:30:49.796Z] 
[2022-09-26T23:30:49.796Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 166 > AdjustStreamThreadCountTest > 
testConcurrentlyAccessThreads() STARTED
[2022-09-26T23:30:53.800Z] 
[2022-09-26T23:30:53.800Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 166 > AdjustStreamThreadCountTest > 
testConcurrentlyAccessThreads() PASSED
[2022-09-26T23:30:53.800Z] 
[2022-09-26T23:30:53.800Z] Gradle Test Run :streams:integrationTest > Gradle 
Test 

Re: [DISCUSS] KIP-871: Fix ByteBufferSerializer#serialize(String, ByteBuffer) compatible problem

[Guozhang Wang]

Hello ShunKang,

Thanks for filing the report. I looked at the source code and I agree it's
a bug. The reason we did not get it in the past is probably just because we
were inefficiently keep byte-copying and hence for almost all cases, the
offset is set to 0 when serialization happens.

Note that this fix does not really require a KIP since it does not change
any public APIs (for details on what kind of changes require a KIP
discussion, please see
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals#KafkaImprovementProposals-Whatisconsidereda%22majorchange%22thatneedsaKIP
?)


Guozhang

On Sun, Sep 25, 2022 at 6:52 AM ShunKang Lin 
wrote:

> Hi all,
>
> I'd like to start a new discussion thread on KIP-871 (Kafka Client) which
> proposes that fix ByteBufferSerializer#serialize(String, ByteBuffer)
> compatible problem.
>
> Links:
>
> KIP:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=228495816
>
> Jira: https://issues.apache.org/jira/browse/KAFKA-4852
>
> Thanks,
> ShunKang
>


-- 
-- Guozhang

[jira] [Created] (KAFKA-14261) Dependency Vulnerability Scan Results (Mend/WhiteSource)

[VeeVee Wang (Jira)]

VeeVee Wang created KAFKA-14261:
---

 Summary: Dependency Vulnerability Scan Results (Mend/WhiteSource)
 Key: KAFKA-14261
 URL: https://issues.apache.org/jira/browse/KAFKA-14261
 Project: Kafka
  Issue Type: Bug
  Components: security
Affects Versions: 3.2.3
Reporter: VeeVee Wang
 Attachments: GH_kafka-vulnerability-report.xlsx

The Kafka repository was scanned with Mend's (formerly WhiteSource) SCA 
(software composition analysis) tool for 3rd party dependency vulnerabilities. 
We scanned Kafka version 3.2.3 on 9/20. 

The scan result detected the following instances of vulnerability severities:
 * 12 highs
 * 12 mediums
 * 1 low

We would like to submit the Mend findings (attached to this ticket) as a bug 
with the request to update to non-vulnerable library versions. In the attached 
spreadsheet, column W "Top Fix" has notes on non-vulnerable versions to upgrade 
to.

Is there an SLA or typical amount of time to remediate vulnerabilities in the 
Kafka repo? 

Thank you. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.3 #87

[Apache Jenkins Server]

See 

Re: [VOTE] 3.3.0 RC2

[David Arthur]

Thanks for the votes, everyone!

Here is the best recent run of the system tests on 3.3
http://confluent-kafka-system-test-results.s3-us-west-2.amazonaws.com/3.3/2022-09-24--001.system-test-kafka-3.3--1664037736--confluentinc--3.3--b084e9068c/report.html.
I'm currently re-running the failed tests to confirm that they are
merely flaky and not broken. One exception is the failing delegation
token test that is consistently failing. This appears to be an issue
with the test itself due to changes in the command output introduced
recently.

Similarly, the unit/integration tests are mostly passing, with some
flaky failures. Here is a recent run that has two out of three jobs
passing https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.3/86/.

Once I verify the system tests are good, I'll update this thread and
close out the vote.

Thanks!
David


On Mon, Sep 26, 2022 at 2:24 PM Ismael Juma  wrote:
>
> +1 provided that the system test results are good. Can you please post them
> along with the JUnit test results (these seem ok although there are some
> flakes)?
>
> I tested the kraft quick start with the Scala 2.13 binary and ran the tests
> on the source release. I noticed a non-blocker issue with the KRaft readme
> and submitted a PR:
>
> https://github.com/apache/kafka/pull/12688
>
> Ismael
>
> On Tue, Sep 20, 2022 at 4:17 PM David Arthur  wrote:
>
> > Hello Kafka users, developers and client-developers,
> >
> > This is the second release candidate for Apache Kafka 3.3.0. Many new
> > features and bug fixes are included in this major release of Kafka. A
> > significant number of the issues in this release are related to KRaft,
> > which will be considered "production ready" as part of this release
> > (KIP-833)
> >
> > KRaft improvements:
> > * KIP-778: Online KRaft to KRaft Upgrades
> > * KIP-833: Mark KRaft as Production Ready
> > * KIP-835: Monitor Quorum health (many new KRaft metrics)
> > * KIP-836: Expose voter lag via kafka-metadata-quorum.sh
> > * KIP-841: Fenced replicas should not be allowed to join the ISR in KRaft
> > * KIP-859: Add Metadata Log Processing Error Related Metrics
> >
> > Other major improvements include:
> > * KIP-618: Exactly-Once Support for Source Connectors
> > * KIP-831: Add metric for log recovery progress
> > * KIP-827: Expose logdirs total and usable space via Kafka API
> > * KIP-834: Add ability to Pause / Resume KafkaStreams Topologies
> >
> > The full release notes are available here:
> > https://home.apache.org/~davidarthur/kafka-3.3.0-rc2/RELEASE_NOTES.html
> >
> > Please download, test and vote by Monday, Sep 26 at 5pm EDT
> >
> > Also, huge thanks to José for running the release so far. He has done
> > the vast majority of the work to prepare this rather large release :)
> >
> > -
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > https://kafka.apache.org/KEYS
> >
> > * Release artifacts to be voted upon (source and binary):
> > https://home.apache.org/~davidarthur/kafka-3.3.0-rc2/
> >
> > * Maven artifacts to be voted upon:
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >
> > * Javadoc: https://home.apache.org/~davidarthur/kafka-3.3.0-rc2/javadoc/
> >
> > * Tag to be voted upon (off 3.3 branch) is the 3.3.0 tag:
> > https://github.com/apache/kafka/releases/tag/3.3.0-rc2
> >
> > * Documentation:  https://kafka.apache.org/33/documentation.html
> >
> > * Protocol: https://kafka.apache.org/33/protocol.html
> >
> >
> >
> >
> > Successful Jenkins builds to follow in a future update to this email.
> >
> >
> > Thanks!
> > David Arthur
> >

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1256

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 508863 lines...]
[2022-09-26T21:03:29.661Z] > Task :connect:api:testJar
[2022-09-26T21:03:29.661Z] > Task :connect:api:testSrcJar
[2022-09-26T21:03:29.661Z] > Task 
:connect:api:publishMavenJavaPublicationToMavenLocal
[2022-09-26T21:03:29.661Z] > Task :connect:api:publishToMavenLocal
[2022-09-26T21:03:29.661Z] 
[2022-09-26T21:03:29.661Z] > Task :streams:javadoc
[2022-09-26T21:03:29.661Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk@2/streams/src/main/java/org/apache/kafka/streams/processor/StreamPartitioner.java:50:
 warning - Tag @link: reference not found: 
org.apache.kafka.clients.producer.internals.DefaultPartitioner
[2022-09-26T21:03:30.609Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk@2/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:890:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-09-26T21:03:30.609Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk@2/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:919:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-09-26T21:03:30.609Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk@2/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:939:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-09-26T21:03:30.609Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk@2/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:854:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-09-26T21:03:30.609Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk@2/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:890:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-09-26T21:03:30.609Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk@2/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:919:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-09-26T21:03:30.609Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk@2/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:939:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-09-26T21:03:31.558Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk@2/streams/src/main/java/org/apache/kafka/streams/kstream/Produced.java:84:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-09-26T21:03:31.558Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk@2/streams/src/main/java/org/apache/kafka/streams/kstream/Produced.java:136:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-09-26T21:03:31.558Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk@2/streams/src/main/java/org/apache/kafka/streams/kstream/Produced.java:147:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-09-26T21:03:31.558Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk@2/streams/src/main/java/org/apache/kafka/streams/kstream/Repartitioned.java:101:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-09-26T21:03:31.558Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk@2/streams/src/main/java/org/apache/kafka/streams/kstream/Repartitioned.java:167:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-09-26T21:03:32.505Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk@2/streams/src/main/java/org/apache/kafka/streams/TopologyConfig.java:58:
 warning - Tag @link: missing '#': "org.apache.kafka.streams.StreamsBuilder()"
[2022-09-26T21:03:32.505Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk@2/streams/src/main/java/org/apache/kafka/streams/TopologyConfig.java:58:
 warning - Tag @link: can't find org.apache.kafka.streams.StreamsBuilder() in 
org.apache.kafka.streams.TopologyConfig
[2022-09-26T21:03:32.505Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk@2/streams/src/main/java/org/apache/kafka/streams/TopologyDescription.java:38:
 warning - Tag @link: reference not found: ProcessorContext#forward(Object, 
Object) forwards
[2022-09-26T21:03:32.505Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk@2/streams/src/main/java/org/apache/kafka/streams/query/Position.java:44:
 warning - Tag @link: can't find query(Query,
[2022-09-26T21:03:32.505Z]  PositionBound, boolean) in 
org.apache.kafka.streams.processor.StateStore
[2022-09-26T21:03:32.505Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk@2/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:109:
 warning - Tag @link: reference not found: this#getResult()
[2022-09-26T21:03:32.505Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk@2/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:116:
 warning - Tag @link: reference not found: this#getFailureReason()

Re: [DISCUSS] KIP-848: The Next Generation of the Consumer Rebalance Protocol

[Guozhang Wang]

Re versions: fair enough, I think it's okay to keep it as strings. Just to
clarify the concern I had is that if we do want to augment it in the
future, it will be harder to change a `string` field into a `struct` :P

Re onAssignment: actually even with the current proposal, since we are
giving the final encoded metadata upon the first `onAssignment` call for
the generation already, Streams would infer all the standby tasks that need
are migrated in / out, so for Streams' purposes, we'd need to handle such
decoupling anyways, and it's actually the opposite: if the first
`onAssignment` does not include those yet-to-be-assigned then Streams would
not know if a migrating out standby would really be promoting to active
later or should it be completely removed, until later when next
`onAssignment` is called. And that's where I mentioned "we'd need to figure
out which onAssignment is the final call" etc. If we have all the
partitions on the `onAssignment`, then we can infer such actions and decide
whether we should close a standby task immediately or just recycle it and
wait for the active task to be assigned eventually.

On the other hand, if we call onAssignment incrementally similar to
onPartitionsAssigned, in which we include both assigned and
yet-to-be-assigned partitions, then for people who implement both
interfaces, they can just ignore the `onPartitionsAssigned` since they have
all knowledge they need from onAssignment already. And that's what I'm
trying to avoid by making the functionality of the two more separated.


Guozhang




On Mon, Sep 26, 2022 at 11:30 AM David Jacot 
wrote:

> Regarding the version, I would rather add this later when we have a
> clear use case for it. It seems to me that we are speculating here. I
> understand your point but I am not fully convinced by the solution at
> the moment. Would you agree with this?
>
> Regarding the onAssignment, I was thinking about the case where a task
> is promoted from standby to active. In this case, having onAssignment
> with the combined partitions could make it difficult, no? I am
> thinking that the assignor will have to remove the standby based on
> the metadata but it won't know what to do after that. If the partition
> is assigned directly, it can convert it to an active task. On the
> other hand, if the partition is not available yet, it would have to
> keep it as a standby until the partition is really assigned. It seems
> to be that the assignor won't have the information to make this
> decision, no? In this case, decouple the "why" from the "when" seems
> to make things harder. I am not so familiar with Streams though so my
> intuition could be wrong here.
>
> David
>
> On Mon, Sep 26, 2022 at 7:26 PM Guozhang Wang  wrote:
> >
> > Regarding the version, what I was thinking is that in the HB request, for
> > "serverAssignor" field, instead of just having it as a single string
> field,
> > maybe we could consider also making it a structure that includes: name,
> > minimumVersion, maximumVersion. Where the minimumVersion/maximumVersion
> > means the versions of the server assignor that the client work best with.
> > That being said, I agree with you that such information may also be
> > inferred elsewhere e.g. by looking into the "rackId" field, and see if it
> > contains a hyphen or not etc. All I was wondering is that, if such
> version
> > information would be useful for the server assignors to determine its
> > actual assignment logic. I do not feel very strong about this one though
> > --- even if we do not add it now, we can potentially add later, it's just
> > that changing a single string field to a structure would be hard for
> > compatibility and we'd then probably have to add top-level fields.
> >
> > Regarding the `onAssignment` logic, again my train of thoughts is that,
> if
> > users want to know exactly when a partition is assigned / revoked, they
> > would be leveraging on the rebalance callbacks, as that's what people
> > should rely on to determine "when" partitions are assigned. The
> > `onAssignment` should be used for getting "why" such partition assignment
> > decision is made, and hence returning `combined partitions` would be
> okay.
> > Streams e.g. implement both rebalance callbacks and the assignors, and it
> > gets the "when" from the former (and create/close active tasks
> accordingly)
> > and the "why" from the latter (and update its global info bookkeeping as
> > well as standby maintenance accordingly). Most users would be just
> > interested in the rebalance callback, and not implement their own
> assignor
> > at all if they do not care about "why" as they trust the server assignors
> > would take good care of those, and only about "when". So if we did build
> > such two types of APIs from scratch, I'd indeed feel that not providing
> the
> > partitions but only the metadata for `onAssignment` may be less confusing
> > and push users to separate the usage of these two more clearly, but since
> > we 

[jira] [Resolved] (KAFKA-14207) Add a 6.10 section for KRaft

[Jose Armando Garcia Sancio (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jose Armando Garcia Sancio resolved KAFKA-14207.

Resolution: Fixed

> Add a 6.10 section for KRaft
> 
>
> Key: KAFKA-14207
> URL: https://issues.apache.org/jira/browse/KAFKA-14207
> Project: Kafka
>  Issue Type: Sub-task
>  Components: documentation
>Affects Versions: 3.3.0
>Reporter: Jose Armando Garcia Sancio
>Assignee: Jose Armando Garcia Sancio
>Priority: Major
>  Labels: documentation, kraft
>
> The section should talk about:
>  # Limitation
>  # Recommended deployment: external controller
>  # How to start a KRaft cluster.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-848: The Next Generation of the Consumer Rebalance Protocol

[David Jacot]

Regarding the version, I would rather add this later when we have a
clear use case for it. It seems to me that we are speculating here. I
understand your point but I am not fully convinced by the solution at
the moment. Would you agree with this?

Regarding the onAssignment, I was thinking about the case where a task
is promoted from standby to active. In this case, having onAssignment
with the combined partitions could make it difficult, no? I am
thinking that the assignor will have to remove the standby based on
the metadata but it won't know what to do after that. If the partition
is assigned directly, it can convert it to an active task. On the
other hand, if the partition is not available yet, it would have to
keep it as a standby until the partition is really assigned. It seems
to be that the assignor won't have the information to make this
decision, no? In this case, decouple the "why" from the "when" seems
to make things harder. I am not so familiar with Streams though so my
intuition could be wrong here.

David

On Mon, Sep 26, 2022 at 7:26 PM Guozhang Wang  wrote:
>
> Regarding the version, what I was thinking is that in the HB request, for
> "serverAssignor" field, instead of just having it as a single string field,
> maybe we could consider also making it a structure that includes: name,
> minimumVersion, maximumVersion. Where the minimumVersion/maximumVersion
> means the versions of the server assignor that the client work best with.
> That being said, I agree with you that such information may also be
> inferred elsewhere e.g. by looking into the "rackId" field, and see if it
> contains a hyphen or not etc. All I was wondering is that, if such version
> information would be useful for the server assignors to determine its
> actual assignment logic. I do not feel very strong about this one though
> --- even if we do not add it now, we can potentially add later, it's just
> that changing a single string field to a structure would be hard for
> compatibility and we'd then probably have to add top-level fields.
>
> Regarding the `onAssignment` logic, again my train of thoughts is that, if
> users want to know exactly when a partition is assigned / revoked, they
> would be leveraging on the rebalance callbacks, as that's what people
> should rely on to determine "when" partitions are assigned. The
> `onAssignment` should be used for getting "why" such partition assignment
> decision is made, and hence returning `combined partitions` would be okay.
> Streams e.g. implement both rebalance callbacks and the assignors, and it
> gets the "when" from the former (and create/close active tasks accordingly)
> and the "why" from the latter (and update its global info bookkeeping as
> well as standby maintenance accordingly). Most users would be just
> interested in the rebalance callback, and not implement their own assignor
> at all if they do not care about "why" as they trust the server assignors
> would take good care of those, and only about "when". So if we did build
> such two types of APIs from scratch, I'd indeed feel that not providing the
> partitions but only the metadata for `onAssignment` may be less confusing
> and push users to separate the usage of these two more clearly, but since
> we already introduced partitions in `onAssignment` for compatibility I'm
> less keen on removing them.
>
>
> Guozhang
>
> On Mon, Sep 26, 2022 at 6:55 AM David Jacot 
> wrote:
>
> > Hi Guozhang,
> >
> > Regarding the version, my understanding is that the version would be
> > either the client software version or the request version, is this
> > correct? If so, we could indeed pass this information down to the
> > assignor via the interface. One way would be to pass a "server
> > context" to the assignor and that context would include that
> > information (and perhaps more). Is this what you are looking for?
> >
> > Regarding the onAssignment, I think that I understand your point. I
> > suppose that the assignor could also be clever and keep track of the
> > last metadata to decide whether it has to do something or not. One
> > question that is still not clear to me is whether the assignor needs
> > to know all the assigned partitions upfront regardless of whether they
> > are already revoked or not. Do you think that we need this as well?
> >
> > From an API perspective, we could have something like
> > onAssignment(Metadata(version, reason, metadata, assigned partitions,
> > pending partitions)). Where the assigned partitions are the partitions
> > ready to be used and the pending partitions are the one assigned to
> > the member but not revoked yet. I find it a bit weird that this method
> > would be called only once because the assignor would not know when the
> > pending partitions changes. That does not look like a clean API. An
> > alternative would be to use onAssignment(Metadata(version, reason,
> > metadata, combined partitions)) but this seems error prone because it
> > is not clear whether a 

Re: [VOTE] 3.3.0 RC2

[Ismael Juma]

+1 provided that the system test results are good. Can you please post them
along with the JUnit test results (these seem ok although there are some
flakes)?

I tested the kraft quick start with the Scala 2.13 binary and ran the tests
on the source release. I noticed a non-blocker issue with the KRaft readme
and submitted a PR:

https://github.com/apache/kafka/pull/12688

Ismael

On Tue, Sep 20, 2022 at 4:17 PM David Arthur  wrote:

> Hello Kafka users, developers and client-developers,
>
> This is the second release candidate for Apache Kafka 3.3.0. Many new
> features and bug fixes are included in this major release of Kafka. A
> significant number of the issues in this release are related to KRaft,
> which will be considered "production ready" as part of this release
> (KIP-833)
>
> KRaft improvements:
> * KIP-778: Online KRaft to KRaft Upgrades
> * KIP-833: Mark KRaft as Production Ready
> * KIP-835: Monitor Quorum health (many new KRaft metrics)
> * KIP-836: Expose voter lag via kafka-metadata-quorum.sh
> * KIP-841: Fenced replicas should not be allowed to join the ISR in KRaft
> * KIP-859: Add Metadata Log Processing Error Related Metrics
>
> Other major improvements include:
> * KIP-618: Exactly-Once Support for Source Connectors
> * KIP-831: Add metric for log recovery progress
> * KIP-827: Expose logdirs total and usable space via Kafka API
> * KIP-834: Add ability to Pause / Resume KafkaStreams Topologies
>
> The full release notes are available here:
> https://home.apache.org/~davidarthur/kafka-3.3.0-rc2/RELEASE_NOTES.html
>
> Please download, test and vote by Monday, Sep 26 at 5pm EDT
>
> Also, huge thanks to José for running the release so far. He has done
> the vast majority of the work to prepare this rather large release :)
>
> -
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> https://home.apache.org/~davidarthur/kafka-3.3.0-rc2/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>
> * Javadoc: https://home.apache.org/~davidarthur/kafka-3.3.0-rc2/javadoc/
>
> * Tag to be voted upon (off 3.3 branch) is the 3.3.0 tag:
> https://github.com/apache/kafka/releases/tag/3.3.0-rc2
>
> * Documentation:  https://kafka.apache.org/33/documentation.html
>
> * Protocol: https://kafka.apache.org/33/protocol.html
>
>
>
>
> Successful Jenkins builds to follow in a future update to this email.
>
>
> Thanks!
> David Arthur
>

Re: [VOTE] 3.3.0 RC2

[Jakub Scholz]

+1 (non-binding). I used the staged binaries (Scala 2.13) and Maven
artifacts and ran my tests. All seems to work fine. Thanks for running the
release.

Jakub

On Wed, Sep 21, 2022 at 1:17 AM David Arthur  wrote:

> Hello Kafka users, developers and client-developers,
>
> This is the second release candidate for Apache Kafka 3.3.0. Many new
> features and bug fixes are included in this major release of Kafka. A
> significant number of the issues in this release are related to KRaft,
> which will be considered "production ready" as part of this release
> (KIP-833)
>
> KRaft improvements:
> * KIP-778: Online KRaft to KRaft Upgrades
> * KIP-833: Mark KRaft as Production Ready
> * KIP-835: Monitor Quorum health (many new KRaft metrics)
> * KIP-836: Expose voter lag via kafka-metadata-quorum.sh
> * KIP-841: Fenced replicas should not be allowed to join the ISR in KRaft
> * KIP-859: Add Metadata Log Processing Error Related Metrics
>
> Other major improvements include:
> * KIP-618: Exactly-Once Support for Source Connectors
> * KIP-831: Add metric for log recovery progress
> * KIP-827: Expose logdirs total and usable space via Kafka API
> * KIP-834: Add ability to Pause / Resume KafkaStreams Topologies
>
> The full release notes are available here:
> https://home.apache.org/~davidarthur/kafka-3.3.0-rc2/RELEASE_NOTES.html
>
> Please download, test and vote by Monday, Sep 26 at 5pm EDT
>
> Also, huge thanks to José for running the release so far. He has done
> the vast majority of the work to prepare this rather large release :)
>
> -
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> https://home.apache.org/~davidarthur/kafka-3.3.0-rc2/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>
> * Javadoc: https://home.apache.org/~davidarthur/kafka-3.3.0-rc2/javadoc/
>
> * Tag to be voted upon (off 3.3 branch) is the 3.3.0 tag:
> https://github.com/apache/kafka/releases/tag/3.3.0-rc2
>
> * Documentation:  https://kafka.apache.org/33/documentation.html
>
> * Protocol: https://kafka.apache.org/33/protocol.html
>
>
>
>
> Successful Jenkins builds to follow in a future update to this email.
>
>
> Thanks!
> David Arthur
>

Re: [DISCUSS] KIP-848: The Next Generation of the Consumer Rebalance Protocol

[Guozhang Wang]

Regarding the version, what I was thinking is that in the HB request, for
"serverAssignor" field, instead of just having it as a single string field,
maybe we could consider also making it a structure that includes: name,
minimumVersion, maximumVersion. Where the minimumVersion/maximumVersion
means the versions of the server assignor that the client work best with.
That being said, I agree with you that such information may also be
inferred elsewhere e.g. by looking into the "rackId" field, and see if it
contains a hyphen or not etc. All I was wondering is that, if such version
information would be useful for the server assignors to determine its
actual assignment logic. I do not feel very strong about this one though
--- even if we do not add it now, we can potentially add later, it's just
that changing a single string field to a structure would be hard for
compatibility and we'd then probably have to add top-level fields.

Regarding the `onAssignment` logic, again my train of thoughts is that, if
users want to know exactly when a partition is assigned / revoked, they
would be leveraging on the rebalance callbacks, as that's what people
should rely on to determine "when" partitions are assigned. The
`onAssignment` should be used for getting "why" such partition assignment
decision is made, and hence returning `combined partitions` would be okay.
Streams e.g. implement both rebalance callbacks and the assignors, and it
gets the "when" from the former (and create/close active tasks accordingly)
and the "why" from the latter (and update its global info bookkeeping as
well as standby maintenance accordingly). Most users would be just
interested in the rebalance callback, and not implement their own assignor
at all if they do not care about "why" as they trust the server assignors
would take good care of those, and only about "when". So if we did build
such two types of APIs from scratch, I'd indeed feel that not providing the
partitions but only the metadata for `onAssignment` may be less confusing
and push users to separate the usage of these two more clearly, but since
we already introduced partitions in `onAssignment` for compatibility I'm
less keen on removing them.


Guozhang

On Mon, Sep 26, 2022 at 6:55 AM David Jacot 
wrote:

> Hi Guozhang,
>
> Regarding the version, my understanding is that the version would be
> either the client software version or the request version, is this
> correct? If so, we could indeed pass this information down to the
> assignor via the interface. One way would be to pass a "server
> context" to the assignor and that context would include that
> information (and perhaps more). Is this what you are looking for?
>
> Regarding the onAssignment, I think that I understand your point. I
> suppose that the assignor could also be clever and keep track of the
> last metadata to decide whether it has to do something or not. One
> question that is still not clear to me is whether the assignor needs
> to know all the assigned partitions upfront regardless of whether they
> are already revoked or not. Do you think that we need this as well?
>
> From an API perspective, we could have something like
> onAssignment(Metadata(version, reason, metadata, assigned partitions,
> pending partitions)). Where the assigned partitions are the partitions
> ready to be used and the pending partitions are the one assigned to
> the member but not revoked yet. I find it a bit weird that this method
> would be called only once because the assignor would not know when the
> pending partitions changes. That does not look like a clean API. An
> alternative would be to use onAssignment(Metadata(version, reason,
> metadata, combined partitions)) but this seems error prone because it
> is not clear whether a partition is usable or not. Or do you think
> that we should not provide the partitions but only the metadata?
>
> Best,
> David
>
> On Fri, Sep 23, 2022 at 9:40 PM Guozhang Wang  wrote:
> >
> > Hello David,
> >
> > On Fri, Sep 23, 2022 at 2:00 AM David Jacot  >
> > wrote:
> >
> > > Hey,
> > >
> > > > Just to clarify I was asking about the `version` of the assignor
> (i.e. up
> > > to what version that the client would support), and I do agree we
> would not
> > > need metadata. What I have in mind is that, for some specific built-in
> > > broker-assignors, e.g. rack-aware assignors, if it's possible that in a
> > > newer version we would have a hierarchical rack ID string format, like
> > > "tier1-tier2" etc, but if some client has not upgraded their rack ID
> > > would still be in old format. In this case, the broker then needs to
> choose
> > > the old versioned assignor. I'm probably making something up here for
> rack
> > > aware assignors, but I'm wondering if in general such an
> "auto-downgrade"
> > > behavior would be needed still for broker-side assignor, and if yes
> would
> > > "version" still be useful.
> > >
> > > Got it. That's an interesting thought. I think that the issue is that
> > > the 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.3 #86

[Apache Jenkins Server]

See 

Re: [VOTE] 3.3.0 RC2

[Bill Bejeck]

Hi Jose and David,

Thanks for running the release!
I did the following verification steps:
I performed the following validations:

   1.   Verified all checksums and signatures.
   2.   Built from source and ran unit tests.
   3.   Ran the first quickstart steps
  1. Zookeeper quick start
  2. KRaft quick start
  3. The raft test script
   4.  Spot checked the docs


I noticed the docs page is a little mal-formed, but this doesn't need to
block the release we can fix it separately.  Also, the quickstart page
still references 3.2, but this can be fixed on a followup docs PR.

+1(binding)

Thanks,
Bill

On Mon, Sep 26, 2022 at 12:01 PM David Jacot 
wrote:

> Thanks for running the release, José and David.
>
> I performed the following validations:
> * Verified all checksums and signatures.
> * Built from source and ran unit tests.
> * Ran the first quickstart steps for both ZK and KRaft.
> * Spotchecked the Javadocs.
>
> I have also noticed that the doc is malformed but we can fix this
> separately.
>
> I am +1 (binding), assuming that the system tests look good. It would
> be great if you could share them.
>
> Best,
> David
>
> On Mon, Sep 26, 2022 at 5:36 PM John Roesler  wrote:
> >
> > Thanks for running this, David!
> >
> > I've verified the signatures, looked at the docs, and run the quickstart
> (ZK and KRaft). I also ran the unit tests, as well as all the tests for
> Streams locally.
> >
> > The docs look a little malformed (the "collapse/expand" button floats
> over the text, the collapsed doc tree is only halfway collapsed, and
> there's a weird empty panel on the right).
> >
> > We can fix the docs site independent of this release, so I'm +1
> (binding).
> >
> > Thanks,
> > -John
> >
> > On Tue, Sep 20, 2022, at 18:17, David Arthur wrote:
> > > Hello Kafka users, developers and client-developers,
> > >
> > > This is the second release candidate for Apache Kafka 3.3.0. Many new
> > > features and bug fixes are included in this major release of Kafka. A
> > > significant number of the issues in this release are related to KRaft,
> > > which will be considered "production ready" as part of this release
> > > (KIP-833)
> > >
> > > KRaft improvements:
> > > * KIP-778: Online KRaft to KRaft Upgrades
> > > * KIP-833: Mark KRaft as Production Ready
> > > * KIP-835: Monitor Quorum health (many new KRaft metrics)
> > > * KIP-836: Expose voter lag via kafka-metadata-quorum.sh
> > > * KIP-841: Fenced replicas should not be allowed to join the ISR in
> KRaft
> > > * KIP-859: Add Metadata Log Processing Error Related Metrics
> > >
> > > Other major improvements include:
> > > * KIP-618: Exactly-Once Support for Source Connectors
> > > * KIP-831: Add metric for log recovery progress
> > > * KIP-827: Expose logdirs total and usable space via Kafka API
> > > * KIP-834: Add ability to Pause / Resume KafkaStreams Topologies
> > >
> > > The full release notes are available here:
> > >
> https://home.apache.org/~davidarthur/kafka-3.3.0-rc2/RELEASE_NOTES.html
> > >
> > > Please download, test and vote by Monday, Sep 26 at 5pm EDT
> > >
> > > Also, huge thanks to José for running the release so far. He has done
> > > the vast majority of the work to prepare this rather large release :)
> > >
> > > -
> > >
> > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > https://kafka.apache.org/KEYS
> > >
> > > * Release artifacts to be voted upon (source and binary):
> > > https://home.apache.org/~davidarthur/kafka-3.3.0-rc2/
> > >
> > > * Maven artifacts to be voted upon:
> > > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > >
> > > * Javadoc:
> https://home.apache.org/~davidarthur/kafka-3.3.0-rc2/javadoc/
> > >
> > > * Tag to be voted upon (off 3.3 branch) is the 3.3.0 tag:
> > > https://github.com/apache/kafka/releases/tag/3.3.0-rc2
> > >
> > > * Documentation:  https://kafka.apache.org/33/documentation.html
> > >
> > > * Protocol: https://kafka.apache.org/33/protocol.html
> > >
> > >
> > >
> > >
> > > Successful Jenkins builds to follow in a future update to this email.
> > >
> > >
> > > Thanks!
> > > David Arthur
>

Re: [VOTE] 3.3.0 RC2

[David Jacot]

Thanks for running the release, José and David.

I performed the following validations:
* Verified all checksums and signatures.
* Built from source and ran unit tests.
* Ran the first quickstart steps for both ZK and KRaft.
* Spotchecked the Javadocs.

I have also noticed that the doc is malformed but we can fix this separately.

I am +1 (binding), assuming that the system tests look good. It would
be great if you could share them.

Best,
David

On Mon, Sep 26, 2022 at 5:36 PM John Roesler  wrote:
>
> Thanks for running this, David!
>
> I've verified the signatures, looked at the docs, and run the quickstart (ZK 
> and KRaft). I also ran the unit tests, as well as all the tests for Streams 
> locally.
>
> The docs look a little malformed (the "collapse/expand" button floats over 
> the text, the collapsed doc tree is only halfway collapsed, and there's a 
> weird empty panel on the right).
>
> We can fix the docs site independent of this release, so I'm +1 (binding).
>
> Thanks,
> -John
>
> On Tue, Sep 20, 2022, at 18:17, David Arthur wrote:
> > Hello Kafka users, developers and client-developers,
> >
> > This is the second release candidate for Apache Kafka 3.3.0. Many new
> > features and bug fixes are included in this major release of Kafka. A
> > significant number of the issues in this release are related to KRaft,
> > which will be considered "production ready" as part of this release
> > (KIP-833)
> >
> > KRaft improvements:
> > * KIP-778: Online KRaft to KRaft Upgrades
> > * KIP-833: Mark KRaft as Production Ready
> > * KIP-835: Monitor Quorum health (many new KRaft metrics)
> > * KIP-836: Expose voter lag via kafka-metadata-quorum.sh
> > * KIP-841: Fenced replicas should not be allowed to join the ISR in KRaft
> > * KIP-859: Add Metadata Log Processing Error Related Metrics
> >
> > Other major improvements include:
> > * KIP-618: Exactly-Once Support for Source Connectors
> > * KIP-831: Add metric for log recovery progress
> > * KIP-827: Expose logdirs total and usable space via Kafka API
> > * KIP-834: Add ability to Pause / Resume KafkaStreams Topologies
> >
> > The full release notes are available here:
> > https://home.apache.org/~davidarthur/kafka-3.3.0-rc2/RELEASE_NOTES.html
> >
> > Please download, test and vote by Monday, Sep 26 at 5pm EDT
> >
> > Also, huge thanks to José for running the release so far. He has done
> > the vast majority of the work to prepare this rather large release :)
> >
> > -
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > https://kafka.apache.org/KEYS
> >
> > * Release artifacts to be voted upon (source and binary):
> > https://home.apache.org/~davidarthur/kafka-3.3.0-rc2/
> >
> > * Maven artifacts to be voted upon:
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >
> > * Javadoc: https://home.apache.org/~davidarthur/kafka-3.3.0-rc2/javadoc/
> >
> > * Tag to be voted upon (off 3.3 branch) is the 3.3.0 tag:
> > https://github.com/apache/kafka/releases/tag/3.3.0-rc2
> >
> > * Documentation:  https://kafka.apache.org/33/documentation.html
> >
> > * Protocol: https://kafka.apache.org/33/protocol.html
> >
> >
> >
> >
> > Successful Jenkins builds to follow in a future update to this email.
> >
> >
> > Thanks!
> > David Arthur

Re: [VOTE] 3.3.0 RC2

[John Roesler]

Thanks for running this, David!

I've verified the signatures, looked at the docs, and run the quickstart (ZK 
and KRaft). I also ran the unit tests, as well as all the tests for Streams 
locally.

The docs look a little malformed (the "collapse/expand" button floats over the 
text, the collapsed doc tree is only halfway collapsed, and there's a weird 
empty panel on the right).

We can fix the docs site independent of this release, so I'm +1 (binding).

Thanks,
-John

On Tue, Sep 20, 2022, at 18:17, David Arthur wrote:
> Hello Kafka users, developers and client-developers,
>
> This is the second release candidate for Apache Kafka 3.3.0. Many new
> features and bug fixes are included in this major release of Kafka. A
> significant number of the issues in this release are related to KRaft,
> which will be considered "production ready" as part of this release
> (KIP-833)
>
> KRaft improvements:
> * KIP-778: Online KRaft to KRaft Upgrades
> * KIP-833: Mark KRaft as Production Ready
> * KIP-835: Monitor Quorum health (many new KRaft metrics)
> * KIP-836: Expose voter lag via kafka-metadata-quorum.sh
> * KIP-841: Fenced replicas should not be allowed to join the ISR in KRaft
> * KIP-859: Add Metadata Log Processing Error Related Metrics
>
> Other major improvements include:
> * KIP-618: Exactly-Once Support for Source Connectors
> * KIP-831: Add metric for log recovery progress
> * KIP-827: Expose logdirs total and usable space via Kafka API
> * KIP-834: Add ability to Pause / Resume KafkaStreams Topologies
>
> The full release notes are available here:
> https://home.apache.org/~davidarthur/kafka-3.3.0-rc2/RELEASE_NOTES.html
>
> Please download, test and vote by Monday, Sep 26 at 5pm EDT
>
> Also, huge thanks to José for running the release so far. He has done
> the vast majority of the work to prepare this rather large release :)
>
> -
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> https://home.apache.org/~davidarthur/kafka-3.3.0-rc2/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>
> * Javadoc: https://home.apache.org/~davidarthur/kafka-3.3.0-rc2/javadoc/
>
> * Tag to be voted upon (off 3.3 branch) is the 3.3.0 tag:
> https://github.com/apache/kafka/releases/tag/3.3.0-rc2
>
> * Documentation:  https://kafka.apache.org/33/documentation.html
>
> * Protocol: https://kafka.apache.org/33/protocol.html
>
>
>
>
> Successful Jenkins builds to follow in a future update to this email.
>
>
> Thanks!
> David Arthur

Re: [kafka-clients] Re: [VOTE] 3.3.0 RC2

[David Arthur]

Happy Monday, everyone! I've uploaded the 3.3 docs to the website to
make that part of the release validation easier

https://kafka.apache.org/33/documentation.html

These docs are not referenced by the top level docs page, so they can
only be accessed directly via the "33" url above.

I'd like to close out the vote by the end of day today, so please take a look.

Thanks!
David

On Thu, Sep 22, 2022 at 9:06 AM David Arthur
 wrote:
>
> Josep, thanks for the note. We will mention the CVEs fixed in this release
> in the announcement email. I believe we can also update the release notes
> HTML after the vote is complete.
>
> -David
>
> On Wed, Sep 21, 2022 at 2:51 AM 'Josep Prat' via kafka-clients <
> kafka-clie...@googlegroups.com> wrote:
>
> > Hi David,
> >
> > Thanks for driving this. One question, should we include in the release
> > notes the recently fixed CVE vulnerability? I understand this not being
> > explicitly mentioned on the recently released versions to not cause an
> > unintentional 0-day, but I think it could be mentioned for this release.
> > What do you think?
> >
> > Best,
> >
> > On Wed, Sep 21, 2022 at 1:17 AM David Arthur 
> > wrote:
> >
> >> Hello Kafka users, developers and client-developers,
> >>
> >> This is the second release candidate for Apache Kafka 3.3.0. Many new
> >> features and bug fixes are included in this major release of Kafka. A
> >> significant number of the issues in this release are related to KRaft,
> >> which will be considered "production ready" as part of this release
> >> (KIP-833)
> >>
> >> KRaft improvements:
> >> * KIP-778: Online KRaft to KRaft Upgrades
> >> * KIP-833: Mark KRaft as Production Ready
> >> * KIP-835: Monitor Quorum health (many new KRaft metrics)
> >> * KIP-836: Expose voter lag via kafka-metadata-quorum.sh
> >> * KIP-841: Fenced replicas should not be allowed to join the ISR in KRaft
> >> * KIP-859: Add Metadata Log Processing Error Related Metrics
> >>
> >> Other major improvements include:
> >> * KIP-618: Exactly-Once Support for Source Connectors
> >> * KIP-831: Add metric for log recovery progress
> >> * KIP-827: Expose logdirs total and usable space via Kafka API
> >> * KIP-834: Add ability to Pause / Resume KafkaStreams Topologies
> >>
> >> The full release notes are available here:
> >> https://home.apache.org/~davidarthur/kafka-3.3.0-rc2/RELEASE_NOTES.html
> >>
> >> Please download, test and vote by Monday, Sep 26 at 5pm EDT
> >>
> >> Also, huge thanks to José for running the release so far. He has done
> >> the vast majority of the work to prepare this rather large release :)
> >>
> >> -
> >>
> >> Kafka's KEYS file containing PGP keys we use to sign the release:
> >> https://kafka.apache.org/KEYS
> >>
> >> * Release artifacts to be voted upon (source and binary):
> >> https://home.apache.org/~davidarthur/kafka-3.3.0-rc2/
> >>
> >> * Maven artifacts to be voted upon:
> >> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >>
> >> * Javadoc: https://home.apache.org/~davidarthur/kafka-3.3.0-rc2/javadoc/
> >>
> >> * Tag to be voted upon (off 3.3 branch) is the 3.3.0 tag:
> >> https://github.com/apache/kafka/releases/tag/3.3.0-rc2
> >>
> >> * Documentation:  https://kafka.apache.org/33/documentation.html
> >>
> >> * Protocol: https://kafka.apache.org/33/protocol.html
> >>
> >>
> >>
> >>
> >> Successful Jenkins builds to follow in a future update to this email.
> >>
> >>
> >> Thanks!
> >> David Arthur
> >>
> >
> >
> > --
> > [image: Aiven] 
> >
> > *Josep Prat*
> > Open Source Engineering Director, *Aiven*
> > josep.p...@aiven.io   |   +491715557497
> > aiven.io    |
> > 
> >    
> > *Aiven Deutschland GmbH*
> > Immanuelkirchstraße 26, 10405 Berlin
> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > Amtsgericht Charlottenburg, HRB 209739 B
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "kafka-clients" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to kafka-clients+unsubscr...@googlegroups.com.
> > To view this discussion on the web visit
> > https://groups.google.com/d/msgid/kafka-clients/CAOJ18G4DE9Q_DYyZTbDLF6J6MRj30WrCNj6njrYRV3SQeThs-w%40mail.gmail.com
> > 
> > .
> >
>
>
> --
> -David

Re: [DISCUSS] KIP-793: Sink Connectors: Support topic-mutating SMTs for async connectors (preCommit users)

[Yash Mayya]

Hi Randall,

Thanks for reviewing the KIP!

> That latter logic can get quite ugly.

I'm not sure I quite understand why it would be "easier" for connector
developers to account for implementing two different overloaded `put`
methods (assuming that they want to use this new feature) versus using a
try-catch block around `SinkRecord` access methods? In both cases, a
connector developer would need to write additional code in order to ensure
that their connector continues working with older Connect runtimes.
Furthermore, we would probably need to carefully document how the
implementation for the older `put` method should look like for connectors
that want to use this new feature. I think the advantage of going with the
proposed approach in the KIP is that it wouldn't require extra book-keeping
(the Map in `WorkerSinkTask` in your proposed approach) and also the
fact that the try-catch based logic is an already established pattern
through
https://cwiki.apache.org/confluence/display/KAFKA/KIP-610%3A+Error+Reporting+in+Sink+Connectors
and other KIPs which added methods to source/sink connector/task contexts.

Let me know if you still feel that having a new overloaded put method is a
cleaner solution and I'd be happy to reconsider!

Thanks,
Yash

On Thu, Sep 22, 2022 at 11:18 PM Randall Hauch  wrote:

> Hi, Yash. Thanks for picking up this KIP and discussion.
>
> The KIP includes this rejected alternative:
>
> > 4. Update SinkTask.put in any way to pass the new information outside
> > SinkRecord (e.g. a Map or a derived class)
> >
> >-
> >
> >Much more disruptive change without considerable pros
> >
> >
> One advantage about doing this is that sink connector implementations can
> more easily implement two different "put(...)" methods to handle running in
> a variety of runtimes, without having to use try-catch logic around the
> newer SinkRecord access methods. That latter logic can get quite ugly.
>
> For example, the existing `put` method has this signature:
>
> public abstract void put(Collection records);
>
> If we added an overloaded method that passed in a map of the old
> topic+partition for each record (and defined the absence of an entry as
> having an unchanged topic and partition):
>
> public void put(Collection records, Map TopicPartition> updatedTopicPartitions) {
> put(records);
> }
>
> then a `SinkTask` implementation that wants to use this new feature could
> simply implement both methods:
>
> public void put(Collection records) {
> // Running in an older runtime, so no tracking of SMT-modified topic names
> or partitions
> put(records, Map.of());
> }
>
> public void put(Collection records, Map TopicPartition> updatedTopicPartitions) {
> // real logic here
> }
>
> This seems a lot easier than having to use try-catch logic, yet still
> allows sink connectors to utilize the new functionality and still work with
> older Connect runtimes.
>
> WDYT?
>
> Randall
>
>
> On Thu, Sep 8, 2022 at 7:03 AM Yash Mayya  wrote:
>
> > Hi all,
> >
> > I would like to (re)start a new discussion thread on KIP-793 (Kafka
> > Connect) which proposes some additions to the public SinkRecord interface
> > in order to support topic mutating SMTs for sink connectors that do their
> > own offset tracking.
> >
> > Links:
> >
> > KIP:
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=191336830
> >
> > Older discussion thread:
> > https://lists.apache.org/thread/00kcth6057jdcsyzgy1x8nb2s1cymy8h,
> > https://lists.apache.org/thread/rzqkm0q5y5v3vdjhg8wqppxbkw7nyopj
> >
> > Jira: https://issues.apache.org/jira/browse/KAFKA-13431
> >
> >
> > Thanks,
> > Yash
> >
>

[GitHub] [kafka-site] mumrah merged pull request #445: Pre-release of Kafka 3.3 documentation

[GitBox]

mumrah merged PR #445:
URL: https://github.com/apache/kafka-site/pull/445


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: PR for CVE-2022-34917

[Tom Bentley]

Hi Swathi,

In this case the PR reviews happened on a private repo because the CVE
wasn't public at that time. On the 3.3 branches you can look at/cherry-pick
commits 015d7aede6cbd350d56d75006930dd2bf89a4a5a and
b2b928338c7226b41a73786df27a2127eaa32ab2.

Kind regards,

Tom


On Mon, 26 Sept 2022 at 15:19, Swathi Mocharla 
wrote:

> Hi,
> CVE: https://nvd.nist.gov/vuln/detail/CVE-2022-34917
> Could you please help with the PR that fixed this vulnerability? We are
> looking to apply the patch that fixes this and we are unable to find it.
> Thanks,
> Swathi
>

[GitHub] [kafka-site] mumrah commented on pull request #445: Pre-release of Kafka 3.3 documentation

[GitBox]

mumrah commented on PR #445:
URL: https://github.com/apache/kafka-site/pull/445#issuecomment-1258126700

   @ijuma yea, I just did:
   
   * untar
   * mv site-docs 33


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: PR for CVE-2022-34917

[Manikumar]

https://issues.apache.org/jira/browse/KAFKA-14063?focusedCommentId=17608137=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17608137

On Mon, Sep 26, 2022 at 7:42 PM Swathi Mocharla
 wrote:
>
> Hi,
> CVE: https://nvd.nist.gov/vuln/detail/CVE-2022-34917
> Could you please help with the PR that fixed this vulnerability? We are
> looking to apply the patch that fixes this and we are unable to find it.
> Thanks,
> Swathi

[GitHub] [kafka-site] mumrah opened a new pull request, #445: Pre-release of Kafka 3.3 documentation

[GitBox]

mumrah opened a new pull request, #445:
URL: https://github.com/apache/kafka-site/pull/445

   This adds the documentation for 3.3 generated at 
https://github.com/apache/kafka/commit/6174f95d61c54fb228fc23e5111e2941f3947cad.
 This PR does _not_ include the updated reference in the top-level 
documentation.html file. This PR also does not include any javadocs for 3.3


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

PR for CVE-2022-34917

[Swathi Mocharla]

Hi,
CVE: https://nvd.nist.gov/vuln/detail/CVE-2022-34917
Could you please help with the PR that fixed this vulnerability? We are
looking to apply the patch that fixes this and we are unable to find it.
Thanks,
Swathi

Re: [DISCUSS] KIP-848: The Next Generation of the Consumer Rebalance Protocol

[David Jacot]

Hi Guozhang,

Regarding the version, my understanding is that the version would be
either the client software version or the request version, is this
correct? If so, we could indeed pass this information down to the
assignor via the interface. One way would be to pass a "server
context" to the assignor and that context would include that
information (and perhaps more). Is this what you are looking for?

Regarding the onAssignment, I think that I understand your point. I
suppose that the assignor could also be clever and keep track of the
last metadata to decide whether it has to do something or not. One
question that is still not clear to me is whether the assignor needs
to know all the assigned partitions upfront regardless of whether they
are already revoked or not. Do you think that we need this as well?

>From an API perspective, we could have something like
onAssignment(Metadata(version, reason, metadata, assigned partitions,
pending partitions)). Where the assigned partitions are the partitions
ready to be used and the pending partitions are the one assigned to
the member but not revoked yet. I find it a bit weird that this method
would be called only once because the assignor would not know when the
pending partitions changes. That does not look like a clean API. An
alternative would be to use onAssignment(Metadata(version, reason,
metadata, combined partitions)) but this seems error prone because it
is not clear whether a partition is usable or not. Or do you think
that we should not provide the partitions but only the metadata?

Best,
David

On Fri, Sep 23, 2022 at 9:40 PM Guozhang Wang  wrote:
>
> Hello David,
>
> On Fri, Sep 23, 2022 at 2:00 AM David Jacot 
> wrote:
>
> > Hey,
> >
> > > Just to clarify I was asking about the `version` of the assignor (i.e. up
> > to what version that the client would support), and I do agree we would not
> > need metadata. What I have in mind is that, for some specific built-in
> > broker-assignors, e.g. rack-aware assignors, if it's possible that in a
> > newer version we would have a hierarchical rack ID string format, like
> > "tier1-tier2" etc, but if some client has not upgraded their rack ID
> > would still be in old format. In this case, the broker then needs to choose
> > the old versioned assignor. I'm probably making something up here for rack
> > aware assignors, but I'm wondering if in general such an "auto-downgrade"
> > behavior would be needed still for broker-side assignor, and if yes would
> > "version" still be useful.
> >
> > Got it. That's an interesting thought. I think that the issue is that
> > the client will never tell you which version of the server-side
> > assignor should be used. Do you think that the coordinator would
> > downgrade the version if the assignment fails with a higher version? I
> > tend to believe that this should be handled within the assignor
> > itself. In the example that you mentioned, the assignor would have to
> > handle all the cases. I am not really convinced that we need this at
> > the moment.
> >
> > The version from the client side would not be indicating the broker which
> version to use, but rather which version the client would "work best with".
> Such a "version" field would not be settible by the users, since they will
> be hard-codedly bumped when the Kafka byte code version bumped.
> Back to the rack aware assignor example, if the older versioned client does
> not have a hierarchical rack ID, however if the assignment returned to them
> is assuming a hierarchical rack structure, it may not reflect the best
> workload balance among those new and old versioned clients. That means,
> when receiving the members subscriptions at the server side, if the
> versions from all these members are different, the broker's assignor may
> need to consider using the lower version logic to do the assignment. So yes
> the assignor would indeed have to handle all such cases, but it needs to do
> so such that if there are clients who would not work with certain new
> logic, it would then handle such cases automatically by e.g. still using an
> older versioned logic.
>
>
>
> > > Okay, my understanding is that the calling ordering of these callbacks
> > would be like the following:
> >
> > Yes, your examples look right.
> >
> > > I'm wondering if we would still call onAssignment just once, that encodes
> > all the assignment for this rebalance, including all the partitions that
> > should be assigned to the member but not yet assigned since they have not
> > been revoked by others. In that case the call ordering would be:
> >
> > Interesting. Is there a case for Streams where having the full
> > assignment is beneficial? For instance, I can think of the following
> > case. When a standby task is promoted to an active task, the metadata
> > would not contain the standby task anymore and the assignment may not
> > have the partition yet. In this case, Streams would stop the standby
> > tasks but not have the active task 

Jenkins build is unstable: Kafka » Kafka Branch Builder » 3.3 #85

[Apache Jenkins Server]

See 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1255

[Apache Jenkins Server]

See