Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2023

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 292357 lines...]
Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
RestoreIntegrationTest > shouldRestoreStateFromSourceTopic(boolean) > [2] false 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
RestoreIntegrationTest > shouldRestoreStateFromSourceTopic(boolean) > [2] false 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
RestoreIntegrationTest > shouldSuccessfullyStartWhenLoggingDisabled(boolean) > 
[1] true STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
RestoreIntegrationTest > shouldSuccessfullyStartWhenLoggingDisabled(boolean) > 
[1] true PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
RestoreIntegrationTest > shouldSuccessfullyStartWhenLoggingDisabled(boolean) > 
[2] false STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
RestoreIntegrationTest > shouldSuccessfullyStartWhenLoggingDisabled(boolean) > 
[2] false PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
RestoreIntegrationTest > shouldRestoreStateFromChangelogTopic(boolean) > [1] 
true STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
RestoreIntegrationTest > shouldRestoreStateFromChangelogTopic(boolean) > [1] 
true PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
RestoreIntegrationTest > shouldRestoreStateFromChangelogTopic(boolean) > [2] 
false STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
RestoreIntegrationTest > shouldRestoreStateFromChangelogTopic(boolean) > [2] 
false PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
SmokeTestDriverIntegrationTest > shouldWorkWithRebalance(boolean) > [1] true 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
RestoreIntegrationTest > shouldRestoreNullRecord() PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
RestoreIntegrationTest > shouldRestoreStateFromSourceTopic(boolean) > [1] true 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
RestoreIntegrationTest > shouldRestoreStateFromSourceTopic(boolean) > [1] true 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
RestoreIntegrationTest > shouldRestoreStateFromSourceTopic(boolean) > [2] false 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
RestoreIntegrationTest > shouldRestoreStateFromSourceTopic(boolean) > [2] false 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
RestoreIntegrationTest > shouldSuccessfullyStartWhenLoggingDisabled(boolean) > 
[1] true STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
RestoreIntegrationTest > shouldSuccessfullyStartWhenLoggingDisabled(boolean) > 
[1] true PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
RestoreIntegrationTest > shouldSuccessfullyStartWhenLoggingDisabled(boolean) > 
[2] false STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
RestoreIntegrationTest > shouldSuccessfullyStartWhenLoggingDisabled(boolean) > 
[2] false PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
RestoreIntegrationTest > shouldRestoreStateFromChangelogTopic(boolean) > [1] 
true STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
RestoreIntegrationTest > shouldRestoreStateFromChangelogTopic(boolean) > [1] 
true PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
RestoreIntegrationTest > shouldRestoreStateFromChangelogTopic(boolean) > [2] 
false STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
RestoreIntegrationTest > shouldRestoreStateFromChangelogTopic(boolean) > [2] 
false PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
SmokeTestDriverIntegrationTest > shouldWorkWithRebalance(boolean) > [1] true 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
SmokeTestDriverIntegrationTest > shouldWorkWithRebalance(boolean) > [1] true 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
SmokeTestDriverIntegrationTest > shouldWorkWithRebalance(boolean) > [2] false 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
SmokeTestDriverIntegrationTest > shouldWorkWithRebalance(boolean) > [1] true 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
SmokeTestDriverIntegrationTest > shouldWorkWithRebalance(boolean) > [2] false 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
SmokeTestDriverIntegrationTest > 

Re: [DISCUSS] KIP-954: expand default DSL store configuration to custom types

[Sophie Blee-Goldman]

Hey Almog, first off, thanks for the KIP! I (and others) raised concerns
over how restrictive the default.dsl.store config would be if not
extendable to custom store types, especially given that this seems to be
the primary userbase of such a feature. At the time we didn't really have
any better ideas for a clean way to achieve that, but what you proposed
makes a lot of sense to me. Happy to see a good solution to this, and
hopefully others will share my satisfaction :P

I did have one quick piece of feedback which arose from an unrelated
question posed to the dev mailing list w/ subject line
"ReadOnlyKeyValueStore#range()
Semantics"
. I
recommend checking out the full thread for context, but it made me think
about how we can leverage the new StoreTypeSpec concept as an answer to the
long-standing question in Streams: where can we put guarantees of the
public contract for RocksDB (or other store implementations) when all the
RocksDB stuff is technically internal.

Basically, I'm suggesting two things: first, call out in some way (perhaps
the StoreTypeSpec javadocs) that each StoreTypeSpec is considered a public
contract in itself and should outline any semantic guarantees it does, or
does not, make. Second, we should add a note on ordering guarantees in the
two OOTB specs: for RocksDB we assert that range queries will honor
serialized byte ordering, whereas the InMemory flavor gives no ordering
guarantee whatsoever at this time.

Thoughts?

-Sophie

On Thu, Jul 20, 2023 at 4:28 PM Almog Gavra  wrote:

> Hi All,
>
> I would like to propose a KIP to expand support for default store types
> (KIP-591) to encompass custom store implementations:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-954%3A+expand+default+DSL+store+configuration+to+custom+types
>
> Looking forward to your feedback!
>
> Cheers,
> Almog
>

Re: ReadOnlyKeyValueStore#range() Semantics

[Sophie Blee-Goldman]

Hey Colt, I honestly don't really know whether a KIP would be strictly
required for this or not, but let me offer an alternative to what you
proposed that would both address this particular ambiguity and, to me,
provide a better mechanism for guaranteeing this as public contract of
RocksDB in general.

Earlier today someone proposed a KIP to expand the default DSL store type
specification to allow for custom state store implementations.
Fundamentally this involves introducing a new interface that basically
defines a single store type spec, such as RocksDB or in-memory, or whatever
other things people are plugging in. I think it's difficult to reason about
where to place this guarantee in the existing codebase, but it would be a
trivial addition to KIP-954 to include this as a public contract for all
RocksDB stores. To me, the assertion that a particular store honors
serialized byte ordering is a characteristic of the underlying store type
(eg rocks vs IM), which today is not really exposed anywhere in the public
API and barely a concept internally -- making it hard to see where an
appropriate place for this would be. However, if we expand the concept of a
store type spec in KIP-954 to include things like this, ie semantic
guarantees about the type itself, we'll have a clean solution to this
without the need or question of an additional KIP.

I actually think this idea of a semantic contract as part of the store spec
makes sense to add/call out in that KIP anyways, so I'll probably suggest
this on the discussion thread regardless of whether it's decided to be the
right solution for this particular concern. I like the idea that if/when
something like this comes up again in the future, we'll have a clear place
to make any changes to the public contract of internal store
implementations. Going forward, this should be done as a quick KIP.

Let me know what you think, and feel free to add your thoughts on this
(and/or the proposal itself) over on the KIP-954 [DISCUSS] thread

Cheers

On Thu, Jul 20, 2023 at 8:51 PM Colt McNealy  wrote:

> Hi all,
>
> The [current documentation](
>
> https://kafka.apache.org/35/javadoc/org/apache/kafka/streams/state/ReadOnlyKeyValueStore.html#range(K,K)
> )  for ReadOnlyKeyValueStore#range() states that:
>
> > Order is not guaranteed as bytes lexicographical ordering might not
> represent key order.
>
> That makes sense—a the ordering of the two keys inserted via `store.put()`
> as determined by the `compareTo()` method is not what determines the
> ordering in the store; rather, it's the compareTo() of the serialized
> byte[] array that matters.
>
> Some observations after playing with it for over a year:
>
> A ) The behavior when you open a store for IQ and don't specify a specific
> partition is that (behind the scenes) a store is opened for one partition,
> and when that store is exhausted, then the next partition is opened. No
> guarantees about which partition is opened in what order. As such, if you
> just System.out.println() all the keys from the iterator, they are not
> ordered properly.
>
> B) WITHIN a partition, such as if you do a .withPartition() when requesting
> the ReadOnlyKeyValueStore, keys are indeed ordered properly according to
> the bytes produced by the key serializer.
>
> We at LittleHorse rely upon that behavior for API pagination, and if that
> behavior were to change it would break some things.
>
> After some digging, it turns out that the reason why we *do* indeed get
> lexicographical ordering of results according to the byte[] array of the
> keys is because that is a public contract exposed by RocksDB.
>
> I had asked Matthias offline if it would be possible to open a PR to
> clarify on the documentation that all results *within a partition of the
> Store* are ordered by the byte[] representation of the key, since I would
> feel more comfortable relying upon a publicly documented API.
>
> However, there are a few counterpoints to this:
>
> - ReadOnlyKeyValueStore is an *interface*, not an implementation. The
> lexicographical ordering is something we observe from the RocksDB
> implementation. If the store were implemented with, for example, a HashMap,
> this would not work.
>
> - The semantics of ordering thus seem to be more associated with the
> *implementation* rather than with the *interface*.
>
> - Is it possible at all to add a clarification on the RocksDB store that
> this behavior is a guarantee? Would that require a KIP?
>
> I'd be super-happy if I could open a PR to put a public documentation note
> somewhere on some implementation of a State Store that documents that this
> ordering by byte[] representation is guaranteed for range scans, but I do
> recognize that making a public documentation note is a contract, and as
> such may require a KIP and/or not be accepted.
>
> Any thoughts?
>
> Thanks for reading,
> Colt McNealy
>
> *Founder, LittleHorse.dev*
>

ReadOnlyKeyValueStore#range() Semantics

[Colt McNealy]

Hi all,

The [current documentation](
https://kafka.apache.org/35/javadoc/org/apache/kafka/streams/state/ReadOnlyKeyValueStore.html#range(K,K)
)  for ReadOnlyKeyValueStore#range() states that:

> Order is not guaranteed as bytes lexicographical ordering might not
represent key order.

That makes sense—a the ordering of the two keys inserted via `store.put()`
as determined by the `compareTo()` method is not what determines the
ordering in the store; rather, it's the compareTo() of the serialized
byte[] array that matters.

Some observations after playing with it for over a year:

A ) The behavior when you open a store for IQ and don't specify a specific
partition is that (behind the scenes) a store is opened for one partition,
and when that store is exhausted, then the next partition is opened. No
guarantees about which partition is opened in what order. As such, if you
just System.out.println() all the keys from the iterator, they are not
ordered properly.

B) WITHIN a partition, such as if you do a .withPartition() when requesting
the ReadOnlyKeyValueStore, keys are indeed ordered properly according to
the bytes produced by the key serializer.

We at LittleHorse rely upon that behavior for API pagination, and if that
behavior were to change it would break some things.

After some digging, it turns out that the reason why we *do* indeed get
lexicographical ordering of results according to the byte[] array of the
keys is because that is a public contract exposed by RocksDB.

I had asked Matthias offline if it would be possible to open a PR to
clarify on the documentation that all results *within a partition of the
Store* are ordered by the byte[] representation of the key, since I would
feel more comfortable relying upon a publicly documented API.

However, there are a few counterpoints to this:

- ReadOnlyKeyValueStore is an *interface*, not an implementation. The
lexicographical ordering is something we observe from the RocksDB
implementation. If the store were implemented with, for example, a HashMap,
this would not work.

- The semantics of ordering thus seem to be more associated with the
*implementation* rather than with the *interface*.

- Is it possible at all to add a clarification on the RocksDB store that
this behavior is a guarantee? Would that require a KIP?

I'd be super-happy if I could open a PR to put a public documentation note
somewhere on some implementation of a State Store that documents that this
ordering by byte[] representation is guaranteed for range scans, but I do
recognize that making a public documentation note is a contract, and as
such may require a KIP and/or not be accepted.

Any thoughts?

Thanks for reading,
Colt McNealy

*Founder, LittleHorse.dev*

Re: Requesting permission to contribute to Apache Kafka

[Luke Chen]

Hi Kenneth,

Your accounts are all set.

Thanks.
Luke


On Fri, Jul 21, 2023 at 1:25 AM Kenneth Eversole
 wrote:

> Hello,
>
> I am requesting permission to contribute to Apache Kafka
>
> wiki id: keversole
> jira id: keversole
>
>
> Thank you,
>
> Kenneth Eversole
>

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #2022

[Apache Jenkins Server]

See 

[jira] [Resolved] (KAFKA-15176) Add missing tests for remote storage metrics

[Luke Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen resolved KAFKA-15176.
---
Fix Version/s: 3.6.0
   Resolution: Fixed

> Add missing tests for remote storage metrics
> 
>
> Key: KAFKA-15176
> URL: https://issues.apache.org/jira/browse/KAFKA-15176
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Luke Chen
>Assignee: Luke Chen
>Priority: Major
> Fix For: 3.6.0
>
>
> {{RemoteLogReaderTaskQueueSize}}
> {{RemoteLogReaderAvgIdlePercent}}
> {{RemoteLogManagerTasksAvgIdlePercent}}
> {{}}
> https://github.com/apache/kafka/pull/13944#pullrequestreview-1513943273{{{}{}}}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Errors related to Request METADATA failed on brokers List kafka1:9092 disconnected

[odashima.tatsuya]

Hi, there,
I have a problem when I routinely run the following command line.
In addition, I am experiencing process hangs on one Broker at irregular 
intervals, but I do not know the relevance of the error content and causing of 
the error.
Has anyone had the same problem to help me?

○Kafka Broker Server Information
・3-unit cluster configuration
・Apache kafka version 3.2.1

○Commandline
bin/kafka-broker-api-versions.sh --bootstrap-server 
kafka1:9092,kafka2:9092,kafka3:9092

○Error contents

WARN [LegacyAdminClient clientId=admin-1] Bootstrap broker kafka1:9092 (id: -1 
rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
WARN [LegacyAdminClient clientId=admin-1] Bootstrap broker kafka1:9092 (id: -1 
rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
Exception in thread "main" java.lang.RuntimeException: Request METADATA failed 
on brokers List(kafka1:9092 (id: -1 rack: null))
  at 
kafka.admin.BrokerApiVersionsCommand$AdminClient.sendAnyNode(BrokerApiVersionsCommand.scala:156)
  at 
kafka.admin.BrokerApiVersionsCommand$AdminClient.findAllBrokers(BrokerApiVersionsCommand.scala:179)
  at 
kafka.admin.BrokerApiVersionsCommand$AdminClient.awaitBrokers(BrokerApiVersionsCommand.scala:171)
  at 
kafka.admin.BrokerApiVersionsCommand$.execute(BrokerApiVersionsCommand.scala:61)
  at 
kafka.admin.BrokerApiVersionsCommand$.main(BrokerApiVersionsCommand.scala:55)
  at 
kafka.admin.BrokerApiVersionsCommand.main(BrokerApiVersionsCommand.scala)

Best regards
Sharp Odashima

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.5 #47

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 560413 lines...]

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
StreamsAssignmentScaleTest > testStickyTaskAssignorManyThreadsPerClient PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
StreamsAssignmentScaleTest > testFallbackPriorTaskAssignorManyThreadsPerClient 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
StreamsAssignmentScaleTest > testFallbackPriorTaskAssignorManyThreadsPerClient 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
StreamsAssignmentScaleTest > testFallbackPriorTaskAssignorLargePartitionCount 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
StreamsAssignmentScaleTest > testFallbackPriorTaskAssignorLargePartitionCount 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
StreamsAssignmentScaleTest > testStickyTaskAssignorLargePartitionCount STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
StreamsAssignmentScaleTest > testStickyTaskAssignorLargePartitionCount PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
StreamsAssignmentScaleTest > testFallbackPriorTaskAssignorManyStandbys STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
StreamsAssignmentScaleTest > testFallbackPriorTaskAssignorManyStandbys PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
StreamsAssignmentScaleTest > testHighAvailabilityTaskAssignorManyStandbys 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
StreamsAssignmentScaleTest > testHighAvailabilityTaskAssignorManyStandbys PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
StreamsAssignmentScaleTest > testFallbackPriorTaskAssignorLargeNumConsumers 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
StreamsAssignmentScaleTest > testFallbackPriorTaskAssignorLargeNumConsumers 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
StreamsAssignmentScaleTest > testStickyTaskAssignorLargeNumConsumers STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
StreamsAssignmentScaleTest > testStickyTaskAssignorLargeNumConsumers PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
EmitOnChangeIntegrationTest > shouldEmitSameRecordAfterFailover() STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
EmitOnChangeIntegrationTest > shouldEmitSameRecordAfterFailover() PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
HighAvailabilityTaskAssignorIntegrationTest > 
shouldScaleOutWithWarmupTasksAndPersistentStores(TestInfo) STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
HighAvailabilityTaskAssignorIntegrationTest > 
shouldScaleOutWithWarmupTasksAndPersistentStores(TestInfo) PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
HighAvailabilityTaskAssignorIntegrationTest > 
shouldScaleOutWithWarmupTasksAndInMemoryStores(TestInfo) STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
HighAvailabilityTaskAssignorIntegrationTest > 
shouldScaleOutWithWarmupTasksAndInMemoryStores(TestInfo) PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
KStreamAggregationDedupIntegrationTest > shouldReduce(TestInfo) STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
KStreamAggregationDedupIntegrationTest > shouldReduce(TestInfo) PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
KStreamAggregationDedupIntegrationTest > shouldGroupByKey(TestInfo) STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
KStreamAggregationDedupIntegrationTest > shouldGroupByKey(TestInfo) PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
KStreamAggregationDedupIntegrationTest > shouldReduceWindowed(TestInfo) STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
KStreamAggregationDedupIntegrationTest > shouldReduceWindowed(TestInfo) PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
KStreamKStreamIntegrationTest > shouldOuterJoin() STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
KStreamKStreamIntegrationTest > shouldOuterJoin() PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
KTableSourceTopicRestartIntegrationTest > 
shouldRestoreAndProgressWhenTopicNotWrittenToDuringRestoration() STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
KTableSourceTopicRestartIntegrationTest > 

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2021

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 392509 lines...]

Gradle Test Run :streams:integrationTest > Gradle Test Executor 178 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testInnerOuter[caching enabled = false] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 178 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testInnerOuter[caching enabled = false] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 178 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testInnerLeft[caching enabled = false] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 178 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testInnerLeft[caching enabled = false] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 178 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testOuterInner[caching enabled = false] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 178 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testOuterInner[caching enabled = false] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 178 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testOuterOuter[caching enabled = false] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 178 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testOuterOuter[caching enabled = false] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 178 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testInnerWithRightVersionedOnly[caching enabled = false] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 178 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testInnerWithRightVersionedOnly[caching enabled = false] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 178 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testLeftWithLeftVersionedOnly[caching enabled = false] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 178 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testLeftWithLeftVersionedOnly[caching enabled = false] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 178 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testInnerWithLeftVersionedOnly[caching enabled = false] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 178 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testInnerWithLeftVersionedOnly[caching enabled = false] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 178 > 
TaskAssignorIntegrationTest > shouldProperlyConfigureTheAssignor STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 178 > 
TaskAssignorIntegrationTest > shouldProperlyConfigureTheAssignor PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 178 > 
TaskMetadataIntegrationTest > shouldReportCorrectEndOffsetInformation STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 178 > 
TaskMetadataIntegrationTest > shouldReportCorrectEndOffsetInformation PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 178 > 
TaskMetadataIntegrationTest > shouldReportCorrectCommittedOffsetInformation 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 178 > 
TaskMetadataIntegrationTest > shouldReportCorrectCommittedOffsetInformation 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 178 > 
EmitOnChangeIntegrationTest > shouldEmitSameRecordAfterFailover() STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 178 > 
EmitOnChangeIntegrationTest > shouldEmitSameRecordAfterFailover() PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 178 > 
HighAvailabilityTaskAssignorIntegrationTest > 
shouldScaleOutWithWarmupTasksAndPersistentStores(TestInfo) STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 178 > 
HighAvailabilityTaskAssignorIntegrationTest > 
shouldScaleOutWithWarmupTasksAndPersistentStores(TestInfo) PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 178 > 
HighAvailabilityTaskAssignorIntegrationTest > 
shouldScaleOutWithWarmupTasksAndInMemoryStores(TestInfo) STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 178 > 
HighAvailabilityTaskAssignorIntegrationTest > 
shouldScaleOutWithWarmupTasksAndInMemoryStores(TestInfo) PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 178 > 
KStreamAggregationDedupIntegrationTest > shouldReduce(TestInfo) STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 178 > 
KStreamAggregationDedupIntegrationTest > 

[DISCUSS] KIP-954: expand default DSL store configuration to custom types

[Almog Gavra]

Hi All,

I would like to propose a KIP to expand support for default store types
(KIP-591) to encompass custom store implementations:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-954%3A+expand+default+DSL+store+configuration+to+custom+types

Looking forward to your feedback!

Cheers,
Almog

Re: KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum

[Colin McCabe]

On Tue, Jul 18, 2023, at 09:30, Mickael Maison wrote:
> H Colin,
>
> Thanks for the KIP.
>
> Just a few points:
> 1. As Tom mentioned it would be good to clarify the APIs we expect
> available on controllers. I assume we want to add DESCRIBE_CONFIGS as
> part of this KIP.

Hi Mickael,

Yes, this is a good point. I added a table describing the APIs that will now be 
added.

> 2. Currently we have no way of retrieving the list of configs that
> apply to controllers. It would be good to have an object, so we can
> add that to the docs but also use that in kafka-configs.

I think this is out of scope.

> 3. Should we have a new entity-type in kafka-configs for setting
> controller configs?

The BROKER entity type already applies to controllers. It probably needs a new 
name (NODE would be better) but that's out of scope for this KIP, I think.

best,
Colin


>
> Thanks,
> Mickael
>
> On Tue, Jul 4, 2023 at 2:20 PM Luke Chen  wrote:
>>
>> Hi Colin,
>>
>> Thanks for the answers to my previous questions.
>>
>> > Yes, the common thread here is that all of these shell commands perform
>> operations can be done without the broker. So it's reasonable to allow them
>> to be done without going through the broker. I don't know if we need a
>> separate note for each since the rationale is really the same for all (is
>> it reasonable? if so allow it.)
>>
>> Yes, it makes sense. Could we make a note about the main rationale for
>> selecting these command-line tools in the KIP to make it clear?
>> Ex: The following command-line tools will get a new --bootstrap-controllers
>> argument (because these shell commands perform operations can be done
>> without the broker):
>>
>> > kafka-reassign-partitions.sh cannot be used to move the
>> __cluster_metadata topic. However, it can be used to move partitions that
>> reside on the brokers, even when using --bootstrap-controllers to talk
>> directly to the quorum.
>>
>> Fair enough.
>>
>>
>> 4. Does all the command-line tools with `--bootstrap-controllers` support
>> all the options in the tool?
>> For example, kafka-configs.sh, In addition to the `--alter` option you
>> mentioned in the example, do we also support `--describe` or `--delete`
>> option?
>> If so, do we also support setting "quota" for users/clients/topics... via
>> `--bootstrap-controllers`? (not intuitive, but maybe we just directly
>> commit the change into the metadata from controller?)
>>
>> 5. Do we have any plan for this feature to be completed? v3.6.0?
>>
>>
>> Thank you.
>> Luke
>>
>>
>> On Fri, Apr 28, 2023 at 1:42 AM Colin McCabe  wrote:
>>
>> > On Wed, Apr 26, 2023, at 22:08, Luke Chen wrote:
>> > > Hi Colin,
>> > >
>> > > Some comments:
>> > > 1. I agree we should set "top-level" errors for metadata response
>> > >
>> > > 2. In the "brokers" field of metadata response from controller, it'll
>> > > respond with "Controller endpoint information as given in
>> > > controller.quorum.voters", instead of the "alive" controllers(voters).
>> > That
>> > > will break the existing admin client because in admin client, we'll rely
>> > on
>> > > the metadata response to build the "current alive brokers" list, and
>> > choose
>> > > one from them to connect (either least load or other criteria). That
>> > means,
>> > > if now, we return the value in `controller.quorum.voters`, but one of
>> > them
>> > > is down. We might choose it to connect and get connection errors. Should
>> > we
>> > > return the "alive" controllers(voters) to client?
>> >
>> > Hi Luke,
>> >
>> > Good question. When talking to the controllers directly, the AdminClient
>> > needs to always send its RPCs to the active controller. There is one
>> > exception: configuring ephemeral log4j settings with
>> > incrementalAlterConfigs must be done by sending them to the specified
>> > controller node.
>> >
>> > I will add this to a section called "AdminClient Implementation Notes" so
>> > that it's captured in the KIP.
>> >
>> > >
>> > > 3. In the KIP, we list the command-line tools will get a new
>> > > --bootstrap-controllers argument, but without explaining why these tools
>> > > need to talk to controller directly. Could we add some explanation about
>> > > them? I tried but cannot know why some tools are listed here:
>> > > - kafka-acls.sh -> Allow clients to update ACLs via controller before
>> > > brokers up
>> > >
>> > > - kafka-cluster.sh -> Reasonable to get/update cluster info via
>> > > controller
>> > >
>> > > - kafka-configs.sh -> Allow clients to dynamically update
>> > > configs/describe configs from controller. But in this script, client can
>> > > still set quota for users/clients/topics... is client also able to update
>> > > via controllers? Or we only allow partial actions in the script to talk
>> > to
>> > > controllers?
>> > >
>> > > - kafka-delegation-tokens.sh -> Reasonable to update
>> > delegation-tokens
>> > > via controllers
>> > >
>> > > - kafka-features.sh -> Reasonable
>> > > - 

Re: KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum

[Colin McCabe]

On Tue, Jul 4, 2023, at 05:19, Luke Chen wrote:
> 4. Does all the command-line tools with `--bootstrap-controllers` support
> all the options in the tool?
> For example, kafka-configs.sh, In addition to the `--alter` option you
> mentioned in the example, do we also support `--describe` or `--delete`
> option?

Hi Luke,

Yes, we do.

> If so, do we also support setting "quota" for users/clients/topics... via
> `--bootstrap-controllers`? (not intuitive, but maybe we just directly
> commit the change into the metadata from controller?)

Yes. This would be supported via the alterClientQuotas API.

>
> 5. Do we have any plan for this feature to be completed? v3.6.0?
>

Yes, I would like to target 3.6. If time gets tight one or two command 
conversions might slip. The most critical command line tools to support are the 
ones for getting the quorum status itself.

best,
Colin

>
> Thank you.
> Luke
>
>
> On Fri, Apr 28, 2023 at 1:42 AM Colin McCabe  wrote:
>
>> On Wed, Apr 26, 2023, at 22:08, Luke Chen wrote:
>> > Hi Colin,
>> >
>> > Some comments:
>> > 1. I agree we should set "top-level" errors for metadata response
>> >
>> > 2. In the "brokers" field of metadata response from controller, it'll
>> > respond with "Controller endpoint information as given in
>> > controller.quorum.voters", instead of the "alive" controllers(voters).
>> That
>> > will break the existing admin client because in admin client, we'll rely
>> on
>> > the metadata response to build the "current alive brokers" list, and
>> choose
>> > one from them to connect (either least load or other criteria). That
>> means,
>> > if now, we return the value in `controller.quorum.voters`, but one of
>> them
>> > is down. We might choose it to connect and get connection errors. Should
>> we
>> > return the "alive" controllers(voters) to client?
>>
>> Hi Luke,
>>
>> Good question. When talking to the controllers directly, the AdminClient
>> needs to always send its RPCs to the active controller. There is one
>> exception: configuring ephemeral log4j settings with
>> incrementalAlterConfigs must be done by sending them to the specified
>> controller node.
>>
>> I will add this to a section called "AdminClient Implementation Notes" so
>> that it's captured in the KIP.
>>
>> >
>> > 3. In the KIP, we list the command-line tools will get a new
>> > --bootstrap-controllers argument, but without explaining why these tools
>> > need to talk to controller directly. Could we add some explanation about
>> > them? I tried but cannot know why some tools are listed here:
>> > - kafka-acls.sh -> Allow clients to update ACLs via controller before
>> > brokers up
>> >
>> > - kafka-cluster.sh -> Reasonable to get/update cluster info via
>> > controller
>> >
>> > - kafka-configs.sh -> Allow clients to dynamically update
>> > configs/describe configs from controller. But in this script, client can
>> > still set quota for users/clients/topics... is client also able to update
>> > via controllers? Or we only allow partial actions in the script to talk
>> to
>> > controllers?
>> >
>> > - kafka-delegation-tokens.sh -> Reasonable to update
>> delegation-tokens
>> > via controllers
>> >
>> > - kafka-features.sh -> Reasonable
>> > - kafka-metadata-quorum.sh -> Reasonable
>> > - kafka-metadata-shell.sh -> Reasonable
>> >
>> > - kafka-reassign-partitions.sh -> Why should we allow clients to move
>> > metadata log partitions in controller nodes? What's the use-case?
>> >
>>
>> Yes, the common thread here is that all of these shell commands perform
>> operations can be done without the broker. So it's reasonable to allow them
>> to be done without going through the broker. I don't know if we need a
>> separate note for each since the rationale is really the same for all (is
>> it reasonable? if so allow it.)
>>
>> kafka-reassign-partitions.sh cannot be used to move the __cluster_metadata
>> topic. However, it can be used to move partitions that reside on the
>> brokers, even when using --bootstrap-controllers to talk directly to the
>> quorum.
>>
>> Colin
>>
>> >
>> > Thank you.
>> > Luke
>> >
>> > On Thu, Apr 27, 2023 at 8:04 AM Colin McCabe  wrote:
>> >
>> >> On Tue, Apr 25, 2023, at 04:59, Divij Vaidya wrote:
>> >> > Thank you for the KIP Colin.
>> >> >
>> >> > In general, I like the idea of having the ability to interact directly
>> >> with
>> >> > the controllers. I agree with your observation that it helps in
>> >> situations
>> >> > where you would want to get data directly from the controller instead
>> of
>> >> > going via a broker. I have some general comments but the main concern
>> I
>> >> > have is with the piggy-backing of error code with response of
>> >> > __cluster_metadata topic.
>> >> >
>> >> > 1. With this change, how are we guarding against the possibility of
>> >> > misbehaving client traffic from disrupting the controller (that you
>> >> > mentioned as a motivation of earlier behaviour)? One solution could
>> be to
>> >> > 

Re: KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum

[Colin McCabe]

On Tue, Jul 4, 2023, at 04:46, Tom Bentley wrote:
> Hi Colin,
>
> Thanks for the KIP.
>
> 1. It mentions kafka-configs.sh as one of the affected tools, but doesn't
> mention that ControllerApis doesn't currently support DESCRIBE_CONFIGS. I
> think this is worth noting as it is, in effect, a change to the wire
> protocol accepted by the controller, even if it's an existing RPC.

Hi Tom,

Good point. I added a section mentioning that we will support DESCRIBE_CONFIGS 
on the controller.

> 2. The diff you show for the MetadataRequest.son doesn't show a change to
> the top-level "listeners" key, presumably this should add "controller"?
> Similarly, per the above point, I guess we'd also be updating the JSON for
> DescribeConfigs.

I have reworked this section to reflect the fact that I am now using 
DESCRIBE_CLUSTER to give AdminClient information about the controller nodes.

> 3. Do you have any timeline for calling a vote for this KIP?
>

Yes, good question. I will call a vote soon since I know the KIP freeze is 
coming up.

best,
Colin

> Many thanks,
>
> Tom
>
> On Thu, 27 Apr 2023 at 18:51, Colin McCabe  wrote:
>
>> On Wed, Apr 26, 2023, at 22:08, Luke Chen wrote:
>> > Hi Colin,
>> >
>> > Some comments:
>> > 1. I agree we should set "top-level" errors for metadata response
>> >
>> > 2. In the "brokers" field of metadata response from controller, it'll
>> > respond with "Controller endpoint information as given in
>> > controller.quorum.voters", instead of the "alive" controllers(voters).
>> That
>> > will break the existing admin client because in admin client, we'll rely
>> on
>> > the metadata response to build the "current alive brokers" list, and
>> choose
>> > one from them to connect (either least load or other criteria). That
>> means,
>> > if now, we return the value in `controller.quorum.voters`, but one of
>> them
>> > is down. We might choose it to connect and get connection errors. Should
>> we
>> > return the "alive" controllers(voters) to client?
>>
>> Hi Luke,
>>
>> Good question. When talking to the controllers directly, the AdminClient
>> needs to always send its RPCs to the active controller. There is one
>> exception: configuring ephemeral log4j settings with
>> incrementalAlterConfigs must be done by sending them to the specified
>> controller node.
>>
>> I will add this to a section called "AdminClient Implementation Notes" so
>> that it's captured in the KIP.
>>
>> >
>> > 3. In the KIP, we list the command-line tools will get a new
>> > --bootstrap-controllers argument, but without explaining why these tools
>> > need to talk to controller directly. Could we add some explanation about
>> > them? I tried but cannot know why some tools are listed here:
>> > - kafka-acls.sh -> Allow clients to update ACLs via controller before
>> > brokers up
>> >
>> > - kafka-cluster.sh -> Reasonable to get/update cluster info via
>> > controller
>> >
>> > - kafka-configs.sh -> Allow clients to dynamically update
>> > configs/describe configs from controller. But in this script, client can
>> > still set quota for users/clients/topics... is client also able to update
>> > via controllers? Or we only allow partial actions in the script to talk
>> to
>> > controllers?
>> >
>> > - kafka-delegation-tokens.sh -> Reasonable to update
>> delegation-tokens
>> > via controllers
>> >
>> > - kafka-features.sh -> Reasonable
>> > - kafka-metadata-quorum.sh -> Reasonable
>> > - kafka-metadata-shell.sh -> Reasonable
>> >
>> > - kafka-reassign-partitions.sh -> Why should we allow clients to move
>> > metadata log partitions in controller nodes? What's the use-case?
>> >
>>
>> Yes, the common thread here is that all of these shell commands perform
>> operations can be done without the broker. So it's reasonable to allow them
>> to be done without going through the broker. I don't know if we need a
>> separate note for each since the rationale is really the same for all (is
>> it reasonable? if so allow it.)
>>
>> kafka-reassign-partitions.sh cannot be used to move the __cluster_metadata
>> topic. However, it can be used to move partitions that reside on the
>> brokers, even when using --bootstrap-controllers to talk directly to the
>> quorum.
>>
>> Colin
>>
>> >
>> > Thank you.
>> > Luke
>> >
>> > On Thu, Apr 27, 2023 at 8:04 AM Colin McCabe  wrote:
>> >
>> >> On Tue, Apr 25, 2023, at 04:59, Divij Vaidya wrote:
>> >> > Thank you for the KIP Colin.
>> >> >
>> >> > In general, I like the idea of having the ability to interact directly
>> >> with
>> >> > the controllers. I agree with your observation that it helps in
>> >> situations
>> >> > where you would want to get data directly from the controller instead
>> of
>> >> > going via a broker. I have some general comments but the main concern
>> I
>> >> > have is with the piggy-backing of error code with response of
>> >> > __cluster_metadata topic.
>> >> >
>> >> > 1. With this change, how are we guarding against the possibility of
>> >> > 

RE: Re: [DISCUSS] KIP-951: Leader discovery optimisations for the client

[Crispin Bernier]

Benchmark numbers have been posted on the KIP, please review.

On 2023/07/20 13:03:00 Mayank Shekhar Narula wrote:
> Jun
> 
> Thanks for the feedback.
> 
> Numbers to follow.
> 
> If we don't plan to
> > bump up the FetchResponse version, we could just remove the reference to
> > version 16.
> 
> Fixed.
> 
> On Thu, Jul 20, 2023 at 1:28 AM Jun Rao  wrote:
> 
> > Hi, Mayank,
> >
> > Thanks for the KIP. I agree with others that it would be useful to see the
> > performance results. Otherwise, just a minor comment. If we don't plan to
> > bump up the FetchResponse version, we could just remove the reference to
> > version 16.
> >
> > Jun
> >
> > On Wed, Jul 19, 2023 at 2:31 PM Mayank Shekhar Narula <
> > mayanks.nar...@gmail.com> wrote:
> >
> > > Luke
> > >
> > > Thanks for the interest in the KIP.
> > >
> > > But what if the consumer was fetching from the follower?
> > >
> > > We already include `PreferredReadReplica` in the fetch response.
> > > > Should we put the node info of PreferredReadReplica under this case,
> > > > instead of the leader's info?
> > > >
> > >
> > > PreferredReadReplica is the decided on the leader. Looking at the Java
> > > client code, AbstractFetch::selectReadReplica, first fetch request goes
> > to
> > > Leader of the partition -> Sends back PreferredReadReplica -> Next fetch
> > > uses PreferredReadReplica. So as long as leader is available,
> > > PreferredReadReplica would be found in subsequent fetches.
> > >
> > > Also, under this case, should we include the leader's info in the
> > response?
> > >
> > >
> > > In this case, I think the follower would fail the fetch if it knows a
> > > different leader. If the follower knows a newer leader, it would return
> > new
> > > leader information in the response, for the client to act on.
> > >
> > >
> > > Will we include the leader/node info in the response when having
> > > > `UNKNOWN_LEADER_EPOCH` error?
> > >
> > >
> > > My understanding is UNKNOWN_LEADER_EPOCH when a request from a client
> > has a
> > > newer epoch than the broker. So the client is already up to date on new
> > > leader information, it's the broker that has the catching up to do. I
> > think
> > > there might be some optimisations to make sure the broker refreshes its
> > > metadata quickly, so it can quickly recover to handle requests that
> > > previously returned UNKNOWN_LEADER_EPOCH. But this work is outside the
> > > scope of this KIP, as for now this KIP focusses on client-side
> > > optimisations.
> > >
> > > Mayank
> > >
> > > On Tue, Jul 18, 2023 at 8:51 AM Luke Chen  wrote:
> > >
> > > > Hi Mayank,
> > > >
> > > > Thanks for the KIP!
> > > >
> > > > Some questions:
> > > > 1. I can see most of the cases we only care about consumer fetch from
> > the
> > > > leader.
> > > > But what if the consumer was fetching from the follower?
> > > > We already include `PreferredReadReplica` in the fetch response.
> > > > Should we put the node info of PreferredReadReplica under this case,
> > > > instead of the leader's info?
> > > > Also, under this case, should we include the leader's info in the
> > > response?
> > > >
> > > > 2. Will we include the leader/node info in the response when having
> > > > `UNKNOWN_LEADER_EPOCH` error?
> > > > I think it's fine we ignore the `UNKNOWN_LEADER_EPOCH` error since when
> > > > this happens, the node might have some error which should refresh the
> > > > metadata. On the other hand, it might also be good if we can heal the
> > > node
> > > > soon to do produce/consume works.
> > > >
> > > >
> > > > Thank you.
> > > > Luke
> > > >
> > > > On Tue, Jul 18, 2023 at 2:00 AM Philip Nee 
> > wrote:
> > > >
> > > > > Hey Mayank:
> > > > >
> > > > > For #1: I think fetch and produce behave a bit differently on
> > metadata.
> > > > > Maybe it is worth highlighting the changes for each client in detail.
> > > In
> > > > > producer did you mean by the metadata timeout before sending out
> > > produce
> > > > > requests? For consumer: I think for fetches it requires user to retry
> > > if
> > > > > the position does not exist on the leader. I don't have the detail on
> > > top
> > > > > of my head, but I think we should lay out these behavioral changes.
> > > > >
> > > > > For #3: Thanks for the clarification.
> > > > >
> > > > > On Mon, Jul 17, 2023 at 10:39 AM Mayank Shekhar Narula <
> > > > > mayanks.nar...@gmail.com> wrote:
> > > > >
> > > > > > Philip
> > > > > >
> > > > > > 1. Good call out about "poll" behaviour, my understanding is the
> > > same.
> > > > I
> > > > > am
> > > > > > assuming it's about the motivation of the KIP. There with async, my
> > > > > > intention was to convey that the client doesn't wait for the
> > > > > > metadata-refresh before a subsequent retry of the produce or fetch
> > > > > request
> > > > > > that failed due to stale metadata(i.e. going to an old leader). The
> > > > only
> > > > > > wait client has is the configured retry-delay.
> > > > > >
> > > > > > 2. Yes, in theory other APIs could 

[GitHub] [kafka-site] mjsax commented on a diff in pull request #528: MINOR: Add statmenet about ZK deprecation to 3.5 release blog post

[via GitHub]

mjsax commented on code in PR #528:
URL: https://github.com/apache/kafka-site/pull/528#discussion_r1269989787


##
35/ops.html:
##
@@ -3444,6 +3444,18 @@ 6.9
   Stable version
   The current stable branch is 3.5. Kafka is regularly updated to include the 
latest release in the 3.5 series.
 
+  ZooKeeper 
Deprecation

Review Comment:
   Thanks. I did c from the PR against `trunk` -- if this does not render as 
expected, we need to part the change to `trunk` and `3.5` branches, too. (I can 
do another PR...)
   
   Good idea to update TOC.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Created] (KAFKA-15228) Add sync-manifests subcommand to connect-plugin-path tool

[Greg Harris (Jira)]

Greg Harris created KAFKA-15228:
---

 Summary: Add sync-manifests subcommand to connect-plugin-path tool
 Key: KAFKA-15228
 URL: https://issues.apache.org/jira/browse/KAFKA-15228
 Project: Kafka
  Issue Type: New Feature
  Components: KafkaConnect, tools
Reporter: Greg Harris
Assignee: Greg Harris
 Fix For: 3.6.0






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2020

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-15227) Use plugin.discovery=SERVICE_LOAD in all plugin test suites

[Greg Harris (Jira)]

Greg Harris created KAFKA-15227:
---

 Summary: Use plugin.discovery=SERVICE_LOAD in all plugin test 
suites
 Key: KAFKA-15227
 URL: https://issues.apache.org/jira/browse/KAFKA-15227
 Project: Kafka
  Issue Type: Test
  Components: KafkaConnect
Reporter: Greg Harris
Assignee: Greg Harris


To speed up these tests where we know all plugins are migrated, use 
SERVICE_LOAD mode in all known test suites.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15226) System tests for plugin.discovery worker configuration

[Greg Harris (Jira)]

Greg Harris created KAFKA-15226:
---

 Summary: System tests for plugin.discovery worker configuration
 Key: KAFKA-15226
 URL: https://issues.apache.org/jira/browse/KAFKA-15226
 Project: Kafka
  Issue Type: Test
  Components: KafkaConnect
Reporter: Greg Harris
Assignee: Greg Harris


Add system tests as described in KIP-898, targeting the startup behavior of the 
connect worker, various states of plugin migration, and the migration script.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15225) Define constants for record types

[David Jacot (Jira)]

David Jacot created KAFKA-15225:
---

 Summary: Define constants for record types
 Key: KAFKA-15225
 URL: https://issues.apache.org/jira/browse/KAFKA-15225
 Project: Kafka
  Issue Type: Sub-task
Reporter: David Jacot
Assignee: David Jacot


Define constants for all the record types. Ideally, this should be defined in 
the record definitions and the constants should be auto-generated (e.g. like 
ApiKeys).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Apache Kafka 3.6.0 release

[Hao Li]

Hi Satish,

KIP-925 was accepted and currently under implementation. I just added it to
the release plan.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-925%3A+Rack+aware+task+assignment+in+Kafka+Streams

Thanks,
Hao

On Thu, Jul 20, 2023 at 6:18 AM Christo Lolov 
wrote:

> Hello!
>
> A couple of days ago I opened a new KIP for discussion - KIP-952 [1]. I
> believe it might be a blocker for the release of 3.6.0, but I wanted to
> bring it up here for a decision on its urgency with the current set of
> people who are looking at Tiered Storage (Satish, Luke, Ivan, Divij) given
> that the date for KIP freeze is fast approaching.
> What are your thoughts on the matter?
>
> [1]
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-952%3A+Regenerate+segment-aligned+producer+snapshots+when+upgrading+to+a+Kafka+version+supporting+Tiered+Storage
>
> Best,
> Christo
>
> On Sat, 8 Jul 2023 at 13:06, Satish Duggana 
> wrote:
>
> > Hi Yash,
> > Thanks for the update. Added KIP-793 to the release plan. Please feel
> > free to update the release wiki with any other updates on the KIP.
> >
> > ~Satish.
> >
> > On Fri, 7 Jul 2023 at 10:52, Yash Mayya  wrote:
> > >
> > > Hi Satish,
> > >
> > > KIP-793 [1] just passed voting and we should be able to wrap up the
> > > implementation in time for the 3.6.0 feature freeze. Could we add it to
> > the
> > > release plan?
> > >
> > > [1] -
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-793%3A+Allow+sink+connectors+to+be+used+with+topic-mutating+SMTs
> > >
> > > Thanks,
> > > Yash
> > >
> > > On Mon, Jun 12, 2023 at 3:52 PM Satish Duggana <
> satish.dugg...@gmail.com
> > >
> > > wrote:
> > >
> > > > Hi,
> > > > I have created a release plan for Apache Kafka version 3.6.0 on the
> > > > wiki. You can access the release plan and all related information by
> > > > following this link:
> > > > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.6.0
> > > >
> > > > The release plan outlines the key milestones and important dates for
> > > > version 3.6.0. Currently, the following dates have been set for the
> > > > release:
> > > >
> > > > KIP Freeze: 26th July 23
> > > > Feature Freeze : 16th Aug 23
> > > > Code Freeze : 30th Aug 23
> > > >
> > > > Please review the release plan and provide any additional information
> > > > or updates regarding KIPs targeting version 3.6.0. If you have
> > > > authored any KIPs that are missing a status or if there are incorrect
> > > > status details, please make the necessary updates and inform me so
> > > > that I can keep the plan accurate and up to date.
> > > >
> > > > Thanks,
> > > > Satish.
> > > >
> > > > On Mon, 17 Apr 2023 at 21:17, Luke Chen  wrote:
> > > > >
> > > > > Thanks for volunteering!
> > > > >
> > > > > +1
> > > > >
> > > > > Luke
> > > > >
> > > > > On Mon, Apr 17, 2023 at 2:03 AM Ismael Juma 
> > wrote:
> > > > >
> > > > > > Thanks for volunteering Satish. +1.
> > > > > >
> > > > > > Ismael
> > > > > >
> > > > > > On Sun, Apr 16, 2023 at 10:08 AM Satish Duggana <
> > > > satish.dugg...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > > I would like to volunteer as release manager for the next
> > release,
> > > > > > > which will be Apache Kafka 3.6.0.
> > > > > > >
> > > > > > > If there are no objections, I will start a release plan a week
> > after
> > > > > > > 3.5.0 release(around early May).
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Satish.
> > > > > > >
> > > > > >
> > > >
> >
>

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.4 #151

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 526662 lines...]
> Task :connect:api:publishToMavenLocal

> Task :clients:javadoc
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.4/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/OAuthBearerLoginCallbackHandler.java:151:
 warning - Tag @link: reference not found: 

> Task :streams:javadoc
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:890:
 warning - Tag @link: reference not found: DefaultPartitioner
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:919:
 warning - Tag @link: reference not found: DefaultPartitioner
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:939:
 warning - Tag @link: reference not found: DefaultPartitioner
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:854:
 warning - Tag @link: reference not found: DefaultPartitioner
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:890:
 warning - Tag @link: reference not found: DefaultPartitioner
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:919:
 warning - Tag @link: reference not found: DefaultPartitioner
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:939:
 warning - Tag @link: reference not found: DefaultPartitioner
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/kstream/Produced.java:84:
 warning - Tag @link: reference not found: DefaultPartitioner
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/kstream/Produced.java:136:
 warning - Tag @link: reference not found: DefaultPartitioner
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/kstream/Produced.java:147:
 warning - Tag @link: reference not found: DefaultPartitioner
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/kstream/Repartitioned.java:101:
 warning - Tag @link: reference not found: DefaultPartitioner
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/kstream/Repartitioned.java:167:
 warning - Tag @link: reference not found: DefaultPartitioner
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/TopologyConfig.java:62:
 warning - Tag @link: missing '#': "org.apache.kafka.streams.StreamsBuilder()"
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/TopologyConfig.java:62:
 warning - Tag @link: can't find org.apache.kafka.streams.StreamsBuilder() in 
org.apache.kafka.streams.TopologyConfig
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/TopologyDescription.java:38:
 warning - Tag @link: reference not found: ProcessorContext#forward(Object, 
Object) forwards
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/query/Position.java:44:
 warning - Tag @link: can't find query(Query,
 PositionBound, boolean) in org.apache.kafka.streams.processor.StateStore
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:109:
 warning - Tag @link: reference not found: this#getResult()
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:116:
 warning - Tag @link: reference not found: this#getFailureReason()
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:116:
 warning - Tag @link: reference not found: this#getFailureMessage()
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:154:
 warning - Tag @link: reference not found: this#isSuccess()
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:154:
 warning - Tag @link: reference not found: this#isFailure()
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:890:
 warning - Tag @link: reference not found: DefaultPartitioner
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:919:
 warning - Tag @link: reference not found: 

Requesting permission to contribute to Apache Kafka

[Kenneth Eversole]

Hello,

I am requesting permission to contribute to Apache Kafka

wiki id: keversole
jira id: keversole


Thank you,

Kenneth Eversole

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2019

[Apache Jenkins Server]

See 

Re: [ANNOUNCE] New committer: Greg Harris

[Boudjelda Mohamed Said]

Congrats Greg!


On Thu 20 Jul 2023 at 18:25, Joobi S.B  wrote:

> Congrats Greg!
>
> On Tue, Jul 11, 2023 at 11:40 PM Anna McDonald 
> wrote:
>
> > Congratulations Greg
> >
> > On Tue, Jul 11, 2023 at 9:02 AM Jorge Esteban Quilcate Otoya <
> > quilcate.jo...@gmail.com> wrote:
> >
> > > Congrats Greg!!
> > >
> > > On Tue 11. Jul 2023 at 15.20, Federico Valeri 
> > > wrote:
> > >
> > > > Congrats Greg!
> > > >
> > > > On Tue, Jul 11, 2023 at 3:55 AM Luke Chen  wrote:
> > > > >
> > > > > Congrats Greg!
> > > > >
> > > > > Luke
> > > > >
> > > > > On Tue, Jul 11, 2023 at 8:19 AM Matthew de Detrich
> > > > >  wrote:
> > > > >
> > > > > > Congratulations, well deserved!
> > > > > >
> > > > > > On Mon, Jul 10, 2023 at 5:45 PM Chris Egerton <
> ceger...@apache.org
> > >
> > > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > The PMC for Apache Kafka has invited Greg Harris to become a
> > > > committer,
> > > > > > and
> > > > > > > we are happy to announce that he has accepted!
> > > > > > >
> > > > > > > Greg has been contributing to Kafka since 2019. He has made
> over
> > 50
> > > > > > commits
> > > > > > > mostly around Kafka Connect and Mirror Maker 2. His most
> notable
> > > > > > > contributions include KIP-898: "Modernize Connect plugin
> > discovery"
> > > > and a
> > > > > > > deep overhaul of the offset syncing logic in MM2 that addressed
> > > > several
> > > > > > > technically-difficult, long-standing, high
> -impact
> issues.
> > > > > > >
> > > > > > > He has also been an active participant in discussions and
> reviews
> > > on
> > > > the
> > > > > > > mailing lists and on GitHub.
> > > > > > >
> > > > > > > Thanks for all of your contributions, Greg. Congratulations!
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > > Matthew de Detrich
> > > > > >
> > > > > > *Aiven Deutschland GmbH*
> > > > > >
> > > > > > Immanuelkirchstraße 26, 10405 Berlin
> > > > > >
> > > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > > >
> > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > > >
> > > > > > *m:* +491603708037
> > > > > >
> > > > > > *w:* aiven.io *e:* matthew.dedetr...@aiven.io
> > > > > >
> > > >
> > >
> >
>

Re: [ANNOUNCE] New committer: Greg Harris

[Joobi S.B]

Congrats Greg!

On Tue, Jul 11, 2023 at 11:40 PM Anna McDonald 
wrote:

> Congratulations Greg
>
> On Tue, Jul 11, 2023 at 9:02 AM Jorge Esteban Quilcate Otoya <
> quilcate.jo...@gmail.com> wrote:
>
> > Congrats Greg!!
> >
> > On Tue 11. Jul 2023 at 15.20, Federico Valeri 
> > wrote:
> >
> > > Congrats Greg!
> > >
> > > On Tue, Jul 11, 2023 at 3:55 AM Luke Chen  wrote:
> > > >
> > > > Congrats Greg!
> > > >
> > > > Luke
> > > >
> > > > On Tue, Jul 11, 2023 at 8:19 AM Matthew de Detrich
> > > >  wrote:
> > > >
> > > > > Congratulations, well deserved!
> > > > >
> > > > > On Mon, Jul 10, 2023 at 5:45 PM Chris Egerton  >
> > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > The PMC for Apache Kafka has invited Greg Harris to become a
> > > committer,
> > > > > and
> > > > > > we are happy to announce that he has accepted!
> > > > > >
> > > > > > Greg has been contributing to Kafka since 2019. He has made over
> 50
> > > > > commits
> > > > > > mostly around Kafka Connect and Mirror Maker 2. His most notable
> > > > > > contributions include KIP-898: "Modernize Connect plugin
> discovery"
> > > and a
> > > > > > deep overhaul of the offset syncing logic in MM2 that addressed
> > > several
> > > > > > technically-difficult, long-standing, high-impact issues.
> > > > > >
> > > > > > He has also been an active participant in discussions and reviews
> > on
> > > the
> > > > > > mailing lists and on GitHub.
> > > > > >
> > > > > > Thanks for all of your contributions, Greg. Congratulations!
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Matthew de Detrich
> > > > >
> > > > > *Aiven Deutschland GmbH*
> > > > >
> > > > > Immanuelkirchstraße 26, 10405 Berlin
> > > > >
> > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > >
> > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > >
> > > > > *m:* +491603708037
> > > > >
> > > > > *w:* aiven.io *e:* matthew.dedetr...@aiven.io
> > > > >
> > >
> >
>

Re: [DISCUSS] KIP-910: Update Source offsets for Source Connectors without producing records

[Sagar]

Hey All,

Please let me know how the KIP looks now. Is it at a stage where I can
start with the Voting phase? Of course I am still open to
feedback/suggestions but planning to start the Vote for it.

Thanks!
Sagar.

On Tue, Jul 11, 2023 at 10:00 PM Sagar  wrote:

> Hi Yash/Chris,
>
> Thanks for the feedback! I have updated the KIP with the suggestions
> provided. I would also update the PR with the suggestions.
>
> Also, I was hoping that this could make it to the 3.6 release given that
> it would benefit source connectors which have some of the problems listed
> in the Motivation Section.
>
> Responses Inline:
>
> Yash:
>
> 1) In the proposed changes section where you talk about modifying the
>> offsets, could you please clarify that tasks shouldn't modify the offsets
>> map that is passed as an argument? Currently, the distinction between the
>> offsets map passed as an argument and the offsets map that is returned is
>> not very clear in numerous places.
>
>
>
> Added
>
> 2) The default return value of Optional.empty() seems to be fairly
>> non-intuitive considering that the return value is supposed to be the
>> offsets that are to be committed. Can we consider simply returning the
>> offsets argument itself by default instead?
>
>
>
> Chris is suggesting returning null for the default case. I am thinking to
> make null
> as the default return type. If the returned map is null, there won't be
> any further
> processing otherwise we will contonue with the existing logic.
>
> 3) The KIP states that "It is also possible that a task might choose to
>> send a tombstone record as an offset. This is not recommended and to
>> prevent connectors shooting themselves in the foot due to this" - could
>> you
>> please clarify why this is not recommended / supported?
>
>
>
> I have added that a better way of doing that would be via KIP-875. Also, I
> didn't want to include
> any mechamisms for users to meddle with the offsets topic. Allowing
> tombstone records via this method
> would be akin to publishing tombstone records directly to the offsets
> topic which is not recommended
> generally.
>
> 4) The KIP states that "If a task returns an Optional of a null object or
>> an Optional of an empty map, even for such cases the behaviour would would
>> be disabled." - since this is an optional API that source task
>> implementations don't necessarily need to implement, I don't think I fully
>> follow why the return type of the proposed "updateOffsets" method is an
>> Optional? Can we not simply use the Map as the return type instead?
>
>
>
> Yeah, I updated the return type to be a Map.
>
>
> 5) The KIP states that "The offsets passed to the updateOffsets  method
>> would be the offset from the latest source record amongst all source
>> records per partition. This way, if the source offset for a given source
>> partition is updated, that offset is the one that gets committed for the
>> source partition." - we should clarify that the "latest" offset refers to
>> the offsets that are about to be committed, and not the latest offsets
>> returned from SourceTask::poll so far (see related discussion in
>> https://issues.apache.org/jira/browse/KAFKA-15091 and
>> https://issues.apache.org/jira/browse/KAFKA-5716).
>
>
>
> Done
>
>
> 6) We haven't used the terminology of "Atleast Once Semantics" elsewhere in
>> Connect since the framework itself does not (and cannot) make any
>> guarantees on the delivery semantics. Depending on the source connector
>> and
>> the source system, both at-least once and at-most once semantics (for
>> example - a source system where reads are destructive) are possible. We
>> should avoid introducing this terminology in the KIP and instead refer to
>> this scenario as exactly-once support being disabled.
>
>
>
> Done
>
>
> 7) Similar to the above point, we should remove the use of the term
>> "Exactly Once Semantics" and instead refer to exactly-once support being
>> enabled since the framework can't guarantee exactly-once semantics for all
>> possible source connectors (for example - a message queue source connector
>> where offsets are essentially managed in the source system via an ack
>> mechanism).
>
>
> Done
>
> 8) In a previous attempt to fix this gap in functionality, a significant
>> concern was raised on offsets ordering guarantees when we retry sending a
>> batch of records (ref -
>> https://github.com/apache/kafka/pull/5553/files#r213329307). It doesn't
>> look like this KIP addresses that concern either? In the case where
>> exactly-once support is disabled - if we update the committableOffsets
>> with
>> the offsets provided by the task through the new updateOffsets method,
>> these offsets could be committed before older "regular" offsets are
>> committed due to producer retries which could then lead to an
>> inconsistency
>> if the send operation eventually succeeds.
>
>
>
>
> Thanks for bringing this up. I went through the comment shared above. If
> you see the implementation
> that I 

Re: [DISCUSS] KIP-953: partition method to be overloaded to accept headers as well.

[Sagar]

Thanks Jack for the updates.

Some more feedback:

1) It would be better if you can add the Javadoc in the Public interfaces
section. That is a general practice used which gives the readers of the KIP
a high level idea of the Public Interfaces.

2) In the proposed section, the bit about marking headers as read only
seems like an implementation detail This can generally be avoided in KIPs.

3) Also, in the Deprecation section, can you mention again that this is a
backward compatible change and the reason for it (already done in the
Proposed Changes section).

4) In the Testing Plan section, there is still the KIP template bit copied
over. That can be removed.

Thanks!
Sagar.


On Thu, Jul 20, 2023 at 2:48 PM Jack Tomy  wrote:

> Hey Everyone,
>
> Please consider this as a reminder and share your feedback. Thank you.
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263424937
>
> On Tue, Jul 18, 2023 at 5:43 PM Jack Tomy  wrote:
>
> > Hey @Sagar,
> >
> > Thank you for the response and feedback.
> >
> >1. Done
> >2. Yeah, that was a mistake from my end. Corrected.
> >3. Can you please elaborate this, I have added the java doc along with
> >the code changes. Should I paste the same in KIP too?
> >4. Moved.
> >5. I have added one more use case, it is actually helpful in any
> >situation where you want to pass some information to partition method
> but
> >don't have to have it in the key or value.
> >6. Added.
> >
> >
> > Hey @Sagar and everyone,
> > Please have a look at the new version and share your thoughts.
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263424937
> >
> >
> > On Tue, Jul 18, 2023 at 9:53 AM Sagar  wrote:
> >
> >> Hi Jack,
> >>
> >> Thanks for the KIP! Seems like an interesting idea. I have some
> feedback:
> >>
> >> 1) It would be great if you could clean up the text that seems to mimic
> >> the
> >> KIP template. It is generally not required in the KIP.
> >>
> >> 2) In the Public Interfaces where you mentioned *Partitioner method in
> >> **org/apache/kafka/clients/producer
> >> will have the following update*, I believe you meant the Partitioner
> >> *interface*?
> >>
> >> 3) Staying on Public Interface, it is generally preferable to add a
> >> Javadocs section along with the newly added method. You could also
> >> describe
> >> the behaviour of it invoking the default existing method.
> >>
> >> 4) The option that is mentioned in the Rejected Alternatives, seems more
> >> like a workaround to the current problem that you are describing. That
> >> could be added to the Motivation section IMO.
> >>
> >> 5) Can you also add some more examples of scenarios where this would be
> >> helpful? The only scenario mentioned seems to have a workaround. Just
> >> trying to ensure that we have a strong enough motivation before adding a
> >> public API.
> >>
> >> 6) One thing which should also be worth noting down would be what
> happens
> >> if users override both methods, only one method (new or old) and no
> >> methods
> >> (the default behaviour). It would help in understanding the proposal
> >> better.
> >>
> >> Thanks!
> >> Sagar.
> >>
> >>
> >> On Mon, Jul 17, 2023 at 9:19 PM Jack Tomy 
> wrote:
> >>
> >> > Hey everyone,
> >> >
> >> > Not seeing much discussion on the KPI. Might be because it is too
> >> > obvious .
> >> >
> >> > If there are no more comments, I will start the VOTE in the coming
> days.
> >> >
> >> > On Sat, Jul 15, 2023 at 8:48 PM Jack Tomy 
> >> wrote:
> >> >
> >> > > Hey everyone,
> >> > >
> >> > > Please take a look at the KPI below and provide your suggestions and
> >> > > feedback. TIA.
> >> > >
> >> > >
> >> >
> >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263424937
> >> > >
> >> > >
> >> > > --
> >> > > Best Regards
> >> > > *Jack*
> >> > >
> >> >
> >> >
> >> > --
> >> > Best Regards
> >> > *Jack*
> >> >
> >>
> >
> >
> > --
> > Best Regards
> > *Jack*
> >
>
>
> --
> Best Regards
> *Jack*
>

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.5 #46

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 561624 lines...]
[INFO] streams-quickstart-java[maven-archetype]
[INFO] 
[INFO] < org.apache.kafka:streams-quickstart >-
[INFO] Building Kafka Streams :: Quickstart 3.5.2-SNAPSHOT[1/2]
[INFO]   from pom.xml
[INFO] [ pom ]-
[INFO] 
[INFO] --- clean:3.0.0:clean (default-clean) @ streams-quickstart ---
[INFO] 
[INFO] --- remote-resources:1.5:process (process-resource-bundles) @ 
streams-quickstart ---
[INFO] 
[INFO] --- site:3.5.1:attach-descriptor (attach-descriptor) @ 
streams-quickstart ---
[INFO] 
[INFO] --- gpg:1.6:sign (sign-artifacts) @ streams-quickstart ---
[INFO] 
[INFO] --- install:2.5.2:install (default-install) @ streams-quickstart ---
[INFO] Installing 
/home/jenkins/workspace/Kafka_kafka_3.5/streams/quickstart/pom.xml to 
/home/jenkins/.m2/repository/org/apache/kafka/streams-quickstart/3.5.2-SNAPSHOT/streams-quickstart-3.5.2-SNAPSHOT.pom
[INFO] 
[INFO] --< org.apache.kafka:streams-quickstart-java >--
[INFO] Building streams-quickstart-java 3.5.2-SNAPSHOT[2/2]
[INFO]   from java/pom.xml
[INFO] --[ maven-archetype ]---
[INFO] 
[INFO] --- clean:3.0.0:clean (default-clean) @ streams-quickstart-java ---
[INFO] 
[INFO] --- remote-resources:1.5:process (process-resource-bundles) @ 
streams-quickstart-java ---
[INFO] 
[INFO] --- resources:2.7:resources (default-resources) @ 
streams-quickstart-java ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 6 resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- resources:2.7:testResources (default-testResources) @ 
streams-quickstart-java ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- archetype:2.2:jar (default-jar) @ streams-quickstart-java ---
[INFO] Building archetype jar: 
/home/jenkins/workspace/Kafka_kafka_3.5/streams/quickstart/java/target/streams-quickstart-java-3.5.2-SNAPSHOT
[INFO] 
[INFO] --- site:3.5.1:attach-descriptor (attach-descriptor) @ 
streams-quickstart-java ---
[INFO] 
[INFO] --- archetype:2.2:integration-test (default-integration-test) @ 
streams-quickstart-java ---
[WARNING]  Parameter 'skip' (user property 'archetype.test.skip') is read-only, 
must not be used in configuration
[INFO] 
[INFO] --- gpg:1.6:sign (sign-artifacts) @ streams-quickstart-java ---
[INFO] 
[INFO] --- install:2.5.2:install (default-install) @ streams-quickstart-java ---
[INFO] Installing 
/home/jenkins/workspace/Kafka_kafka_3.5/streams/quickstart/java/target/streams-quickstart-java-3.5.2-SNAPSHOT.jar
 to 
/home/jenkins/.m2/repository/org/apache/kafka/streams-quickstart-java/3.5.2-SNAPSHOT/streams-quickstart-java-3.5.2-SNAPSHOT.jar
[INFO] Installing 
/home/jenkins/workspace/Kafka_kafka_3.5/streams/quickstart/java/pom.xml to 
/home/jenkins/.m2/repository/org/apache/kafka/streams-quickstart-java/3.5.2-SNAPSHOT/streams-quickstart-java-3.5.2-SNAPSHOT.pom
[INFO] 
[INFO] --- archetype:2.2:update-local-catalog (default-update-local-catalog) @ 
streams-quickstart-java ---
[INFO] 
[INFO] Reactor Summary for Kafka Streams :: Quickstart 3.5.2-SNAPSHOT:
[INFO] 
[INFO] Kafka Streams :: Quickstart  SUCCESS [  2.577 s]
[INFO] streams-quickstart-java  SUCCESS [  1.417 s]
[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time:  4.448 s
[INFO] Finished at: 2023-07-20T15:59:31Z
[INFO] 
[Pipeline] dir
Running in 
/home/jenkins/workspace/Kafka_kafka_3.5/streams/quickstart/test-streams-archetype
[Pipeline] {
[Pipeline] sh
+ echo Y
+ mvn archetype:generate -DarchetypeCatalog=local 
-DarchetypeGroupId=org.apache.kafka 
-DarchetypeArtifactId=streams-quickstart-java -DarchetypeVersion=3.5.2-SNAPSHOT 
-DgroupId=streams.examples -DartifactId=streams.examples -Dversion=0.1 
-Dpackage=myapps
[INFO] Scanning for projects...
[INFO] 
[INFO] --< org.apache.maven:standalone-pom >---
[INFO] Building Maven Stub Project (No POM) 1
[INFO] [ pom ]-
[INFO] 
[INFO] >>> archetype:3.2.1:generate (default-cli) > generate-sources @ 
standalone-pom >>>
[INFO] 
[INFO] <<< archetype:3.2.1:generate (default-cli) < generate-sources @ 
standalone-pom <<<
[INFO] 
[INFO] 
[INFO] --- archetype:3.2.1:generate (default-cli) @ standalone-pom ---
[INFO] Generating project in Interactive mode

Re: Requesting permission to contribute to Apache Kafka

[Divij Vaidya]

You should be all set now.

--
Divij Vaidya



On Thu, Jul 20, 2023 at 4:43 PM Crispin Bernier
 wrote:

> Hi,
> I'm following this wiki to request permission to contribute to Apache Kafka
>
> https://cwiki.apache.org/confluence/display/kafka/kafka+improvement+proposals#KafkaImprovementProposals-GettingStarted
> My wiki ID and jira ID are both: crispinbernier
> Can I get permission please?
>
> Thank you,
> Crispin
>

Requesting permission to contribute to Apache Kafka

[Crispin Bernier]

Hi,
I'm following this wiki to request permission to contribute to Apache Kafka
https://cwiki.apache.org/confluence/display/kafka/kafka+improvement+proposals#KafkaImprovementProposals-GettingStarted
My wiki ID and jira ID are both: crispinbernier
Can I get permission please?

Thank you,
Crispin

[jira] [Resolved] (KAFKA-14669) Include MirrorMaker connector configurations in docs

[Chris Egerton (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Egerton resolved KAFKA-14669.
---
Resolution: Done

> Include MirrorMaker connector configurations in docs
> 
>
> Key: KAFKA-14669
> URL: https://issues.apache.org/jira/browse/KAFKA-14669
> Project: Kafka
>  Issue Type: Improvement
>  Components: docs
>Reporter: Mickael Maison
>Assignee: Gantigmaa Selenge
>Priority: Major
> Fix For: 3.6.0
>
>
> In the https://kafka.apache.org/documentation/#georeplication-flow-configure 
> section we list some of the MirrorMaker connectors configurations. These are 
> hardcoded in the docs: 
> https://github.com/apache/kafka/blob/trunk/docs/ops.html#L768-L788
> Instead we should used the generated docs (added as part of 
> https://github.com/apache/kafka/commit/40af3a74507cce9155f4fb4fca317d3c68235d78)
>  like we do for the file connectors.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2018

[Apache Jenkins Server]

See 

Re: Apache Kafka 3.6.0 release

[Christo Lolov]

Hello!

A couple of days ago I opened a new KIP for discussion - KIP-952 [1]. I
believe it might be a blocker for the release of 3.6.0, but I wanted to
bring it up here for a decision on its urgency with the current set of
people who are looking at Tiered Storage (Satish, Luke, Ivan, Divij) given
that the date for KIP freeze is fast approaching.
What are your thoughts on the matter?

[1]
https://cwiki.apache.org/confluence/display/KAFKA/KIP-952%3A+Regenerate+segment-aligned+producer+snapshots+when+upgrading+to+a+Kafka+version+supporting+Tiered+Storage

Best,
Christo

On Sat, 8 Jul 2023 at 13:06, Satish Duggana 
wrote:

> Hi Yash,
> Thanks for the update. Added KIP-793 to the release plan. Please feel
> free to update the release wiki with any other updates on the KIP.
>
> ~Satish.
>
> On Fri, 7 Jul 2023 at 10:52, Yash Mayya  wrote:
> >
> > Hi Satish,
> >
> > KIP-793 [1] just passed voting and we should be able to wrap up the
> > implementation in time for the 3.6.0 feature freeze. Could we add it to
> the
> > release plan?
> >
> > [1] -
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-793%3A+Allow+sink+connectors+to+be+used+with+topic-mutating+SMTs
> >
> > Thanks,
> > Yash
> >
> > On Mon, Jun 12, 2023 at 3:52 PM Satish Duggana  >
> > wrote:
> >
> > > Hi,
> > > I have created a release plan for Apache Kafka version 3.6.0 on the
> > > wiki. You can access the release plan and all related information by
> > > following this link:
> > > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.6.0
> > >
> > > The release plan outlines the key milestones and important dates for
> > > version 3.6.0. Currently, the following dates have been set for the
> > > release:
> > >
> > > KIP Freeze: 26th July 23
> > > Feature Freeze : 16th Aug 23
> > > Code Freeze : 30th Aug 23
> > >
> > > Please review the release plan and provide any additional information
> > > or updates regarding KIPs targeting version 3.6.0. If you have
> > > authored any KIPs that are missing a status or if there are incorrect
> > > status details, please make the necessary updates and inform me so
> > > that I can keep the plan accurate and up to date.
> > >
> > > Thanks,
> > > Satish.
> > >
> > > On Mon, 17 Apr 2023 at 21:17, Luke Chen  wrote:
> > > >
> > > > Thanks for volunteering!
> > > >
> > > > +1
> > > >
> > > > Luke
> > > >
> > > > On Mon, Apr 17, 2023 at 2:03 AM Ismael Juma 
> wrote:
> > > >
> > > > > Thanks for volunteering Satish. +1.
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Sun, Apr 16, 2023 at 10:08 AM Satish Duggana <
> > > satish.dugg...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > > I would like to volunteer as release manager for the next
> release,
> > > > > > which will be Apache Kafka 3.6.0.
> > > > > >
> > > > > > If there are no objections, I will start a release plan a week
> after
> > > > > > 3.5.0 release(around early May).
> > > > > >
> > > > > > Thanks,
> > > > > > Satish.
> > > > > >
> > > > >
> > >
>

[GitHub] [kafka-site] divijvaidya commented on pull request #510: MINOR: Add collaborator policy

[via GitHub]

divijvaidya commented on PR #510:
URL: https://github.com/apache/kafka-site/pull/510#issuecomment-1643908265

   > I get the same page not found errors for both the links. Could it be a 
permissions issue? @divijvaidya if you're able to view the page now, maybe it's 
only visible to committers?
   
   Nope still can't see. And I am logged-in.
   
   ![Screenshot 2023-07-20 at 15 13 
54](https://github.com/apache/kafka-site/assets/71267/cebe9ac5-a40b-40ac-acd9-e07aa04307ad)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Created] (KAFKA-15224) Automate version change to snapshot

[Divij Vaidya (Jira)]

Divij Vaidya created KAFKA-15224:


 Summary: Automate version change to snapshot 
 Key: KAFKA-15224
 URL: https://issues.apache.org/jira/browse/KAFKA-15224
 Project: Kafka
  Issue Type: Sub-task
Reporter: Divij Vaidya


We require changing to SNAPSHOT version as part of the release process [1]. The 
specific manual steps are:

Update version on the branch to 0.10.0.1-SNAPSHOT in the following places:
 * 
 ** docs/js/templateData.js
 ** gradle.properties
 ** kafka-merge-pr.py
 ** streams/quickstart/java/pom.xml
 ** streams/quickstart/java/src/main/resources/archetype-resources/pom.xml
 ** streams/quickstart/pom.xml
 ** tests/kafkatest/__init__.py (note: this version name can't follow the 
-SNAPSHOT convention due to python version naming restrictions, instead
update it to 0.10.0.1.dev0)
 ** tests/kafkatest/version.py

It would be nice if we could run a script to automatically do it. Note that 
release.py (line 550) already does something similar where it replaces SNAPSHOT 
with actual version. We need to do the opposite here. We can repurpose that 
code in release.py and extract into a new script to perform this opertaion.

[1] https://cwiki.apache.org/confluence/display/KAFKA/Release+Process



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-951: Leader discovery optimisations for the client

[Mayank Shekhar Narula]

Jun

Thanks for the feedback.

Numbers to follow.

If we don't plan to
> bump up the FetchResponse version, we could just remove the reference to
> version 16.

Fixed.

On Thu, Jul 20, 2023 at 1:28 AM Jun Rao  wrote:

> Hi, Mayank,
>
> Thanks for the KIP. I agree with others that it would be useful to see the
> performance results. Otherwise, just a minor comment. If we don't plan to
> bump up the FetchResponse version, we could just remove the reference to
> version 16.
>
> Jun
>
> On Wed, Jul 19, 2023 at 2:31 PM Mayank Shekhar Narula <
> mayanks.nar...@gmail.com> wrote:
>
> > Luke
> >
> > Thanks for the interest in the KIP.
> >
> > But what if the consumer was fetching from the follower?
> >
> > We already include `PreferredReadReplica` in the fetch response.
> > > Should we put the node info of PreferredReadReplica under this case,
> > > instead of the leader's info?
> > >
> >
> > PreferredReadReplica is the decided on the leader. Looking at the Java
> > client code, AbstractFetch::selectReadReplica, first fetch request goes
> to
> > Leader of the partition -> Sends back PreferredReadReplica -> Next fetch
> > uses PreferredReadReplica. So as long as leader is available,
> > PreferredReadReplica would be found in subsequent fetches.
> >
> > Also, under this case, should we include the leader's info in the
> response?
> >
> >
> > In this case, I think the follower would fail the fetch if it knows a
> > different leader. If the follower knows a newer leader, it would return
> new
> > leader information in the response, for the client to act on.
> >
> >
> > Will we include the leader/node info in the response when having
> > > `UNKNOWN_LEADER_EPOCH` error?
> >
> >
> > My understanding is UNKNOWN_LEADER_EPOCH when a request from a client
> has a
> > newer epoch than the broker. So the client is already up to date on new
> > leader information, it's the broker that has the catching up to do. I
> think
> > there might be some optimisations to make sure the broker refreshes its
> > metadata quickly, so it can quickly recover to handle requests that
> > previously returned UNKNOWN_LEADER_EPOCH. But this work is outside the
> > scope of this KIP, as for now this KIP focusses on client-side
> > optimisations.
> >
> > Mayank
> >
> > On Tue, Jul 18, 2023 at 8:51 AM Luke Chen  wrote:
> >
> > > Hi Mayank,
> > >
> > > Thanks for the KIP!
> > >
> > > Some questions:
> > > 1. I can see most of the cases we only care about consumer fetch from
> the
> > > leader.
> > > But what if the consumer was fetching from the follower?
> > > We already include `PreferredReadReplica` in the fetch response.
> > > Should we put the node info of PreferredReadReplica under this case,
> > > instead of the leader's info?
> > > Also, under this case, should we include the leader's info in the
> > response?
> > >
> > > 2. Will we include the leader/node info in the response when having
> > > `UNKNOWN_LEADER_EPOCH` error?
> > > I think it's fine we ignore the `UNKNOWN_LEADER_EPOCH` error since when
> > > this happens, the node might have some error which should refresh the
> > > metadata. On the other hand, it might also be good if we can heal the
> > node
> > > soon to do produce/consume works.
> > >
> > >
> > > Thank you.
> > > Luke
> > >
> > > On Tue, Jul 18, 2023 at 2:00 AM Philip Nee 
> wrote:
> > >
> > > > Hey Mayank:
> > > >
> > > > For #1: I think fetch and produce behave a bit differently on
> metadata.
> > > > Maybe it is worth highlighting the changes for each client in detail.
> > In
> > > > producer did you mean by the metadata timeout before sending out
> > produce
> > > > requests? For consumer: I think for fetches it requires user to retry
> > if
> > > > the position does not exist on the leader. I don't have the detail on
> > top
> > > > of my head, but I think we should lay out these behavioral changes.
> > > >
> > > > For #3: Thanks for the clarification.
> > > >
> > > > On Mon, Jul 17, 2023 at 10:39 AM Mayank Shekhar Narula <
> > > > mayanks.nar...@gmail.com> wrote:
> > > >
> > > > > Philip
> > > > >
> > > > > 1. Good call out about "poll" behaviour, my understanding is the
> > same.
> > > I
> > > > am
> > > > > assuming it's about the motivation of the KIP. There with async, my
> > > > > intention was to convey that the client doesn't wait for the
> > > > > metadata-refresh before a subsequent retry of the produce or fetch
> > > > request
> > > > > that failed due to stale metadata(i.e. going to an old leader). The
> > > only
> > > > > wait client has is the configured retry-delay.
> > > > >
> > > > > 2. Yes, in theory other APIs could benefit from this too. But that
> is
> > > > > outside of the scope of the KIP.
> > > > >
> > > > > 3. Do you mean the response for the Metadata RPC? I think brokers
> > > always
> > > > > have a view of the cluster, although it can be stale,it would
> always
> > > > return
> > > > > a leader(whether old or new).
> > > > >
> > > > > Mayank
> > > > >
> > > > > On Fri, Jul 14, 2023 at 8:53 PM 

[jira] [Created] (KAFKA-15223) Need clarity in documentation for upgrade/downgrade across releases.

[kaushik srinivas (Jira)]

kaushik srinivas created KAFKA-15223:


 Summary: Need clarity in documentation for upgrade/downgrade 
across releases.
 Key: KAFKA-15223
 URL: https://issues.apache.org/jira/browse/KAFKA-15223
 Project: Kafka
  Issue Type: Improvement
Reporter: kaushik srinivas


Referring to the upgrade documentation for apache kafka.

[https://kafka.apache.org/34/documentation.html#upgrade_3_4_0]

There is confusion with respect to below statements from the above sectioned 
link of apache docs.

"If you are upgrading from a version prior to 2.1.x, please see the note below 
about the change to the schema used to store consumer offsets. *Once you have 
changed the inter.broker.protocol.version to the latest version, it will not be 
possible to downgrade to a version prior to 2.1."*

The above statement mentions that the downgrade would not be possible to 
version prior to "2.1" in case of "upgrading the inter.broker.protocol.version 
to the latest version".

But, there is another statement made in the documentation in *point 4* as below

"Restart the brokers one by one for the new protocol version to take effect. 
{*}Once the brokers begin using the latest protocol version, it will no longer 
be possible to downgrade the cluster to an older version.{*}"

 

These two statements are repeated across a lot of prior release of kafka and is 
confusing.

Below are the questions:
 # Is downgrade not at all possible to *"any"* older version of kafka once the 
inter.broker.protocol.version is updated to latest version *OR* downgrades are 
not possible only to versions *"<2.1"* ?
 # Suppose one takes an approach similar to upgrade even for the downgrade 
path. i.e. downgrade the inter.broker.protocol.version first to the previous 
version, next downgrade the software/code of kafka to previous release 
revision. Does downgrade work with this approach ?

Can these two questions be documented if the results are already known ?

Maybe a downgrade guide can be created too similar to the existing upgrade 
guide ?



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

The description of Kafka project in github (About section)

[Boudjelda Mohamed Said]

Hello,

   On github repository home page https://github.com/apache/kafka, the
about section mention that the repository is a  "Mirror of Apache Kafka", I
think it's a better to put a description about what is kafka and a linked
to Apache Kafka official website: https://kafka.apache.org

  Suggestion to use an already existing in Kafka website

 "Apache Kafka is an open-source distributed event streaming platform used
by thousands of companies for high-performance data pipelines, streaming
analytics, data integration, and mission-critical applications."
https://kafka.apache.org

  Because the world mirror can give the impression a non active repository


Thanks

~
Said BOUDJELDA

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2017

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-15222) Upgrade zinc scala incremental compiler plugin version to a latests stable fit version

[Said BOUDJELDA (Jira)]

Said BOUDJELDA created KAFKA-15222:
--

 Summary: Upgrade zinc scala incremental compiler plugin version to 
a latests stable fit version 
 Key: KAFKA-15222
 URL: https://issues.apache.org/jira/browse/KAFKA-15222
 Project: Kafka
  Issue Type: Improvement
  Components: build, tools
Reporter: Said BOUDJELDA
Assignee: Said BOUDJELDA






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-12525) Inaccurate task status due to status record interleaving in fast rebalances in Connect

[Sagar Rao (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-12525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sagar Rao resolved KAFKA-12525.
---
Resolution: Fixed

> Inaccurate task status due to status record interleaving in fast rebalances 
> in Connect
> --
>
> Key: KAFKA-12525
> URL: https://issues.apache.org/jira/browse/KAFKA-12525
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 2.3.1, 2.4.1, 2.5.1, 2.7.0, 2.6.1
>Reporter: Konstantine Karantasis
>Assignee: Sagar Rao
>Priority: Major
>
> When a task is stopped in Connect it produces an {{UNASSIGNED}} status 
> record. 
> Equivalently, when a task is started or restarted in Connect it produces an 
> {{RUNNING}} status record in the Connect status topic.
> At the same time rebalances are decoupled from task start and stop. These 
> operations happen in separate executor outside of the main worker thread that 
> performs the rebalance.
> Normally, any delayed and stale {{UNASSIGNED}} status records are fenced by 
> the worker that is sending them. This worker is using the 
> {{StatusBackingStore#putSafe}} method that will reject any stale status 
> messages (called only for {{UNASSIGNED}} or {{FAILED}}) as long as the worker 
> is aware of the newer status record that declares a task as {{RUNNING}}.
> In cases of fast consecutive rebalances where a task is revoked from one 
> worker and assigned to another one, it has been observed that there is a 
> small time window and thus a race condition during which a {{RUNNING}} status 
> record in the new generation is produced and is immediately followed by a 
> delayed {{UNASSIGNED}} status record belonging to the same or a previous 
> generation before the worker that sends this message reads the {{RUNNING}} 
> status record that corresponds to the latest generation.
> A couple of options are available to remediate this race condition. 
> For example a worker that is has started a task can re-write the {{RUNNING}} 
> status message in the topic if it reads a stale {{UNASSIGNED}} message from a 
> previous generation (that should have been fenced). 
> Another option is to ignore stale {{UNASSIGNED}} message (messages from an 
> earlier generation than the one in which the task had {{RUNNING}} status).
> Worth noting that when this race condition takes place, besides the 
> inaccurate status representation, the actual execution of the tasks remains 
> unaffected (e.g. the tasks are running correctly even though they appear as 
> {{UNASSIGNED}}). 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: FYI - Timeout waiting to lock zinc...It is currently in use by another Gradle instance

[Boudjelda Mohamed Said]

Hello,

  Thanks much Divij for the reports and for links

  This is issue is not related to a the current used version of Gradle,
it's occurs even on older versions, and every time it happens with zinc
incremental scala compiler, and this because a timeout
on a lock of zinc caches, possible reason can since there is no
configurable timeout on Gradle side,  for now there is a script retry_sinc
on the root of kafka project that avoid retrying to run a build manually
on jenkins

   Avoiding these errors  for now can cost much time by getting a seperate
workplace and cache for each build
 Stopping all running gradle instances and clearing cache before running a
new one by using ./gradlew --stop (I am not sure this will solve the issue)

  I am trying to understand more about this issue, I'll update you as soon
as I have news about it


~
Said BOUDJELDA


On Thu, Jul 20, 2023 at 11:34 AM Divij Vaidya 
wrote:

> I added a link to the PR above, should have added it to this email as well.
>
> See the following link fot the CI errors (it also lists the PRs on top
> right corner (top tags) which are impacted by the error):
>
> https://ge.apache.org/scans/failures?failures.failureMessage=Execution%20failed%20for%20task%20*%0A%3E%20Timeout%20waiting%20to%20lock%20*%20cache%20(/home/jenkins/.gradle/caches/*/*%20It%20is%20currently%20in%20use%20by%20another%20Gradle%20instance.%0A%20%20Owner%20PID:%20*%0A%20%20Our%20PID:%20*%0A%20%20Owner%20Operation:%20%0A%20%20Our%20operation:%20%0A%20%20Lock%20file:%20/home/jenkins/.gradle/caches/*/*/*=kafka=Europe/Berlin
>
>
> It seems to be settling down on it's own so hopefully we may not have to do
> anything here. Let's observe for another 24 hours.
>
> --
> Divij Vaidya
>
>
>
> On Thu, Jul 20, 2023 at 4:56 AM Luke Chen  wrote:
>
> > Hi Divij,
> >
> > Thanks for the info.
> >
> > I was trying to investigate it, but cannot find the failed CI build with
> > the zinc error.
> > The PR: https://github.com/apache/kafka/pull/14032 is merged in build
> > #2013
> > .
> > From build #2013 to the latest one, I cannot find any build with the
> error.
> > If there's more info, please share with us.
> >
> > Thank you.
> > Luke
> >
> > On Thu, Jul 20, 2023 at 4:41 AM Divij Vaidya 
> > wrote:
> >
> > > If you see CI failures with the error above, it is possible caused by
> > > recent gradle upgrade we merged at
> > > https://github.com/apache/kafka/pull/14032
> > >
> > > I will take a look at it tomorrow and revert the commit if required.
> > > Meanwhile, if someone else gets a chance, please investigate this.
> > >
> > > --
> > > Divij Vaidya
> > >
> >
>

Re: [DISCUSS] KIP-694: Support Reducing Partitions for Topics

[Xiangyuan LI]

Does the progress of the KIP have any news? I think this feature is
reasonable

Guoqiang Shu  于2021年3月9日周二 22:31写道：

>
> Thanks Guozhang for the comments! Again sorry for the very late response.
> We took time to further verify the implementation internally and rebased
> the proposal on top of our perceived KIP-500 architecture. Embedded please
> find some detailed reply.
>
> On 2020/12/15 05:25:13, Guozhang Wang  wrote:
> > Hello George,
> >
> > Thanks for submitting this KIP. On the high-level I think I agree that
> > handling keyed messages is a very complicated issue and maybe we can just
> > start with the easy scenario that does not involve them. Pushing the
> burden
> > to admin users to determine if it is really safe to delete partitions
> (i.e.
> > there should be no key-ed messages, OR message keys are never used in the
> > partitioner). Regarding the detailed proposal, I have some clarification
> > questions / comments above:
> >
> > 1) Compatibility wise, we need to clarify when talking to the old
> versioned
> > clients who do not recognize the added `status` field, what the brokers
> > should return for those read-only / offline topics. My guess is that, for
> > old versioned brokers we would not include the status field, but would
> > exclude / include the partitions for producers / consumers accordingly.
>
> [GS] We added a compatibility session to the KIP.  We start with a naive
> implementation given that we have reasonable control to internal Kafka use
> cases. We suggest the admin explicit turns on the global feature flag
> delete.topic.partition.enable, and hence limit the Metadata version. The
> lower version client will get a LEADER_NOT_AVAILABLE response. Clients with
> higher version will not have problem as the 'mode' filed has default
> ReadWrite.
>
> >
> > 2) In the upcoming release, with KIP-500/691 we will have a
> zookeeper-free
> > quorum-based controller mechanism. So I'd suggest we "rebase" the
> proposal
> > on top of that given the timing of this KIP, i.e. consider moving those
> > zk-paths as for how the new controller could handle the requests. I'd
> > recommend incorporating the proposal with KIP-691.
>
> [GS] Indeed. We rewrite the proposal based on KIP-500, with zk related
> metadata changes replaced with the perspective new approach. Kindly take a
> look please.
>
> >
> > 3) For the read-only partitions, who's responsible for finally switching
> > the status of those partitions to offline when the retention period
> passed?
> > I cannot tell for sure from the diagram in the KIP. Note that in practice
> > that retention period could be days or even weeks. Also, could we still
> > force delete a read-only partition before its retention period?
> >
>
> [GS] We added a scheduled task that periodically checks the topic that
> passes the retention period. Admittedly this proposal is suited in
> environment like ours where topics have shorter retention. yes we support
> force delete a read-only partition at any time.
>
> > 4) Another thing to consider is how deleting partitions work with adding
> > partitions. For example, if a topic has 3 partitions with 0/1 online and
> 2
> > read-only, and a new admin request is received to add a new partition to
> > that topic, how would this be handled?
> >
>
> [GS] For sake simplicity we mark any topic with pending changes and forbid
> further add and delayed removal.
>
> >
> > Guozhang
> >
> >
> > On Tue, Dec 8, 2020 at 8:05 AM georgeshu(舒国强) 
> wrote:
> >
> > > Hello,
> > >
> > > We write up a KIP based on a straightforward mechanism implemented and
> > > tested in order to solve a practical issue in production.
> > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-694%3A+Support+Reducing+Partitions+for+Topics
> > > Look forward to hearing feedback and suggestions.
> > >
> > > Thanks!
> > >
> > >
> > >
> >
> > --
> > -- Guozhang
> >
>

[RESULTS] [VOTE] Release Kafka version 3.5.1

[Divij Vaidya]

This vote passes with 4 +1 votes (3 bindings) and no 0 or -1 votes.

+1 votes
PMC Members:
* Mickael Maison
* Luke Chen
* Chris Egerton

Community:
* Federico Valeri

0 votes
* No votes

-1 votes
* No votes


Vote thread:https://lists.apache.org/thread/jp82w4rw8l24gm5wh0fgnp5370pyq6s8


I'll continue with the release process and the release announcement
will follow in the next few days.


--

Divij Vaidya
Release Manager for Apache Kafka 3.5.1

Re: FYI - Timeout waiting to lock zinc...It is currently in use by another Gradle instance

[Divij Vaidya]

I added a link to the PR above, should have added it to this email as well.

See the following link fot the CI errors (it also lists the PRs on top
right corner (top tags) which are impacted by the error):
https://ge.apache.org/scans/failures?failures.failureMessage=Execution%20failed%20for%20task%20*%0A%3E%20Timeout%20waiting%20to%20lock%20*%20cache%20(/home/jenkins/.gradle/caches/*/*%20It%20is%20currently%20in%20use%20by%20another%20Gradle%20instance.%0A%20%20Owner%20PID:%20*%0A%20%20Our%20PID:%20*%0A%20%20Owner%20Operation:%20%0A%20%20Our%20operation:%20%0A%20%20Lock%20file:%20/home/jenkins/.gradle/caches/*/*/*=kafka=Europe/Berlin


It seems to be settling down on it's own so hopefully we may not have to do
anything here. Let's observe for another 24 hours.

--
Divij Vaidya



On Thu, Jul 20, 2023 at 4:56 AM Luke Chen  wrote:

> Hi Divij,
>
> Thanks for the info.
>
> I was trying to investigate it, but cannot find the failed CI build with
> the zinc error.
> The PR: https://github.com/apache/kafka/pull/14032 is merged in build
> #2013
> .
> From build #2013 to the latest one, I cannot find any build with the error.
> If there's more info, please share with us.
>
> Thank you.
> Luke
>
> On Thu, Jul 20, 2023 at 4:41 AM Divij Vaidya 
> wrote:
>
> > If you see CI failures with the error above, it is possible caused by
> > recent gradle upgrade we merged at
> > https://github.com/apache/kafka/pull/14032
> >
> > I will take a look at it tomorrow and revert the commit if required.
> > Meanwhile, if someone else gets a chance, please investigate this.
> >
> > --
> > Divij Vaidya
> >
>

[jira] [Resolved] (KAFKA-15212) Remove unneeded classgraph license file

[Divij Vaidya (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Divij Vaidya resolved KAFKA-15212.
--
  Reviewer: Divij Vaidya
Resolution: Fixed

> Remove unneeded classgraph license file
> ---
>
> Key: KAFKA-15212
> URL: https://issues.apache.org/jira/browse/KAFKA-15212
> Project: Kafka
>  Issue Type: Bug
>Reporter: Divij Vaidya
>Assignee: Tanay Karmarkar
>Priority: Major
>  Labels: newbie
> Fix For: 3.6.0
>
>
> The license file for classgraph can be completely removed from here: 
> [https://github.com/apache/kafka/blob/trunk/licenses/classgraph-MIT] since it 
> is not a dependency of Kafka any more.
> The associated package was removed from license at 
> [https://github.com/apache/kafka/commit/6cf4a2eaa7a436f0233aece49ed81bafe64262c4]
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-953: partition method to be overloaded to accept headers as well.

[Jack Tomy]

Hey Everyone,

Please consider this as a reminder and share your feedback. Thank you.
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263424937

On Tue, Jul 18, 2023 at 5:43 PM Jack Tomy  wrote:

> Hey @Sagar,
>
> Thank you for the response and feedback.
>
>1. Done
>2. Yeah, that was a mistake from my end. Corrected.
>3. Can you please elaborate this, I have added the java doc along with
>the code changes. Should I paste the same in KIP too?
>4. Moved.
>5. I have added one more use case, it is actually helpful in any
>situation where you want to pass some information to partition method but
>don't have to have it in the key or value.
>6. Added.
>
>
> Hey @Sagar and everyone,
> Please have a look at the new version and share your thoughts.
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263424937
>
>
> On Tue, Jul 18, 2023 at 9:53 AM Sagar  wrote:
>
>> Hi Jack,
>>
>> Thanks for the KIP! Seems like an interesting idea. I have some feedback:
>>
>> 1) It would be great if you could clean up the text that seems to mimic
>> the
>> KIP template. It is generally not required in the KIP.
>>
>> 2) In the Public Interfaces where you mentioned *Partitioner method in
>> **org/apache/kafka/clients/producer
>> will have the following update*, I believe you meant the Partitioner
>> *interface*?
>>
>> 3) Staying on Public Interface, it is generally preferable to add a
>> Javadocs section along with the newly added method. You could also
>> describe
>> the behaviour of it invoking the default existing method.
>>
>> 4) The option that is mentioned in the Rejected Alternatives, seems more
>> like a workaround to the current problem that you are describing. That
>> could be added to the Motivation section IMO.
>>
>> 5) Can you also add some more examples of scenarios where this would be
>> helpful? The only scenario mentioned seems to have a workaround. Just
>> trying to ensure that we have a strong enough motivation before adding a
>> public API.
>>
>> 6) One thing which should also be worth noting down would be what happens
>> if users override both methods, only one method (new or old) and no
>> methods
>> (the default behaviour). It would help in understanding the proposal
>> better.
>>
>> Thanks!
>> Sagar.
>>
>>
>> On Mon, Jul 17, 2023 at 9:19 PM Jack Tomy  wrote:
>>
>> > Hey everyone,
>> >
>> > Not seeing much discussion on the KPI. Might be because it is too
>> > obvious .
>> >
>> > If there are no more comments, I will start the VOTE in the coming days.
>> >
>> > On Sat, Jul 15, 2023 at 8:48 PM Jack Tomy 
>> wrote:
>> >
>> > > Hey everyone,
>> > >
>> > > Please take a look at the KPI below and provide your suggestions and
>> > > feedback. TIA.
>> > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263424937
>> > >
>> > >
>> > > --
>> > > Best Regards
>> > > *Jack*
>> > >
>> >
>> >
>> > --
>> > Best Regards
>> > *Jack*
>> >
>>
>
>
> --
> Best Regards
> *Jack*
>


-- 
Best Regards
*Jack*

[GitHub] [kafka-site] mimaison commented on a diff in pull request #528: MINOR: Add statmenet about ZK deprecation to 3.5 release blog post

[via GitHub]

mimaison commented on code in PR #528:
URL: https://github.com/apache/kafka-site/pull/528#discussion_r1269168738


##
35/ops.html:
##
@@ -3444,6 +3444,18 @@ 6.9
   Stable version
   The current stable branch is 3.5. Kafka is regularly updated to include the 
latest release in the 3.5 series.
 
+  ZooKeeper 
Deprecation

Review Comment:
   This does not render well. You need to use the same syntax than the other 
headers.
   My diff to make it render correctly:
   ```
   diff --git 35/ops.html 35/ops.html
   index 37e6a9b9..37d1ffdf 100644
   --- 35/ops.html
   +++ 35/ops.html
   @@ -3444,16 +3444,16 @@ for built-in state stores, currently we have:
  Stable version
  The current stable branch is 3.5. Kafka is regularly updated to include 
the latest release in the 3.5 series.
   
   -  ZooKeeper Deprecation
   -  With the release of Apache Kafka 3.5, Zookeeper is now marked 
deprecated. Removal of ZooKeeper is planned in the next major release of Apache 
Kafka (version 4.0), which is scheduled to happen no sooner than April 2024. 
During the deprecation phase, ZooKeeper is still supported for metadata 
management of Kafka clusters, but it is not recommended for new deployments. 
There is a small subset of features that remain to be implemented in KRaft, see 
current missing features for 
more information.
   +  ZooKeeper Deprecation
   +  With the release of Apache Kafka 3.5, Zookeeper is now marked 
deprecated. Removal of ZooKeeper is planned in the next major release of Apache 
Kafka (version 4.0), which is scheduled to happen no sooner than April 2024. 
During the deprecation phase, ZooKeeper is still supported for metadata 
management of Kafka clusters, but it is not recommended for new deployments. 
There is a small subset of features that remain to be implemented in KRaft, see 
current missing features for more information.
   
   -Migration
   +Migration
Migration of an existing ZooKeeper based Kafka cluster to KRaft is 
currently Preview and we expect it to be ready for production usage in version 
3.6. Users are recommended to begin planning for migration to KRaft and also 
begin testing to provide any feedback. Refer to ZooKeeper to KRaft Migration for details on how 
to perform a live migration from ZooKeeper to KRaft and current limitations.
   
   -3.x and ZooKeeper Support
   +3.x and ZooKeeper 
Support
The final 3.x minor release, that supports ZooKeeper mode, will 
receive critical bug fixes and security fixes for 12 months after its 
release.
   
   -ZooKeeper and KRaft timeline
   +ZooKeeper and KRaft 
timeline
For details and updates on tentative timelines for ZooKeeper removal 
and planned KRaft feature releases, refer to https://cwiki.apache.org/confluence/display/KAFKA/KIP-833%3A+Mark+KRaft+as+Production+Ready;>KIP-833.
   
  Operationalizing ZooKeeper
   ```



##
blog.html:
##
@@ -31,6 +31,7 @@ 
 We are proud to announce the release of Apache Kafka 3.5.0. 
This release contains many new features and improvements. This blog post will 
highlight some of the more prominent features. For a full list of changes, be 
sure to check the https://downloads.apache.org/kafka/3.5.0/RELEASE_NOTES.html;>release 
notes.
 See the https://kafka.apache.org/35/documentation.html#upgrade_3_5_0;>Upgrading 
to 3.5.0 from any version 0.8.x through 3.4.x section in the documentation 
for the list of notable changes and detailed upgrade steps.
 The ability to migrate Kafka clusters from ZK to KRaft mode 
with no downtime is still an early access feature. It is currently only 
suitable for testing in non production environments. See https://cwiki.apache.org/confluence/display/KAFKA/KIP-866+ZooKeeper+to+KRaft+Migration;>KIP-866
 for more details.
+   Note: ZooKeeper is now marked deprecated in this 
release. ZooKeeper is planned to be removed in Apache Kafka 4.0. (Cf ZooKeeper Deprecation)

Review Comment:
   The anchor is `zk_depr`



##
35/ops.html:
##
@@ -3444,6 +3444,18 @@ 6.9
   Stable version
   The current stable branch is 3.5. Kafka is regularly updated to include the 
latest release in the 3.5 series.
 
+  ZooKeeper 
Deprecation

Review Comment:
   Should we also add an entry to the ToC since we're adding a section? 
Currently the ToC only has:
   ```
   6.9 ZooKeeper
   - Stable Version
   - Operationalization
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: Request permission to contribute

[Bruno Cadonna]

Hi Igor,

You should be all set, now.

Thank you for your interest in Apache Kafka!

Best,
Bruno

On 7/18/23 1:53 PM, Igor Fomenko wrote:

Hello,

I would like to get permission to contribute to the Apache Kafka project.
Specifically I would like to create KIP for Kafka Streams and to work on
the implementation for that KIP.

My JIRA and Confluence ID is the same and it is: igorf211

Regards,
Igor Fomenko