Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2312

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 327276 lines...]
Gradle Test Run :core:test > Gradle Test Executor 88 > ZkMigrationClientTest > 
testEmptyWrite() STARTED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZkMigrationClientTest > 
testEmptyWrite() PASSED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZkMigrationClientTest > 
testReadMigrateAndWriteProducerId() STARTED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZkMigrationClientTest > 
testReadMigrateAndWriteProducerId() PASSED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZkMigrationClientTest > 
testExistingKRaftControllerClaim() STARTED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZkMigrationClientTest > 
testExistingKRaftControllerClaim() PASSED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZkMigrationClientTest > 
testMigrateTopicConfigs() STARTED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZkMigrationClientTest > 
testMigrateTopicConfigs() PASSED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZkMigrationClientTest > 
testNonIncreasingKRaftEpoch() STARTED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZkMigrationClientTest > 
testNonIncreasingKRaftEpoch() PASSED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZkMigrationClientTest > 
testMigrateEmptyZk() STARTED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZkMigrationClientTest > 
testMigrateEmptyZk() PASSED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZkMigrationClientTest > 
testTopicAndBrokerConfigsMigrationWithSnapshots() STARTED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZkMigrationClientTest > 
testTopicAndBrokerConfigsMigrationWithSnapshots() PASSED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZkMigrationClientTest > 
testClaimAndReleaseExistingController() STARTED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZkMigrationClientTest > 
testClaimAndReleaseExistingController() PASSED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZkMigrationClientTest > 
testClaimAbsentController() STARTED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZkMigrationClientTest > 
testClaimAbsentController() PASSED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZkMigrationClientTest > 
testIdempotentCreateTopics() STARTED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZkMigrationClientTest > 
testIdempotentCreateTopics() PASSED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZkMigrationClientTest > 
testCreateNewTopic() STARTED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZkMigrationClientTest > 
testCreateNewTopic() PASSED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZkMigrationClientTest > 
testUpdateExistingTopicWithNewAndChangedPartitions() STARTED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZkMigrationClientTest > 
testUpdateExistingTopicWithNewAndChangedPartitions() PASSED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZooKeeperClientTest > 
testZNodeChangeHandlerForDataChange() STARTED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZooKeeperClientTest > 
testZNodeChangeHandlerForDataChange() PASSED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZooKeeperClientTest > 
testZooKeeperSessionStateMetric() STARTED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZooKeeperClientTest > 
testZooKeeperSessionStateMetric() PASSED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZooKeeperClientTest > 
testExceptionInBeforeInitializingSession() STARTED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZooKeeperClientTest > 
testExceptionInBeforeInitializingSession() PASSED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZooKeeperClientTest > 
testGetChildrenExistingZNode() STARTED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZooKeeperClientTest > 
testGetChildrenExistingZNode() PASSED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZooKeeperClientTest > 
testConnection() STARTED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZooKeeperClientTest > 
testConnection() PASSED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZooKeeperClientTest > 
testZNodeChangeHandlerForCreation() STARTED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZooKeeperClientTest > 
testZNodeChangeHandlerForCreation() PASSED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZooKeeperClientTest > 
testGetAclExistingZNode() STARTED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZooKeeperClientTest > 
testGetAclExistingZNode() PASSED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZooKeeperClientTest > 
testSessionExpiryDuringClose() STARTED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZooKeeperClientTest > 
testSessionExpiryDuringClose() PASSED

Gradle Test Run :core:test > 

Re: [VOTE] KIP-982: Enhance Custom KafkaPrincipalBuilder to Access SslPrincipalMapper and KerberosShortNamer

[Manikumar]

Hi,

Thanks for the KIP.

+1 (binding)

Thanks,
Manikumar

On Fri, Oct 20, 2023 at 4:26 AM Raghu B  wrote:

> Hi everyone,
>
> I would like to start a vote on KIP-982, which proposed enhancements to
> the Custom KafkaPrincipalBuilder to allow access to SslPrincipalMapper and
> KerberosShortNamer.
>
> This KIP
> 
> aims to improve the flexibility and usability of custom
> KafkaPrincipalBuilder implementations by enabling support for Mapping Rules
> and enhancing the overall security configuration of Kafka brokers.
>
> Thank you for your participation!
>
> Sincerely,
> Raghu
>

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

[Manikumar]

Hi Krishna, Vedarth,

Thanks for the KIP.

1. Can we add directory structure of Docker Image related files in Kafka
repo.

2. > Steps for the Docker image release will be included in the Release
Process doc of Apache Kafka

Can we list down the requirements (repos, accounts) for releasing images to
docker hub. I am mainly asking because PMC needs to request docker hub
access/repos.
I can help in getting required repos/accounts.
https://infra.apache.org/docker-hub-policy.html


Thanks,
Manikumar

On Thu, Oct 19, 2023 at 8:22 PM Krishna Agarwal <
krishna0608agar...@gmail.com> wrote:

> Hi Viktor,
>
> I've noticed there are two types of custom jar configurations:
>
>1. *Type 1*: In this case, only the class name is required(e.g
> *authorizer.class.name
>**)* This can be configured by the
>following steps:
>   - Mount the jar in the container.
>   - Configure the *CLASSPATH* environment variable (used by
>   *kafka-run-class.sh*) by providing the mounted path to it. This can
>   be passed as an environment variable to the docker container.
>2. *Type 2*: Here, in addition to the class name, classpath can also be
>configured (eg *remote.log.metadata.manager.class.name
> *and
>*remote.log.metadata.manager.class.path*). This can be configured by the
>following steps:
>   - Mount the jar in the container.
>   - Configure the respective *class.path* property.
>
> Regards,
> Krishna
>
> On Mon, Sep 25, 2023 at 11:41 PM Krishna Agarwal <
> krishna0608agar...@gmail.com> wrote:
>
> > Hi Viktor,
> > Thanks for the questions.
> >
> >1. While the docker image outlined in KIP-975 is designed for
> >production environments, it is equally suitable for development and
> testing
> >purposes. We will furnish the docker image, allowing users the
> flexibility
> >to employ it according to their specific needs.
> >2. The configs will be injected into the docker container through
> >environment variables. These environment variables will have a prefix
> >allowing for efficient parsing to extract the relevant
> properties.(Will add
> >this implementation in the KIP as well once we converge on this.)
> >3. Regarding this question, I'll conduct a test on my end after
> >gaining a better understanding, and then provide you with a response.
> >
> > Regards,
> > Krishna
> >
> >
> > On Tue, Sep 19, 2023 at 3:42 PM Viktor Somogyi-Vass
> >  wrote:
> >
> >> Hi Ismael,
> >>
> >> I'm not trying to advocate against the docker image, I just pointed out
> >> that the current scoping of the KIP may be a bit too generic and thought
> >> that KIP-974 and KIP-975 were aiming for mostly the same thing and can
> be
> >> discussed under one umbrella. Apologies if this was rooted in a
> >> misunderstanding.
> >>
> >> Kirshna,
> >>
> >> I think we need to refine the KIP a bit more. I think there are some
> >> interfaces that we need to include in the KIP as Kafka has plugins in
> >> certain cases where users are expected to provide implementation and I
> >> think it's worth discussing this in the KIP as they're kind of
> interfaces
> >> for users. Here are my questions in order:
> >> 1. In what environments do you want the image to be used? As I
> understand
> >> it would replace the current testing image and serve as a basis for
> >> development, but would it aim at production use cases too
> (docker-compose,
> >> Kubernetes, etc.)?
> >> 2. How do you plan to forward configs to the broker? Do we expect a
> >> populated server.properties file placed in a certain location or should
> >> the
> >> docker image create this file based on some input (like env vars)?
> >> 3. Certain parts can be pluggable, like metric reporters or remote log
> >> implementations that were just introduced by KIP-405. These manifest in
> >> jar
> >> files that must be put on the classpath of Kafka while certain
> classnames
> >> have to be configured. How do you plan to implement this, how do we
> >> allow users to configure such things?
> >>
> >> Thanks,
> >> Viktor
> >>
> >>
> >>
> >>
> >> On Thu, Sep 14, 2023 at 4:59 PM Kenneth Eversole
> >>  wrote:
> >>
> >> > Hello,
> >> >
> >> > I think this would be a wonderful improvement to the ecosystem. While
> >> > Viktor is correct that most Docker pipelines eventually lead to a
> >> > kubernetes deployment, that should not stop us from creating an
> >> > Official Docker Image. Creating a Docker image would allow us to
> ensure
> >> a
> >> > level of quality and support for people who want to deploy Kafka as a
> >> > container on baremetal machines, it could allow us to create
> >> > a sandbox/developer environment for new contributors and developers to
> >> test
> >> > and have a single agreed upon environment that kafka works in for
> future
> >> > KIPs and would most likely spawn more contributions from people
> wanting
> >> to
> >> > optimize kafka for k8s.
> 

Jenkins build is unstable: Kafka » Kafka Branch Builder » 3.5 #80

[Apache Jenkins Server]

See 

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #2311

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-974 Docker Image for GraalVM based Native Kafka Broker

[Manikumar]

Hi,

> For the native AK docker image, we are considering '*kafka-local*' as it
clearly signifies that this image is intended exclusively for local

I am not sure, if there is any naming pattern for graalvm based images. Can
we include "graalvm" to the image name like "kafka-graalvm-native".
This will clearly indicate this is graalvm based image.


Thanks. Regards




On Wed, Oct 18, 2023 at 9:26 PM Krishna Agarwal <
krishna0608agar...@gmail.com> wrote:

> Hi Federico,
> Thanks for the feedback and apologies for the delay.
>
> I've included a section in the KIP on the release process. I would greatly
> appreciate your insights after reviewing it.
>
> Regards,
> Krishna
>
> On Fri, Sep 8, 2023 at 3:08 PM Federico Valeri 
> wrote:
>
> > Hi Krishna, thanks for opening this discussion.
> >
> > I see you created two separate KIPs (974 and 975), but there are some
> > common points (build system and test plan).
> >
> > Currently, the Docker image used for system tests is only supported in
> > that limited scope, so the maintenance burden is minimal. Providing
> > official Kafka images would be much more complicated. Have you
> > considered how the image rebuild process would work in case a high
> > severity CVE comes out for a non Kafka image dependency? In that case,
> > there will be no Kafka release.
> >
> > Br
> > Fede
> >
> > On Fri, Sep 8, 2023 at 9:17 AM Krishna Agarwal
> >  wrote:
> > >
> > > Hi,
> > > I want to submit a KIP to deliver an experimental Apache Kafka docker
> > image.
> > > The proposed docker image can launch brokers with sub-second startup
> time
> > > and minimal memory footprint by leveraging a GraalVM based native Kafka
> > > binary.
> > >
> > > KIP-974: Docker Image for GraalVM based Native Kafka Broker
> > > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-974%3A+Docker+Image+for+GraalVM+based+Native+Kafka+Broker
> > >
> > >
> > > Regards,
> > > Krishna
> >
>

Re: [DISCUSS] KIP-988 Streams Standby Task Update Listener

[Bruno Cadonna]

Hi,

Matthias is correct that the end offsets are stored somewhere in the 
metadata of the consumer. More precisely, they are stored in the 
`TopicPartitionState`. However, I could not find public API on the 
consumer other than currentLag() that uses the stored end offsets. If I 
understand the code correctly, method endOffSets() always triggers a 
remote call.


I am a bit concerned about doing remote calls every commit.interval.ms 
(by default 200ms under EOS). At the moment the remote calls are only 
issued if an optimization for KTables is turned on where changelog 
topics are replaced with the input topic of the KTable. The current 
remote calls retrieve all committed offsets of the group at once. If I 
understand correctly, that is one single remote call. Remote calls for 
getting end offsets of changelog topics -- as I understand you are 
planning to issue -- will probably result in multiple remote calls to 
multiple leaders of the changelog topic partitions.


Please correct me if I misunderstood anything of the above.

If my understanding is correct, I propose to modify the consumer in such 
a way to get the end offset from the locally stored metadata whenever 
possible as part of the implementation of this KIP. I do not know what 
the implications are of such a change of the consumer and if a KIP is 
needed for it. Maybe, endOffsets() guarantees to return the freshest end 
offsets possible, which would not be satisfied with the modification.


Regarding the naming, I do not completely agree with Matthias. While the 
pattern might be consistent with onBatchUpdated, what is the meaning of 
onBatchUpdated? Is the batch updated? The names onBatchLoaded or 
onBatchWritten or onBatchAdded are more clear IMO.
With "restore" the pattern works better. If I restore a batch of records 
in a state, the records are not there although they should be there and 
I add them. If I update a batch of records in a state. This sounds like 
the batch of records is in the state and I modify the existing records 
within the state. That is clearly not the meaning of the event for which 
the listener should be called.


Best,
Bruno



On 10/19/23 2:12 AM, Matthias J. Sax wrote:

Thanks for the KIP. Seems I am almost late to the party.

About naming (fun, fun, fun): I like the current proposal overall, 
except `onBachLoaded`, but would prefer `onBatchUpdated`. It better 
aligns to everything else:


  - it's an update-listener, not loaded-listener
  - `StateRestoreListener` has `onRestoreStart`, `onRestoreEnd`, 
`onRestoreSuspended, and `onBachRestored` (it's very consistent
  - `StandbyUpdateListener` should have `onUpdateStart`, 
`onUpdateSuspended` and `onBatchUpdated`  to be equally consistent 
(using "loaded" breaks the pattern)



About the end-offset question: I am relatively sure that the consumer 
gets the latest end-offset as attached metadata in every fetch response. 
(We exploit this behavior to track end-offsets for input topic with 
regard to `max.task.idle.ms` without overhead -- it was also a concern 
when we did the corresponding KIP how we could track lag with no overhead).


Thus, I believe we would "just" need to modify the code accordingly to 
get this information from the restore-consumer 
(`restorConsumer.endOffsets(...)`; should be served w/o RPC but from 
internal metadata cache) for free, and pass into the listener.


Please double check / verify this claim and keep me honest about it.


-Matthias

On 10/17/23 6:38 AM, Eduwer Camacaro wrote:

Hi Bruno,

Thanks for your observation; surely it will require a network call using
the admin client in order to know this "endOffset" and that will have an
impact on performance. We can either find a solution that has a low 
impact
on performance or ideally zero impact; unfortunately, I don't see a 
way to

have zero impact on performance. However, we can leverage the existing
#maybeUpdateLimitOffsetsForStandbyChangelogs method, which uses the admin
client to ask for these "endOffset"s. As far I can understand, this 
update

is done periodically using the "commit.interval.ms" configuration. I
believe this option will force us to invoke StandbyUpdateLister once this
interval is reached.

On Mon, Oct 16, 2023 at 8:52 AM Bruno Cadonna  wrote:


Thanks for the KIP, Colt and Eduwer,

Are you sure there is also not a significant performance impact for
passing into the callback `currentEndOffset`?

I am asking because the comment here:

https://github.com/apache/kafka/blob/c32d2338a7e0079e539b74eb16f0095380a1ce85/streams/src/main/java/org/apache/kafka/streams/processor/internals/StoreChangelogReader.java#L129

says that the end-offset is only updated once for standby tasks whose
changelog topic is not piggy-backed on input topics. I could also not
find the update of end-offset for those standbys.


Best,
Bruno

On 10/16/23 10:55 AM, Lucas Brutschy wrote:

Hi all,

it's a nice improvement! I don't have anything to add on top of the
previous comments, just came 

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

[Vedarth Sharma]

Hi Manikumar,

Thanks for the feedback!

1. We propose the addition of a new directory named "docker" at the root of
the repository, where all Docker-related code will be stored. A detailed
directory structure has been added in the KIP.
2. We request the creation of an Apache Kafka repository (apache/kafka) on
DockerHub, to be administered under the The Apache Software Foundation
. The PMC members should have the
necessary permissions for pushing updates to the docker repo.

Thanks and regards,
Vedarth


On Fri, Oct 20, 2023 at 2:44 PM Manikumar  wrote:

> Hi Krishna, Vedarth,
>
> Thanks for the KIP.
>
> 1. Can we add directory structure of Docker Image related files in Kafka
> repo.
>
> 2. > Steps for the Docker image release will be included in the Release
> Process doc of Apache Kafka
>
> Can we list down the requirements (repos, accounts) for releasing images to
> docker hub. I am mainly asking because PMC needs to request docker hub
> access/repos.
> I can help in getting required repos/accounts.
> https://infra.apache.org/docker-hub-policy.html
>
>
> Thanks,
> Manikumar
>
> On Thu, Oct 19, 2023 at 8:22 PM Krishna Agarwal <
> krishna0608agar...@gmail.com> wrote:
>
> > Hi Viktor,
> >
> > I've noticed there are two types of custom jar configurations:
> >
> >1. *Type 1*: In this case, only the class name is required(e.g
> > *authorizer.class.name
> >**)* This can be configured by the
> >following steps:
> >   - Mount the jar in the container.
> >   - Configure the *CLASSPATH* environment variable (used by
> >   *kafka-run-class.sh*) by providing the mounted path to it. This can
> >   be passed as an environment variable to the docker container.
> >2. *Type 2*: Here, in addition to the class name, classpath can also
> be
> >configured (eg *remote.log.metadata.manager.class.name
> > *and
> >*remote.log.metadata.manager.class.path*). This can be configured by
> the
> >following steps:
> >   - Mount the jar in the container.
> >   - Configure the respective *class.path* property.
> >
> > Regards,
> > Krishna
> >
> > On Mon, Sep 25, 2023 at 11:41 PM Krishna Agarwal <
> > krishna0608agar...@gmail.com> wrote:
> >
> > > Hi Viktor,
> > > Thanks for the questions.
> > >
> > >1. While the docker image outlined in KIP-975 is designed for
> > >production environments, it is equally suitable for development and
> > testing
> > >purposes. We will furnish the docker image, allowing users the
> > flexibility
> > >to employ it according to their specific needs.
> > >2. The configs will be injected into the docker container through
> > >environment variables. These environment variables will have a
> prefix
> > >allowing for efficient parsing to extract the relevant
> > properties.(Will add
> > >this implementation in the KIP as well once we converge on this.)
> > >3. Regarding this question, I'll conduct a test on my end after
> > >gaining a better understanding, and then provide you with a
> response.
> > >
> > > Regards,
> > > Krishna
> > >
> > >
> > > On Tue, Sep 19, 2023 at 3:42 PM Viktor Somogyi-Vass
> > >  wrote:
> > >
> > >> Hi Ismael,
> > >>
> > >> I'm not trying to advocate against the docker image, I just pointed
> out
> > >> that the current scoping of the KIP may be a bit too generic and
> thought
> > >> that KIP-974 and KIP-975 were aiming for mostly the same thing and can
> > be
> > >> discussed under one umbrella. Apologies if this was rooted in a
> > >> misunderstanding.
> > >>
> > >> Kirshna,
> > >>
> > >> I think we need to refine the KIP a bit more. I think there are some
> > >> interfaces that we need to include in the KIP as Kafka has plugins in
> > >> certain cases where users are expected to provide implementation and I
> > >> think it's worth discussing this in the KIP as they're kind of
> > interfaces
> > >> for users. Here are my questions in order:
> > >> 1. In what environments do you want the image to be used? As I
> > understand
> > >> it would replace the current testing image and serve as a basis for
> > >> development, but would it aim at production use cases too
> > (docker-compose,
> > >> Kubernetes, etc.)?
> > >> 2. How do you plan to forward configs to the broker? Do we expect a
> > >> populated server.properties file placed in a certain location or
> should
> > >> the
> > >> docker image create this file based on some input (like env vars)?
> > >> 3. Certain parts can be pluggable, like metric reporters or remote log
> > >> implementations that were just introduced by KIP-405. These manifest
> in
> > >> jar
> > >> files that must be put on the classpath of Kafka while certain
> > classnames
> > >> have to be configured. How do you plan to implement this, how do we
> > >> allow users to configure such things?
> > >>
> > >> Thanks,
> > >> Viktor
> > >>
> > >>
> > >>
> > >>
> > >> On 

Re: [DISCUSS] KIP-987: Connect Static Assignments

[Mickael Maison]

Hi Greg,

Thanks for the reply.

I still find the proposed mechanism limited and I'm not sure it really
addressed the pain points I've experienced with Connect.
As you said different tasks from a connector may have different
workload. Connectors may also change the assignment of tasks at
runtime so for example it task-2 is really busy (because it's assigned
a partition with high throughput), this may not be true in 10 minutes
as this partition is now assigned to task-1. So having to put which
tasks can run on each worker does not really help in this case.

I think the "hints" where to place a connector/tasks should come from
the connector configuration as it's the engineers building a pipeline
that knows best the requirements (in terms of isolation, resources) of
their workload. This is basically point 3) in my initial email. The
mechanism you propose puts this burden on the cluster administrators
who may well not know the workloads and also have to guess/know in
advance to properly configure workers.

I've not looked into the feasibility but I wonder if a simplified
taint/selector approach could give us enough flexibility to make
Connect behave better in containerized environments. I understand it's
an alternative you rejected but I think could have some benefits. Here
is my green field thinking:
Add 2 new fields in the connector config: placement and tags.
Placement defines the degree of isolation a task requires, it accept 3
values: any (can be placed anywhere like today, the default),
colocated (can run on a worker with other tasks from this connector),
isolated (requires a dedicated worker). I think these degrees of
isolation should cover most use cases. Tags accepts a collections of
key=value pair. These can have arbitrary values and are meant to mean
something to the management system (for example Strimzi). The accepted
values could be configured on the workers by the administrators as
they also operate the management system.

When a connector is created, the runtime tries to place tasks on the
available brokers by matching the placement and tags. If no suitable
workers are found, the tasks stay in unassigned state and the runtime
waits for the management system to create the necessary workers.

We could even envisage to start with only the placement field as in my
opinion this is what brings the most value to users.

Thanks,
Mickael

On Wed, Oct 18, 2023 at 8:12 PM Greg Harris
 wrote:
>
> Hey Sagar,
>
> Thanks for the questions. I hope you find the answers satisfying:
>
> 1. This is detailed in the KIP two sentences earlier: "If the
> connect.protocol is set to static, each worker will send it's
> static.connectors and static.tasks to the coordinator during
> rebalances."
>
> 2. If you have a static worker and a wildcard worker, the static
> worker will be assigned the work preferentially. If the static worker
> goes offline, the wildcard worker will be used as a backup.
>
> 3. I don't think that new Connect users will make use of this feature,
> but I've added that clarification.
>
> 4. Users can implement the strategy you're describing by leaving the
> static.connectors field unset. I think that Connect should include
> static.connectors for users that do want to control the placement of
> connectors.
>
> 5. Yes. Arbitrary here just means that the assignment is not
> influenced by the static assignment.
>
> 6. Yes. There are no guardrails that ensure that the balance of the
> static assignments is better than the builtin algorithm because we
> have no method to compare them.
>
> 7. If the whole cluster uses static assignments with each job only
> specified on one worker, the assignments are completely sticky. If a
> worker goes offline, those tasks will be offline until that worker
> comes back.
> If there are multiple workers for a single job, that is specified as
> "arbitrary". We could choose to wait for the delay to elapse or
> immediately reassign it, the KIP as written could be implemented by
> either.
> If the assignment would land on a wildcard worker, that should use
> cooperative rules, so we would need to respect the rebalance delay.
>
> Thanks!
> Greg
>
> On Wed, Oct 18, 2023 at 6:21 AM Sagar  wrote:
> >
> > Hi Greg,
> >
> > Thanks for the KIP. I have a few questions/comments:
> >
> > 1) You mentioned that during a rebalance if all the members of the cluster
> > support the static protocol, then it would use the steps outlined in the
> > Proposed Section to do the assignments. In those steps, the leader
> > identifies the static/wildcard jobs. It is not clear to me how the leader
> > makes that distinction? Are we going to enhance the embedded protocol to
> > also write the static jobs that the worker owns as part of it's assignment?
> > As of today, the workers just write only the owned/revoked connectors/tasks
> > in case of incremental and above and only owned connectors/tasks in case of
> > eager.
> >
> > 2) Could you also elaborate this statement a bit:
> >
> > > A cluster with 

[jira] [Created] (KAFKA-15659) Flaky test RestoreIntegrationTest.shouldInvokeUserDefinedGlobalStateRestoreListener

[Divij Vaidya (Jira)]

Divij Vaidya created KAFKA-15659:


 Summary: Flaky test 
RestoreIntegrationTest.shouldInvokeUserDefinedGlobalStateRestoreListener 
 Key: KAFKA-15659
 URL: https://issues.apache.org/jira/browse/KAFKA-15659
 Project: Kafka
  Issue Type: Test
Reporter: Divij Vaidya
 Attachments: Screenshot 2023-10-20 at 13.19.20.png

The test added in the PR [https://github.com/apache/kafka/pull/14519] 
{{shouldInvokeUserDefinedGlobalStateRestoreListener}} has been flaky since it 
was added. You can find the flaky build on trunk using the link 
[https://ge.apache.org/scans/tests?search.relativeStartTime=P28D=kafka=trunk=Europe%2FBerlin=org.apache.kafka.streams.integration.RestoreIntegrationTest=shouldInvokeUserDefinedGlobalStateRestoreListener()]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-992 Proposal to introduce IQv2 Query Types: TimestampedKeyQuery and TimestampedRangeQuery

[Alieh Saeedi]

Hey Hanyu,

Thanks for the KIP. It seems good to me.
Just one point: AFAIK, we are going to remove "get" from the name of all
getter methods.

Cheers,
Alieh

On Thu, Oct 19, 2023 at 5:44 PM Hanyu (Peter) Zheng
 wrote:

> Hello everyone,
>
> I would like to start the discussion for KIP-992: Proposal to introduce
> IQv2 Query Types: TimestampedKeyQuery and TimestampedRangeQuery
>
> The KIP can be found here:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-992%3A+Proposal+to+introduce+IQv2+Query+Types%3A+TimestampedKeyQuery+and+TimestampedRangeQuery
>
> Any suggestions are more than welcome.
>
> Many thanks,
> Hanyu
>
> On Thu, Oct 19, 2023 at 8:17 AM Hanyu (Peter) Zheng 
> wrote:
>
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-992%3A+Proposal+to+introduce+IQv2+Query+Types%3A+TimestampedKeyQuery+and+TimestampedRangeQuery
> >
> > --
> >
> > [image: Confluent] 
> > Hanyu (Peter) Zheng he/him/his
> > Software Engineer Intern
> > +1 (213) 431-7193 <+1+(213)+431-7193>
> > Follow us: [image: Blog]
> > <
> https://www.confluent.io/blog?utm_source=footer_medium=email_campaign=ch.email-signature_type.community_content.blog
> >[image:
> > Twitter] [image: LinkedIn]
> > [image: Slack]
> > [image: YouTube]
> > 
> >
> > [image: Try Confluent Cloud for Free]
> > <
> https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound_source=gmail_medium=organic
> >
> >
>
>
> --
>
> [image: Confluent] 
> Hanyu (Peter) Zheng he/him/his
> Software Engineer Intern
> +1 (213) 431-7193 <+1+(213)+431-7193>
> Follow us: [image: Blog]
> <
> https://www.confluent.io/blog?utm_source=footer_medium=email_campaign=ch.email-signature_type.community_content.blog
> >[image:
> Twitter] [image: LinkedIn]
> [image: Slack]
> [image: YouTube]
> 
>
> [image: Try Confluent Cloud for Free]
> <
> https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound_source=gmail_medium=organic
> >
>

Re: [VOTE] KIP-982: Enhance Custom KafkaPrincipalBuilder to Access SslPrincipalMapper and KerberosShortNamer

[Doğuşcan Namal]

Hi Raghu,

Thanks for the short KIP.

+1(non-binding)

On Thu, 19 Oct 2023 at 23:56, Raghu B  wrote:

> Hi everyone,
>
> I would like to start a vote on KIP-982, which proposed enhancements to
> the Custom KafkaPrincipalBuilder to allow access to SslPrincipalMapper and
> KerberosShortNamer.
>
> This KIP
> 
> aims to improve the flexibility and usability of custom
> KafkaPrincipalBuilder implementations by enabling support for Mapping Rules
> and enhancing the overall security configuration of Kafka brokers.
>
> Thank you for your participation!
>
> Sincerely,
> Raghu
>

[jira] [Created] (KAFKA-15658) Zookeeper 3.6.3 jar | CVE-2023-44981

[masood (Jira)]

masood created KAFKA-15658:
--

 Summary: Zookeeper 3.6.3 jar | CVE-2023-44981 
 Key: KAFKA-15658
 URL: https://issues.apache.org/jira/browse/KAFKA-15658
 Project: Kafka
  Issue Type: Bug
Reporter: masood


The [CVE-2023-44981|https://www.mend.io/vulnerability-database/CVE-2023-44981]  
vulnerability has been reported in the zookeeper.jar. 

It's worth noting that the latest version of Kafka has a dependency on version 
3.8.2 of Zookeeper, which is also impacted by this vulnerability. 

[https://mvnrepository.com/artifact/org.apache.zookeeper/zookeeper/3.8.2|https://mvnrepository.com/artifact/org.apache.zookeeper/zookeeper/3.8.2.]

could you please verify its impact on the Kafka.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

[Mickael Maison]

Hi Krishna,

Overall I'm supportive of having an official docker image.
I have a few questions:
- Can you clarify the process of selecting the Java version? Is the
proposal to only pick LTS versions? or to pick the highest version
supported by Kafka?
- Once a new Kafka version is released, what happens to the image
containing the previous release? Do we expect to still update it in
case of CVEs? If so for how long?
- How will we get notified that the base image has a CVE?
- Rather than having scripts PMC members have to run from their
machines, would it e possible to have a Jenkins job or GitHub action?

Thanks,
Mickael



On Fri, Oct 20, 2023 at 12:51 PM Vedarth Sharma
 wrote:
>
> Hi Manikumar,
>
> Thanks for the feedback!
>
> 1. We propose the addition of a new directory named "docker" at the root of
> the repository, where all Docker-related code will be stored. A detailed
> directory structure has been added in the KIP.
> 2. We request the creation of an Apache Kafka repository (apache/kafka) on
> DockerHub, to be administered under the The Apache Software Foundation
> . The PMC members should have the
> necessary permissions for pushing updates to the docker repo.
>
> Thanks and regards,
> Vedarth
>
>
> On Fri, Oct 20, 2023 at 2:44 PM Manikumar  wrote:
>
> > Hi Krishna, Vedarth,
> >
> > Thanks for the KIP.
> >
> > 1. Can we add directory structure of Docker Image related files in Kafka
> > repo.
> >
> > 2. > Steps for the Docker image release will be included in the Release
> > Process doc of Apache Kafka
> >
> > Can we list down the requirements (repos, accounts) for releasing images to
> > docker hub. I am mainly asking because PMC needs to request docker hub
> > access/repos.
> > I can help in getting required repos/accounts.
> > https://infra.apache.org/docker-hub-policy.html
> >
> >
> > Thanks,
> > Manikumar
> >
> > On Thu, Oct 19, 2023 at 8:22 PM Krishna Agarwal <
> > krishna0608agar...@gmail.com> wrote:
> >
> > > Hi Viktor,
> > >
> > > I've noticed there are two types of custom jar configurations:
> > >
> > >1. *Type 1*: In this case, only the class name is required(e.g
> > > *authorizer.class.name
> > >**)* This can be configured by the
> > >following steps:
> > >   - Mount the jar in the container.
> > >   - Configure the *CLASSPATH* environment variable (used by
> > >   *kafka-run-class.sh*) by providing the mounted path to it. This can
> > >   be passed as an environment variable to the docker container.
> > >2. *Type 2*: Here, in addition to the class name, classpath can also
> > be
> > >configured (eg *remote.log.metadata.manager.class.name
> > > *and
> > >*remote.log.metadata.manager.class.path*). This can be configured by
> > the
> > >following steps:
> > >   - Mount the jar in the container.
> > >   - Configure the respective *class.path* property.
> > >
> > > Regards,
> > > Krishna
> > >
> > > On Mon, Sep 25, 2023 at 11:41 PM Krishna Agarwal <
> > > krishna0608agar...@gmail.com> wrote:
> > >
> > > > Hi Viktor,
> > > > Thanks for the questions.
> > > >
> > > >1. While the docker image outlined in KIP-975 is designed for
> > > >production environments, it is equally suitable for development and
> > > testing
> > > >purposes. We will furnish the docker image, allowing users the
> > > flexibility
> > > >to employ it according to their specific needs.
> > > >2. The configs will be injected into the docker container through
> > > >environment variables. These environment variables will have a
> > prefix
> > > >allowing for efficient parsing to extract the relevant
> > > properties.(Will add
> > > >this implementation in the KIP as well once we converge on this.)
> > > >3. Regarding this question, I'll conduct a test on my end after
> > > >gaining a better understanding, and then provide you with a
> > response.
> > > >
> > > > Regards,
> > > > Krishna
> > > >
> > > >
> > > > On Tue, Sep 19, 2023 at 3:42 PM Viktor Somogyi-Vass
> > > >  wrote:
> > > >
> > > >> Hi Ismael,
> > > >>
> > > >> I'm not trying to advocate against the docker image, I just pointed
> > out
> > > >> that the current scoping of the KIP may be a bit too generic and
> > thought
> > > >> that KIP-974 and KIP-975 were aiming for mostly the same thing and can
> > > be
> > > >> discussed under one umbrella. Apologies if this was rooted in a
> > > >> misunderstanding.
> > > >>
> > > >> Kirshna,
> > > >>
> > > >> I think we need to refine the KIP a bit more. I think there are some
> > > >> interfaces that we need to include in the KIP as Kafka has plugins in
> > > >> certain cases where users are expected to provide implementation and I
> > > >> think it's worth discussing this in the KIP as they're kind of
> > > interfaces
> > > >> for users. Here are my questions in order:
> > > >> 1. In what environments do you want the 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.5 #81

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-15660) File-based Tiered Storage should delete folders upon topic deletion

[Christo Lolov (Jira)]

Christo Lolov created KAFKA-15660:
-

 Summary: File-based Tiered Storage should delete folders upon 
topic deletion
 Key: KAFKA-15660
 URL: https://issues.apache.org/jira/browse/KAFKA-15660
 Project: Kafka
  Issue Type: Bug
Affects Versions: 3.6.0
Reporter: Christo Lolov


We have added a quick-start guide for Tiered Storage as part of Apache Kafka 
3.6 - [https://kafka.apache.org/documentation/#tiered_storage_config_ex.]

When interacting with it, however, it appears that when topics are deleted 
while remote segments and their indecies are deleted the folders are not:
{code:java}
> ls /tmp/kafka-remote-storage/kafka-tiered-storage 
A-0-ApBdPOE1SOOw-Ie8RQLuAA  B-0-2omLZKw1Tiu2-EUKsIzj9Q  
C-0-FXdccGWXQJCj-RQynsOK3Q  D-0-vqfdYtYLSlWEyXp6cwwmpg

> ls /tmp/kafka-remote-storage/kafka-tiered-storage/A-0-ApBdPOE1SOOw-Ie8RQLuAA

{code}
I think that the file-based implementation shipping with Kafka should delete 
the folders as well.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Apache Kafka 3.5.2 release

[Luke Chen]

Hi Matthias,

I'm planning to have the 1st RC next week.
Does that work for you?
Should I defer one more week?

Thanks.
Luke

On Wed, Oct 18, 2023 at 1:52 AM Matthias J. Sax  wrote:

> Thanks -- there is a few fixed for Kafka Streams we are considering to
> cherry-pick to get into 3.5.2 release -- what timeline do you target for
> the release?
>
>
> -Matthias
>
> On 10/17/23 8:47 AM, Divij Vaidya wrote:
> > Thank you for volunteering Luke.
> >
> > --
> > Divij Vaidya
> >
> >
> >
> > On Tue, Oct 17, 2023 at 3:26 PM Bill Bejeck  wrote:
> >
> >> Thanks for driving the release, Luke.
> >>
> >> +1
> >> -Bill
> >>
> >> On Tue, Oct 17, 2023 at 5:05 AM Satish Duggana <
> satish.dugg...@gmail.com>
> >> wrote:
> >>
> >>> Thanks Luke for volunteering for 3.5.2 release.
> >>>
> >>> On Tue, 17 Oct 2023 at 11:58, Josep Prat 
> >>> wrote:
> 
>  Hi Luke,
> 
>  Thanks for taking this one!
> 
>  Best,
> 
>  On Tue, Oct 17, 2023 at 8:12 AM Luke Chen  wrote:
> 
> > Hi all,
> >
> > I'd like to volunteer as release manager for the Apache Kafka 3.5.2,
> >> to
> > have an important bug/vulnerability fix release for 3.5.1.
> >
> > If there are no objections, I'll start building a release plan in
> >>> thewiki
> > in the next couple of weeks.
> >
> > Thanks,
> > Luke
> >
> 
> 
>  --
>  [image: Aiven] 
> 
>  *Josep Prat*
>  Open Source Engineering Director, *Aiven*
>  josep.p...@aiven.io   |   +491715557497
>  aiven.io    |   <
> >>> https://www.facebook.com/aivencloud>
>    <
> >>> https://twitter.com/aiven_io>
>  *Aiven Deutschland GmbH*
>  Alexanderufer 3-7, 10117 Berlin
>  Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>  Amtsgericht Charlottenburg, HRB 209739 B
> >>>
> >>
> >
>

[jira] [Resolved] (KAFKA-15428) Cluster-wide dynamic log adjustments for Connect

[Chris Egerton (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Egerton resolved KAFKA-15428.
---
Fix Version/s: 3.7.0
   Resolution: Done

> Cluster-wide dynamic log adjustments for Connect
> 
>
> Key: KAFKA-15428
> URL: https://issues.apache.org/jira/browse/KAFKA-15428
> Project: Kafka
>  Issue Type: New Feature
>  Components: KafkaConnect
>Reporter: Chris Egerton
>Assignee: Chris Egerton
>Priority: Major
>  Labels: kip
> Fix For: 3.7.0
>
>
> [KIP-495|https://cwiki.apache.org/confluence/display/KAFKA/KIP-495%3A+Dynamically+Adjust+Log+Levels+in+Connect]
>  added REST APIs to view and adjust the logging levels of Kafka Connect 
> workers at runtime. This has been tremendously valuable (thank you 
> [~wicknicks]!), but one frequently-observed area for improvement is that the 
> API requires a REST request to be issued to each to-be-adjusted worker.
> If possible, we should add support for adjusting the logging level of all 
> workers in a cluster with a single REST request.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[VOTE] KIP-967: Support custom SSL configuration for Kafka Connect RestServer

[Taras Ledkov]

Hi Kafka Team.

II'd like to call a vote on KIP-967: Support custom SSL configuration for Kafka 
Connect RestServer [1].
Discussion thread [2] was started more then 2 month ago and there was not any 
negative or critical comments.

[1]. 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-967%3A+Support+custom+SSL+configuration+for+Kafka+Connect+RestServer
[2]. https://lists.apache.org/thread/w0vmbf1yzgjo7hkzyyzjjnb509x6s9qq

--
With best regards,
Taras Ledkov

RE: [DISCUSS] KIP-905: Broker interceptors

[Ivan Yurchenko]

Hi David,

I wonder if you considered interceptors that work not only on produce request, 
but on any request type the broker is receiving (apart from maybe internal 
request types). For example, being able to rewrite topic names in all the 
request types is essential to create a virtual clusters solutions (many virtual 
clusters, invisible for each other, inside one physical).

Best,
Ivan


On 2023/02/09 19:28:21 David Mariassy wrote:
> Hi everyone,
> 
> I'd like to get a discussion going for KIP-905
> ,
> which proposes the addition of broker interceptors to the stack.
> 
> The KIP contains the motivation, and lists the new public interfaces that
> this change would entail. Since my company had its quarterly hack days this
> week, I also took the liberty to throw together a first prototype of the
> proposed new feature here: https://github.com/apache/kafka/pull/13224.
> 
> Looking forward to the group's feedback!
> 
> Thanks,
> David
> 

Re: Re: [DISCUSS] Apache Kafka 3.5.2 release

[Luke Chen]

Hi Ryan,

OK, I've backported it to 3.5 branch.
I'll be included in v3.5.2.

Thanks.
Luke

On Fri, Oct 20, 2023 at 7:43 AM Ryan Leslie (BLP/ NEW YORK (REMOT) <
rles...@bloomberg.net> wrote:

> Hi Luke,
>
> Hope you are well. Can you please include
> https://issues.apache.org/jira/browse/KAFKA-15106 in 3.5.2?
>
> Thanks,
>
> Ryan
>
> From: dev@kafka.apache.org At: 10/17/23 05:05:24 UTC-4:00
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] Apache Kafka 3.5.2 release
>
> Thanks Luke for volunteering for 3.5.2 release.
>
> On Tue, 17 Oct 2023 at 11:58, Josep Prat 
> wrote:
> >
> > Hi Luke,
> >
> > Thanks for taking this one!
> >
> > Best,
> >
> > On Tue, Oct 17, 2023 at 8:12 AM Luke Chen  wrote:
> >
> > > Hi all,
> > >
> > > I'd like to volunteer as release manager for the Apache Kafka 3.5.2, to
> > > have an important bug/vulnerability fix release for 3.5.1.
> > >
> > > If there are no objections, I'll start building a release plan in
> thewiki
> > > in the next couple of weeks.
> > >
> > > Thanks,
> > > Luke
> > >
> >
> >
> > --
> > [image: Aiven] 
> >
> > *Josep Prat*
> > Open Source Engineering Director, *Aiven*
> > josep.p...@aiven.io | +491715557497
> > aiven.io  | 
> >  
> > *Aiven Deutschland GmbH*
> > Alexanderufer 3-7, 10117 Berlin
> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > Amtsgericht Charlottenburg, HRB 209739 B
>
>
>

Re: [DISCUSS] KIP-978: Allow dynamic reloading of certificates with different DN / SANs

[Jakub Scholz]

Please let me know if anyone has some more comments on this. If not, I will
start the vote next week.

Thanks & Regards
Jakub

On Wed, Sep 13, 2023 at 9:59 PM Jakub Scholz  wrote:

> Hi all,
>
> I would like to start the discussion about the KIP-978: Allow dynamic
> reloading of certificates with different DN / SANs
> .
> It proposes adding an option to disable the current validation of the DN
> and SANs when dynamically changing the keystore. Please have a look and let
> me know your thoughts ...
>
> Thanks & Regards
> Jakub
>

Re: [DISCUSS] KIP-987: Connect Static Assignments

[Hector Geraldino (BLOOMBERG/ 919 3RD A)]

Hi,

I think different algorithms might work for different workload/scenarios. I 
have some thoughts that are somewhat tangential to this KIP: it might be a good 
idea to elevate the ConnectAssignor to the category of plugin, so users can 
provide their own implementation. 

The fact that there's a public o.a.k.c.r.distributed.ConnectAssignor interface 
is brilliant (I actually wanted the same thing on the Kafka client side, alas  
https://cwiki.apache.org/confluence/display/KAFKA/KIP-795%3A+Add+public+APIs+for+AbstractCoordinator).
 I think it should play well with the future Connect's counterpart of KIP-848 
(new consumer rebalance protocol).

I don't want to hijack this thread, but will definitely raise a KIP and start a 
discussion around this idea.

From: dev@kafka.apache.org At: 10/20/23 07:21:11 UTC-4:00To:  
dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-987: Connect Static Assignments

Hi Greg,

Thanks for the reply.

I still find the proposed mechanism limited and I'm not sure it really
addressed the pain points I've experienced with Connect.
As you said different tasks from a connector may have different
workload. Connectors may also change the assignment of tasks at
runtime so for example it task-2 is really busy (because it's assigned
a partition with high throughput), this may not be true in 10 minutes
as this partition is now assigned to task-1. So having to put which
tasks can run on each worker does not really help in this case.

I think the "hints" where to place a connector/tasks should come from
the connector configuration as it's the engineers building a pipeline
that knows best the requirements (in terms of isolation, resources) of
their workload. This is basically point 3) in my initial email. The
mechanism you propose puts this burden on the cluster administrators
who may well not know the workloads and also have to guess/know in
advance to properly configure workers.

I've not looked into the feasibility but I wonder if a simplified
taint/selector approach could give us enough flexibility to make
Connect behave better in containerized environments. I understand it's
an alternative you rejected but I think could have some benefits. Here
is my green field thinking:
Add 2 new fields in the connector config: placement and tags.
Placement defines the degree of isolation a task requires, it accept 3
values: any (can be placed anywhere like today, the default),
colocated (can run on a worker with other tasks from this connector),
isolated (requires a dedicated worker). I think these degrees of
isolation should cover most use cases. Tags accepts a collections of
key=value pair. These can have arbitrary values and are meant to mean
something to the management system (for example Strimzi). The accepted
values could be configured on the workers by the administrators as
they also operate the management system.

When a connector is created, the runtime tries to place tasks on the
available brokers by matching the placement and tags. If no suitable
workers are found, the tasks stay in unassigned state and the runtime
waits for the management system to create the necessary workers.

We could even envisage to start with only the placement field as in my
opinion this is what brings the most value to users.

Thanks,
Mickael

On Wed, Oct 18, 2023 at 8:12 PM Greg Harris
 wrote:
>
> Hey Sagar,
>
> Thanks for the questions. I hope you find the answers satisfying:
>
> 1. This is detailed in the KIP two sentences earlier: "If the
> connect.protocol is set to static, each worker will send it's
> static.connectors and static.tasks to the coordinator during
> rebalances."
>
> 2. If you have a static worker and a wildcard worker, the static
> worker will be assigned the work preferentially. If the static worker
> goes offline, the wildcard worker will be used as a backup.
>
> 3. I don't think that new Connect users will make use of this feature,
> but I've added that clarification.
>
> 4. Users can implement the strategy you're describing by leaving the
> static.connectors field unset. I think that Connect should include
> static.connectors for users that do want to control the placement of
> connectors.
>
> 5. Yes. Arbitrary here just means that the assignment is not
> influenced by the static assignment.
>
> 6. Yes. There are no guardrails that ensure that the balance of the
> static assignments is better than the builtin algorithm because we
> have no method to compare them.
>
> 7. If the whole cluster uses static assignments with each job only
> specified on one worker, the assignments are completely sticky. If a
> worker goes offline, those tasks will be offline until that worker
> comes back.
> If there are multiple workers for a single job, that is specified as
> "arbitrary". We could choose to wait for the delay to elapse or
> immediately reassign it, the KIP as written could be implemented by
> either.
> If the assignment would land on a wildcard worker, that should use
> cooperative rules, so we 

[jira] [Created] (KAFKA-15661) KIP-951: Server side and protocol changes

[Crispin Bernier (Jira)]

Crispin Bernier created KAFKA-15661:
---

 Summary: KIP-951: Server side and protocol changes
 Key: KAFKA-15661
 URL: https://issues.apache.org/jira/browse/KAFKA-15661
 Project: Kafka
  Issue Type: Task
  Components: protocol
Reporter: Crispin Bernier


Server side and protocol changes for implementing KIP-951, passing back the new 
leader to the client on NOT_LEADER_OR_FOLLOWER errors for fetch and produce 
requests.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

RE: Re: [DISCUSS] KIP-905: Broker interceptors

[Ivan Yurchenko]

Hi David and Ahmed,

First, thank you David for the KIP. It would be very valuable for multiple use 
cases. Products like Conduktor Gateway [1] validate the demand and offer many 
potential use cases [2].

Now, I understand Ahmed's concerns about possible in-band interruptions, the 
are valid. However, certain use cases cannot be handled without intercepting 
the request flow to Kafka brokers (for example, the broker-side schema 
validation.) A number of open source and proprietary proxy solutions exist and 
they have their user base, for which the benefits outweigh the risks. In the 
current state, the broker itself already has injection points for custom code 
executed in the hot path of message handling, namely the Authorizer.

One benefit of pluggable interceptors is that they don't affect users who don't 
need and don't use them, so the Kafka robustness remains at the baseline. Those 
who need this functionality, can make their conscious decision. So to me it 
seems this will be positive to Kafka community and ecosystem.

Best regards,
Ivan

[1] https://docs.conduktor.io/gateway/
[2] https://marketplace.conduktor.io/

On 2023/02/10 16:41:01 David Mariassy wrote:
> Hi Ahmed,
> 
> Thanks for taking a look at the KIP, and for your insightful feedback!
> 
> I don't disagree with the sentiment that in-band interceptors could be a
> potential source of bugs in a cluster.
> 
> Having said that, I don't necessarily think that an in-band interceptor is
> significantly riskier than an out-of-band pre-processor. Let's take the
> example of platform-wide privacy scrubbing. In my opinion it doesn't really
> matter if this feature is deployed as an out-of-band stream processor app
> that consumes from all topics OR if the logic is implemented as an in-ban
> interceptor. Either way, a faulty release of the scrubber will result in
> the platform-wide disruption of data flows. Thus, I'd argue that from the
> perspective of the platform's overall health, the level of risk is very
> comparable in both cases. However in-band interceptors have a couple of
> advantages in my opinion:
> 1. They are significantly cheaper (don't require duplicating data between
> raw and sanitized topics. There are also a lot of potential savings in
> network costs)
> 2. They are easier to maintain (no need to set up additional infrastructure
> for out-of-band processing)
> 3. They can provide accurate produce responses to clients (since there is
> no downstream processing that could render a client's messages invalid
> async)
> 
> Also, in-band interceptors could be as safe or risky as their authors
> design them to be. There's nothing stopping someone from catching all
> exceptions in a `processRecord` method, and letting all unprocessed
> messages go through or sending them to a DLQ. Once the interceptor is
> fixed, those unprocessed messages could get re-ingested into Kafka to
> re-attempt pre-processing.
> 
> Thanks and happy Friday,
> David
> 
> 
> 
> 
> 
> On Fri, Feb 10, 2023 at 8:23 AM Ahmed Abdalla 
> wrote:
> 
> > Hi David,
> >
> > That's a very interesting KIP and I wanted to share my two cents. I believe
> > there's a lot of value and use cases for the ability to intercept, mutate
> > and filter Kafka's messages, however I'm not sure if trying to achieve that
> > via in-band interceptors is the best approach for this.
> >
> >- My mental model around one of Kafka's core values is the brokers'
> >focus on a single functionality (more or less): highly available and
> > fault
> >tolerant commit log. I see this in many design decisions such as
> >off-loading responsibilities to the clients (partitioner, assignor,
> >consumer groups coordination etc).
> >- And the impact of this KIP on the Kafka server would be adding another
> >moving part to their "state of the world" that they try to maintain.
> > What
> >if an interceptor goes bad? What if there're version-mismatch? etc, a
> > lot
> >of responsibilities that can be managed very efficiently out-of-band
> > IMHO.
> >- The comparison to NginX and Kubernetes is IMHO comparing apples to
> >oranges
> >   - NginX
> >  - Doesn't maintain persisted data.
> >  - It's designed as a middleware, it's an interceptor by nature.
> >   - Kubernetes
> >  - CRDs extend the API surface, they don't "augment" existing APIs.
> >  I think admission webhooks
> >  <
> > https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/
> > >
> > is
> >  Kubernetes' solution for providing interceptors.
> >  - The admission webhooks are out-of-band, and in fact they're a
> >  great example of "opening up your cluster for extensibility"
> > going wrong.
> >  Installing a misbehaving webhook can brick the whole cluster.
> >
> > As I mentioned, I see a value for users being able to intercept and
> > transform Kafka's messages. But I'm worried that having this as a 

Re: [DISCUSS] KIP-987: Connect Static Assignments

[Greg Harris]

Mickael,

Thank you for discussing that rejected alternative, I was almost going
to propose it.

> I still find the proposed mechanism limited and I'm not sure it really 
> addressed the pain points I've experienced with Connect.

I think that this KIP on its own is insufficient to solve the
operational difficulties of Connect, and an external management layer
is necessary. In this KIP i'm trying to find the minimum viable
abstraction to allow a management layer to make decisions about
placement, knowing that the abstraction may be non-ergonomic for
"direct users" without a management layer mediating.

> Connectors may also change the assignment of tasks at runtime so for example 
> it task-2 is really busy (because it's assigned a partition with high 
> throughput), this may not be true in 10 minutes as this partition is now 
> assigned to task-1

I think this is similar to a concern (#5) that Tom raised, and a
limitation of the "task index" abstraction. I don't know if there is a
way for us to manage this sort of fine-grained dynamic utilization of
tasks. Once we start a task, it has some static resources assigned to
it (via the JVM). If you notice the resource requirements expand, it
will need to stop in order to move JVMs and change its resource
allocation, and stopping the task may cause assignments to change and
the workload to be distributed elsewhere.

> I think the "hints" where to place a connector/tasks should come from the 
> connector configuration as it's the engineers building a pipeline that knows 
> best the requirements (in terms of isolation, resources) of their workload. 
> This is basically point 3) in my initial email. The mechanism you propose 
> puts this burden on the cluster administrators who may well not know the 
> workloads and also have to guess/know in advance to properly configure 
> workers.

In the Strimzi proposal I imagined that the resource limits would be
chosen by the users creating the connector configurations, and the
fact that they were beside the connector configuration rather than
inside of it didn't change the ownership. I completely agree that the
users defining configurations should also be responsible for setting
resource limits.

> I've not looked into the feasibility but I wonder if a simplified 
> taint/selector approach could give us enough flexibility to make Connect 
> behave better in containerized environments.

Yes, a taint/selector system would be more helpful for Connect
clusters without an external management layer, but significantly more
complex. Do you have a strategy for specifying unique selectors for an
arbitrary size set of tasks?
A connector config wishes to give a different selector to each task,
like `task-id=connector-id-N` where N is the index of the task. My
best solution was a special $TASK_ID placeholder that would be filled
in with the task number at runtime, but that felt like an inelegant
carve-out.

Thanks for your thoughts.
Greg

On Fri, Oct 20, 2023 at 6:25 AM Hector Geraldino (BLOOMBERG/ 919 3RD
A)  wrote:
>
> Hi,
>
> I think different algorithms might work for different workload/scenarios. I 
> have some thoughts that are somewhat tangential to this KIP: it might be a 
> good idea to elevate the ConnectAssignor to the category of plugin, so users 
> can provide their own implementation.
>
> The fact that there's a public o.a.k.c.r.distributed.ConnectAssignor 
> interface is brilliant (I actually wanted the same thing on the Kafka client 
> side, alas  
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-795%3A+Add+public+APIs+for+AbstractCoordinator).
>  I think it should play well with the future Connect's counterpart of KIP-848 
> (new consumer rebalance protocol).
>
> I don't want to hijack this thread, but will definitely raise a KIP and start 
> a discussion around this idea.
>
> From: dev@kafka.apache.org At: 10/20/23 07:21:11 UTC-4:00To:  
> dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-987: Connect Static Assignments
>
> Hi Greg,
>
> Thanks for the reply.
>
> I still find the proposed mechanism limited and I'm not sure it really
> addressed the pain points I've experienced with Connect.
> As you said different tasks from a connector may have different
> workload. Connectors may also change the assignment of tasks at
> runtime so for example it task-2 is really busy (because it's assigned
> a partition with high throughput), this may not be true in 10 minutes
> as this partition is now assigned to task-1. So having to put which
> tasks can run on each worker does not really help in this case.
>
> I think the "hints" where to place a connector/tasks should come from
> the connector configuration as it's the engineers building a pipeline
> that knows best the requirements (in terms of isolation, resources) of
> their workload. This is basically point 3) in my initial email. The
> mechanism you propose puts this burden on the cluster administrators
> who may well not know the workloads and also have to 

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2313

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 420448 lines...]
Gradle Test Run :core:test > Gradle Test Executor 88 > ZkAclMigrationClientTest 
> testAclsMigrateAndDualWrite() STARTED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZkAclMigrationClientTest 
> testAclsMigrateAndDualWrite() PASSED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZkMigrationClientTest > 
testUpdateExistingPartitions() STARTED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZkMigrationClientTest > 
testUpdateExistingPartitions() PASSED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZkMigrationClientTest > 
testEmptyWrite() STARTED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZkMigrationClientTest > 
testEmptyWrite() PASSED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZkMigrationClientTest > 
testReadMigrateAndWriteProducerId() STARTED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZkMigrationClientTest > 
testReadMigrateAndWriteProducerId() PASSED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZkMigrationClientTest > 
testExistingKRaftControllerClaim() STARTED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZkMigrationClientTest > 
testExistingKRaftControllerClaim() PASSED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZkMigrationClientTest > 
testMigrateTopicConfigs() STARTED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZkMigrationClientTest > 
testMigrateTopicConfigs() PASSED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZkMigrationClientTest > 
testNonIncreasingKRaftEpoch() STARTED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZkMigrationClientTest > 
testNonIncreasingKRaftEpoch() PASSED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZkMigrationClientTest > 
testMigrateEmptyZk() STARTED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZkMigrationClientTest > 
testMigrateEmptyZk() PASSED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZkMigrationClientTest > 
testTopicAndBrokerConfigsMigrationWithSnapshots() STARTED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZkMigrationClientTest > 
testTopicAndBrokerConfigsMigrationWithSnapshots() PASSED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZkMigrationClientTest > 
testClaimAndReleaseExistingController() STARTED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZkMigrationClientTest > 
testClaimAndReleaseExistingController() PASSED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZkMigrationClientTest > 
testClaimAbsentController() STARTED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZkMigrationClientTest > 
testClaimAbsentController() PASSED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZkMigrationClientTest > 
testIdempotentCreateTopics() STARTED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZkMigrationClientTest > 
testIdempotentCreateTopics() PASSED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZkMigrationClientTest > 
testCreateNewTopic() STARTED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZkMigrationClientTest > 
testCreateNewTopic() PASSED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZkMigrationClientTest > 
testUpdateExistingTopicWithNewAndChangedPartitions() STARTED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZkMigrationClientTest > 
testUpdateExistingTopicWithNewAndChangedPartitions() PASSED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZooKeeperClientTest > 
testZNodeChangeHandlerForDataChange() STARTED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZooKeeperClientTest > 
testZNodeChangeHandlerForDataChange() PASSED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZooKeeperClientTest > 
testZooKeeperSessionStateMetric() STARTED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZooKeeperClientTest > 
testZooKeeperSessionStateMetric() PASSED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZooKeeperClientTest > 
testExceptionInBeforeInitializingSession() STARTED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZooKeeperClientTest > 
testExceptionInBeforeInitializingSession() PASSED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZooKeeperClientTest > 
testGetChildrenExistingZNode() STARTED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZooKeeperClientTest > 
testGetChildrenExistingZNode() PASSED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZooKeeperClientTest > 
testConnection() STARTED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZooKeeperClientTest > 
testConnection() PASSED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZooKeeperClientTest > 
testZNodeChangeHandlerForCreation() STARTED

Gradle Test Run :core:test > Gradle Test Executor 88 > ZooKeeperClientTest > 
testZNodeChangeHandlerForCreation() PASSED

Gradle 

Re: [DISCUSS] KIP-988 Streams Standby Task Update Listener

[Matthias J. Sax]

Thanks for digging into this Bruno.

The JavaDoc on the consumer does not say anything specific about 
`endOffset` guarantees:



Get the end offsets for the given partitions. In the default {@code 
read_uncommitted} isolation level, the end
offset is the high watermark (that is, the offset of the last successfully 
replicated message plus one). For
{@code read_committed} consumers, the end offset is the last stable offset 
(LSO), which is the minimum of
the high watermark and the smallest offset of any open transaction. Finally, if 
the partition has never been
written to, the end offset is 0.


Thus, I actually believe that it would be ok to change the 
implementation and serve the answer from the `TopicPartitionState`?


Another idea would be, to use `currentLag()` in combination with 
`position()` (or the offset of the last read record) to compute the 
end-offset of the fly?



-Matthias

On 10/20/23 4:00 AM, Bruno Cadonna wrote:

Hi,

Matthias is correct that the end offsets are stored somewhere in the 
metadata of the consumer. More precisely, they are stored in the 
`TopicPartitionState`. However, I could not find public API on the 
consumer other than currentLag() that uses the stored end offsets. If I 
understand the code correctly, method endOffSets() always triggers a 
remote call.


I am a bit concerned about doing remote calls every commit.interval.ms 
(by default 200ms under EOS). At the moment the remote calls are only 
issued if an optimization for KTables is turned on where changelog 
topics are replaced with the input topic of the KTable. The current 
remote calls retrieve all committed offsets of the group at once. If I 
understand correctly, that is one single remote call. Remote calls for 
getting end offsets of changelog topics -- as I understand you are 
planning to issue -- will probably result in multiple remote calls to 
multiple leaders of the changelog topic partitions.


Please correct me if I misunderstood anything of the above.

If my understanding is correct, I propose to modify the consumer in such 
a way to get the end offset from the locally stored metadata whenever 
possible as part of the implementation of this KIP. I do not know what 
the implications are of such a change of the consumer and if a KIP is 
needed for it. Maybe, endOffsets() guarantees to return the freshest end 
offsets possible, which would not be satisfied with the modification.


Regarding the naming, I do not completely agree with Matthias. While the 
pattern might be consistent with onBatchUpdated, what is the meaning of 
onBatchUpdated? Is the batch updated? The names onBatchLoaded or 
onBatchWritten or onBatchAdded are more clear IMO.
With "restore" the pattern works better. If I restore a batch of records 
in a state, the records are not there although they should be there and 
I add them. If I update a batch of records in a state. This sounds like 
the batch of records is in the state and I modify the existing records 
within the state. That is clearly not the meaning of the event for which 
the listener should be called.


Best,
Bruno



On 10/19/23 2:12 AM, Matthias J. Sax wrote:

Thanks for the KIP. Seems I am almost late to the party.

About naming (fun, fun, fun): I like the current proposal overall, 
except `onBachLoaded`, but would prefer `onBatchUpdated`. It better 
aligns to everything else:


  - it's an update-listener, not loaded-listener
  - `StateRestoreListener` has `onRestoreStart`, `onRestoreEnd`, 
`onRestoreSuspended, and `onBachRestored` (it's very consistent
  - `StandbyUpdateListener` should have `onUpdateStart`, 
`onUpdateSuspended` and `onBatchUpdated`  to be equally consistent 
(using "loaded" breaks the pattern)



About the end-offset question: I am relatively sure that the consumer 
gets the latest end-offset as attached metadata in every fetch 
response. (We exploit this behavior to track end-offsets for input 
topic with regard to `max.task.idle.ms` without overhead -- it was 
also a concern when we did the corresponding KIP how we could track 
lag with no overhead).


Thus, I believe we would "just" need to modify the code accordingly to 
get this information from the restore-consumer 
(`restorConsumer.endOffsets(...)`; should be served w/o RPC but from 
internal metadata cache) for free, and pass into the listener.


Please double check / verify this claim and keep me honest about it.


-Matthias

On 10/17/23 6:38 AM, Eduwer Camacaro wrote:

Hi Bruno,

Thanks for your observation; surely it will require a network call using
the admin client in order to know this "endOffset" and that will have an
impact on performance. We can either find a solution that has a low 
impact
on performance or ideally zero impact; unfortunately, I don't see a 
way to

have zero impact on performance. However, we can leverage the existing
#maybeUpdateLimitOffsetsForStandbyChangelogs method, which uses the 
admin
client to ask for these "endOffset"s. As far I can understand, this 
update

Re: [DISCUSS] KIP-987: Connect Static Assignments

[Greg Harris]

Hey Hector,

That's a cool idea for the ConnectAssignor plugin.

While this proposal could be viewed as an "assignor" problem that a
custom assignor could solve, it's really about providing additional
context to the assignor which isn't present currently. This lack of
context would prevent a custom assignor from solving the resource
utilization problem adequately.

Thanks!
Greg

On Fri, Oct 20, 2023 at 9:58 AM Greg Harris  wrote:
>
> Mickael,
>
> Thank you for discussing that rejected alternative, I was almost going
> to propose it.
>
> > I still find the proposed mechanism limited and I'm not sure it really 
> > addressed the pain points I've experienced with Connect.
>
> I think that this KIP on its own is insufficient to solve the
> operational difficulties of Connect, and an external management layer
> is necessary. In this KIP i'm trying to find the minimum viable
> abstraction to allow a management layer to make decisions about
> placement, knowing that the abstraction may be non-ergonomic for
> "direct users" without a management layer mediating.
>
> > Connectors may also change the assignment of tasks at runtime so for 
> > example it task-2 is really busy (because it's assigned a partition with 
> > high throughput), this may not be true in 10 minutes as this partition is 
> > now assigned to task-1
>
> I think this is similar to a concern (#5) that Tom raised, and a
> limitation of the "task index" abstraction. I don't know if there is a
> way for us to manage this sort of fine-grained dynamic utilization of
> tasks. Once we start a task, it has some static resources assigned to
> it (via the JVM). If you notice the resource requirements expand, it
> will need to stop in order to move JVMs and change its resource
> allocation, and stopping the task may cause assignments to change and
> the workload to be distributed elsewhere.
>
> > I think the "hints" where to place a connector/tasks should come from the 
> > connector configuration as it's the engineers building a pipeline that 
> > knows best the requirements (in terms of isolation, resources) of their 
> > workload. This is basically point 3) in my initial email. The mechanism you 
> > propose puts this burden on the cluster administrators who may well not 
> > know the workloads and also have to guess/know in advance to properly 
> > configure workers.
>
> In the Strimzi proposal I imagined that the resource limits would be
> chosen by the users creating the connector configurations, and the
> fact that they were beside the connector configuration rather than
> inside of it didn't change the ownership. I completely agree that the
> users defining configurations should also be responsible for setting
> resource limits.
>
> > I've not looked into the feasibility but I wonder if a simplified 
> > taint/selector approach could give us enough flexibility to make Connect 
> > behave better in containerized environments.
>
> Yes, a taint/selector system would be more helpful for Connect
> clusters without an external management layer, but significantly more
> complex. Do you have a strategy for specifying unique selectors for an
> arbitrary size set of tasks?
> A connector config wishes to give a different selector to each task,
> like `task-id=connector-id-N` where N is the index of the task. My
> best solution was a special $TASK_ID placeholder that would be filled
> in with the task number at runtime, but that felt like an inelegant
> carve-out.
>
> Thanks for your thoughts.
> Greg
>
> On Fri, Oct 20, 2023 at 6:25 AM Hector Geraldino (BLOOMBERG/ 919 3RD
> A)  wrote:
> >
> > Hi,
> >
> > I think different algorithms might work for different workload/scenarios. I 
> > have some thoughts that are somewhat tangential to this KIP: it might be a 
> > good idea to elevate the ConnectAssignor to the category of plugin, so 
> > users can provide their own implementation.
> >
> > The fact that there's a public o.a.k.c.r.distributed.ConnectAssignor 
> > interface is brilliant (I actually wanted the same thing on the Kafka 
> > client side, alas  
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-795%3A+Add+public+APIs+for+AbstractCoordinator).
> >  I think it should play well with the future Connect's counterpart of 
> > KIP-848 (new consumer rebalance protocol).
> >
> > I don't want to hijack this thread, but will definitely raise a KIP and 
> > start a discussion around this idea.
> >
> > From: dev@kafka.apache.org At: 10/20/23 07:21:11 UTC-4:00To:  
> > dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-987: Connect Static Assignments
> >
> > Hi Greg,
> >
> > Thanks for the reply.
> >
> > I still find the proposed mechanism limited and I'm not sure it really
> > addressed the pain points I've experienced with Connect.
> > As you said different tasks from a connector may have different
> > workload. Connectors may also change the assignment of tasks at
> > runtime so for example it task-2 is really busy (because it's assigned
> > a partition 

Re: [DISCUSS] Apache Kafka 3.5.2 release

[Matthias J. Sax]

Thanks for the info Luke.

We did backport all but one PR in the mean time. The missing PR is a 
RocksDB version bump. We want to consider it for 3.5.2, because it 
addresses a CVE.


Cf https://github.com/apache/kafka/pull/14216

However, RocksDB versions bumps are a little bit more tricky, and we 
would like to test this properly on 3.5 branch, what would take at least 
one week; we could do the cherry-pick on Monday and start testing.


Please let us know if such a delay for 3.5.2 is acceptable or not.

Thanks.

-Matthias


On 10/20/23 5:44 AM, Luke Chen wrote:

Hi Ryan,

OK, I've backported it to 3.5 branch.
I'll be included in v3.5.2.

Thanks.
Luke

On Fri, Oct 20, 2023 at 7:43 AM Ryan Leslie (BLP/ NEW YORK (REMOT) <
rles...@bloomberg.net> wrote:


Hi Luke,

Hope you are well. Can you please include
https://issues.apache.org/jira/browse/KAFKA-15106 in 3.5.2?

Thanks,

Ryan

From: dev@kafka.apache.org At: 10/17/23 05:05:24 UTC-4:00
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] Apache Kafka 3.5.2 release

Thanks Luke for volunteering for 3.5.2 release.

On Tue, 17 Oct 2023 at 11:58, Josep Prat 
wrote:


Hi Luke,

Thanks for taking this one!

Best,

On Tue, Oct 17, 2023 at 8:12 AM Luke Chen  wrote:


Hi all,

I'd like to volunteer as release manager for the Apache Kafka 3.5.2, to
have an important bug/vulnerability fix release for 3.5.1.

If there are no objections, I'll start building a release plan in

thewiki

in the next couple of weeks.

Thanks,
Luke




--
[image: Aiven] 

*Josep Prat*
Open Source Engineering Director, *Aiven*
josep.p...@aiven.io | +491715557497
aiven.io  | 
 
*Aiven Deutschland GmbH*
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
Amtsgericht Charlottenburg, HRB 209739 B

Re: [VOTE] KIP-988 Streams StandbyUpdateListener

[Almog Gavra]

+1 (non-binding) - great improvement, thanks Colt & Eduwer!

On Tue, Oct 17, 2023 at 11:25 AM Guozhang Wang 
wrote:

> +1 from me.
>
> On Mon, Oct 16, 2023 at 1:56 AM Lucas Brutschy
>  wrote:
> >
> > Hi,
> >
> > thanks again for the KIP!
> >
> > +1 (binding)
> >
> > Cheers,
> > Lucas
> >
> >
> >
> > On Sun, Oct 15, 2023 at 9:13 AM Colt McNealy 
> wrote:
> > >
> > > Hello there,
> > >
> > > I'd like to call a vote on KIP-988 (co-authored by my friend and
> colleague
> > > Eduwer Camacaro). We are hoping to get it in before the 3.7.0 release.
> > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-988%3A+Streams+Standby+Task+Update+Listener
> > >
> > > Cheers,
> > > Colt McNealy
> > >
> > > *Founder, LittleHorse.dev*
>

Re: [DISCUSS] KIP-992 Proposal to introduce IQv2 Query Types: TimestampedKeyQuery and TimestampedRangeQuery

[Matthias J. Sax]

Thanks for the KIP Hanyu,

One questions:


To address this inconsistency, we propose that KeyQuery  should be restricted 
to querying kv-stores  only, ensuring that it always returns a plain V  type, 
making the behavior of the aforementioned code more predictable. Similarly, 
RangeQuery  should be dedicated to querying kv-stores , consistently returning 
only the plain V .


Why do you want to restrict `KeyQuery` and `RangeQuery` to kv-stores? I 
think it would be possible to still allow both queries for ts-kv-stores, 
but change the implementation to return "plain V" instead of 
`ValueAndTimestamp`, ie, the implementation would automatically 
unwrap the value.




-Matthias

On 10/20/23 2:32 AM, Alieh Saeedi wrote:

Hey Hanyu,

Thanks for the KIP. It seems good to me.
Just one point: AFAIK, we are going to remove "get" from the name of all
getter methods.

Cheers,
Alieh

On Thu, Oct 19, 2023 at 5:44 PM Hanyu (Peter) Zheng
 wrote:


Hello everyone,

I would like to start the discussion for KIP-992: Proposal to introduce
IQv2 Query Types: TimestampedKeyQuery and TimestampedRangeQuery

The KIP can be found here:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-992%3A+Proposal+to+introduce+IQv2+Query+Types%3A+TimestampedKeyQuery+and+TimestampedRangeQuery

Any suggestions are more than welcome.

Many thanks,
Hanyu

On Thu, Oct 19, 2023 at 8:17 AM Hanyu (Peter) Zheng 
wrote:





https://cwiki.apache.org/confluence/display/KAFKA/KIP-992%3A+Proposal+to+introduce+IQv2+Query+Types%3A+TimestampedKeyQuery+and+TimestampedRangeQuery


--

[image: Confluent] 
Hanyu (Peter) Zheng he/him/his
Software Engineer Intern
+1 (213) 431-7193 <+1+(213)+431-7193>
Follow us: [image: Blog]
<

https://www.confluent.io/blog?utm_source=footer_medium=email_campaign=ch.email-signature_type.community_content.blog

[image:
Twitter] [image: LinkedIn]
[image: Slack]
[image: YouTube]


[image: Try Confluent Cloud for Free]
<

https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound_source=gmail_medium=organic






--

[image: Confluent] 
Hanyu (Peter) Zheng he/him/his
Software Engineer Intern
+1 (213) 431-7193 <+1+(213)+431-7193>
Follow us: [image: Blog]
<
https://www.confluent.io/blog?utm_source=footer_medium=email_campaign=ch.email-signature_type.community_content.blog

[image:

Twitter] [image: LinkedIn]
[image: Slack]
[image: YouTube]


[image: Try Confluent Cloud for Free]
<
https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound_source=gmail_medium=organic

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.5 #82

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 562216 lines...]

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
TableTableJoinIntegrationTest > [caching enabled = true] > 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest.testInner[caching
 enabled = true] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
TableTableJoinIntegrationTest > [caching enabled = true] > 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest.testOuter[caching
 enabled = true] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
TableTableJoinIntegrationTest > [caching enabled = true] > 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest.testOuter[caching
 enabled = true] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
TableTableJoinIntegrationTest > [caching enabled = true] > 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest.testInnerWithVersionedStores[caching
 enabled = true] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
TableTableJoinIntegrationTest > [caching enabled = true] > 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest.testInnerWithVersionedStores[caching
 enabled = true] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
TableTableJoinIntegrationTest > [caching enabled = true] > 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest.testLeft[caching
 enabled = true] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
TableTableJoinIntegrationTest > [caching enabled = true] > 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest.testLeft[caching
 enabled = true] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
TableTableJoinIntegrationTest > [caching enabled = true] > 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest.testOuterWithVersionedStores[caching
 enabled = true] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
TableTableJoinIntegrationTest > [caching enabled = true] > 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest.testOuterWithVersionedStores[caching
 enabled = true] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
TableTableJoinIntegrationTest > [caching enabled = true] > 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest.testOuterWithRightVersionedOnly[caching
 enabled = true] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
TableTableJoinIntegrationTest > [caching enabled = true] > 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest.testOuterWithRightVersionedOnly[caching
 enabled = true] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
TableTableJoinIntegrationTest > [caching enabled = true] > 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest.testLeftWithVersionedStores[caching
 enabled = true] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
TableTableJoinIntegrationTest > [caching enabled = true] > 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest.testLeftWithVersionedStores[caching
 enabled = true] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
TableTableJoinIntegrationTest > [caching enabled = true] > 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest.testOuterWithLeftVersionedOnly[caching
 enabled = true] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
TableTableJoinIntegrationTest > [caching enabled = true] > 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest.testOuterWithLeftVersionedOnly[caching
 enabled = true] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
TableTableJoinIntegrationTest > [caching enabled = true] > 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest.testLeftWithRightVersionedOnly[caching
 enabled = true] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
TableTableJoinIntegrationTest > [caching enabled = true] > 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest.testLeftWithRightVersionedOnly[caching
 enabled = true] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
TableTableJoinIntegrationTest > [caching enabled = true] > 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest.testInnerInner[caching
 enabled = true] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
TableTableJoinIntegrationTest > [caching enabled = true] > 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest.testInnerInner[caching
 enabled = true] 

Re: [DISCUSS] KIP-988 Streams Standby Task Update Listener

[Matthias J. Sax]

Forgot one thing:

We could also pass `currentLag()` into `onBachLoaded()` instead of 
end-offset.



-Matthias

On 10/20/23 10:56 AM, Matthias J. Sax wrote:

Thanks for digging into this Bruno.

The JavaDoc on the consumer does not say anything specific about 
`endOffset` guarantees:


Get the end offsets for the given partitions. In the default {@code 
read_uncommitted} isolation level, the end
offset is the high watermark (that is, the offset of the last 
successfully replicated message plus one). For
{@code read_committed} consumers, the end offset is the last stable 
offset (LSO), which is the minimum of
the high watermark and the smallest offset of any open transaction. 
Finally, if the partition has never been

written to, the end offset is 0.


Thus, I actually believe that it would be ok to change the 
implementation and serve the answer from the `TopicPartitionState`?


Another idea would be, to use `currentLag()` in combination with 
`position()` (or the offset of the last read record) to compute the 
end-offset of the fly?



-Matthias

On 10/20/23 4:00 AM, Bruno Cadonna wrote:

Hi,

Matthias is correct that the end offsets are stored somewhere in the 
metadata of the consumer. More precisely, they are stored in the 
`TopicPartitionState`. However, I could not find public API on the 
consumer other than currentLag() that uses the stored end offsets. If 
I understand the code correctly, method endOffSets() always triggers a 
remote call.


I am a bit concerned about doing remote calls every commit.interval.ms 
(by default 200ms under EOS). At the moment the remote calls are only 
issued if an optimization for KTables is turned on where changelog 
topics are replaced with the input topic of the KTable. The current 
remote calls retrieve all committed offsets of the group at once. If I 
understand correctly, that is one single remote call. Remote calls for 
getting end offsets of changelog topics -- as I understand you are 
planning to issue -- will probably result in multiple remote calls to 
multiple leaders of the changelog topic partitions.


Please correct me if I misunderstood anything of the above.

If my understanding is correct, I propose to modify the consumer in 
such a way to get the end offset from the locally stored metadata 
whenever possible as part of the implementation of this KIP. I do not 
know what the implications are of such a change of the consumer and if 
a KIP is needed for it. Maybe, endOffsets() guarantees to return the 
freshest end offsets possible, which would not be satisfied with the 
modification.


Regarding the naming, I do not completely agree with Matthias. While 
the pattern might be consistent with onBatchUpdated, what is the 
meaning of onBatchUpdated? Is the batch updated? The names 
onBatchLoaded or onBatchWritten or onBatchAdded are more clear IMO.
With "restore" the pattern works better. If I restore a batch of 
records in a state, the records are not there although they should be 
there and I add them. If I update a batch of records in a state. This 
sounds like the batch of records is in the state and I modify the 
existing records within the state. That is clearly not the meaning of 
the event for which the listener should be called.


Best,
Bruno



On 10/19/23 2:12 AM, Matthias J. Sax wrote:

Thanks for the KIP. Seems I am almost late to the party.

About naming (fun, fun, fun): I like the current proposal overall, 
except `onBachLoaded`, but would prefer `onBatchUpdated`. It better 
aligns to everything else:


  - it's an update-listener, not loaded-listener
  - `StateRestoreListener` has `onRestoreStart`, `onRestoreEnd`, 
`onRestoreSuspended, and `onBachRestored` (it's very consistent
  - `StandbyUpdateListener` should have `onUpdateStart`, 
`onUpdateSuspended` and `onBatchUpdated`  to be equally consistent 
(using "loaded" breaks the pattern)



About the end-offset question: I am relatively sure that the consumer 
gets the latest end-offset as attached metadata in every fetch 
response. (We exploit this behavior to track end-offsets for input 
topic with regard to `max.task.idle.ms` without overhead -- it was 
also a concern when we did the corresponding KIP how we could track 
lag with no overhead).


Thus, I believe we would "just" need to modify the code accordingly 
to get this information from the restore-consumer 
(`restorConsumer.endOffsets(...)`; should be served w/o RPC but from 
internal metadata cache) for free, and pass into the listener.


Please double check / verify this claim and keep me honest about it.


-Matthias

On 10/17/23 6:38 AM, Eduwer Camacaro wrote:

Hi Bruno,

Thanks for your observation; surely it will require a network call 
using
the admin client in order to know this "endOffset" and that will 
have an
impact on performance. We can either find a solution that has a low 
impact
on performance or ideally zero impact; unfortunately, I don't see a 
way to

have zero impact on performance. However, we can leverage 

Re: [DISCUSS] KIP-992 Proposal to introduce IQv2 Query Types: TimestampedKeyQuery and TimestampedRangeQuery

[Hanyu (Peter) Zheng]

Thank you Matthias,

I will modify the KIP to eliminate this restriction.

Sincerely,
Hanyu

On Fri, Oct 20, 2023 at 2:04 PM Hanyu (Peter) Zheng 
wrote:

> Thank you Alieh,
>
> In these two new query types, I will remove 'get' from all getter method
> names.
>
> Sincerely,
> Hanyu
>
> On Fri, Oct 20, 2023 at 10:40 AM Matthias J. Sax  wrote:
>
>> Thanks for the KIP Hanyu,
>>
>> One questions:
>>
>> > To address this inconsistency, we propose that KeyQuery  should be
>> restricted to querying kv-stores  only, ensuring that it always returns a
>> plain V  type, making the behavior of the aforementioned code more
>> predictable. Similarly, RangeQuery  should be dedicated to querying
>> kv-stores , consistently returning only the plain V .
>>
>> Why do you want to restrict `KeyQuery` and `RangeQuery` to kv-stores? I
>> think it would be possible to still allow both queries for ts-kv-stores,
>> but change the implementation to return "plain V" instead of
>> `ValueAndTimestamp`, ie, the implementation would automatically
>> unwrap the value.
>>
>>
>>
>> -Matthias
>>
>> On 10/20/23 2:32 AM, Alieh Saeedi wrote:
>> > Hey Hanyu,
>> >
>> > Thanks for the KIP. It seems good to me.
>> > Just one point: AFAIK, we are going to remove "get" from the name of all
>> > getter methods.
>> >
>> > Cheers,
>> > Alieh
>> >
>> > On Thu, Oct 19, 2023 at 5:44 PM Hanyu (Peter) Zheng
>> >  wrote:
>> >
>> >> Hello everyone,
>> >>
>> >> I would like to start the discussion for KIP-992: Proposal to introduce
>> >> IQv2 Query Types: TimestampedKeyQuery and TimestampedRangeQuery
>> >>
>> >> The KIP can be found here:
>> >>
>> >>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-992%3A+Proposal+to+introduce+IQv2+Query+Types%3A+TimestampedKeyQuery+and+TimestampedRangeQuery
>> >>
>> >> Any suggestions are more than welcome.
>> >>
>> >> Many thanks,
>> >> Hanyu
>> >>
>> >> On Thu, Oct 19, 2023 at 8:17 AM Hanyu (Peter) Zheng <
>> pzh...@confluent.io>
>> >> wrote:
>> >>
>> >>>
>> >>>
>> >>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-992%3A+Proposal+to+introduce+IQv2+Query+Types%3A+TimestampedKeyQuery+and+TimestampedRangeQuery
>> >>>
>> >>> --
>> >>>
>> >>> [image: Confluent] 
>> >>> Hanyu (Peter) Zheng he/him/his
>> >>> Software Engineer Intern
>> >>> +1 (213) 431-7193 <+1+(213)+431-7193>
>> >>> Follow us: [image: Blog]
>> >>> <
>> >>
>> https://www.confluent.io/blog?utm_source=footer_medium=email_campaign=ch.email-signature_type.community_content.blog
>> >>> [image:
>> >>> Twitter] [image: LinkedIn]
>> >>> [image: Slack]
>> >>> [image: YouTube]
>> >>> 
>> >>>
>> >>> [image: Try Confluent Cloud for Free]
>> >>> <
>> >>
>> https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound_source=gmail_medium=organic
>> >>>
>> >>>
>> >>
>> >>
>> >> --
>> >>
>> >> [image: Confluent] 
>> >> Hanyu (Peter) Zheng he/him/his
>> >> Software Engineer Intern
>> >> +1 (213) 431-7193 <+1+(213)+431-7193>
>> >> Follow us: [image: Blog]
>> >> <
>> >>
>> https://www.confluent.io/blog?utm_source=footer_medium=email_campaign=ch.email-signature_type.community_content.blog
>> >>> [image:
>> >> Twitter] [image: LinkedIn]
>> >> [image: Slack]
>> >> [image: YouTube]
>> >> 
>> >>
>> >> [image: Try Confluent Cloud for Free]
>> >> <
>> >>
>> https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound_source=gmail_medium=organic
>> >>>
>> >>
>> >
>>
>
>
> --
>
> [image: Confluent] 
> Hanyu (Peter) Zheng he/him/his
> Software Engineer Intern
> +1 (213) 431-7193 <+1+(213)+431-7193>
> Follow us: [image: Blog]
> [image:
> Twitter] [image: LinkedIn]
> [image: Slack]
> [image: YouTube]
> 
>
> [image: Try Confluent Cloud for Free]
> 
>


-- 

[image: Confluent] 
Hanyu (Peter) Zheng he/him/his
Software Engineer Intern
+1 (213) 431-7193 <+1+(213)+431-7193>
Follow us: [image: Blog]
[image:
Twitter] [image: LinkedIn]
[image: Slack]
[image: YouTube]


[image: Try Confluent Cloud for Free]

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2314

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 217596 lines...]

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultStateUpdaterTest > shouldThrowIfAddingActiveTasksWithSameId() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultStateUpdaterTest > 
shouldRestoreActiveStatefulTasksAndUpdateStandbyTasks() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultStateUpdaterTest > 
shouldRestoreActiveStatefulTasksAndUpdateStandbyTasks() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultStateUpdaterTest > shouldPauseStandbyTask() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultStateUpdaterTest > shouldPauseStandbyTask() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultStateUpdaterTest > shouldThrowIfStatefulTaskNotInStateRestoring() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultStateUpdaterTest > shouldThrowIfStatefulTaskNotInStateRestoring() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskExecutorTest > shouldSetUncaughtStreamsException() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskExecutorTest > shouldSetUncaughtStreamsException() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskExecutorTest > shouldClearTaskTimeoutOnProcessed() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskExecutorTest > shouldClearTaskTimeoutOnProcessed() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskExecutorTest > shouldUnassignTaskWhenRequired() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskExecutorTest > shouldUnassignTaskWhenRequired() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskExecutorTest > shouldClearTaskReleaseFutureOnShutdown() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskExecutorTest > shouldClearTaskReleaseFutureOnShutdown() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskExecutorTest > shouldProcessTasks() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskExecutorTest > shouldProcessTasks() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskExecutorTest > shouldPunctuateStreamTime() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskExecutorTest > shouldPunctuateStreamTime() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskExecutorTest > shouldShutdownTaskExecutor() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskExecutorTest > shouldShutdownTaskExecutor() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskExecutorTest > shouldAwaitProcessableTasksIfNoneAssignable() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskExecutorTest > shouldAwaitProcessableTasksIfNoneAssignable() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskExecutorTest > 
shouldRespectPunctuationDisabledByTaskExecutionMetadata() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskExecutorTest > 
shouldRespectPunctuationDisabledByTaskExecutionMetadata() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskExecutorTest > shouldSetTaskTimeoutOnTimeoutException() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskExecutorTest > shouldSetTaskTimeoutOnTimeoutException() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskExecutorTest > shouldPunctuateSystemTime() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskExecutorTest > shouldPunctuateSystemTime() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskExecutorTest > shouldUnassignTaskWhenNotProgressing() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskExecutorTest > shouldUnassignTaskWhenNotProgressing() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskExecutorTest > shouldNotFlushOnException() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskExecutorTest > shouldNotFlushOnException() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskExecutorTest > 
shouldRespectProcessingDisabledByTaskExecutionMetadata() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskExecutorTest > 
shouldRespectProcessingDisabledByTaskExecutionMetadata() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldLockAnEmptySetOfTasks() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest 

Re: [DISCUSS] KIP-992 Proposal to introduce IQv2 Query Types: TimestampedKeyQuery and TimestampedRangeQuery

[Hanyu (Peter) Zheng]

Thank you Alieh,

In these two new query types, I will remove 'get' from all getter method
names.

Sincerely,
Hanyu

On Fri, Oct 20, 2023 at 10:40 AM Matthias J. Sax  wrote:

> Thanks for the KIP Hanyu,
>
> One questions:
>
> > To address this inconsistency, we propose that KeyQuery  should be
> restricted to querying kv-stores  only, ensuring that it always returns a
> plain V  type, making the behavior of the aforementioned code more
> predictable. Similarly, RangeQuery  should be dedicated to querying
> kv-stores , consistently returning only the plain V .
>
> Why do you want to restrict `KeyQuery` and `RangeQuery` to kv-stores? I
> think it would be possible to still allow both queries for ts-kv-stores,
> but change the implementation to return "plain V" instead of
> `ValueAndTimestamp`, ie, the implementation would automatically
> unwrap the value.
>
>
>
> -Matthias
>
> On 10/20/23 2:32 AM, Alieh Saeedi wrote:
> > Hey Hanyu,
> >
> > Thanks for the KIP. It seems good to me.
> > Just one point: AFAIK, we are going to remove "get" from the name of all
> > getter methods.
> >
> > Cheers,
> > Alieh
> >
> > On Thu, Oct 19, 2023 at 5:44 PM Hanyu (Peter) Zheng
> >  wrote:
> >
> >> Hello everyone,
> >>
> >> I would like to start the discussion for KIP-992: Proposal to introduce
> >> IQv2 Query Types: TimestampedKeyQuery and TimestampedRangeQuery
> >>
> >> The KIP can be found here:
> >>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-992%3A+Proposal+to+introduce+IQv2+Query+Types%3A+TimestampedKeyQuery+and+TimestampedRangeQuery
> >>
> >> Any suggestions are more than welcome.
> >>
> >> Many thanks,
> >> Hanyu
> >>
> >> On Thu, Oct 19, 2023 at 8:17 AM Hanyu (Peter) Zheng <
> pzh...@confluent.io>
> >> wrote:
> >>
> >>>
> >>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-992%3A+Proposal+to+introduce+IQv2+Query+Types%3A+TimestampedKeyQuery+and+TimestampedRangeQuery
> >>>
> >>> --
> >>>
> >>> [image: Confluent] 
> >>> Hanyu (Peter) Zheng he/him/his
> >>> Software Engineer Intern
> >>> +1 (213) 431-7193 <+1+(213)+431-7193>
> >>> Follow us: [image: Blog]
> >>> <
> >>
> https://www.confluent.io/blog?utm_source=footer_medium=email_campaign=ch.email-signature_type.community_content.blog
> >>> [image:
> >>> Twitter] [image: LinkedIn]
> >>> [image: Slack]
> >>> [image: YouTube]
> >>> 
> >>>
> >>> [image: Try Confluent Cloud for Free]
> >>> <
> >>
> https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound_source=gmail_medium=organic
> >>>
> >>>
> >>
> >>
> >> --
> >>
> >> [image: Confluent] 
> >> Hanyu (Peter) Zheng he/him/his
> >> Software Engineer Intern
> >> +1 (213) 431-7193 <+1+(213)+431-7193>
> >> Follow us: [image: Blog]
> >> <
> >>
> https://www.confluent.io/blog?utm_source=footer_medium=email_campaign=ch.email-signature_type.community_content.blog
> >>> [image:
> >> Twitter] [image: LinkedIn]
> >> [image: Slack]
> >> [image: YouTube]
> >> 
> >>
> >> [image: Try Confluent Cloud for Free]
> >> <
> >>
> https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound_source=gmail_medium=organic
> >>>
> >>
> >
>


-- 

[image: Confluent] 
Hanyu (Peter) Zheng he/him/his
Software Engineer Intern
+1 (213) 431-7193 <+1+(213)+431-7193>
Follow us: [image: Blog]
[image:
Twitter] [image: LinkedIn]
[image: Slack]
[image: YouTube]


[image: Try Confluent Cloud for Free]

[jira] [Resolved] (KAFKA-15651) Investigate auto commit guarantees during Consumer.assign()

[Jun Rao (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao resolved KAFKA-15651.
-
Resolution: Not A Problem

> Investigate auto commit guarantees during Consumer.assign()
> ---
>
> Key: KAFKA-15651
> URL: https://issues.apache.org/jira/browse/KAFKA-15651
> Project: Kafka
>  Issue Type: Sub-task
>  Components: clients, consumer
>Reporter: Kirk True
>Assignee: Kirk True
>Priority: Major
>  Labels: consumer-threading-refactor, kip-848-preview
>
> In the {{assign()}} method implementation, both {{KafkaConsumer}} and 
> {{PrototypeAsyncConsumer}} commit offsets asynchronously. Is this 
> intentional? [~junrao] asks in a [recent PR 
> review|https://github.com/apache/kafka/pull/14406/files/193af8230d0c61853d764cbbe29bca2fc6361af9#r1349023459]:
> {quote}Do we guarantee that the new owner of the unsubscribed partitions 
> could pick up the latest committed offset?
> {quote}
> Let's confirm whether the asynchronous approach is acceptable and correct. If 
> it is, great, let's enhance the documentation to briefly explain why. If it 
> is not, let's correct the behavior if it's within the API semantic 
> expectations.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15626) Replace verification guard object with an specific type

[Justine Olshan (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Justine Olshan resolved KAFKA-15626.

Resolution: Fixed

> Replace verification guard object with an specific type
> ---
>
> Key: KAFKA-15626
> URL: https://issues.apache.org/jira/browse/KAFKA-15626
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Justine Olshan
>Assignee: Justine Olshan
>Priority: Major
>
>  https://github.com/apache/kafka/pull/13787#discussion_r1361468169



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.6 #95

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 409138 lines...]

Gradle Test Run :connect:runtime:test > Gradle Test Executor 43 > 
org.apache.kafka.connect.storage.MemoryStatusBackingStoreTest > 
putAndGetTaskStatus STARTED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 43 > 
org.apache.kafka.connect.storage.MemoryStatusBackingStoreTest > 
putAndGetTaskStatus PASSED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 43 > 
org.apache.kafka.connect.storage.MemoryStatusBackingStoreTest > 
deleteTaskStatus STARTED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 43 > 
org.apache.kafka.connect.storage.MemoryStatusBackingStoreTest > 
deleteTaskStatus PASSED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 43 > 
org.apache.kafka.connect.storage.MemoryStatusBackingStoreTest > 
deleteConnectorStatus STARTED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 43 > 
org.apache.kafka.connect.storage.MemoryStatusBackingStoreTest > 
deleteConnectorStatus PASSED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 43 > 
org.apache.kafka.connect.storage.OffsetUtilsTest > 
testValidateFormatWithValidFormat STARTED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 43 > 
org.apache.kafka.connect.storage.OffsetUtilsTest > 
testValidateFormatWithValidFormat PASSED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 43 > 
org.apache.kafka.connect.storage.OffsetUtilsTest > 
testValidateFormatMapWithNonStringKeys STARTED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 43 > 
org.apache.kafka.connect.storage.OffsetUtilsTest > 
testValidateFormatMapWithNonStringKeys PASSED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 43 > 
org.apache.kafka.connect.storage.OffsetUtilsTest > 
testProcessPartitionKeyListWithElementsOfWrongType STARTED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 43 > 
org.apache.kafka.connect.storage.OffsetUtilsTest > 
testProcessPartitionKeyListWithElementsOfWrongType PASSED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 43 > 
org.apache.kafka.connect.storage.OffsetUtilsTest > 
testProcessPartitionKeyNotList STARTED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 43 > 
org.apache.kafka.connect.storage.OffsetUtilsTest > 
testProcessPartitionKeyNotList PASSED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 43 > 
org.apache.kafka.connect.storage.OffsetUtilsTest > 
testValidateFormatMapWithNonPrimitiveKeys STARTED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 43 > 
org.apache.kafka.connect.storage.OffsetUtilsTest > 
testValidateFormatMapWithNonPrimitiveKeys PASSED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 43 > 
org.apache.kafka.connect.storage.OffsetUtilsTest > 
testProcessPartitionKeyListWithOneElement STARTED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 43 > 
org.apache.kafka.connect.storage.OffsetUtilsTest > 
testProcessPartitionKeyListWithOneElement PASSED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 43 > 
org.apache.kafka.connect.storage.OffsetUtilsTest > testValidateFormatNotMap 
STARTED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 43 > 
org.apache.kafka.connect.storage.OffsetUtilsTest > testValidateFormatNotMap 
PASSED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 43 > 
org.apache.kafka.connect.storage.OffsetUtilsTest > 
testProcessPartitionKeyValidList STARTED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 43 > 
org.apache.kafka.connect.storage.OffsetUtilsTest > 
testProcessPartitionKeyValidList PASSED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 43 > 
org.apache.kafka.connect.util.ConvertingFutureCallbackTest > 
shouldNotConvertBeforeGetOnFailedCompletion STARTED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 43 > 
org.apache.kafka.connect.util.ConvertingFutureCallbackTest > 
shouldNotConvertBeforeGetOnFailedCompletion PASSED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 43 > 
org.apache.kafka.connect.util.ConvertingFutureCallbackTest > 
shouldBlockUntilCancellation STARTED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 43 > 
org.apache.kafka.connect.util.ConvertingFutureCallbackTest > 
shouldBlockUntilCancellation PASSED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 43 > 
org.apache.kafka.connect.util.ConvertingFutureCallbackTest > 
shouldConvertOnlyOnceBeforeGetOnSuccessfulCompletion STARTED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 43 > 
org.apache.kafka.connect.util.ConvertingFutureCallbackTest > 
shouldConvertOnlyOnceBeforeGetOnSuccessfulCompletion PASSED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 43 > 
org.apache.kafka.connect.util.ConvertingFutureCallbackTest > 

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #2315

[Apache Jenkins Server]

See 

[jira] [Resolved] (KAFKA-15378) Rolling upgrade system tests are failing

[Matthias J. Sax (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-15378.
-
Fix Version/s: 3.4.2
   3.5.2
   3.7.0
   3.6.1
   Resolution: Fixed

> Rolling upgrade system tests are failing
> 
>
> Key: KAFKA-15378
> URL: https://issues.apache.org/jira/browse/KAFKA-15378
> Project: Kafka
>  Issue Type: Task
>  Components: streams, system tests
>Affects Versions: 3.5.1
>Reporter: Lucas Brutschy
>Assignee: Matthias J. Sax
>Priority: Major
> Fix For: 3.4.2, 3.5.2, 3.7.0, 3.6.1
>
>
> The system tests are having failures for these tests:
> {noformat}
> kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.1.2.to_version=3.6.0-SNAPSHOT
> kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.2.3.to_version=3.6.0-SNAPSHOT
> kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.3.1.to_version=3.6.0-SNAPSHOT
> {noformat}
> See 
> [https://jenkins.confluent.io/job/system-test-kafka-branch-builder/5801/console]
>  for logs and other test data.
> Note that system tests currently only run with [this 
> fix](https://github.com/apache/kafka/commit/24d1780061a645bb2fbeefd8b8f50123c28ca94e),
>  I think some CVE python library update broke the system tests... 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.5 #83

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 563475 lines...]

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
TableTableJoinIntegrationTest > [caching enabled = true] > 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest.testInner[caching
 enabled = true] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
TableTableJoinIntegrationTest > [caching enabled = true] > 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest.testOuter[caching
 enabled = true] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
TableTableJoinIntegrationTest > [caching enabled = true] > 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest.testOuter[caching
 enabled = true] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
TableTableJoinIntegrationTest > [caching enabled = true] > 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest.testInnerWithVersionedStores[caching
 enabled = true] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
TableTableJoinIntegrationTest > [caching enabled = true] > 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest.testInnerWithVersionedStores[caching
 enabled = true] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
TableTableJoinIntegrationTest > [caching enabled = true] > 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest.testLeft[caching
 enabled = true] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
TableTableJoinIntegrationTest > [caching enabled = true] > 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest.testLeft[caching
 enabled = true] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
TableTableJoinIntegrationTest > [caching enabled = true] > 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest.testOuterWithVersionedStores[caching
 enabled = true] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
TableTableJoinIntegrationTest > [caching enabled = true] > 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest.testOuterWithVersionedStores[caching
 enabled = true] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
TableTableJoinIntegrationTest > [caching enabled = true] > 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest.testOuterWithRightVersionedOnly[caching
 enabled = true] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
TableTableJoinIntegrationTest > [caching enabled = true] > 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest.testOuterWithRightVersionedOnly[caching
 enabled = true] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
TableTableJoinIntegrationTest > [caching enabled = true] > 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest.testLeftWithVersionedStores[caching
 enabled = true] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
TableTableJoinIntegrationTest > [caching enabled = true] > 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest.testLeftWithVersionedStores[caching
 enabled = true] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
TableTableJoinIntegrationTest > [caching enabled = true] > 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest.testOuterWithLeftVersionedOnly[caching
 enabled = true] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
TableTableJoinIntegrationTest > [caching enabled = true] > 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest.testOuterWithLeftVersionedOnly[caching
 enabled = true] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
TableTableJoinIntegrationTest > [caching enabled = true] > 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest.testLeftWithRightVersionedOnly[caching
 enabled = true] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
TableTableJoinIntegrationTest > [caching enabled = true] > 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest.testLeftWithRightVersionedOnly[caching
 enabled = true] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
TableTableJoinIntegrationTest > [caching enabled = true] > 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest.testInnerInner[caching
 enabled = true] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
TableTableJoinIntegrationTest > [caching enabled = true] > 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest.testInnerInner[caching
 enabled = true] 

[jira] [Created] (KAFKA-15662) Implement support for clientInstanceIds in Kafka Stream

[Apoorv Mittal (Jira)]

Apoorv Mittal created KAFKA-15662:
-

 Summary: Implement support for clientInstanceIds in Kafka Stream
 Key: KAFKA-15662
 URL: https://issues.apache.org/jira/browse/KAFKA-15662
 Project: Kafka
  Issue Type: Sub-task
Reporter: Apoorv Mittal
Assignee: Matthias J. Sax


The KIP requires Kafka Stream to support below method to give access to the 
client instance ids of the producers, consumers and admin clients used by Kafka 
Streams.

 

This method is only permitted when Kafka Streams is in state RUNNING or 
REBALANCING. In the event that Kafka Streams is not in state RUNNING or 
REBALANCING, the method throws 
{{org.apache.kafka.streams.errors.StreamsNotRunningException}} , which is a new 
subclass of {{InvalidStateStoreException}} .

```

{{public}} {{ClientInstanceIds clientInstanceIds(Duration timeout);}}

{{```}}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15663) Implement ClientTelemetryReporter which manages telemetry lifecyclye

[Apoorv Mittal (Jira)]

Apoorv Mittal created KAFKA-15663:
-

 Summary: Implement ClientTelemetryReporter which manages telemetry 
lifecyclye
 Key: KAFKA-15663
 URL: https://issues.apache.org/jira/browse/KAFKA-15663
 Project: Kafka
  Issue Type: Sub-task
Reporter: Apoorv Mittal
Assignee: Apoorv Mittal


The KIP requires metrics to be collected and emitted to broker over OTLP 
format. The client telemetry reporter acts as a central piece to manage 
lifecycle of all telemetry components.

 

The changes should include:
 # Implementation of ClientTelemetry reporter binding together all components.
 # Placeholder to implement telemetry updater. 
 # Supporting classes.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.6 #96

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 308761 lines...]
> Task :connect:api:compileTestJava UP-TO-DATE
> Task :connect:api:testClasses UP-TO-DATE
> Task :clients:generateMetadataFileForMavenJavaPublication
> Task :connect:json:copyDependantLibs UP-TO-DATE
> Task :connect:json:jar UP-TO-DATE
> Task :connect:json:generateMetadataFileForMavenJavaPublication
> Task :connect:api:testJar
> Task :connect:api:testSrcJar
> Task :connect:api:publishMavenJavaPublicationToMavenLocal
> Task :connect:api:publishToMavenLocal
> Task :connect:json:publishMavenJavaPublicationToMavenLocal
> Task :connect:json:publishToMavenLocal
> Task :storage:api:compileTestJava
> Task :storage:api:testClasses
> Task :server-common:compileTestJava
> Task :server-common:testClasses
> Task :raft:compileTestJava
> Task :raft:testClasses
> Task :group-coordinator:compileTestJava
> Task :group-coordinator:testClasses

> Task :clients:javadoc
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.6/clients/src/main/java/org/apache/kafka/common/config/ConfigDef.java:81:
 warning - Tag @link:illegal character: "60" in "#define(String, Type, 
Importance, String, String, int, Width, String, List)"
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.6/clients/src/main/java/org/apache/kafka/common/config/ConfigDef.java:81:
 warning - Tag @link:illegal character: "62" in "#define(String, Type, 
Importance, String, String, int, Width, String, List)"
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.6/clients/src/main/java/org/apache/kafka/common/config/ConfigDef.java:81:
 warning - Tag @link: can't find define(String, Type, Importance, String, 
String, int, Width, String, List) in 
org.apache.kafka.common.config.ConfigDef
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.6/clients/src/main/java/org/apache/kafka/clients/admin/ScramMechanism.java:32:
 warning - Tag @see: missing final '>': "https://cwiki.apache.org/confluence/display/KAFKA/KIP-554%3A+Add+Broker-side+SCRAM+Config+API;>KIP-554:
 Add Broker-side SCRAM Config API

 This code is duplicated in 
org.apache.kafka.common.security.scram.internals.ScramMechanism.
 The type field in both files must match and must not change. The type field
 is used both for passing ScramCredentialUpsertion and for the internal
 UserScramCredentialRecord. Do not change the type field."

> Task :metadata:compileTestJava
> Task :metadata:testClasses

> Task :clients:javadoc
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.6/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/secured/package-info.java:21:
 warning - Tag @link: reference not found: 
org.apache.kafka.common.security.oauthbearer
5 warnings

> Task :clients:javadocJar
> Task :clients:srcJar
> Task :clients:testJar
> Task :clients:testSrcJar
> Task :clients:publishMavenJavaPublicationToMavenLocal
> Task :clients:publishToMavenLocal
> Task :core:classes
> Task :core:compileTestJava NO-SOURCE
> Task :core:compileTestScala
> Task :core:testClasses
> Task :streams:compileTestJava
> Task :streams:testClasses
> Task :streams:testJar
> Task :streams:testSrcJar
> Task :streams:publishMavenJavaPublicationToMavenLocal
> Task :streams:publishToMavenLocal

Deprecated Gradle features were used in this build, making it incompatible with 
Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings 
and determine if they come from your own scripts or plugins.

For more on this, please refer to 
https://docs.gradle.org/8.2.1/userguide/command_line_interface.html#sec:command_line_warnings
 in the Gradle documentation.

BUILD SUCCESSFUL in 10m 20s
94 actionable tasks: 41 executed, 53 up-to-date

Publishing build scan...
https://ge.apache.org/s/vqmjuod4sn7ui

[Pipeline] sh
+ grep ^version= gradle.properties
+ cut -d= -f 2
[Pipeline] dir
Running in 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.6/streams/quickstart
[Pipeline] {
[Pipeline] sh
+ mvn clean install -Dgpg.skip
[INFO] Scanning for projects...
[INFO] 
[INFO] Reactor Build Order:
[INFO] 
[INFO] Kafka Streams :: Quickstart[pom]
[INFO] streams-quickstart-java[maven-archetype]
[INFO] 
[INFO] < org.apache.kafka:streams-quickstart >-
[INFO] Building Kafka Streams :: Quickstart 3.6.1-SNAPSHOT[1/2]
[INFO]   from pom.xml
[INFO] [ pom ]-
[INFO] 
[INFO] --- clean:3.0.0:clean (default-clean) @ streams-quickstart ---
[INFO] 
[INFO] --- remote-resources:1.5:process (process-resource-bundles) @ 
streams-quickstart ---
[INFO] 
[INFO] --- site:3.5.1:attach-descriptor (attach-descriptor) @ 
streams-quickstart ---
[INFO] 
[INFO] --- gpg:1.6:sign (sign-artifacts) @ streams-quickstart ---
[INFO] 
[INFO] --- 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2316

[Apache Jenkins Server]

See 

Re: [DISCUSS] Apache Kafka 3.5.2 release

[Luke Chen]

Hi Matthias,

I agree it's indeed a blocker for 3.5.2 to address CVE in RocksDB.
Please let me know when the test is completed.

Thank you.
Luke

On Sat, Oct 21, 2023 at 2:12 AM Matthias J. Sax  wrote:

> Thanks for the info Luke.
>
> We did backport all but one PR in the mean time. The missing PR is a
> RocksDB version bump. We want to consider it for 3.5.2, because it
> addresses a CVE.
>
> Cf https://github.com/apache/kafka/pull/14216
>
> However, RocksDB versions bumps are a little bit more tricky, and we
> would like to test this properly on 3.5 branch, what would take at least
> one week; we could do the cherry-pick on Monday and start testing.
>
> Please let us know if such a delay for 3.5.2 is acceptable or not.
>
> Thanks.
>
> -Matthias
>
>
> On 10/20/23 5:44 AM, Luke Chen wrote:
> > Hi Ryan,
> >
> > OK, I've backported it to 3.5 branch.
> > I'll be included in v3.5.2.
> >
> > Thanks.
> > Luke
> >
> > On Fri, Oct 20, 2023 at 7:43 AM Ryan Leslie (BLP/ NEW YORK (REMOT) <
> > rles...@bloomberg.net> wrote:
> >
> >> Hi Luke,
> >>
> >> Hope you are well. Can you please include
> >> https://issues.apache.org/jira/browse/KAFKA-15106 in 3.5.2?
> >>
> >> Thanks,
> >>
> >> Ryan
> >>
> >> From: dev@kafka.apache.org At: 10/17/23 05:05:24 UTC-4:00
> >> To: dev@kafka.apache.org
> >> Subject: Re: [DISCUSS] Apache Kafka 3.5.2 release
> >>
> >> Thanks Luke for volunteering for 3.5.2 release.
> >>
> >> On Tue, 17 Oct 2023 at 11:58, Josep Prat 
> >> wrote:
> >>>
> >>> Hi Luke,
> >>>
> >>> Thanks for taking this one!
> >>>
> >>> Best,
> >>>
> >>> On Tue, Oct 17, 2023 at 8:12 AM Luke Chen  wrote:
> >>>
>  Hi all,
> 
>  I'd like to volunteer as release manager for the Apache Kafka 3.5.2,
> to
>  have an important bug/vulnerability fix release for 3.5.1.
> 
>  If there are no objections, I'll start building a release plan in
> >> thewiki
>  in the next couple of weeks.
> 
>  Thanks,
>  Luke
> 
> >>>
> >>>
> >>> --
> >>> [image: Aiven] 
> >>>
> >>> *Josep Prat*
> >>> Open Source Engineering Director, *Aiven*
> >>> josep.p...@aiven.io | +491715557497
> >>> aiven.io  |  >
> >>>  <
> https://twitter.com/aiven_io>
> >>> *Aiven Deutschland GmbH*
> >>> Alexanderufer 3-7, 10117 Berlin
> >>> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> >>> Amtsgericht Charlottenburg, HRB 209739 B
> >>
> >>
> >>
> >
>

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2317

[Apache Jenkins Server]

See 

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.5 #84

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 566093 lines...]

Gradle Test Run :streams:integrationTest > Gradle Test Executor 183 > 
KTableSourceTopicRestartIntegrationTest > 
shouldRestoreAndProgressWhenTopicWrittenToDuringRestorationWithEosAlphaEnabled()
 STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
TableTableJoinIntegrationTest > [caching enabled = true] > 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest.testInnerLeft[caching
 enabled = true] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
TableTableJoinIntegrationTest > [caching enabled = true] > 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest.testOuterInner[caching
 enabled = true] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 183 > 
KTableSourceTopicRestartIntegrationTest > 
shouldRestoreAndProgressWhenTopicWrittenToDuringRestorationWithEosAlphaEnabled()
 PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 183 > 
KTableSourceTopicRestartIntegrationTest > 
shouldRestoreAndProgressWhenTopicWrittenToDuringRestorationWithEosDisabled() 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 183 > 
KTableSourceTopicRestartIntegrationTest > 
shouldRestoreAndProgressWhenTopicWrittenToDuringRestorationWithEosDisabled() 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 183 > 
KTableSourceTopicRestartIntegrationTest > 
shouldRestoreAndProgressWhenTopicWrittenToDuringRestorationWithEosV2Enabled() 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
TableTableJoinIntegrationTest > [caching enabled = true] > 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest.testOuterInner[caching
 enabled = true] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
TableTableJoinIntegrationTest > [caching enabled = true] > 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest.testOuterOuter[caching
 enabled = true] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 183 > 
KTableSourceTopicRestartIntegrationTest > 
shouldRestoreAndProgressWhenTopicWrittenToDuringRestorationWithEosV2Enabled() 
PASSED

Deprecated Gradle features were used in this build, making it incompatible with 
Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings 
and determine if they come from your own scripts or plugins.

See 
https://docs.gradle.org/8.0.2/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 2h 15m 12s
230 actionable tasks: 124 executed, 106 up-to-date

See the profiling report at: 
file:///home/jenkins/workspace/Kafka_kafka_3.5/build/reports/profile/profile-2023-10-21-00-04-37.html
A fine-grained performance profile is available: use the --scan option.
[Pipeline] junit
Recording test results

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
TableTableJoinIntegrationTest > [caching enabled = true] > 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest.testOuterOuter[caching
 enabled = true] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
TableTableJoinIntegrationTest > [caching enabled = true] > 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest.testInnerWithRightVersionedOnly[caching
 enabled = true] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
TableTableJoinIntegrationTest > [caching enabled = true] > 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest.testInnerWithRightVersionedOnly[caching
 enabled = true] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
TableTableJoinIntegrationTest > [caching enabled = true] > 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest.testLeftWithLeftVersionedOnly[caching
 enabled = true] STARTED
[Checks API] No suitable checks publisher found.
[Pipeline] echo
Skipping Kafka Streams archetype test for Java 11
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // stage
[Pipeline] }

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
TableTableJoinIntegrationTest > [caching enabled = true] > 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest.testLeftWithLeftVersionedOnly[caching
 enabled = true] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
TableTableJoinIntegrationTest > [caching enabled = true] > 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest.testInnerWithLeftVersionedOnly[caching
 enabled = true] STARTED

Gradle Test Run