Re: [DISCUSS] KIP-924: customizable task assignment for Streams

Thanks for the feedback so far! I think pretty much all of it is reasonable. I'll reply to it inline: > 1. All the API logic is granular at the Task level, except the previousOwnerForPartition func. I’m not clear what’s the motivation behind it, does our controller also want to change how the

Re: EmbeddedKafkaCluster not working with SASL

I've also noticed that it is not possible to use KRaft mode in EmbeddedKafkaCluster. Since in version 4.0 we're planning to remove Zookeeper I think we should also modify EmbeddedKafkaCluster to enable KRaft mode based on the user-provided configuration. If this sounds reasonable to you I can

Re: Understanding FetchOffSet mechanism from log segment

Can anybody help in understanding the above scenario or any related KIP link would be helpful. Thanks and Regards Arpit Goyal 8861094754 On Fri, Nov 10, 2023 at 12:46 AM Arpit Goyal wrote: > Please let me know if anyone can help me understanding the above > behaviour. > > On Fri, Nov 10, 2023,

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.5 #96

See Changes: -- [...truncated 474664 lines...] Gradle Test Run :core:integrationTest > Gradle Test Executor 182 > ZkMigrationClientTest > testNonIncreasingKRaftEpoch()

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2373

See Changes: -- [...truncated 324506 lines...] Gradle Test Run :streams:test > Gradle Test Executor 62 > DefaultTaskExecutorTest > shouldSetUncaughtStreamsException()

Re: [DISCUSS] Apache Kafka 3.5.2 release

Hi all, Greg found a regression issue in Kafka connect: https://issues.apache.org/jira/browse/KAFKA-15800 I'll wait until this fix gets merged and create CR build for v3.5.2. Thanks. Luke On Sat, Nov 4, 2023 at 1:33 AM Matthias J. Sax wrote: > Hey, > > Sorry for late reply. We finished our

Request for reivews for my 2 PRs

Hi there, Can I ask for some feedback for my 2 PRs below ? Thanks a lot for your help. https://github.com/apache/kafka/pull/14136 https://github.com/apache/kafka/pull/13909 Thanks Owen

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2372

See Changes: -- [...truncated 324238 lines...] Gradle Test Run :streams:test > Gradle Test Executor 88 > DefaultTaskExecutorTest > shouldClearTaskTimeoutOnProcessed()

Re: [VOTE] KIP-979 Allow independently stop KRaft processes

Hi Hailey, Thanks for the KIP. It feels clunky to have to pass an absolute path to the configuration file when starting the broker or controller. I think we should consider one of two alternate options: 1. Use JMXtool to examine the running kafka.Kafka processes. I believe ID is available

[DISCUSS] KIP-1001: Add CurrentControllerId Metric

Hi all, I would like to start the discussion on a KIP I wrote about adding a new CurrentControllerIdMetric. See: https://cwiki.apache.org/confluence/display/KAFKA/KIP-1001%3A+Add+CurrentControllerId+Metric best, Colin

[jira] [Created] (KAFKA-15804) Broker leaks ServerSocketChannel when exception is thrown from ZkConfigManager during startup

Greg Harris created KAFKA-15804: --- Summary: Broker leaks ServerSocketChannel when exception is thrown from ZkConfigManager during startup Key: KAFKA-15804 URL: https://issues.apache.org/jira/browse/KAFKA-15804

Re: [DISCUSS] KIP-977: Partition-Level Throughput Metrics

Hi Qichao, Thanks for updating the KIP, all updates look good to me. Looking forward to see this KIP moving forward! Cheers, Jorge. On Wed, 8 Nov 2023 at 08:55, Qichao Chu wrote: > Hi Divij, > > Thank you for the feedback. I updated the KIP to make it a little bit more > generic: filters

Re: [DISCUSS] KIP-963: Upload and delete lag metrics in Tiered Storage

Hi Christo, I'd like to add another suggestion: 7. Adding on TS lag formulas, my understanding is that per pertition: - RemoteCopyLag: difference between: latest local segment candidate for upload - latest remote segment - Represents how Remote Log Manager task is handling backlog of segments.

Re: [DISCUSS] Road to Kafka 4.0

Hi Anton, It rarely makes sense to scale up and down the number of controller nodes in the cluster. Only one controller node will be active at any given time. The main reason to use 5 nodes would be to be able to tolerate 2 failures instead of 1. At Confluent, we generally run KRaft with 3

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #2371

See

Re: [DISCUSS] KIP-924: customizable task assignment for Streams

Hello Rohan, Thanks for the KIP! Thoughts below (seems I have similar comments to Guozhang, but I had already written this before reading his reply haha!). They're basically all minor suggestions for improvements, I wouldn't consider any of them blocking. 1. For the API, thoughts on changing the

Re: [DISCUSS] KIP-924: customizable task assignment for Streams

Hello Rohan, Thanks for the KIP! Overall it looks very nice. Just some quick thoughts : 1. All the API logic is granular at the Task level, except the previousOwnerForPartition func. I’m not clear what’s the motivation behind it, does our controller also want to change how the partitions->tasks

Re: Understanding FetchOffSet mechanism from log segment

Please let me know if anyone can help me understanding the above behaviour. On Fri, Nov 10, 2023, 00:08 Arpit Goyal wrote: > Recently I am trying to understand the fetch offset mechanism of kafka > through code but I have certain doubts which I am still not able to > understand. > > *What I

Jenkins build is unstable: Kafka » Kafka Branch Builder » 3.6 #108

See

Understanding FetchOffSet mechanism from log segment

Recently I am trying to understand the fetch offset mechanism of kafka through code but I have certain doubts which I am still not able to understand. *What I believe Log Segment contains * Log Segment constitutes a list of record batches with key as base offset. Let's take an example and list

[jira] [Created] (KAFKA-15803) Update last seen epcoh during commit

Philip Nee created KAFKA-15803: -- Summary: Update last seen epcoh during commit Key: KAFKA-15803 URL: https://issues.apache.org/jira/browse/KAFKA-15803 Project: Kafka Issue Type: Task

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2370

See Changes: -- [...truncated 324401 lines...] Gradle Test Run :streams:test > Gradle Test Executor 88 > SynchronizedPartitionGroupTest > testUpdatePartitions() STARTED

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.5 #95

See Changes: -- [...truncated 33 lines...] [Pipeline] { [Pipeline] timeout Timeout set to expire in 8 hr 0 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 8 hr 0

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.5 #94

See Changes: -- [...truncated 478794 lines...] [INFO] --- remote-resources:1.5:process (process-resource-bundles) @ streams-quickstart --- [INFO] [INFO] ---

[jira] [Created] (KAFKA-15802) Trying to access uncopied segments metadata on listOffsets

Francois Visconte created KAFKA-15802: - Summary: Trying to access uncopied segments metadata on listOffsets Key: KAFKA-15802 URL: https://issues.apache.org/jira/browse/KAFKA-15802 Project: Kafka

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.6 #107

See Changes: -- [...truncated 410761 lines...] Gradle Test Run :core:test > Gradle Test Executor 91 > KafkaZkClientTest > testGetAllTopicsInClusterTriggersWatch() STARTED

[jira] [Created] (KAFKA-15801) improve Kafka broker/NetworkClient logging for connectivity issues

Alexander Kilian created KAFKA-15801: Summary: improve Kafka broker/NetworkClient logging for connectivity issues Key: KAFKA-15801 URL: https://issues.apache.org/jira/browse/KAFKA-15801 Project:

Re: [DISCUSS] KIP-968: Support single-key_multi-timestamp interactive queries (IQv2) for versioned state stores

Hi, Thanks for the updates! First my take on previous comments: 50) I am in favor of deprecating the constructor that does not take the validTo parameter. That implies that I am in favor of modifying get(key, asOf) to set the correct validTo. 60) I am in favor of renaming ValueIterator

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.5 #93

See Changes: -- [...truncated 5806 lines...] > Task :metadata:testClasses > Task :storage:api:checkstyleTest > Task :metadata:spotbugsTest SKIPPED > Task :storage:api:check >

License Cost for Apache Kafka with Support

Hi Team, We would like to use licensed version of Kafka along with Support. Could you please share the contact details to discuss further. Best Regards, Vinothkumar S Cloud Infra Specialist TATA ELXSI ITPB Road Whitefield Bangalore 560 048 India Cell +91 9686707172

RE: [DISCUSS] Road to Kafka 4.0

Hi Luke, We have been looking into what switching from ZK to KRaft will mean for Aiven. We heavily depend on an “immutable infrastructure” model for deployments. This means that, when we perform upgrades, we introduce new nodes to our clusters, scale the cluster up to incorporate the new nodes,

RE: [DISCUSS] Road to Kafka 4.0

Hi Luke, We have been looking into what switching from ZK to KRaft will mean for Aiven. We heavily depend on an “immutable infrastructure” model for deployments. This means that, when we perform upgrades, we introduce new nodes to our clusters, scale the cluster up to incorporate the new nodes,

EmbeddedKafkaCluster not working with SASL

Hello, I am utilizing "org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster" to write integration tests for my application. In general, things are functioning properly. However, when I modify the default server configurations to use a SASL_PLAINTEXT listener with the PLAIN mechanism,

Re: [DISCUSS] KIP-963: Upload and delete lag metrics in Tiered Storage

Hi Christo, Thanks for the KIP! Some comments: 1. I agree with Kamal that a metric to cover the time taken to read data from remote storage is helpful. 2. I can see there are some metrics are only on topic level, but some are on partition level. Could you explain why some of them are only on