[jira] [Commented] (KAFKA-16426) Initialise KafkaMetricsGroup statically
[
https://issues.apache.org/jira/browse/KAFKA-16426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832901#comment-17832901
]
Ismael Juma commented on KAFKA-16426:
-
It stores state (a class) - how will that be handled statically?
> Initialise KafkaMetricsGroup statically
> ---
>
> Key: KAFKA-16426
> URL: https://issues.apache.org/jira/browse/KAFKA-16426
> Project: Kafka
> Issue Type: Improvement
> Components: metrics
>Affects Versions: 3.7.0, 3.6.1
>Reporter: Gaurav Narula
>Assignee: Gaurav Narula
>Priority: Minor
>
> KAFKA-14524 ([PR-13067|https://github.com/apache/kafka/pull/13067]) migrated
> {{KafkaMetricsGroup}} from Scala to Java. In doing so, we overlooked
> initialising {{KafkaMetricsGroup}} instances statically. This results in
> allocating many more {{KafkaMetricsGroup}} instances than are required.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-16426) Initialise KafkaMetricsGroup statically
[
https://issues.apache.org/jira/browse/KAFKA-16426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismael Juma updated KAFKA-16426:
Fix Version/s: (was: 3.7.1)
> Initialise KafkaMetricsGroup statically
> ---
>
> Key: KAFKA-16426
> URL: https://issues.apache.org/jira/browse/KAFKA-16426
> Project: Kafka
> Issue Type: Improvement
> Components: metrics
>Affects Versions: 3.7.0, 3.6.1
>Reporter: Gaurav Narula
>Assignee: Gaurav Narula
>Priority: Minor
>
> KAFKA-14524 ([PR-13067|https://github.com/apache/kafka/pull/13067]) migrated
> {{KafkaMetricsGroup}} from Scala to Java. In doing so, we overlooked
> initialising {{KafkaMetricsGroup}} instances statically. This results in
> allocating many more {{KafkaMetricsGroup}} instances than are required.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-16310) ListOffsets doesn't report the offset with maxTimestamp anymore
[
https://issues.apache.org/jira/browse/KAFKA-16310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17831848#comment-17831848
]
Ismael Juma commented on KAFKA-16310:
-
Perfect, thanks!
> ListOffsets doesn't report the offset with maxTimestamp anymore
> ---
>
> Key: KAFKA-16310
> URL: https://issues.apache.org/jira/browse/KAFKA-16310
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 3.7.0
>Reporter: Emanuele Sabellico
>Assignee: Chia-Ping Tsai
>Priority: Blocker
> Fix For: 3.8.0, 3.7.1
>
>
> Updated: This is confirmed a regression issue in v3.7.0.
> The impact of this issue is that when there is a batch containing records
> with timestamp not in order, the offset of the timestamp will be wrong.(ex:
> the timestamp for t0 should be mapping to offset 10, but will get offset 12..
> etc). It'll cause the time index is putting the wrong offset, so the result
> will be unexpected.
> ===
> The last offset is reported instead.
> A test in librdkafka (0081/do_test_ListOffsets) is failing an it's checking
> that the offset with the max timestamp is the middle one and not the last
> one. The tests is passing with 3.6.0 and previous versions
> This is the test:
> [https://github.com/confluentinc/librdkafka/blob/a6d85bdbc1023b1a5477b8befe516242c3e182f6/tests/0081-admin.c#L4989]
>
> there are three messages, with timestamps:
> {noformat}
> t0 + 100
> t0 + 400
> t0 + 250{noformat}
> and indices 0,1,2.
> then a ListOffsets with RD_KAFKA_OFFSET_SPEC_MAX_TIMESTAMP is done.
> it should return offset 1 but in 3.7.0 and trunk is returning offset 2
> Even after 5 seconds from producing it's still returning 2 as the offset with
> max timestamp.
> ProduceRequest and ListOffsets were sent to the same broker (2), the leader
> didn't change.
> {code:java}
> %7|1709134230.019|SEND|0081_admin#producer-3|
> [thrd:localhost:39951/bootstrap]: localhost:39951/2: Sent ProduceRequest (v7,
> 206 bytes @ 0, CorrId 2) %7|1709134230.020|RECV|0081_admin#producer-3|
> [thrd:localhost:39951/bootstrap]: localhost:39951/2: Received ProduceResponse
> (v7, 95 bytes, CorrId 2, rtt 1.18ms)
> %7|1709134230.020|MSGSET|0081_admin#producer-3|
> [thrd:localhost:39951/bootstrap]: localhost:39951/2:
> rdkafkatest_rnd22e8d8ec45b53f98_do_test_ListOffsets [0]: MessageSet with 3
> message(s) (MsgId 0, BaseSeq -1) delivered {code}
> {code:java}
> %7|1709134235.021|SEND|0081_admin#producer-2|
> [thrd:localhost:39951/bootstrap]: localhost:39951/2: Sent ListOffsetsRequest
> (v7, 103 bytes @ 0, CorrId 7) %7|1709134235.022|RECV|0081_admin#producer-2|
> [thrd:localhost:39951/bootstrap]: localhost:39951/2: Received
> ListOffsetsResponse (v7, 88 bytes, CorrId 7, rtt 0.54ms){code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-16310) ListOffsets doesn't report the offset with maxTimestamp anymore
[
https://issues.apache.org/jira/browse/KAFKA-16310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17831837#comment-17831837
]
Ismael Juma commented on KAFKA-16310:
-
Team, there are serious issues that are fixed by 3.6.2. We should not delay it
due to a long standing issue. As I said elsewhere, if it's not a regression or
a security issue, it should be avoided. Precisely to avoid the issue that
happened here. Let's revert and move on please.
> ListOffsets doesn't report the offset with maxTimestamp anymore
> ---
>
> Key: KAFKA-16310
> URL: https://issues.apache.org/jira/browse/KAFKA-16310
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 3.7.0
>Reporter: Emanuele Sabellico
>Assignee: Chia-Ping Tsai
>Priority: Blocker
> Fix For: 3.8.0, 3.7.1
>
>
> Updated: This is confirmed a regression issue in v3.7.0.
> The impact of this issue is that when there is a batch containing records
> with timestamp not in order, the offset of the timestamp will be wrong.(ex:
> the timestamp for t0 should be mapping to offset 10, but will get offset 12..
> etc). It'll cause the time index is putting the wrong offset, so the result
> will be unexpected.
> ===
> The last offset is reported instead.
> A test in librdkafka (0081/do_test_ListOffsets) is failing an it's checking
> that the offset with the max timestamp is the middle one and not the last
> one. The tests is passing with 3.6.0 and previous versions
> This is the test:
> [https://github.com/confluentinc/librdkafka/blob/a6d85bdbc1023b1a5477b8befe516242c3e182f6/tests/0081-admin.c#L4989]
>
> there are three messages, with timestamps:
> {noformat}
> t0 + 100
> t0 + 400
> t0 + 250{noformat}
> and indices 0,1,2.
> then a ListOffsets with RD_KAFKA_OFFSET_SPEC_MAX_TIMESTAMP is done.
> it should return offset 1 but in 3.7.0 and trunk is returning offset 2
> Even after 5 seconds from producing it's still returning 2 as the offset with
> max timestamp.
> ProduceRequest and ListOffsets were sent to the same broker (2), the leader
> didn't change.
> {code:java}
> %7|1709134230.019|SEND|0081_admin#producer-3|
> [thrd:localhost:39951/bootstrap]: localhost:39951/2: Sent ProduceRequest (v7,
> 206 bytes @ 0, CorrId 2) %7|1709134230.020|RECV|0081_admin#producer-3|
> [thrd:localhost:39951/bootstrap]: localhost:39951/2: Received ProduceResponse
> (v7, 95 bytes, CorrId 2, rtt 1.18ms)
> %7|1709134230.020|MSGSET|0081_admin#producer-3|
> [thrd:localhost:39951/bootstrap]: localhost:39951/2:
> rdkafkatest_rnd22e8d8ec45b53f98_do_test_ListOffsets [0]: MessageSet with 3
> message(s) (MsgId 0, BaseSeq -1) delivered {code}
> {code:java}
> %7|1709134235.021|SEND|0081_admin#producer-2|
> [thrd:localhost:39951/bootstrap]: localhost:39951/2: Sent ListOffsetsRequest
> (v7, 103 bytes @ 0, CorrId 7) %7|1709134235.022|RECV|0081_admin#producer-2|
> [thrd:localhost:39951/bootstrap]: localhost:39951/2: Received
> ListOffsetsResponse (v7, 88 bytes, CorrId 7, rtt 0.54ms){code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-16426) Initialise KafkaMetricsGroup statically
[
https://issues.apache.org/jira/browse/KAFKA-16426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17830953#comment-17830953
]
Ismael Juma commented on KAFKA-16426:
-
Can you elaborate on why this is an issue?
> Initialise KafkaMetricsGroup statically
> ---
>
> Key: KAFKA-16426
> URL: https://issues.apache.org/jira/browse/KAFKA-16426
> Project: Kafka
> Issue Type: Improvement
> Components: metrics
>Affects Versions: 3.7.0, 3.6.1
>Reporter: Gaurav Narula
>Assignee: Gaurav Narula
>Priority: Minor
> Fix For: 3.7.1
>
>
> KAFKA-14524 ([PR-13067|https://github.com/apache/kafka/pull/13067]) migrated
> {{KafkaMetricsGroup}} from Scala to Java. In doing so, we overlooked
> initialising {{KafkaMetricsGroup}} instances statically. This results in
> allocating many more {{KafkaMetricsGroup}} instances than are required.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-14482) Move LogLoader to storage module
[ https://issues.apache.org/jira/browse/KAFKA-14482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828514#comment-17828514 ] Ismael Juma commented on KAFKA-14482: - [~javakillah] This ticket can't be done until the dependency listed above. I have something in progress, but haven't submitted for that reason. > Move LogLoader to storage module > > > Key: KAFKA-14482 > URL: https://issues.apache.org/jira/browse/KAFKA-14482 > Project: Kafka > Issue Type: Sub-task >Reporter: Ismael Juma >Assignee: Dmitry Werner >Priority: Major > Fix For: 3.8.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16226) Java client: Performance regression in Trogdor benchmark with high partition counts
[
https://issues.apache.org/jira/browse/KAFKA-16226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismael Juma resolved KAFKA-16226.
-
Resolution: Fixed
> Java client: Performance regression in Trogdor benchmark with high partition
> counts
> ---
>
> Key: KAFKA-16226
> URL: https://issues.apache.org/jira/browse/KAFKA-16226
> Project: Kafka
> Issue Type: Bug
> Components: clients
>Affects Versions: 3.7.0, 3.6.1
>Reporter: Mayank Shekhar Narula
>Assignee: Mayank Shekhar Narula
>Priority: Major
> Labels: kip-951
> Fix For: 3.6.2, 3.8.0, 3.7.1
>
> Attachments: baseline_lock_profile.png, kafka_15415_lock_profile.png
>
>
> h1. Background
> https://issues.apache.org/jira/browse/KAFKA-15415 implemented optimisation in
> java-client to skip backoff period if client knows of a newer leader, for
> produce-batch being retried.
> h1. What changed
> The implementation introduced a regression noticed on a trogdor-benchmark
> running with high partition counts(36000!).
> With regression, following metrics changed on the produce side.
> # record-queue-time-avg: increased from 20ms to 30ms.
> # request-latency-avg: increased from 50ms to 100ms.
> h1. Why it happened
> As can be seen from the original
> [PR|https://github.com/apache/kafka/pull/14384]
> RecordAccmulator.partitionReady() & drainBatchesForOneNode() started using
> synchronised method Metadata.currentLeader(). This has led to increased
> synchronization between KafkaProducer's application-thread that call send(),
> and background-thread that actively send producer-batches to leaders.
> Lock profiles clearly show increased synchronisation in KAFKA-15415
> PR(highlighted in {color:#de350b}Red{color}) Vs baseline ( see below ). Note
> the synchronisation is much worse for paritionReady() in this benchmark as
> its called for each partition, and it has 36k partitions!
> h3. Lock Profile: Kafka-15415
> !kafka_15415_lock_profile.png!
> h3. Lock Profile: Baseline
> !baseline_lock_profile.png!
> h1. Fix
> Synchronization has to be reduced between 2 threads in order to address this.
> [https://github.com/apache/kafka/pull/15323] is a fix for it, as it avoids
> using Metadata.currentLeader() instead rely on Cluster.leaderFor().
> With the fix, lock-profile & metrics are similar to baseline.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-16359) kafka-clients-3.7.0.jar published to Maven Central is defective
[
https://issues.apache.org/jira/browse/KAFKA-16359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17825794#comment-17825794
]
Ismael Juma commented on KAFKA-16359:
-
[~norrisjeremy] it's clearly a bug - there is no debate about that. But it's
good to confirm the impact and whether there is a workaround. That will
influence the urgency for the new release (i.e. does it have to be immediate or
can it wait a week or two so that other important issues are fixed as well).
Thanks for confirming the specifics.
> kafka-clients-3.7.0.jar published to Maven Central is defective
> ---
>
> Key: KAFKA-16359
> URL: https://issues.apache.org/jira/browse/KAFKA-16359
> Project: Kafka
> Issue Type: Bug
> Components: clients
>Affects Versions: 3.7.0
>Reporter: Jeremy Norris
>Assignee: Apoorv Mittal
>Priority: Critical
>
> The {{kafka-clients-3.7.0.jar}} that has been published to Maven Central is
> defective: it's {{META-INF/MANIFEST.MF}} bogusly include a {{Class-Path}}
> element:
> {code}
> Manifest-Version: 1.0
> Class-Path: zstd-jni-1.5.5-6.jar lz4-java-1.8.0.jar snappy-java-1.1.10
> .5.jar slf4j-api-1.7.36.jar
> {code}
> This bogus {{Class-Path}} element leads to compiler warnings for projects
> that utilize it as a dependency:
> {code}
> [WARNING] [path] bad path element
> ".../.m2/repository/org/apache/kafka/kafka-clients/3.7.0/zstd-jni-1.5.5-6.jar":
> no such file or directory
> [WARNING] [path] bad path element
> ".../.m2/repository/org/apache/kafka/kafka-clients/3.7.0/lz4-java-1.8.0.jar":
> no such file or directory
> [WARNING] [path] bad path element
> ".../.m2/repository/org/apache/kafka/kafka-clients/3.7.0/snappy-java-1.1.10.5.jar":
> no such file or directory
> [WARNING] [path] bad path element
> ".../.m2/repository/org/apache/kafka/kafka-clients/3.7.0/slf4j-api-1.7.36.jar":
> no such file or directory
> {code}
> Either the {{kafka-clients-3.7.0.jar}} needs to be respun and published
> without the bogus {{Class-Path}} element in it's {{META-INF/MANIFEST.MF}} or
> a new release should be published that corrects this defect.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-16359) kafka-clients-3.7.0.jar published to Maven Central is defective
[
https://issues.apache.org/jira/browse/KAFKA-16359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17825767#comment-17825767
]
Ismael Juma commented on KAFKA-16359:
-
The original message said there were warnings. But the latest says the
artifacts are broken. Can you clarify the impact please?
> kafka-clients-3.7.0.jar published to Maven Central is defective
> ---
>
> Key: KAFKA-16359
> URL: https://issues.apache.org/jira/browse/KAFKA-16359
> Project: Kafka
> Issue Type: Bug
> Components: clients
>Affects Versions: 3.7.0
>Reporter: Jeremy Norris
>Assignee: Apoorv Mittal
>Priority: Critical
>
> The {{kafka-clients-3.7.0.jar}} that has been published to Maven Central is
> defective: it's {{META-INF/MANIFEST.MF}} bogusly include a {{Class-Path}}
> element:
> {code}
> Manifest-Version: 1.0
> Class-Path: zstd-jni-1.5.5-6.jar lz4-java-1.8.0.jar snappy-java-1.1.10
> .5.jar slf4j-api-1.7.36.jar
> {code}
> This bogus {{Class-Path}} element leads to compiler warnings for projects
> that utilize it as a dependency:
> {code}
> [WARNING] [path] bad path element
> ".../.m2/repository/org/apache/kafka/kafka-clients/3.7.0/zstd-jni-1.5.5-6.jar":
> no such file or directory
> [WARNING] [path] bad path element
> ".../.m2/repository/org/apache/kafka/kafka-clients/3.7.0/lz4-java-1.8.0.jar":
> no such file or directory
> [WARNING] [path] bad path element
> ".../.m2/repository/org/apache/kafka/kafka-clients/3.7.0/snappy-java-1.1.10.5.jar":
> no such file or directory
> [WARNING] [path] bad path element
> ".../.m2/repository/org/apache/kafka/kafka-clients/3.7.0/slf4j-api-1.7.36.jar":
> no such file or directory
> {code}
> Either the {{kafka-clients-3.7.0.jar}} needs to be respun and published
> without the bogus {{Class-Path}} element in it's {{META-INF/MANIFEST.MF}} or
> a new release should be published that corrects this defect.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-16349) ShutdownableThread fails build by calling Exit with race condition
[
https://issues.apache.org/jira/browse/KAFKA-16349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17824207#comment-17824207
]
Ismael Juma commented on KAFKA-16349:
-
Good catch.
> ShutdownableThread fails build by calling Exit with race condition
> --
>
> Key: KAFKA-16349
> URL: https://issues.apache.org/jira/browse/KAFKA-16349
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 3.8.0
>Reporter: Greg Harris
>Priority: Minor
>
> `ShutdownableThread` calls `Exit.exit()` when the thread's operation throws
> FatalExitError. In normal operation, this calls System.exit, and exits the
> process. In tests, the exit procedure is masked with Exit.setExitProcedure to
> prevent tests that encounter a FatalExitError from crashing the test JVM.
> Masking of exit procedures is usually done in BeforeEach/AfterEach
> annotations, with the exit procedures cleaned up immediately after the test
> finishes. If the body of the test creates a ShutdownableThread that outlives
> the test, such as by omitting `ShutdownableThread#awaitShutdown`, by having
> `ShutdownableThread#awaitShutdown` be interrupted by a test timeout, or by a
> race condition between `Exit.resetExitProcedure` and `Exit.exit`, then
> System.exit() can be called erroneously.
>
> {noformat}
> // First, in the test thread:
> Exit.setExitProcedure(...)
> try {
> new ShutdownableThread(...).start()
> } finally {
> Exit.resetExitProcedure()
> }
> // Second, in the ShutdownableThread:
> try {
> throw new FatalExitError(...)
> } catch (FatalExitError e) {
> Exit.exit(...) // Calls real System.exit()
> }{noformat}
>
> This can be resolved by one of the following:
> # Eliminate FatalExitError usages in code when setExitProcedure is in-use
> # Eliminate the Exit.exit call from ShutdownableThread, and instead
> propagate this error to another thread to handle without a race-condition
> # Eliminate resetExitProcedure by refactoring Exit to be non-static
> FatalExitError is in use in a small number of places, but may be difficult to
> eliminate:
> * FinalizedFeatureChangeListener
> * InterBrokerSendThread
> * TopicBasedRemoteLogMetadataManager
> It appears that every other use of Exit.setExitProcedure/Exit.exit is done on
> a single thread, so ShutdownableThread is the only place where this race
> condition is present.
> The effect of this bug is that the build is flaky, as race
> conditions/timeouts in tests can cause the gradle executors to exit with
> status code 1, which has happened 26 times in the last 28 days. I have not
> yet been able to confirm this bug is happening in other tests, but I do have
> a deterministic reproduction case with the exact same symptoms:
> {noformat}
> Gradle Test Run :core:test > Gradle Test Executor 38 > ShutdownableThreadTest
> > testShutdownWhenTestTimesOut(boolean) >
> "testShutdownWhenTestTimesOut(boolean).isInterruptible=true" SKIPPED
> FAILURE: Build failed with an exception.
> * What went wrong:
> Execution failed for task ':core:test'.
> > Process 'Gradle Test Executor 38' finished with non-zero exit value 1
> This problem might be caused by incorrect test process configuration.
> For more on test execution, please refer to
> https://docs.gradle.org/8.6/userguide/java_testing.html#sec:test_execution in
> the Gradle documentation.{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (KAFKA-16310) ListOffsets doesn't report the offset with maxTimestamp anymore
[
https://issues.apache.org/jira/browse/KAFKA-16310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17823266#comment-17823266
]
Ismael Juma edited comment on KAFKA-16310 at 3/4/24 5:31 PM:
-
Thanks for confirming that.
[https://github.com/apache/kafka/commit/e905ef1edfb92d8771e8c2a9a668f32210ad7e07]
is indeed the only relevant regression then. We should take the chance to fix
the regression and the previously existing issue.
was (Author: ijuma):
Thanks for confirming that.
[https://github.com/apache/kafka/commit/e905ef1edfb92d8771e8c2a9a668f32210ad7e07]
is indeed the only relevant regression then. We should take the chance to fix
both.
> ListOffsets doesn't report the offset with maxTimestamp anymore
> ---
>
> Key: KAFKA-16310
> URL: https://issues.apache.org/jira/browse/KAFKA-16310
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 3.7.0
>Reporter: Emanuele Sabellico
>Priority: Blocker
> Fix For: 3.7.1
>
>
> Updated: This is confirmed a regression issue in v3.7.0.
> The impact of this issue is that when there is a batch containing records
> with timestamp not in order, the offset of the timestamp will be wrong.(ex:
> the timestamp for t0 should be mapping to offset 10, but will get offset 12..
> etc). It'll cause the time index is putting the wrong offset, so the result
> will be unexpected.
> ===
> The last offset is reported instead.
> A test in librdkafka (0081/do_test_ListOffsets) is failing an it's checking
> that the offset with the max timestamp is the middle one and not the last
> one. The tests is passing with 3.6.0 and previous versions
> This is the test:
> [https://github.com/confluentinc/librdkafka/blob/a6d85bdbc1023b1a5477b8befe516242c3e182f6/tests/0081-admin.c#L4989]
>
> there are three messages, with timestamps:
> {noformat}
> t0 + 100
> t0 + 400
> t0 + 250{noformat}
> and indices 0,1,2.
> then a ListOffsets with RD_KAFKA_OFFSET_SPEC_MAX_TIMESTAMP is done.
> it should return offset 1 but in 3.7.0 and trunk is returning offset 2
> Even after 5 seconds from producing it's still returning 2 as the offset with
> max timestamp.
> ProduceRequest and ListOffsets were sent to the same broker (2), the leader
> didn't change.
> {code:java}
> %7|1709134230.019|SEND|0081_admin#producer-3|
> [thrd:localhost:39951/bootstrap]: localhost:39951/2: Sent ProduceRequest (v7,
> 206 bytes @ 0, CorrId 2) %7|1709134230.020|RECV|0081_admin#producer-3|
> [thrd:localhost:39951/bootstrap]: localhost:39951/2: Received ProduceResponse
> (v7, 95 bytes, CorrId 2, rtt 1.18ms)
> %7|1709134230.020|MSGSET|0081_admin#producer-3|
> [thrd:localhost:39951/bootstrap]: localhost:39951/2:
> rdkafkatest_rnd22e8d8ec45b53f98_do_test_ListOffsets [0]: MessageSet with 3
> message(s) (MsgId 0, BaseSeq -1) delivered {code}
> {code:java}
> %7|1709134235.021|SEND|0081_admin#producer-2|
> [thrd:localhost:39951/bootstrap]: localhost:39951/2: Sent ListOffsetsRequest
> (v7, 103 bytes @ 0, CorrId 7) %7|1709134235.022|RECV|0081_admin#producer-2|
> [thrd:localhost:39951/bootstrap]: localhost:39951/2: Received
> ListOffsetsResponse (v7, 88 bytes, CorrId 7, rtt 0.54ms){code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-16310) ListOffsets doesn't report the offset with maxTimestamp anymore
[
https://issues.apache.org/jira/browse/KAFKA-16310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17823266#comment-17823266
]
Ismael Juma commented on KAFKA-16310:
-
Thanks for confirming that.
[https://github.com/apache/kafka/commit/e905ef1edfb92d8771e8c2a9a668f32210ad7e07]
is indeed the only relevant regression then. We should take the chance to fix
both.
> ListOffsets doesn't report the offset with maxTimestamp anymore
> ---
>
> Key: KAFKA-16310
> URL: https://issues.apache.org/jira/browse/KAFKA-16310
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 3.7.0
>Reporter: Emanuele Sabellico
>Priority: Blocker
> Fix For: 3.7.1
>
>
> Updated: This is confirmed a regression issue in v3.7.0.
> The impact of this issue is that when there is a batch containing records
> with timestamp not in order, the offset of the timestamp will be wrong.(ex:
> the timestamp for t0 should be mapping to offset 10, but will get offset 12..
> etc). It'll cause the time index is putting the wrong offset, so the result
> will be unexpected.
> ===
> The last offset is reported instead.
> A test in librdkafka (0081/do_test_ListOffsets) is failing an it's checking
> that the offset with the max timestamp is the middle one and not the last
> one. The tests is passing with 3.6.0 and previous versions
> This is the test:
> [https://github.com/confluentinc/librdkafka/blob/a6d85bdbc1023b1a5477b8befe516242c3e182f6/tests/0081-admin.c#L4989]
>
> there are three messages, with timestamps:
> {noformat}
> t0 + 100
> t0 + 400
> t0 + 250{noformat}
> and indices 0,1,2.
> then a ListOffsets with RD_KAFKA_OFFSET_SPEC_MAX_TIMESTAMP is done.
> it should return offset 1 but in 3.7.0 and trunk is returning offset 2
> Even after 5 seconds from producing it's still returning 2 as the offset with
> max timestamp.
> ProduceRequest and ListOffsets were sent to the same broker (2), the leader
> didn't change.
> {code:java}
> %7|1709134230.019|SEND|0081_admin#producer-3|
> [thrd:localhost:39951/bootstrap]: localhost:39951/2: Sent ProduceRequest (v7,
> 206 bytes @ 0, CorrId 2) %7|1709134230.020|RECV|0081_admin#producer-3|
> [thrd:localhost:39951/bootstrap]: localhost:39951/2: Received ProduceResponse
> (v7, 95 bytes, CorrId 2, rtt 1.18ms)
> %7|1709134230.020|MSGSET|0081_admin#producer-3|
> [thrd:localhost:39951/bootstrap]: localhost:39951/2:
> rdkafkatest_rnd22e8d8ec45b53f98_do_test_ListOffsets [0]: MessageSet with 3
> message(s) (MsgId 0, BaseSeq -1) delivered {code}
> {code:java}
> %7|1709134235.021|SEND|0081_admin#producer-2|
> [thrd:localhost:39951/bootstrap]: localhost:39951/2: Sent ListOffsetsRequest
> (v7, 103 bytes @ 0, CorrId 7) %7|1709134235.022|RECV|0081_admin#producer-2|
> [thrd:localhost:39951/bootstrap]: localhost:39951/2: Received
> ListOffsetsResponse (v7, 88 bytes, CorrId 7, rtt 0.54ms){code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-16310) ListOffsets doesn't report the offset with maxTimestamp anymore
[
https://issues.apache.org/jira/browse/KAFKA-16310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17823208#comment-17823208
]
Ismael Juma commented on KAFKA-16310:
-
More recent changes to LogValidator:
[https://github.com/apache/kafka/commit/e905ef1edfb92d8771e8c2a9a668f32210ad7e07]
My understanding is that the following change caused the change to the non
compressed case. It looks like the compressed case had the issue already, but
we didn't go further back to see if that has always been the case or not.
> ListOffsets doesn't report the offset with maxTimestamp anymore
> ---
>
> Key: KAFKA-16310
> URL: https://issues.apache.org/jira/browse/KAFKA-16310
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 3.7.0
>Reporter: Emanuele Sabellico
>Priority: Blocker
> Fix For: 3.7.1
>
>
> Updated: This is confirmed a regression issue in v3.7.0.
> The impact of this issue is that when there is a batch containing records
> with timestamp not in order, the offset of the timestamp will be wrong.(ex:
> the timestamp for t0 should be mapping to offset 10, but will get offset 12..
> etc). It'll cause the time index is putting the wrong offset, so the result
> will be unexpected.
> ===
> The last offset is reported instead.
> A test in librdkafka (0081/do_test_ListOffsets) is failing an it's checking
> that the offset with the max timestamp is the middle one and not the last
> one. The tests is passing with 3.6.0 and previous versions
> This is the test:
> [https://github.com/confluentinc/librdkafka/blob/a6d85bdbc1023b1a5477b8befe516242c3e182f6/tests/0081-admin.c#L4989]
>
> there are three messages, with timestamps:
> {noformat}
> t0 + 100
> t0 + 400
> t0 + 250{noformat}
> and indices 0,1,2.
> then a ListOffsets with RD_KAFKA_OFFSET_SPEC_MAX_TIMESTAMP is done.
> it should return offset 1 but in 3.7.0 and trunk is returning offset 2
> Even after 5 seconds from producing it's still returning 2 as the offset with
> max timestamp.
> ProduceRequest and ListOffsets were sent to the same broker (2), the leader
> didn't change.
> {code:java}
> %7|1709134230.019|SEND|0081_admin#producer-3|
> [thrd:localhost:39951/bootstrap]: localhost:39951/2: Sent ProduceRequest (v7,
> 206 bytes @ 0, CorrId 2) %7|1709134230.020|RECV|0081_admin#producer-3|
> [thrd:localhost:39951/bootstrap]: localhost:39951/2: Received ProduceResponse
> (v7, 95 bytes, CorrId 2, rtt 1.18ms)
> %7|1709134230.020|MSGSET|0081_admin#producer-3|
> [thrd:localhost:39951/bootstrap]: localhost:39951/2:
> rdkafkatest_rnd22e8d8ec45b53f98_do_test_ListOffsets [0]: MessageSet with 3
> message(s) (MsgId 0, BaseSeq -1) delivered {code}
> {code:java}
> %7|1709134235.021|SEND|0081_admin#producer-2|
> [thrd:localhost:39951/bootstrap]: localhost:39951/2: Sent ListOffsetsRequest
> (v7, 103 bytes @ 0, CorrId 7) %7|1709134235.022|RECV|0081_admin#producer-2|
> [thrd:localhost:39951/bootstrap]: localhost:39951/2: Received
> ListOffsetsResponse (v7, 88 bytes, CorrId 7, rtt 0.54ms){code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (KAFKA-16310) ListOffsets doesn't report the offset with maxTimestamp anymore
[
https://issues.apache.org/jira/browse/KAFKA-16310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17823205#comment-17823205
]
Ismael Juma edited comment on KAFKA-16310 at 3/4/24 2:23 PM:
-
By the way, the bug attribution doesn't seem completely right (or not right at
all). There were more recent changes to LogValidator after the conversion to
Java that touched the related area.
was (Author: ijuma):
By the way, the bug attribution doesn't seem completely right. There were more
recent changes to LogValidator after the conversion to Java that touched the
related area.
> ListOffsets doesn't report the offset with maxTimestamp anymore
> ---
>
> Key: KAFKA-16310
> URL: https://issues.apache.org/jira/browse/KAFKA-16310
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 3.7.0
>Reporter: Emanuele Sabellico
>Priority: Blocker
> Fix For: 3.7.1
>
>
> Updated: This is confirmed a regression issue in v3.7.0.
> The impact of this issue is that when there is a batch containing records
> with timestamp not in order, the offset of the timestamp will be wrong.(ex:
> the timestamp for t0 should be mapping to offset 10, but will get offset 12..
> etc). It'll cause the time index is putting the wrong offset, so the result
> will be unexpected.
> ===
> The last offset is reported instead.
> A test in librdkafka (0081/do_test_ListOffsets) is failing an it's checking
> that the offset with the max timestamp is the middle one and not the last
> one. The tests is passing with 3.6.0 and previous versions
> This is the test:
> [https://github.com/confluentinc/librdkafka/blob/a6d85bdbc1023b1a5477b8befe516242c3e182f6/tests/0081-admin.c#L4989]
>
> there are three messages, with timestamps:
> {noformat}
> t0 + 100
> t0 + 400
> t0 + 250{noformat}
> and indices 0,1,2.
> then a ListOffsets with RD_KAFKA_OFFSET_SPEC_MAX_TIMESTAMP is done.
> it should return offset 1 but in 3.7.0 and trunk is returning offset 2
> Even after 5 seconds from producing it's still returning 2 as the offset with
> max timestamp.
> ProduceRequest and ListOffsets were sent to the same broker (2), the leader
> didn't change.
> {code:java}
> %7|1709134230.019|SEND|0081_admin#producer-3|
> [thrd:localhost:39951/bootstrap]: localhost:39951/2: Sent ProduceRequest (v7,
> 206 bytes @ 0, CorrId 2) %7|1709134230.020|RECV|0081_admin#producer-3|
> [thrd:localhost:39951/bootstrap]: localhost:39951/2: Received ProduceResponse
> (v7, 95 bytes, CorrId 2, rtt 1.18ms)
> %7|1709134230.020|MSGSET|0081_admin#producer-3|
> [thrd:localhost:39951/bootstrap]: localhost:39951/2:
> rdkafkatest_rnd22e8d8ec45b53f98_do_test_ListOffsets [0]: MessageSet with 3
> message(s) (MsgId 0, BaseSeq -1) delivered {code}
> {code:java}
> %7|1709134235.021|SEND|0081_admin#producer-2|
> [thrd:localhost:39951/bootstrap]: localhost:39951/2: Sent ListOffsetsRequest
> (v7, 103 bytes @ 0, CorrId 7) %7|1709134235.022|RECV|0081_admin#producer-2|
> [thrd:localhost:39951/bootstrap]: localhost:39951/2: Received
> ListOffsetsResponse (v7, 88 bytes, CorrId 7, rtt 0.54ms){code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (KAFKA-16310) ListOffsets doesn't report the offset with maxTimestamp anymore
[
https://issues.apache.org/jira/browse/KAFKA-16310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17823205#comment-17823205
]
Ismael Juma edited comment on KAFKA-16310 at 3/4/24 2:13 PM:
-
By the way, the bug attribution doesn't seem completely right. There were more
recent changes to LogValidator after the conversion to Java that touched the
related area.
was (Author: ijuma):
By the way, the bug attribution doesn't seem completely right. There were more
recent changes to LogValidator after the conversion to Java.
> ListOffsets doesn't report the offset with maxTimestamp anymore
> ---
>
> Key: KAFKA-16310
> URL: https://issues.apache.org/jira/browse/KAFKA-16310
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 3.7.0
>Reporter: Emanuele Sabellico
>Priority: Blocker
> Fix For: 3.7.1
>
>
> Updated: This is confirmed a regression issue in v3.7.0.
> The impact of this issue is that when there is a batch containing records
> with timestamp not in order, the offset of the timestamp will be wrong.(ex:
> the timestamp for t0 should be mapping to offset 10, but will get offset 12..
> etc). It'll cause the time index is putting the wrong offset, so the result
> will be unexpected.
> ===
> The last offset is reported instead.
> A test in librdkafka (0081/do_test_ListOffsets) is failing an it's checking
> that the offset with the max timestamp is the middle one and not the last
> one. The tests is passing with 3.6.0 and previous versions
> This is the test:
> [https://github.com/confluentinc/librdkafka/blob/a6d85bdbc1023b1a5477b8befe516242c3e182f6/tests/0081-admin.c#L4989]
>
> there are three messages, with timestamps:
> {noformat}
> t0 + 100
> t0 + 400
> t0 + 250{noformat}
> and indices 0,1,2.
> then a ListOffsets with RD_KAFKA_OFFSET_SPEC_MAX_TIMESTAMP is done.
> it should return offset 1 but in 3.7.0 and trunk is returning offset 2
> Even after 5 seconds from producing it's still returning 2 as the offset with
> max timestamp.
> ProduceRequest and ListOffsets were sent to the same broker (2), the leader
> didn't change.
> {code:java}
> %7|1709134230.019|SEND|0081_admin#producer-3|
> [thrd:localhost:39951/bootstrap]: localhost:39951/2: Sent ProduceRequest (v7,
> 206 bytes @ 0, CorrId 2) %7|1709134230.020|RECV|0081_admin#producer-3|
> [thrd:localhost:39951/bootstrap]: localhost:39951/2: Received ProduceResponse
> (v7, 95 bytes, CorrId 2, rtt 1.18ms)
> %7|1709134230.020|MSGSET|0081_admin#producer-3|
> [thrd:localhost:39951/bootstrap]: localhost:39951/2:
> rdkafkatest_rnd22e8d8ec45b53f98_do_test_ListOffsets [0]: MessageSet with 3
> message(s) (MsgId 0, BaseSeq -1) delivered {code}
> {code:java}
> %7|1709134235.021|SEND|0081_admin#producer-2|
> [thrd:localhost:39951/bootstrap]: localhost:39951/2: Sent ListOffsetsRequest
> (v7, 103 bytes @ 0, CorrId 7) %7|1709134235.022|RECV|0081_admin#producer-2|
> [thrd:localhost:39951/bootstrap]: localhost:39951/2: Received
> ListOffsetsResponse (v7, 88 bytes, CorrId 7, rtt 0.54ms){code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-16310) ListOffsets doesn't report the offset with maxTimestamp anymore
[
https://issues.apache.org/jira/browse/KAFKA-16310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17823205#comment-17823205
]
Ismael Juma commented on KAFKA-16310:
-
By the way, the bug attribution doesn't seem completely right. There were more
recent changes to LogValidator after the conversion to Java.
> ListOffsets doesn't report the offset with maxTimestamp anymore
> ---
>
> Key: KAFKA-16310
> URL: https://issues.apache.org/jira/browse/KAFKA-16310
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 3.7.0
>Reporter: Emanuele Sabellico
>Priority: Blocker
> Fix For: 3.7.1
>
>
> Updated: This is confirmed a regression issue in v3.7.0.
> The impact of this issue is that when there is a batch containing records
> with timestamp not in order, the offset of the timestamp will be wrong.(ex:
> the timestamp for t0 should be mapping to offset 10, but will get offset 12..
> etc). It'll cause the time index is putting the wrong offset, so the result
> will be unexpected.
> ===
> The last offset is reported instead.
> A test in librdkafka (0081/do_test_ListOffsets) is failing an it's checking
> that the offset with the max timestamp is the middle one and not the last
> one. The tests is passing with 3.6.0 and previous versions
> This is the test:
> [https://github.com/confluentinc/librdkafka/blob/a6d85bdbc1023b1a5477b8befe516242c3e182f6/tests/0081-admin.c#L4989]
>
> there are three messages, with timestamps:
> {noformat}
> t0 + 100
> t0 + 400
> t0 + 250{noformat}
> and indices 0,1,2.
> then a ListOffsets with RD_KAFKA_OFFSET_SPEC_MAX_TIMESTAMP is done.
> it should return offset 1 but in 3.7.0 and trunk is returning offset 2
> Even after 5 seconds from producing it's still returning 2 as the offset with
> max timestamp.
> ProduceRequest and ListOffsets were sent to the same broker (2), the leader
> didn't change.
> {code:java}
> %7|1709134230.019|SEND|0081_admin#producer-3|
> [thrd:localhost:39951/bootstrap]: localhost:39951/2: Sent ProduceRequest (v7,
> 206 bytes @ 0, CorrId 2) %7|1709134230.020|RECV|0081_admin#producer-3|
> [thrd:localhost:39951/bootstrap]: localhost:39951/2: Received ProduceResponse
> (v7, 95 bytes, CorrId 2, rtt 1.18ms)
> %7|1709134230.020|MSGSET|0081_admin#producer-3|
> [thrd:localhost:39951/bootstrap]: localhost:39951/2:
> rdkafkatest_rnd22e8d8ec45b53f98_do_test_ListOffsets [0]: MessageSet with 3
> message(s) (MsgId 0, BaseSeq -1) delivered {code}
> {code:java}
> %7|1709134235.021|SEND|0081_admin#producer-2|
> [thrd:localhost:39951/bootstrap]: localhost:39951/2: Sent ListOffsetsRequest
> (v7, 103 bytes @ 0, CorrId 7) %7|1709134235.022|RECV|0081_admin#producer-2|
> [thrd:localhost:39951/bootstrap]: localhost:39951/2: Received
> ListOffsetsResponse (v7, 88 bytes, CorrId 7, rtt 0.54ms){code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-16310) ListOffsets doesn't report the offset with maxTimestamp anymore
[
https://issues.apache.org/jira/browse/KAFKA-16310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17822991#comment-17822991
]
Ismael Juma commented on KAFKA-16310:
-
Yes, [~jolshan] and [~hachikuji] looked into this and arrived to a similar
conclusion. I think one of them was going to submit a fix soon.
> ListOffsets doesn't report the offset with maxTimestamp anymore
> ---
>
> Key: KAFKA-16310
> URL: https://issues.apache.org/jira/browse/KAFKA-16310
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 3.7.0
>Reporter: Emanuele Sabellico
>Assignee: Cheng-Kai, Zhang
>Priority: Minor
>
> The last offset is reported instead.
> A test in librdkafka (0081/do_test_ListOffsets) is failing an it's checking
> that the offset with the max timestamp is the middle one and not the last
> one. The tests is passing with 3.6.0 and previous versions
> This is the test:
> [https://github.com/confluentinc/librdkafka/blob/a6d85bdbc1023b1a5477b8befe516242c3e182f6/tests/0081-admin.c#L4989]
>
> there are three messages, with timestamps:
> {noformat}
> t0 + 100
> t0 + 400
> t0 + 250{noformat}
> and indices 0,1,2.
> then a ListOffsets with RD_KAFKA_OFFSET_SPEC_MAX_TIMESTAMP is done.
> it should return offset 1 but in 3.7.0 and trunk is returning offset 2
> Even after 5 seconds from producing it's still returning 2 as the offset with
> max timestamp.
> ProduceRequest and ListOffsets were sent to the same broker (2), the leader
> didn't change.
> {code:java}
> %7|1709134230.019|SEND|0081_admin#producer-3|
> [thrd:localhost:39951/bootstrap]: localhost:39951/2: Sent ProduceRequest (v7,
> 206 bytes @ 0, CorrId 2) %7|1709134230.020|RECV|0081_admin#producer-3|
> [thrd:localhost:39951/bootstrap]: localhost:39951/2: Received ProduceResponse
> (v7, 95 bytes, CorrId 2, rtt 1.18ms)
> %7|1709134230.020|MSGSET|0081_admin#producer-3|
> [thrd:localhost:39951/bootstrap]: localhost:39951/2:
> rdkafkatest_rnd22e8d8ec45b53f98_do_test_ListOffsets [0]: MessageSet with 3
> message(s) (MsgId 0, BaseSeq -1) delivered {code}
> {code:java}
> %7|1709134235.021|SEND|0081_admin#producer-2|
> [thrd:localhost:39951/bootstrap]: localhost:39951/2: Sent ListOffsetsRequest
> (v7, 103 bytes @ 0, CorrId 7) %7|1709134235.022|RECV|0081_admin#producer-2|
> [thrd:localhost:39951/bootstrap]: localhost:39951/2: Received
> ListOffsetsResponse (v7, 88 bytes, CorrId 7, rtt 0.54ms){code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-16292) Revamp upgrade.html page
[ https://issues.apache.org/jira/browse/KAFKA-16292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17819258#comment-17819258 ] Ismael Juma commented on KAFKA-16292: - I agree that having the upgrade details for so many versions is excessive, but it's actually much easier to read one page than several different pages (very few people upgrade after every release) - perhaps there is something in the middle where we can keep the releases for a couple of years in a single page. > Revamp upgrade.html page > - > > Key: KAFKA-16292 > URL: https://issues.apache.org/jira/browse/KAFKA-16292 > Project: Kafka > Issue Type: Improvement > Components: documentation >Reporter: Mickael Maison >Priority: Major > > At the moment we keep adding to this page for each release. The upgrade.html > file is now over 2000 line long. It still contains steps for upgrading from > 0.8 to 0.9! These steps are already in the docs for each version which can be > accessed via //documentation.html. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16244) Move code style exceptions from suppressions.xml to the code
[ https://issues.apache.org/jira/browse/KAFKA-16244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17816631#comment-17816631 ] Ismael Juma commented on KAFKA-16244: - This implies adding the checkstyle annotations to the code - worth making sure the license is ok. There was an issue with the annotations for findBugs, but checkstyle may be OK. > Move code style exceptions from suppressions.xml to the code > > > Key: KAFKA-16244 > URL: https://issues.apache.org/jira/browse/KAFKA-16244 > Project: Kafka > Issue Type: Sub-task >Reporter: David Jacot >Assignee: David Jacot >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-10140) Incremental config api excludes plugin config changes
[
https://issues.apache.org/jira/browse/KAFKA-10140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17809511#comment-17809511
]
Ismael Juma commented on KAFKA-10140:
-
[~chia7712] That makes sense to me.
> Incremental config api excludes plugin config changes
> -
>
> Key: KAFKA-10140
> URL: https://issues.apache.org/jira/browse/KAFKA-10140
> Project: Kafka
> Issue Type: Bug
>Reporter: Jason Gustafson
>Priority: Critical
>
> I was trying to alter the jmx metric filters using the incremental alter
> config api and hit this error:
> {code:java}
> java.util.NoSuchElementException: key not found: metrics.jmx.blacklist
> at scala.collection.MapLike.default(MapLike.scala:235)
> at scala.collection.MapLike.default$(MapLike.scala:234)
> at scala.collection.AbstractMap.default(Map.scala:65)
> at scala.collection.MapLike.apply(MapLike.scala:144)
> at scala.collection.MapLike.apply$(MapLike.scala:143)
> at scala.collection.AbstractMap.apply(Map.scala:65)
> at kafka.server.AdminManager.listType$1(AdminManager.scala:681)
> at
> kafka.server.AdminManager.$anonfun$prepareIncrementalConfigs$1(AdminManager.scala:693)
> at kafka.server.AdminManager.prepareIncrementalConfigs(AdminManager.scala:687)
> at
> kafka.server.AdminManager.$anonfun$incrementalAlterConfigs$1(AdminManager.scala:618)
> at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:273)
> at scala.collection.immutable.Map$Map1.foreach(Map.scala:154)
> at scala.collection.TraversableLike.map(TraversableLike.scala:273)
> at scala.collection.TraversableLike.map$(TraversableLike.scala:266)
> at scala.collection.AbstractTraversable.map(Traversable.scala:108)
> at kafka.server.AdminManager.incrementalAlterConfigs(AdminManager.scala:589)
> at
> kafka.server.KafkaApis.handleIncrementalAlterConfigsRequest(KafkaApis.scala:2698)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:188)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:78)
> at java.base/java.lang.Thread.run(Thread.java:834) {code}
> It looks like we are only allowing changes to the keys defined in
> `KafkaConfig` through this API. This excludes config changes to any plugin
> components such as `JmxReporter`.
> Note that I was able to use the regular `alterConfig` API to change this
> config.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-16180) Full metadata request sometimes fails during zk migration
[
https://issues.apache.org/jira/browse/KAFKA-16180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismael Juma updated KAFKA-16180:
Description:
Example:
{code:java}
java.util.NoSuchElementException: topic_name
at
scala.collection.mutable.AnyRefMap$ExceptionDefault.apply(AnyRefMap.scala:508)
at
scala.collection.mutable.AnyRefMap$ExceptionDefault.apply(AnyRefMap.scala:507)
at scala.collection.mutable.AnyRefMap.apply(AnyRefMap.scala:207)
at
kafka.server.metadata.ZkMetadataCache$.$anonfun$maybeInjectDeletedPartitionsFromFullMetadataRequest$2(ZkMetadataCache.scala:112)
at
kafka.server.metadata.ZkMetadataCache$.$anonfun$maybeInjectDeletedPartitionsFromFullMetadataRequest$2$adapted(ZkMetadataCache.scala:105)
at scala.collection.immutable.HashSet.foreach(HashSet.scala:958)
at
kafka.server.metadata.ZkMetadataCache$.maybeInjectDeletedPartitionsFromFullMetadataRequest(ZkMetadataCache.scala:105)
at
kafka.server.metadata.ZkMetadataCache.$anonfun$updateMetadata$1(ZkMetadataCache.scala:506)
at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:183)
at
kafka.server.metadata.ZkMetadataCache.updateMetadata(ZkMetadataCache.scala:496)
at
kafka.server.ReplicaManager.maybeUpdateMetadataCache(ReplicaManager.scala:2482)
at
kafka.server.KafkaApis.handleUpdateMetadataRequest(KafkaApis.scala:733)
at kafka.server.KafkaApis.handle(KafkaApis.scala:349)
at
kafka.server.KafkaRequestHandler.$anonfun$poll$8(KafkaRequestHandler.scala:210)
at
kafka.server.KafkaRequestHandler.$anonfun$poll$8$adapted(KafkaRequestHandler.scala:210)
at
io.confluent.kafka.availability.ThreadCountersManager.wrapEngine(ThreadCountersManager.java:146)
at kafka.server.KafkaRequestHandler.poll(KafkaRequestHandler.scala:210)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:151)
at java.base/java.lang.Thread.run(Thread.java:1583)
at org.apache.kafka.common.utils.KafkaThread.run(KafkaThread.java:66)
{code}
was:
Example:
{code}
java.util.NoSuchElementException:
dev_kafka.stream.detection.bucket3-KTABLE-SUPPRESS-STATE-STORE-08-changelog
at
scala.collection.mutable.AnyRefMap$ExceptionDefault.apply(AnyRefMap.scala:508)
at
scala.collection.mutable.AnyRefMap$ExceptionDefault.apply(AnyRefMap.scala:507)
at scala.collection.mutable.AnyRefMap.apply(AnyRefMap.scala:207)
at
kafka.server.metadata.ZkMetadataCache$.$anonfun$maybeInjectDeletedPartitionsFromFullMetadataRequest$2(ZkMetadataCache.scala:112)
at
kafka.server.metadata.ZkMetadataCache$.$anonfun$maybeInjectDeletedPartitionsFromFullMetadataRequest$2$adapted(ZkMetadataCache.scala:105)
at scala.collection.immutable.HashSet.foreach(HashSet.scala:958)
at
kafka.server.metadata.ZkMetadataCache$.maybeInjectDeletedPartitionsFromFullMetadataRequest(ZkMetadataCache.scala:105)
at
kafka.server.metadata.ZkMetadataCache.$anonfun$updateMetadata$1(ZkMetadataCache.scala:506)
at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:183)
at
kafka.server.metadata.ZkMetadataCache.updateMetadata(ZkMetadataCache.scala:496)
at
kafka.server.ReplicaManager.maybeUpdateMetadataCache(ReplicaManager.scala:2482)
at
kafka.server.KafkaApis.handleUpdateMetadataRequest(KafkaApis.scala:733)
at kafka.server.KafkaApis.handle(KafkaApis.scala:349)
at
kafka.server.KafkaRequestHandler.$anonfun$poll$8(KafkaRequestHandler.scala:210)
at
kafka.server.KafkaRequestHandler.$anonfun$poll$8$adapted(KafkaRequestHandler.scala:210)
at
io.confluent.kafka.availability.ThreadCountersManager.wrapEngine(ThreadCountersManager.java:146)
at kafka.server.KafkaRequestHandler.poll(KafkaRequestHandler.scala:210)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:151)
at java.base/java.lang.Thread.run(Thread.java:1583)
at org.apache.kafka.common.utils.KafkaThread.run(KafkaThread.java:66)
{code}
> Full metadata request sometimes fails during zk migration
> -
>
> Key: KAFKA-16180
> URL: https://issues.apache.org/jira/browse/KAFKA-16180
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 3.7.0
>Reporter: Colin McCabe
>Priority: Blocker
>
> Example:
> {code:java}
> java.util.NoSuchElementException: topic_name
> at
> scala.collection.mutable.AnyRefMap$ExceptionDefault.apply(AnyRefMap.scala:508)
> at
> scala.collection.mutable.AnyRefMap$ExceptionDefault.apply(AnyRefMap.scala:507)
> at scala.collection.mutable.AnyRefMap.apply(AnyRefMap.scala:207)
> at
> kafka.server.metadata.ZkMetadataCache$.$anonfun$maybeInjectDeletedPartitionsFromFullMetadataRequest$2(ZkMetadataCache.scala:112)
[jira] [Commented] (KAFKA-16132) Upgrading from 3.6 to 3.7 in KRaft will have seconds of partitions unavailable
[
https://issues.apache.org/jira/browse/KAFKA-16132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17807046#comment-17807046
]
Ismael Juma commented on KAFKA-16132:
-
OK, marked it as a blocker for now.
> Upgrading from 3.6 to 3.7 in KRaft will have seconds of partitions unavailable
> --
>
> Key: KAFKA-16132
> URL: https://issues.apache.org/jira/browse/KAFKA-16132
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 3.7.0
>Reporter: Luke Chen
>Priority: Blocker
>
> When upgrading from 3.6 to 3.7, we noticed that after upgrade the metadata
> version, all the partitions will be reset at one time, which causes a short
> period of time unavailable. This doesn't happen before.
> {code:java}
> [2024-01-15 20:45:19,757] INFO [BrokerMetadataPublisher id=2] Updating
> metadata.version to 19 at offset OffsetAndEpoch(offset=229, epoch=2).
> (kafka.server.metadata.BrokerMetadataPublisher)
> [2024-01-15 20:45:29,915] INFO [ReplicaFetcherManager on broker 2] Removed
> fetcher for partitions Set(t1-29, t1-25, t1-21, t1-17, t1-46, t1-13, t1-42,
> t1-9, t1-38, t1-5, t1-34, t1-1, t1-30, t1-26, t1-22, t1-18, t1-47, t1-14,
> t1-43, t1-10, t1-39, t1-6, t1-35, t1-2, t1-31, t1-27, t1-23, t1-19, t1-48,
> t1-15, t1-44, t1-11, t1-40, t1-7, t1-36, t1-3, t1-32, t1-28, t1-24, t1-20,
> t1-49, t1-16, t1-45, t1-12, t1-41, t1-8, t1-37, t1-4, t1-33, t1-0)
> (kafka.server.ReplicaFetcherManager)
> {code}
> Complete log:
> https://gist.github.com/showuon/665aa3ce6afd59097a2662f8260ecc10
> Steps:
> 1. start up a 3.6 kafka cluster in KRaft with 1 broker
> 2. create a topic
> 3. upgrade the binary to 3.7
> 4. use kafka-features.sh to upgrade to 3.7 metadata version
> 5. check the log (and metrics if interested)
> Analysis:
> In 3.7, we have JBOD support in KRaft, so the partitionRegistration added a
> new directory field. And it causes diff found while comparing delta. We might
> be able to identify this adding directory change doesn't need to reset the
> leader/follower state, and just update the metadata, to avoid causing
> unavailability.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-16132) Upgrading from 3.6 to 3.7 in KRaft will have seconds of partitions unavailable
[
https://issues.apache.org/jira/browse/KAFKA-16132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismael Juma updated KAFKA-16132:
Priority: Blocker (was: Major)
> Upgrading from 3.6 to 3.7 in KRaft will have seconds of partitions unavailable
> --
>
> Key: KAFKA-16132
> URL: https://issues.apache.org/jira/browse/KAFKA-16132
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 3.7.0
>Reporter: Luke Chen
>Priority: Blocker
>
> When upgrading from 3.6 to 3.7, we noticed that after upgrade the metadata
> version, all the partitions will be reset at one time, which causes a short
> period of time unavailable. This doesn't happen before.
> {code:java}
> [2024-01-15 20:45:19,757] INFO [BrokerMetadataPublisher id=2] Updating
> metadata.version to 19 at offset OffsetAndEpoch(offset=229, epoch=2).
> (kafka.server.metadata.BrokerMetadataPublisher)
> [2024-01-15 20:45:29,915] INFO [ReplicaFetcherManager on broker 2] Removed
> fetcher for partitions Set(t1-29, t1-25, t1-21, t1-17, t1-46, t1-13, t1-42,
> t1-9, t1-38, t1-5, t1-34, t1-1, t1-30, t1-26, t1-22, t1-18, t1-47, t1-14,
> t1-43, t1-10, t1-39, t1-6, t1-35, t1-2, t1-31, t1-27, t1-23, t1-19, t1-48,
> t1-15, t1-44, t1-11, t1-40, t1-7, t1-36, t1-3, t1-32, t1-28, t1-24, t1-20,
> t1-49, t1-16, t1-45, t1-12, t1-41, t1-8, t1-37, t1-4, t1-33, t1-0)
> (kafka.server.ReplicaFetcherManager)
> {code}
> Complete log:
> https://gist.github.com/showuon/665aa3ce6afd59097a2662f8260ecc10
> Steps:
> 1. start up a 3.6 kafka cluster in KRaft with 1 broker
> 2. create a topic
> 3. upgrade the binary to 3.7
> 4. use kafka-features.sh to upgrade to 3.7 metadata version
> 5. check the log (and metrics if interested)
> Analysis:
> In 3.7, we have JBOD support in KRaft, so the partitionRegistration added a
> new directory field. And it causes diff found while comparing delta. We might
> be able to identify this adding directory change doesn't need to reset the
> leader/follower state, and just update the metadata, to avoid causing
> unavailability.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-16132) Upgrading from 3.6 to 3.7 in KRaft will have seconds of partitions unavailable
[
https://issues.apache.org/jira/browse/KAFKA-16132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17807043#comment-17807043
]
Ismael Juma commented on KAFKA-16132:
-
[~cmccabe] Is this a blocker for 3.7.0?
> Upgrading from 3.6 to 3.7 in KRaft will have seconds of partitions unavailable
> --
>
> Key: KAFKA-16132
> URL: https://issues.apache.org/jira/browse/KAFKA-16132
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 3.7.0
>Reporter: Luke Chen
>Priority: Major
>
> When upgrading from 3.6 to 3.7, we noticed that after upgrade the metadata
> version, all the partitions will be reset at one time, which causes a short
> period of time unavailable. This doesn't happen before.
> {code:java}
> [2024-01-15 20:45:19,757] INFO [BrokerMetadataPublisher id=2] Updating
> metadata.version to 19 at offset OffsetAndEpoch(offset=229, epoch=2).
> (kafka.server.metadata.BrokerMetadataPublisher)
> [2024-01-15 20:45:29,915] INFO [ReplicaFetcherManager on broker 2] Removed
> fetcher for partitions Set(t1-29, t1-25, t1-21, t1-17, t1-46, t1-13, t1-42,
> t1-9, t1-38, t1-5, t1-34, t1-1, t1-30, t1-26, t1-22, t1-18, t1-47, t1-14,
> t1-43, t1-10, t1-39, t1-6, t1-35, t1-2, t1-31, t1-27, t1-23, t1-19, t1-48,
> t1-15, t1-44, t1-11, t1-40, t1-7, t1-36, t1-3, t1-32, t1-28, t1-24, t1-20,
> t1-49, t1-16, t1-45, t1-12, t1-41, t1-8, t1-37, t1-4, t1-33, t1-0)
> (kafka.server.ReplicaFetcherManager)
> {code}
> Complete log:
> https://gist.github.com/showuon/665aa3ce6afd59097a2662f8260ecc10
> Steps:
> 1. start up a 3.6 kafka cluster in KRaft with 1 broker
> 2. create a topic
> 3. upgrade the binary to 3.7
> 4. use kafka-features.sh to upgrade to 3.7 metadata version
> 5. check the log (and metrics if interested)
> Analysis:
> In 3.7, we have JBOD support in KRaft, so the partitionRegistration added a
> new directory field. And it causes diff found while comparing delta. We might
> be able to identify this adding directory change doesn't need to reset the
> leader/follower state, and just update the metadata, to avoid causing
> unavailability.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-14616) Topic recreation with offline broker causes permanent URPs
[
https://issues.apache.org/jira/browse/KAFKA-14616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17807042#comment-17807042
]
Ismael Juma commented on KAFKA-14616:
-
[~cmccabe] Is this a blocker for 3.7.0?
> Topic recreation with offline broker causes permanent URPs
> --
>
> Key: KAFKA-14616
> URL: https://issues.apache.org/jira/browse/KAFKA-14616
> Project: Kafka
> Issue Type: Bug
> Components: kraft
>Affects Versions: 3.3.1
>Reporter: Omnia Ibrahim
>Priority: Major
>
> We are facing an odd situation when we delete and recreate a topic while
> broker is offline in KRAFT mode.
> Here’s what we saw step by step
> # Created topic {{foo.test}} with 10 partitions and 4 replicas — Topic
> {{foo.test}} was created with topic ID {{MfuZbwdmSMaiSa0g6__TPg}}
> # Took broker 4 offline — which held replicas for partitions {{0, 3, 4, 5,
> 7, 8, 9}}
> # Deleted topic {{foo.test}} — The deletion process was successful, despite
> the fact that broker 4 still held replicas for partitions {{0, 3, 4, 5, 7, 8,
> 9}} on local disk.
> # Recreated topic {{foo.test}} with 10 partitions and 4 replicas. — Topic
> {{foo.test}} was created with topic ID {{RzalpqQ9Q7ub2M2afHxY4Q}} and
> partitions {{0, 1, 2, 7, 8, 9}} got assigned to broker 4 (which was still
> offline). Notice here that partitions {{0, 7, 8, 9}} are common between the
> assignment of the deleted topic ({{{}topic_id: MfuZbwdmSMaiSa0g6__TPg{}}})
> and the recreated topic ({{{}topic_id: RzalpqQ9Q7ub2M2afHxY4Q{}}}).
> # Brough broker 4 back online.
> # Broker started to create new partition replicas for the recreated topic
> {{foo.test}} ({{{}topic_id: RzalpqQ9Q7ub2M2afHxY4Q{}}})
> # The broker hit the following error {{Tried to assign topic ID
> RzalpqQ9Q7ub2M2afHxY4Q to log for topic partition foo.test-9,but log already
> contained topic ID MfuZbwdmSMaiSa0g6__TPg}} . As a result of this error the
> broker decided to rename log dir for partitions {{0, 3, 4, 5, 7, 8, 9}} to
> {{{}-.-delete{}}}.
> # Ran {{ls }}
> {code:java}
> foo.test-0.658f87fb9a2e42a590b5d7dcc28862b5-delete/
> foo.test-1/
> foo.test-2/
> foo.test-3.a68f05d05bcc4e579087551b539af311-delete/
> foo.test-4.79ce30a5310d4950ad1b28f226f74895-delete/
> foo.test-5.76ed04da75bf46c3a63342be1eb44450-delete/
> foo.test-6/
> foo.test-7.c2d33db3bf844e9ebbcd9ef22f5270da-delete/
> foo.test-8.33836969ac714b41b69b5334a5068ce0-delete/
> foo.test-9.48e1494f4fac48c8aec009bf77d5e4ee-delete/{code}
> 9. Waited until the deletion of the old topic was done and ran {{ls
> }} again, now we were expecting to see log dir for partitions
> {{0, 1, 2, 7, 8, 9}} however the result is:
> {code:java}
> foo.test-1/
> foo.test-2/
> foo.test-6/{code}
> 10. Ran {{kafka-topics.sh --command-config cmd.properties
> --bootstrap-server --describe --topic foo.test}}
> {code:java}
> Topic: foo.test TopicId: RzalpqQ9Q7ub2M2afHxY4Q PartitionCount: 10
> ReplicationFactor: 4 Configs:
> min.insync.replicas=2,segment.bytes=1073741824,max.message.bytes=3145728,unclean.leader.election.enable=false,retention.bytes=10
> Topic: foo.test Partition: 0 Leader: 2 Replicas: 2,3,4,5 Isr: 2,3,5
> Topic: foo.test Partition: 1 Leader: 3 Replicas: 3,4,5,6 Isr: 3,5,6,4
> Topic: foo.test Partition: 2 Leader: 5 Replicas: 5,4,6,1 Isr: 5,6,1,4
> Topic: foo.test Partition: 3 Leader: 5 Replicas: 5,6,1,2 Isr: 5,6,1,2
> Topic: foo.test Partition: 4 Leader: 6 Replicas: 6,1,2,3 Isr: 6,1,2,3
> Topic: foo.test Partition: 5 Leader: 1 Replicas: 1,6,2,5 Isr: 1,6,2,5
> Topic: foo.test Partition: 6 Leader: 6 Replicas: 6,2,5,4 Isr: 6,2,5,4
> Topic: foo.test Partition: 7 Leader: 2 Replicas: 2,5,4,3 Isr: 2,5,3
> Topic: foo.test Partition: 8 Leader: 5 Replicas: 5,4,3,1 Isr: 5,3,1
> Topic: foo.test Partition: 9 Leader: 3 Replicas: 3,4,1,6 Isr: 3,1,6{code}
> Here’s a sample of broker logs
>
> {code:java}
> {"timestamp":"2023-01-11T15:19:53,620Z","level":"INFO","thread":"kafka-scheduler-8","message":"Deleted
> log for partition foo.test-9 in
> /kafka/d1/data/foo.test-9.48e1494f4fac48c8aec009bf77d5e4ee-delete.","logger":"kafka.log.LogManager"}
> {"timestamp":"2023-01-11T15:19:53,617Z","level":"INFO","thread":"kafka-scheduler-8","message":"Deleted
> time index
> /kafka/d1/data/foo.test-9.48e1494f4fac48c8aec009bf77d5e4ee-delete/.timeindex.deleted.","logger":"kafka.log.LogSegment"}
> {"timestamp":"2023-01-11T15:19:53,617Z","level":"INFO","thread":"kafka-scheduler-8","message":"Deleted
> offset index
> /kafka/d1/data/foo.test-9.48e1494f4fac48c8aec009bf77d5e4ee-delete/.index.deleted.","logger":"kafka.log.LogSegment"}
> {"timestamp":"2023-01-11T15:19:53,615Z","level":"INFO","thread":"kafka-scheduler-8","message":"Deleted
> log
>
[jira] [Commented] (KAFKA-16120) Fix partition reassignment during ZK migration
[ https://issues.apache.org/jira/browse/KAFKA-16120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17807041#comment-17807041 ] Ismael Juma commented on KAFKA-16120: - [~cmccabe] If you're working on KAFKA-14616, assign the issue to yourself please. > Fix partition reassignment during ZK migration > -- > > Key: KAFKA-16120 > URL: https://issues.apache.org/jira/browse/KAFKA-16120 > Project: Kafka > Issue Type: Bug >Reporter: David Mao >Assignee: David Mao >Priority: Major > Fix For: 3.7.0 > > > When a reassignment is completed in ZK migration hybrid mode, the > `StopReplica` sent by the kraft quorum migration propagator is sent with > `delete = false` for deleted replicas when processing the topic delta. This > results in stray replicas. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16121) Partition reassignments in ZK migration dual write mode stalled until leader epoch incremented
[ https://issues.apache.org/jira/browse/KAFKA-16121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17807040#comment-17807040 ] Ismael Juma commented on KAFKA-16121: - It's a bit odd to mark this as a duplicate of KAFKA-16120 given that the descriptions are different. I understand that both issues were fixed by the same PR, but we should either update the description of KAFKA-16120 or mark this one as FIXED (instead of duplicate). > Partition reassignments in ZK migration dual write mode stalled until leader > epoch incremented > -- > > Key: KAFKA-16121 > URL: https://issues.apache.org/jira/browse/KAFKA-16121 > Project: Kafka > Issue Type: Bug >Reporter: David Mao >Assignee: David Mao >Priority: Major > Fix For: 3.7.0 > > > I noticed this in an integration test in > https://github.com/apache/kafka/pull/15184 > In ZK mode, partition leaders rely on the LeaderAndIsr request to be notified > of new replicas as part of a reassignment. In ZK mode, we ignore any > LeaderAndIsr request where the partition leader epoch is less than or equal > to the current partition leader epoch. > In KRaft mode, we do not bump the leader epoch when starting a new > reassignment, see: `triggerLeaderEpochBumpIfNeeded`. This means that the > leader will ignore the LISR request initiating the reassignment until a > leader epoch bump is triggered through another means, for instance preferred > leader election. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-16131) Repeated UnsupportedVersionException logged when running Kafka 3.7.0-RC2 KRaft cluster with metadata version 3.6
[
https://issues.apache.org/jira/browse/KAFKA-16131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismael Juma updated KAFKA-16131:
Priority: Blocker (was: Major)
> Repeated UnsupportedVersionException logged when running Kafka 3.7.0-RC2
> KRaft cluster with metadata version 3.6
>
>
> Key: KAFKA-16131
> URL: https://issues.apache.org/jira/browse/KAFKA-16131
> Project: Kafka
> Issue Type: Bug
>Reporter: Jakub Scholz
>Priority: Blocker
> Fix For: 3.7.0
>
>
> When running Kafka 3.7.0-RC2 as a KRaft cluster with metadata version set to
> 3.6-IV2 metadata version, it throws repeated errors like this in the
> controller logs:
> {quote}2024-01-13 16:58:01,197 INFO [QuorumController id=0]
> assignReplicasToDirs: event failed with UnsupportedVersionException in 15
> microseconds. (org.apache.kafka.controller.QuorumController)
> [quorum-controller-0-event-handler]
> 2024-01-13 16:58:01,197 ERROR [ControllerApis nodeId=0] Unexpected error
> handling request RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS, apiVersion=0,
> clientId=1000, correlationId=14, headerVersion=2) –
> AssignReplicasToDirsRequestData(brokerId=1000, brokerEpoch=5,
> directories=[DirectoryData(id=w_uxN7pwQ6eXSMrOKceYIQ,
> topics=[TopicData(topicId=bvAKLSwmR7iJoKv2yZgygQ,
> partitions=[PartitionData(partitionIndex=2),
> PartitionData(partitionIndex=1)]), TopicData(topicId=uNe7f5VrQgO0zST6yH1jDQ,
> partitions=[PartitionData(partitionIndex=0)])])]) with context
> RequestContext(header=RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
> apiVersion=0, clientId=1000, correlationId=14, headerVersion=2),
> connectionId='172.16.14.219:9090-172.16.14.217:53590-7',
> clientAddress=/[172.16.14.217|http://172.16.14.217/],
> principal=User:CN=my-cluster-kafka,O=io.strimzi,
> listenerName=ListenerName(CONTROLPLANE-9090), securityProtocol=SSL,
> clientInformation=ClientInformation(softwareName=apache-kafka-java,
> softwareVersion=3.7.0), fromPrivilegedListener=false,
> principalSerde=Optional[org.apache.kafka.common.security.authenticator.DefaultKafkaPrincipalBuilder@71004ad2])
> (kafka.server.ControllerApis) [quorum-controller-0-event-handler]
> java.util.concurrent.CompletionException:
> org.apache.kafka.common.errors.UnsupportedVersionException: Directory
> assignment is not supported yet.
> at
> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
> at
> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
> at
> java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636)
> at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
> at
> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
> at
> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.complete(QuorumController.java:880)
> at
> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.handleException(QuorumController.java:871)
> at
> org.apache.kafka.queue.KafkaEventQueue$EventContext.completeWithException(KafkaEventQueue.java:148)
> at
> org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:137)
> at
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
> at
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
> at java.base/java.lang.Thread.run(Thread.java:840)
> Caused by: org.apache.kafka.common.errors.UnsupportedVersionException:
> Directory assignment is not supported yet.
> {quote}
>
> With the metadata version set to 3.6-IV2, it makes sense that the request is
> not supported. But the request should in such case not be sent at all.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-16096) Drop broker and tools support for Java 11 in Kafka 4.0 (deprecate in 3.7) (KIP-1013)
[ https://issues.apache.org/jira/browse/KAFKA-16096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-16096: Fix Version/s: 4.0.0 > Drop broker and tools support for Java 11 in Kafka 4.0 (deprecate in 3.7) > (KIP-1013) > > > Key: KAFKA-16096 > URL: https://issues.apache.org/jira/browse/KAFKA-16096 > Project: Kafka > Issue Type: Improvement >Reporter: Ismael Juma >Assignee: Ismael Juma >Priority: Major > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-16096) Drop broker and tools support for Java 11 in Kafka 4.0 (deprecate in 3.7) (KIP-1013)
[ https://issues.apache.org/jira/browse/KAFKA-16096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma reassigned KAFKA-16096: --- Assignee: Ismael Juma > Drop broker and tools support for Java 11 in Kafka 4.0 (deprecate in 3.7) > (KIP-1013) > > > Key: KAFKA-16096 > URL: https://issues.apache.org/jira/browse/KAFKA-16096 > Project: Kafka > Issue Type: Improvement >Reporter: Ismael Juma >Assignee: Ismael Juma >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16096) Drop broker and tools support for Java 11 in Kafka 4.0 (deprecate in 3.7) (KIP-1013)
Ismael Juma created KAFKA-16096: --- Summary: Drop broker and tools support for Java 11 in Kafka 4.0 (deprecate in 3.7) (KIP-1013) Key: KAFKA-16096 URL: https://issues.apache.org/jira/browse/KAFKA-16096 Project: Kafka Issue Type: Improvement Reporter: Ismael Juma -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16072) Create Mockito extension to detect thread leak
[ https://issues.apache.org/jira/browse/KAFKA-16072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17801464#comment-17801464 ] Ismael Juma commented on KAFKA-16072: - I think you mean a JUnit 5 extension (not a mockito extension). > Create Mockito extension to detect thread leak > -- > > Key: KAFKA-16072 > URL: https://issues.apache.org/jira/browse/KAFKA-16072 > Project: Kafka > Issue Type: Improvement > Components: unit tests >Reporter: Divij Vaidya >Priority: Major > Labels: newbie++ > > The objective of this task is to create a Mockito extension that will execute > after every test and verify that there are no lingering threads left over. > An example of how to create a Mockito extension can be found here: > [https://github.com/apache/kafka/pull/14783/files#diff-812cfc2780b6fc0e7a1648ff37912ff13aeda4189ea6b0d4d847b831f66e56d1] > An example on how to find unexpected threads is at > [https://github.com/apache/kafka/blob/d5aa341a185f4df23bf587e55bcda4f16fc511f1/core/src/test/scala/unit/kafka/utils/TestUtils.scala#L2427] > and also at > https://issues.apache.org/jira/browse/KAFKA-16052?focusedCommentId=17800978=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17800978 > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16052) OOM in Kafka test suite
[
https://issues.apache.org/jira/browse/KAFKA-16052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17800875#comment-17800875
]
Ismael Juma commented on KAFKA-16052:
-
It can be done via Mockito.framework().clearInlineMocks().
> OOM in Kafka test suite
> ---
>
> Key: KAFKA-16052
> URL: https://issues.apache.org/jira/browse/KAFKA-16052
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 3.7.0
>Reporter: Divij Vaidya
>Priority: Major
> Attachments: Screenshot 2023-12-27 at 14.04.52.png, Screenshot
> 2023-12-27 at 14.22.21.png, Screenshot 2023-12-27 at 14.45.20.png, Screenshot
> 2023-12-27 at 15.31.09.png, Screenshot 2023-12-27 at 17.44.09.png
>
>
> *Problem*
> Our test suite is failing with frequent OOM. Discussion in the mailing list
> is here: [https://lists.apache.org/thread/d5js0xpsrsvhgjb10mbzo9cwsy8087x4]
> *Setup*
> To find the source of leaks, I ran the :core:test build target with a single
> thread (see below on how to do it) and attached a profiler to it. This Jira
> tracks the list of action items identified from the analysis.
> How to run tests using a single thread:
> {code:java}
> diff --git a/build.gradle b/build.gradle
> index f7abbf4f0b..81df03f1ee 100644
> --- a/build.gradle
> +++ b/build.gradle
> @@ -74,9 +74,8 @@ ext {
> "--add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED"
> )- maxTestForks = project.hasProperty('maxParallelForks') ?
> maxParallelForks.toInteger() : Runtime.runtime.availableProcessors()
> - maxScalacThreads = project.hasProperty('maxScalacThreads') ?
> maxScalacThreads.toInteger() :
> - Math.min(Runtime.runtime.availableProcessors(), 8)
> + maxTestForks = 1
> + maxScalacThreads = 1
> userIgnoreFailures = project.hasProperty('ignoreFailures') ?
> ignoreFailures : false userMaxTestRetries =
> project.hasProperty('maxTestRetries') ? maxTestRetries.toInteger() : 0
> diff --git a/gradle.properties b/gradle.properties
> index 4880248cac..ee4b6e3bc1 100644
> --- a/gradle.properties
> +++ b/gradle.properties
> @@ -30,4 +30,4 @@ scalaVersion=2.13.12
> swaggerVersion=2.2.8
> task=build
> org.gradle.jvmargs=-Xmx2g -Xss4m -XX:+UseParallelGC
> -org.gradle.parallel=true
> +org.gradle.parallel=false {code}
> *Result of experiment*
> This is how the heap memory utilized looks like, starting from tens of MB to
> ending with 1.5GB (with spikes of 2GB) of heap being used as the test
> executes. Note that the total number of threads also increases but it does
> not correlate with sharp increase in heap memory usage. The heap dump is
> available at
> [https://www.dropbox.com/scl/fi/nwtgc6ir6830xlfy9z9cu/GradleWorkerMain_10311_27_12_2023_13_37_08.hprof.zip?rlkey=ozbdgh5vih4rcynnxbatzk7ln=0]
>
> !Screenshot 2023-12-27 at 14.22.21.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-16052) OOM in Kafka test suite
[
https://issues.apache.org/jira/browse/KAFKA-16052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17800874#comment-17800874
]
Ismael Juma commented on KAFKA-16052:
-
Have you tried clearing the mocks during tearDown?
> OOM in Kafka test suite
> ---
>
> Key: KAFKA-16052
> URL: https://issues.apache.org/jira/browse/KAFKA-16052
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 3.7.0
>Reporter: Divij Vaidya
>Priority: Major
> Attachments: Screenshot 2023-12-27 at 14.04.52.png, Screenshot
> 2023-12-27 at 14.22.21.png, Screenshot 2023-12-27 at 14.45.20.png, Screenshot
> 2023-12-27 at 15.31.09.png, Screenshot 2023-12-27 at 17.44.09.png
>
>
> *Problem*
> Our test suite is failing with frequent OOM. Discussion in the mailing list
> is here: [https://lists.apache.org/thread/d5js0xpsrsvhgjb10mbzo9cwsy8087x4]
> *Setup*
> To find the source of leaks, I ran the :core:test build target with a single
> thread (see below on how to do it) and attached a profiler to it. This Jira
> tracks the list of action items identified from the analysis.
> How to run tests using a single thread:
> {code:java}
> diff --git a/build.gradle b/build.gradle
> index f7abbf4f0b..81df03f1ee 100644
> --- a/build.gradle
> +++ b/build.gradle
> @@ -74,9 +74,8 @@ ext {
> "--add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED"
> )- maxTestForks = project.hasProperty('maxParallelForks') ?
> maxParallelForks.toInteger() : Runtime.runtime.availableProcessors()
> - maxScalacThreads = project.hasProperty('maxScalacThreads') ?
> maxScalacThreads.toInteger() :
> - Math.min(Runtime.runtime.availableProcessors(), 8)
> + maxTestForks = 1
> + maxScalacThreads = 1
> userIgnoreFailures = project.hasProperty('ignoreFailures') ?
> ignoreFailures : false userMaxTestRetries =
> project.hasProperty('maxTestRetries') ? maxTestRetries.toInteger() : 0
> diff --git a/gradle.properties b/gradle.properties
> index 4880248cac..ee4b6e3bc1 100644
> --- a/gradle.properties
> +++ b/gradle.properties
> @@ -30,4 +30,4 @@ scalaVersion=2.13.12
> swaggerVersion=2.2.8
> task=build
> org.gradle.jvmargs=-Xmx2g -Xss4m -XX:+UseParallelGC
> -org.gradle.parallel=true
> +org.gradle.parallel=false {code}
> *Result of experiment*
> This is how the heap memory utilized looks like, starting from tens of MB to
> ending with 1.5GB (with spikes of 2GB) of heap being used as the test
> executes. Note that the total number of threads also increases but it does
> not correlate with sharp increase in heap memory usage. The heap dump is
> available at
> [https://www.dropbox.com/scl/fi/nwtgc6ir6830xlfy9z9cu/GradleWorkerMain_10311_27_12_2023_13_37_08.hprof.zip?rlkey=ozbdgh5vih4rcynnxbatzk7ln=0]
>
> !Screenshot 2023-12-27 at 14.22.21.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KAFKA-15493) Ensure system tests work with Java 21
[
https://issues.apache.org/jira/browse/KAFKA-15493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismael Juma resolved KAFKA-15493.
-
Resolution: Fixed
We can mark this as fixed. I ran the system tests with Java 21 and the results
were the same as Java 17.
> Ensure system tests work with Java 21
> -
>
> Key: KAFKA-15493
> URL: https://issues.apache.org/jira/browse/KAFKA-15493
> Project: Kafka
> Issue Type: Improvement
>Reporter: Ismael Juma
>Assignee: Said BOUDJELDA
>Priority: Major
> Fix For: 3.7.0
>
> Attachments: image-2023-09-28-02-11-49-196.png,
> image-2023-09-28-02-12-33-807.png
>
>
> Run the system tests as described below with Java 21:
> [https://github.com/apache/kafka/tree/trunk/tests]
> One relevant portion:
> Run tests with a different JVM (it may be as easy as replacing 11 with 21)
> {code:java}
> bash tests/docker/ducker-ak up -j 'openjdk:11';
> tests/docker/run_tests.sh{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-15493) Ensure system tests work with Java 21
[
https://issues.apache.org/jira/browse/KAFKA-15493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismael Juma updated KAFKA-15493:
Fix Version/s: 3.7.0
(was: 3.8.0)
> Ensure system tests work with Java 21
> -
>
> Key: KAFKA-15493
> URL: https://issues.apache.org/jira/browse/KAFKA-15493
> Project: Kafka
> Issue Type: Improvement
>Reporter: Ismael Juma
>Assignee: Said BOUDJELDA
>Priority: Major
> Fix For: 3.7.0
>
> Attachments: image-2023-09-28-02-11-49-196.png,
> image-2023-09-28-02-12-33-807.png
>
>
> Run the system tests as described below with Java 21:
> [https://github.com/apache/kafka/tree/trunk/tests]
> One relevant portion:
> Run tests with a different JVM (it may be as easy as replacing 11 with 21)
> {code:java}
> bash tests/docker/ducker-ak up -j 'openjdk:11';
> tests/docker/run_tests.sh{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-15853) Move KafkaConfig to server module
[ https://issues.apache.org/jira/browse/KAFKA-15853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17794187#comment-17794187 ] Ismael Juma commented on KAFKA-15853: - [~fvaleri] Why does your work depend on this? Ideally, tools and other modules should not require `KafkaConfig`. They should instead specify interfaces that cover their needs. This is the same we did for the `storage` module. > Move KafkaConfig to server module > - > > Key: KAFKA-15853 > URL: https://issues.apache.org/jira/browse/KAFKA-15853 > Project: Kafka > Issue Type: Sub-task >Reporter: Ismael Juma >Assignee: Omnia Ibrahim >Priority: Major > > The server module is a Java-only module, so this also requires converting > from Scala to Java. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-15959) Replace byte handling classes with synchronized methods
[ https://issues.apache.org/jira/browse/KAFKA-15959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma reassigned KAFKA-15959: --- Assignee: Ismael Juma > Replace byte handling classes with synchronized methods > --- > > Key: KAFKA-15959 > URL: https://issues.apache.org/jira/browse/KAFKA-15959 > Project: Kafka > Issue Type: Improvement >Reporter: Ismael Juma >Assignee: Ismael Juma >Priority: Major > > The JDK has a number of old byte handling classes that have a number of > synchronized methods. This wasn't too bad until biased locking was disabled > by default in Java 17 and removed in Java 21. > The overhead now can be significant if one such method happens to be in the > hot path. And such overhead is unnecessary if the classes are used by a > single thread (which is very common). > The classes we should replace: > # ByteArrayInputStream > # ByteArrayOutputStream > # DataOutputStream (interestingly, DataInputStream doesn't have the same > issue) > # BufferedInputStream > # BufferedOutputStream -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-15959) Replace byte handling classes with synchronized methods
[ https://issues.apache.org/jira/browse/KAFKA-15959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-15959: Description: The JDK has a number of old byte handling classes that have a number of synchronized methods. This wasn't too bad until biased locking was disabled by default in Java 17 and removed in Java 21. The overhead now can be significant if one such method happens to be in the hot path. And such overhead is unnecessary if the classes are used by a single thread (which is very common). The classes we should replace: # ByteArrayInputStream # ByteArrayOutputStream # DataOutputStream (interestingly, DataInputStream doesn't have the same issue) # BufferedInputStream # BufferedOutputStream was: The JDK has a number of old byte handling classes that have a number of synchronized methods. This wasn't too bad until biased locking was disabled by default in Java 17 and removed in Java 21. The overhead now can be significant if one such method happens to be in the hot path. And such overhead is unnecessary if the classes are used by a single thread (which is very common). The classes we should replace: # ByteArrayInputStream # ByteArrayOutputStream # DataOutputStream # BufferedInputStream # BufferedOutputStream > Replace byte handling classes with synchronized methods > --- > > Key: KAFKA-15959 > URL: https://issues.apache.org/jira/browse/KAFKA-15959 > Project: Kafka > Issue Type: Improvement >Reporter: Ismael Juma >Priority: Major > > The JDK has a number of old byte handling classes that have a number of > synchronized methods. This wasn't too bad until biased locking was disabled > by default in Java 17 and removed in Java 21. > The overhead now can be significant if one such method happens to be in the > hot path. And such overhead is unnecessary if the classes are used by a > single thread (which is very common). > The classes we should replace: > # ByteArrayInputStream > # ByteArrayOutputStream > # DataOutputStream (interestingly, DataInputStream doesn't have the same > issue) > # BufferedInputStream > # BufferedOutputStream -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-15959) Replace byte handling classes with synchronized methods
Ismael Juma created KAFKA-15959: --- Summary: Replace byte handling classes with synchronized methods Key: KAFKA-15959 URL: https://issues.apache.org/jira/browse/KAFKA-15959 Project: Kafka Issue Type: Improvement Reporter: Ismael Juma The JDK has a number of old byte handling classes that have a number of synchronized methods. This wasn't too bad until biased locking was disabled by default in Java 17 and removed in Java 21. The overhead now can be significant if one such method happens to be in the hot path. And such overhead is unnecessary if the classes are used by a single thread (which is very common). The classes we should replace: # ByteArrayInputStream # ByteArrayOutputStream # DataOutputStream # BufferedInputStream # BufferedOutputStream -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-14527) Move `kafka.security` from `core` to separate module
[ https://issues.apache.org/jira/browse/KAFKA-14527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-14527. - Resolution: Won't Do Closing as "Won't do" for now. We'll start with KAFKA-15842 and then decide if move needs to be done. > Move `kafka.security` from `core` to separate module > > > Key: KAFKA-14527 > URL: https://issues.apache.org/jira/browse/KAFKA-14527 > Project: Kafka > Issue Type: Improvement >Reporter: Ismael Juma >Assignee: Omnia Ibrahim >Priority: Major > > A possible module name would be `server-security`. We should consider moving > `StandardAuthorizer` and `org.apache.kafka.server.authorizer.Authorizer` to > this module too. > See KAFKA-14524 for more context. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (KAFKA-14527) Move `kafka.security` from `core` to separate module
[ https://issues.apache.org/jira/browse/KAFKA-14527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17789709#comment-17789709 ] Ismael Juma edited comment on KAFKA-14527 at 11/25/23 6:51 PM: --- Closing as "Won't do" for now. We'll start with KAFKA-15899 and then decide if move needs to be done. was (Author: ijuma): Closing as "Won't do" for now. We'll start with KAFKA-15842 and then decide if move needs to be done. > Move `kafka.security` from `core` to separate module > > > Key: KAFKA-14527 > URL: https://issues.apache.org/jira/browse/KAFKA-14527 > Project: Kafka > Issue Type: Improvement >Reporter: Ismael Juma >Assignee: Omnia Ibrahim >Priority: Major > > A possible module name would be `server-security`. We should consider moving > `StandardAuthorizer` and `org.apache.kafka.server.authorizer.Authorizer` to > this module too. > See KAFKA-14524 for more context. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-15899) Move kafka.security package from core to server module
Ismael Juma created KAFKA-15899: --- Summary: Move kafka.security package from core to server module Key: KAFKA-15899 URL: https://issues.apache.org/jira/browse/KAFKA-15899 Project: Kafka Issue Type: Sub-task Reporter: Ismael Juma Assignee: Omnia Ibrahim -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-15853) Move KafkaConfig to server module
[ https://issues.apache.org/jira/browse/KAFKA-15853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17789707#comment-17789707 ] Ismael Juma commented on KAFKA-15853: - [~omnia_h_ibrahim] Sounds reasonable to me. The `ProcessRole` items should go to the server module, yes. > Move KafkaConfig to server module > - > > Key: KAFKA-15853 > URL: https://issues.apache.org/jira/browse/KAFKA-15853 > Project: Kafka > Issue Type: Sub-task >Reporter: Ismael Juma >Assignee: Omnia Ibrahim >Priority: Major > > The server module is a Java-only module, so this also requires converting > from Scala to Java. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-14527) Move `kafka.security` from `core` to separate module
[ https://issues.apache.org/jira/browse/KAFKA-14527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17789690#comment-17789690 ] Ismael Juma commented on KAFKA-14527: - [~omnia_h_ibrahim] Thinking about this some more, I think we should instead move these classes to the `server` module (KAFKA-15852) first and then figure out if it makes sense to further split it into a separate module. If you agree, I can close this and create a subtask under KAFKA-15852 (and assign it to you). Thoughts? > Move `kafka.security` from `core` to separate module > > > Key: KAFKA-14527 > URL: https://issues.apache.org/jira/browse/KAFKA-14527 > Project: Kafka > Issue Type: Improvement >Reporter: Ismael Juma >Assignee: Omnia Ibrahim >Priority: Major > > A possible module name would be `server-security`. We should consider moving > `StandardAuthorizer` and `org.apache.kafka.server.authorizer.Authorizer` to > this module too. > See KAFKA-14524 for more context. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14524) Modularize `core` monolith
[ https://issues.apache.org/jira/browse/KAFKA-14524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-14524: Description: The `core` module has grown too large and it's time to split it into multiple modules. The `core` module will be deleted once it's empty (a KIP will be required for this). Evidence of `core` growing too large is that it takes 1m10s to compile the main code and tests and it takes hours to run all the tests sequentially. As part of this effort, we should rewrite the Scala code in Java to reduce developer friction, reduce compilation time and simplify deployment (i.e. we can remove the scala version suffix from the module name). Scala may have a number of advantages over Java 8 (minimum version we support now) and Java 11 (minimum version we will support in Kafka 4.0), but a mixture of Scala and Java (as we have now) is more complex than just Java. Another benefit is that code dependencies will be strictly enforced, which will hopefully help ensure better abstractions. This pattern was started with the `tools` (but not completed), `metadata` and `raft` modules and we have (when this ticket was filed) a couple more in progress: `group-coordinator` and `storage`. This is an umbrella ticket and it will link to each ticket related to this goal. was: The `core` module has grown too large and it's time to split it into multiple modules. The `core` module will be deleted once it's empty. Evidence of `core` growing too large is that it takes 1m10s to compile the main code and tests and it takes hours to run all the tests sequentially. As part of this effort, we should rewrite the Scala code in Java to reduce developer friction, reduce compilation time and simplify deployment (i.e. we can remove the scala version suffix from the module name). Scala may have a number of advantages over Java 8 (minimum version we support now) and Java 11 (minimum version we will support in Kafka 4.0), but a mixture of Scala and Java (as we have now) is more complex than just Java. Another benefit is that code dependencies will be strictly enforced, which will hopefully help ensure better abstractions. This pattern was started with the `tools` (but not completed), `metadata` and `raft` modules and we have (when this ticket was filed) a couple more in progress: `group-coordinator` and `storage`. This is an umbrella ticket and it will link to each ticket related to this goal. > Modularize `core` monolith > -- > > Key: KAFKA-14524 > URL: https://issues.apache.org/jira/browse/KAFKA-14524 > Project: Kafka > Issue Type: Improvement >Reporter: Ismael Juma >Priority: Major > > The `core` module has grown too large and it's time to split it into multiple > modules. The `core` module will be deleted once it's empty (a KIP will be > required for this). > Evidence of `core` growing too large is that it takes 1m10s to compile the > main code and tests and it takes hours to run all the tests sequentially. > As part of this effort, we should rewrite the Scala code in Java to reduce > developer friction, reduce compilation time and simplify deployment (i.e. we > can remove the scala version suffix from the module name). Scala may have a > number of advantages over Java 8 (minimum version we support now) and Java 11 > (minimum version we will support in Kafka 4.0), but a mixture of Scala and > Java (as we have now) is more complex than just Java. > Another benefit is that code dependencies will be strictly enforced, which > will hopefully help ensure better abstractions. > This pattern was started with the `tools` (but not completed), `metadata` and > `raft` modules and we have (when this ticket was filed) a couple more in > progress: `group-coordinator` and `storage`. > This is an umbrella ticket and it will link to each ticket related to this > goal. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-15852) Move server code from `core` to `server` module
[ https://issues.apache.org/jira/browse/KAFKA-15852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-15852: Description: The relevant packages would be `kafka.server`, `kafka.cluster`, `kafka.network`, etc. See KAFKA-14524 for more context. was: The relevant packages would be `kafka.server`, `kafka.cluster`, etc. See KAFKA-14524 for more context. > Move server code from `core` to `server` module > --- > > Key: KAFKA-15852 > URL: https://issues.apache.org/jira/browse/KAFKA-15852 > Project: Kafka > Issue Type: Improvement >Reporter: Ismael Juma >Priority: Major > > The relevant packages would be `kafka.server`, `kafka.cluster`, > `kafka.network`, etc. > See KAFKA-14524 for more context. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-14526) Move `kafka.network` from `core` to separate module
[ https://issues.apache.org/jira/browse/KAFKA-14526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-14526. - Resolution: Won't Do Marking this as "Won't do" for now in favor of KAFKA-15852. Once that's completed, we can figure out if additional modules make sense. > Move `kafka.network` from `core` to separate module > --- > > Key: KAFKA-14526 > URL: https://issues.apache.org/jira/browse/KAFKA-14526 > Project: Kafka > Issue Type: Improvement >Reporter: Ismael Juma >Priority: Major > > Name to be decided, perhaps `server-network` or something along those lines. > More context in KAFKA-14524. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-14528) Move replication code from `core` to separate module
[ https://issues.apache.org/jira/browse/KAFKA-14528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-14528. - Resolution: Won't Do Marking this as "Won't Do" for now in favor of KAFKA-15852. Once that is completed, we can look at creating additional modules if that makes sense. > Move replication code from `core` to separate module > > > Key: KAFKA-14528 > URL: https://issues.apache.org/jira/browse/KAFKA-14528 > Project: Kafka > Issue Type: Improvement >Reporter: Ismael Juma >Priority: Major > > A potential module name could be `replication`. The relevant classes would be > `ReplicaManager`, `*FetcherThread`, etc. > See KAFKA-14524 for more context. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14552) Remove no longer required server protocol versions in Kafka 4.0
[ https://issues.apache.org/jira/browse/KAFKA-14552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-14552: Description: Kafka 4.0 will remove support for zk mode and kraft mode became production ready in Kafka 3.3. Furthermore, migration from zk mode to kraft mode will require upgrading to the bridge release first (likely 3.5, but could also be 3.6). This provides an opportunity to remove exclusively server side protocols versions that only exist to allow direct upgrades from versions older than 3.n where n is either 0 (KRaft preview), 3 (KRaft production ready) or 5 (bridge release). We should decide on the right `n` and make the change as part of 4.0. Note that this is complementary to the protocols that will be completely removed as part of zk mode removal. Step one would be to create a list of protocols that will be completely removed due to zk mode removal and the list of exclusively server side protocols remaining after that (one example is ControlledShutdown). was: Kafka 4.0 will remove support for zk mode and kraft mode became production ready in Kafka 3.3. Furthermore, migration from zk mode to kraft mode will require upgrading to the bridge release first (likely 3.5, but could also be 3.6). This provides an opportunity to remove exclusively server side protocols versions that only exist to allow direct upgrades from versions older than 3.n where n is either 0 (KRaft preview), 3 (KRaft production ready) or 5 (bridge release). We should decide on the right `n` and make the change before 4.0. Note that this is complementary to the protocols that will be completely removed as part of zk mode removal. Step one would be to create a list of protocols that will be completely removed due to zk mode removal and the list of exclusively server side protocols remaining after that (one example is ControlledShutdown). > Remove no longer required server protocol versions in Kafka 4.0 > --- > > Key: KAFKA-14552 > URL: https://issues.apache.org/jira/browse/KAFKA-14552 > Project: Kafka > Issue Type: Improvement >Reporter: Ismael Juma >Priority: Blocker > Fix For: 4.0.0 > > > Kafka 4.0 will remove support for zk mode and kraft mode became production > ready in Kafka 3.3. Furthermore, migration from zk mode to kraft mode will > require upgrading to the bridge release first (likely 3.5, but could also be > 3.6). > This provides an opportunity to remove exclusively server side protocols > versions that only exist to allow direct upgrades from versions older than > 3.n where n is either 0 (KRaft preview), 3 (KRaft production ready) or 5 > (bridge release). We should decide on the right `n` and make the change as > part of 4.0. > Note that this is complementary to the protocols that will be completely > removed as part of zk mode removal. Step one would be to create a list of > protocols that will be completely removed due to zk mode removal and the list > of exclusively server side protocols remaining after that (one example is > ControlledShutdown). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-14552) Remove no longer required server protocol versions in Kafka 4.0
[ https://issues.apache.org/jira/browse/KAFKA-14552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17788593#comment-17788593 ] Ismael Juma commented on KAFKA-14552: - [~cmccabe] I noticed your work on [KIP-1001|https://cwiki.apache.org/confluence/display/KAFKA/KIP-1001%3A+Some+New+Metrics+and+Config+Key+Deprecations]. Do you think it would make sense to include this in the same KIP or shall we create a separate one? > Remove no longer required server protocol versions in Kafka 4.0 > --- > > Key: KAFKA-14552 > URL: https://issues.apache.org/jira/browse/KAFKA-14552 > Project: Kafka > Issue Type: Improvement >Reporter: Ismael Juma >Priority: Blocker > Fix For: 4.0.0 > > > Kafka 4.0 will remove support for zk mode and kraft mode became production > ready in Kafka 3.3. Furthermore, migration from zk mode to kraft mode will > require upgrading to the bridge release first (likely 3.5, but could also be > 3.6). > This provides an opportunity to remove exclusively server side protocols > versions that only exist to allow direct upgrades from versions older than > 3.n where n is either 0 (KRaft preview), 3 (KRaft production ready) or 5 > (bridge release). We should decide on the right `n` and make the change > before 4.0. > Note that this is complementary to the protocols that will be completely > removed as part of zk mode removal. Step one would be to create a list of > protocols that will be completely removed due to zk mode removal and the list > of exclusively server side protocols remaining after that (one example is > ControlledShutdown). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-15874) Add metric and request log attribute for deprecated request api versions
Ismael Juma created KAFKA-15874: --- Summary: Add metric and request log attribute for deprecated request api versions Key: KAFKA-15874 URL: https://issues.apache.org/jira/browse/KAFKA-15874 Project: Kafka Issue Type: Sub-task Reporter: Ismael Juma Assignee: Ismael Juma Fix For: 3.7.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-15874) Add metric and request log attribute for deprecated request api versions (KIP-896)
[ https://issues.apache.org/jira/browse/KAFKA-15874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-15874: Summary: Add metric and request log attribute for deprecated request api versions (KIP-896) (was: Add metric and request log attribute for deprecated request api versions) > Add metric and request log attribute for deprecated request api versions > (KIP-896) > -- > > Key: KAFKA-15874 > URL: https://issues.apache.org/jira/browse/KAFKA-15874 > Project: Kafka > Issue Type: Sub-task >Reporter: Ismael Juma >Assignee: Ismael Juma >Priority: Major > Fix For: 3.7.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-15854) Move Java classes from kafka.server to the server module
[ https://issues.apache.org/jira/browse/KAFKA-15854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-15854. - Fix Version/s: 3.7.0 Resolution: Fixed > Move Java classes from kafka.server to the server module > > > Key: KAFKA-15854 > URL: https://issues.apache.org/jira/browse/KAFKA-15854 > Project: Kafka > Issue Type: Sub-task >Reporter: Ismael Juma >Assignee: Ismael Juma >Priority: Major > Fix For: 3.7.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-15853) Move KafkaConfig to server module
[ https://issues.apache.org/jira/browse/KAFKA-15853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17787674#comment-17787674 ] Ismael Juma commented on KAFKA-15853: - Feel free to take this. KAFKA-15854 is a dependency and will be merged soon. > Move KafkaConfig to server module > - > > Key: KAFKA-15853 > URL: https://issues.apache.org/jira/browse/KAFKA-15853 > Project: Kafka > Issue Type: Sub-task >Reporter: Ismael Juma >Priority: Major > > The server module is a Java-only module, so this also requires converting > from Scala to Java. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-14819) Publish a single kafka (aka core) Maven artifact in Apache Kafka 4.0 (KIP-897)
[ https://issues.apache.org/jira/browse/KAFKA-14819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-14819. - Fix Version/s: (was: 4.0.0) Resolution: Won't Fix As described in the KIP, we're taking a different approach. > Publish a single kafka (aka core) Maven artifact in Apache Kafka 4.0 (KIP-897) > -- > > Key: KAFKA-14819 > URL: https://issues.apache.org/jira/browse/KAFKA-14819 > Project: Kafka > Issue Type: Improvement >Reporter: Ismael Juma >Assignee: Ismael Juma >Priority: Blocker > > Please see KIP for details: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-897%3A+Publish+a+single+kafka+%28aka+core%29+Maven+artifact+in+Apache+Kafka+4.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14524) Modularize `core` monolith
[ https://issues.apache.org/jira/browse/KAFKA-14524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-14524: Description: The `core` module has grown too large and it's time to split it into multiple modules. The `core` module will be deleted once it's empty. Evidence of `core` growing too large is that it takes 1m10s to compile the main code and tests and it takes hours to run all the tests sequentially. As part of this effort, we should rewrite the Scala code in Java to reduce developer friction, reduce compilation time and simplify deployment (i.e. we can remove the scala version suffix from the module name). Scala may have a number of advantages over Java 8 (minimum version we support now) and Java 11 (minimum version we will support in Kafka 4.0), but a mixture of Scala and Java (as we have now) is more complex than just Java. Another benefit is that code dependencies will be strictly enforced, which will hopefully help ensure better abstractions. This pattern was started with the `tools` (but not completed), `metadata` and `raft` modules and we have (when this ticket was filed) a couple more in progress: `group-coordinator` and `storage`. This is an umbrella ticket and it will link to each ticket related to this goal. was: The `core` module has grown too large and it's time to split it into multiple modules. A much slimmer `core` module will remain in the end. Evidence of `core` growing too large is that it takes 1m10s to compile the main code and tests and it takes hours to run all the tests sequentially. As part of this effort, we should rewrite the Scala code in Java to reduce developer friction, reduce compilation time and simplify deployment (i.e. we can remove the scala version suffix from the module name). Scala may have a number of advantages over Java 8 (minimum version we support now) and Java 11 (minimum version we will support in Kafka 4.0), but a mixture of Scala and Java (as we have now) is more complex than just Java. Another benefit is that code dependencies will be strictly enforced, which will hopefully help ensure better abstractions. This pattern was started with the `tools` (but not completed), `metadata` and `raft` modules and we have (when this ticket was filed) a couple more in progress: `group-coordinator` and `storage`. This is an umbrella ticket and it will link to each ticket related to this goal. > Modularize `core` monolith > -- > > Key: KAFKA-14524 > URL: https://issues.apache.org/jira/browse/KAFKA-14524 > Project: Kafka > Issue Type: Improvement >Reporter: Ismael Juma >Priority: Major > > The `core` module has grown too large and it's time to split it into multiple > modules. The `core` module will be deleted once it's empty. > Evidence of `core` growing too large is that it takes 1m10s to compile the > main code and tests and it takes hours to run all the tests sequentially. > As part of this effort, we should rewrite the Scala code in Java to reduce > developer friction, reduce compilation time and simplify deployment (i.e. we > can remove the scala version suffix from the module name). Scala may have a > number of advantages over Java 8 (minimum version we support now) and Java 11 > (minimum version we will support in Kafka 4.0), but a mixture of Scala and > Java (as we have now) is more complex than just Java. > Another benefit is that code dependencies will be strictly enforced, which > will hopefully help ensure better abstractions. > This pattern was started with the `tools` (but not completed), `metadata` and > `raft` modules and we have (when this ticket was filed) a couple more in > progress: `group-coordinator` and `storage`. > This is an umbrella ticket and it will link to each ticket related to this > goal. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-15854) Move Java classes from kafka.server to the server module
Ismael Juma created KAFKA-15854: --- Summary: Move Java classes from kafka.server to the server module Key: KAFKA-15854 URL: https://issues.apache.org/jira/browse/KAFKA-15854 Project: Kafka Issue Type: Sub-task Reporter: Ismael Juma -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-15854) Move Java classes from kafka.server to the server module
[ https://issues.apache.org/jira/browse/KAFKA-15854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma reassigned KAFKA-15854: --- Assignee: Ismael Juma > Move Java classes from kafka.server to the server module > > > Key: KAFKA-15854 > URL: https://issues.apache.org/jira/browse/KAFKA-15854 > Project: Kafka > Issue Type: Sub-task >Reporter: Ismael Juma >Assignee: Ismael Juma >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-15853) Move KafkaConfig to server module
Ismael Juma created KAFKA-15853: --- Summary: Move KafkaConfig to server module Key: KAFKA-15853 URL: https://issues.apache.org/jira/browse/KAFKA-15853 Project: Kafka Issue Type: Sub-task Reporter: Ismael Juma The server module is a Java-only module, so this also requires converting from Scala to Java. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-15852) Move broker code from `core` to `server` module
[ https://issues.apache.org/jira/browse/KAFKA-15852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-15852: Summary: Move broker code from `core` to `server` module (was: Move broker code from `core` to `broker` module) > Move broker code from `core` to `server` module > --- > > Key: KAFKA-15852 > URL: https://issues.apache.org/jira/browse/KAFKA-15852 > Project: Kafka > Issue Type: Improvement >Reporter: Ismael Juma >Priority: Major > > The relevant packages would be `kafka.server`, `kafka.cluster`, etc. > See KAFKA-14524 for more context. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-15852) Move server code from `core` to `server` module
[ https://issues.apache.org/jira/browse/KAFKA-15852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-15852: Summary: Move server code from `core` to `server` module (was: Move broker code from `core` to `server` module) > Move server code from `core` to `server` module > --- > > Key: KAFKA-15852 > URL: https://issues.apache.org/jira/browse/KAFKA-15852 > Project: Kafka > Issue Type: Improvement >Reporter: Ismael Juma >Priority: Major > > The relevant packages would be `kafka.server`, `kafka.cluster`, etc. > See KAFKA-14524 for more context. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-15852) Move broker code from `core` to `broker` module
[ https://issues.apache.org/jira/browse/KAFKA-15852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-15852: Issue Type: Improvement (was: Bug) > Move broker code from `core` to `broker` module > --- > > Key: KAFKA-15852 > URL: https://issues.apache.org/jira/browse/KAFKA-15852 > Project: Kafka > Issue Type: Improvement >Reporter: Ismael Juma >Priority: Major > > The relevant packages would be `kafka.server`, `kafka.cluster`, etc. > See KAFKA-14524 for more context. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-15852) Move broker code from `core` to `broker` module
Ismael Juma created KAFKA-15852: --- Summary: Move broker code from `core` to `broker` module Key: KAFKA-15852 URL: https://issues.apache.org/jira/browse/KAFKA-15852 Project: Kafka Issue Type: Bug Reporter: Ismael Juma The relevant packages would be `kafka.server`, `kafka.cluster`, etc. See KAFKA-14524 for more context. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-15836) KafkaConsumer subscribes to multiple topics does not respect max.poll.records
[ https://issues.apache.org/jira/browse/KAFKA-15836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-15836: Fix Version/s: 3.6.1 > KafkaConsumer subscribes to multiple topics does not respect max.poll.records > - > > Key: KAFKA-15836 > URL: https://issues.apache.org/jira/browse/KAFKA-15836 > Project: Kafka > Issue Type: Bug >Affects Versions: 3.6.0 >Reporter: Philip Nee >Assignee: Kirk True >Priority: Blocker > Labels: consumer > Fix For: 3.6.1 > > > We discovered that when KafkaConsumer subscribes to multiple topics with > max.poll.record configured. The max.poll.record is not properly respected > for all poll() invocation. > > I was able to reproduce it with the AK example, here is how I ran my tests: > [https://github.com/apache/kafka/pull/14772] > > 1. start zookeeper and kafka server (or kraft mode should be fine too) > 2. Run: examples/bin/java-producer-consumer-demo.sh 1000 > 3. Polled records > 400 will be printed to stdout > > Here is what the program does: > The produce produces a large number of records to multiple topics. We > configure the consumer using a max.poll.record = 400, and subscribed to > multiple topics. The consumer poll, and the returned records can sometimes > be larger than 400. > > This is an issue in AK 3.6 but 3.5 was fine. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-15605) Topics marked for deletion in ZK are incorrectly migrated to KRaft
[
https://issues.apache.org/jira/browse/KAFKA-15605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17785526#comment-17785526
]
Ismael Juma commented on KAFKA-15605:
-
The PR was merged, can this be closed [~davidarthur] ?
> Topics marked for deletion in ZK are incorrectly migrated to KRaft
> --
>
> Key: KAFKA-15605
> URL: https://issues.apache.org/jira/browse/KAFKA-15605
> Project: Kafka
> Issue Type: Bug
> Components: controller, kraft
>Affects Versions: 3.6.0
>Reporter: David Arthur
>Assignee: David Arthur
>Priority: Major
> Fix For: 3.6.1
>
>
> When migrating topics from ZooKeeper, the KRaft controller reads all the
> topic and partition metadata from ZK directly. This includes topics which
> have been marked for deletion by the ZK controller. After being migrated to
> KRaft, the pending topic deletions are never completed, so it is as if the
> delete topic request never happened.
> Since the client request to delete these topics has already been returned as
> successful, it would be confusing to the client that the topic still existed.
> An operator or application would need to issue another topic deletion to
> remove these topics once the controller had moved to KRaft. If they tried to
> create a new topic with the same name, they would receive a
> TOPIC_ALREADY_EXISTS error.
> The migration logic should carry over pending topic deletions and resolve
> them either as part of the migration or shortly after.
> *Note to operators:*
> To determine if a migration was affected by this, an operator can check the
> contents of {{/admin/delete_topics}} after the KRaft controller has migrated
> the metadata. If any topics are listed under this ZNode, they were not
> deleted and will still be present in KRaft. At this point the operator can
> make a determination if the topics should be re-deleted (using
> "kafka-topics.sh --delete") or left in place. In either case, the topics
> should be removed from {{/admin/delete_topics}} to prevent unexpected topic
> deletion in the event of a fallback to ZK.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-15552) Duplicate Producer ID blocks during ZK migration
[
https://issues.apache.org/jira/browse/KAFKA-15552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17785522#comment-17785522
]
Ismael Juma commented on KAFKA-15552:
-
[~showuon] Can this be closed then?
> Duplicate Producer ID blocks during ZK migration
>
>
> Key: KAFKA-15552
> URL: https://issues.apache.org/jira/browse/KAFKA-15552
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 3.4.0, 3.5.0, 3.4.1, 3.6.0, 3.5.1
>Reporter: David Arthur
>Assignee: David Arthur
>Priority: Critical
> Fix For: 3.5.2, 3.6.1
>
>
> When migrating producer ID blocks from ZK to KRaft, we are taking the current
> producer ID block from ZK and writing it's "firstProducerId" into the
> producer IDs KRaft record. However, in KRaft we store the _next_ producer ID
> block in the log rather than storing the current block like ZK does. The end
> result is that the first block given to a caller of AllocateProducerIds is a
> duplicate of the last block allocated in ZK mode.
>
> This can result in duplicate producer IDs being given to transactional or
> idempotent producers. In the case of transactional producers, this can cause
> long term problems since the producer IDs are persisted and reused for a long
> time.
> The time between the last producer ID block being allocated by the ZK
> controller and all the brokers being restarted following the metadata
> migration is when this bug is possible.
>
> Symptoms of this bug will include ReplicaManager OutOfOrderSequenceException
> and possibly some producer epoch validation errors. To see if a cluster is
> affected by this bug, search for the offending producer ID and see if it is
> being used by more than one producer.
>
> For example, the following error was observed
> {code}
> Out of order sequence number for producer 376000 at offset 381338 in
> partition REDACTED: 0 (incoming seq. number), 21 (current end sequence
> number)
> {code}
> Then searching for "376000" on
> org.apache.kafka.clients.producer.internals.TransactionManager logs, two
> brokers both show the same producer ID being provisioned
> {code}
> Broker 0 [Producer clientId=REDACTED-0] ProducerId set to 376000 with epoch 1
> Broker 5 [Producer clientId=REDACTED-1] ProducerId set to 376000 with epoch 1
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-15609) Corrupted index uploaded to remote tier
[ https://issues.apache.org/jira/browse/KAFKA-15609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17782160#comment-17782160 ] Ismael Juma commented on KAFKA-15609: - [~divijvaidya] the indexes write immediately to page cache, right? `flush` forces it to _disk._ not to page cache. > Corrupted index uploaded to remote tier > --- > > Key: KAFKA-15609 > URL: https://issues.apache.org/jira/browse/KAFKA-15609 > Project: Kafka > Issue Type: Bug > Components: Tiered-Storage >Affects Versions: 3.6.0 >Reporter: Divij Vaidya >Priority: Minor > > While testing Tiered Storage, we have observed corrupt indexes being present > in remote tier. One such situation is covered here at > https://issues.apache.org/jira/browse/KAFKA-15401. This Jira presents another > such possible case of corruption. > Potential cause of index corruption: > We want to ensure that the file we are passing to RSM plugin contains all the > data which is present in MemoryByteBuffer i.e. we should have flushed the > MemoryByteBuffer to the file using force(). In Kafka, when we close a > segment, indexes are flushed asynchronously [1]. Hence, it might be possible > that when we are passing the file to RSM, the file doesn't contain flushed > data. Hence, we may end up uploading indexes which haven't been flushed yet. > Ideally, the contract should enforce that we force flush the content of > MemoryByteBuffer before we give the file for RSM. This will ensure that > indexes are not corrupted/incomplete. > [1] > [https://github.com/apache/kafka/blob/4150595b0a2e0f45f2827cebc60bcb6f6558745d/core/src/main/scala/kafka/log/UnifiedLog.scala#L1613] > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-15754) The kafka-storage tool can generate UUID starting with "-"
[
https://issues.apache.org/jira/browse/KAFKA-15754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17781069#comment-17781069
]
Ismael Juma commented on KAFKA-15754:
-
[~jolshan] I think this issue can happen if some code does _not_ use
`Uuid.toString()` and instead uses Java's `UUID.toString()` somehow.
> The kafka-storage tool can generate UUID starting with "-"
> --
>
> Key: KAFKA-15754
> URL: https://issues.apache.org/jira/browse/KAFKA-15754
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 3.6.0
>Reporter: Paolo Patierno
>Assignee: Paolo Patierno
>Priority: Major
>
> Using the kafka-storage.sh tool, it seems that it can still generate a UUID
> starting with a dash "-", which then breaks how the argparse4j library works.
> With such an UUID (i.e. -rmdB0m4T4–Y4thlNXk4Q in my case) the tool exits with
> the following error:
> kafka-storage: error: argument --cluster-id/-t: expected one argument
> Said that, it seems that this problem was already addressed in the
> Uuid.randomUuid method which keeps generating a new UUID until it doesn't
> start with "-". This is the commit addressing it
> [https://github.com/apache/kafka/commit/5c1dd493d6f608b566fdad5ab3a896cb13622bce]
> The problem is that when the toString is called on the Uuid instance, it's
> going to do a Base64 encoding on the generated UUID this way:
> {code:java}
> Base64.getUrlEncoder().withoutPadding().encodeToString(getBytesFromUuid());
> {code}
> Not sure why, but the code is using an URL (safe) encoder which, taking a
> look at the Base64 class in Java, is using a RFC4648_URLSAFE encoder using
> the following alphabet:
>
> {code:java}
> private static final char[] toBase64URL = new char[]{'A', 'B', 'C', 'D', 'E',
> 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T',
> 'U', 'V', 'W', 'X', 'Y', 'Z', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i',
> 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x',
> 'y', 'z', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', '-', '_'}; {code}
> which as you can see includes the "-" character.
> So despite the current Uuid.randomUuid is avoiding the generation of a UUID
> containing a dash, the Base64 encoding operation can return a final UUID
> starting with the dash instead.
>
> I was wondering if there is any good reason for using a Base64 URL encoder
> and not just the RFC4648 (not URL safe) which uses the common Base64 alphabet
> not containing the "-".
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-14133) Remaining EasyMock to Mockito tests
[
https://issues.apache.org/jira/browse/KAFKA-14133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17780406#comment-17780406
]
Ismael Juma commented on KAFKA-14133:
-
It looks like all the remaining tests are in review, but there is no PR link.
It might be useful to add one.
> Remaining EasyMock to Mockito tests
> ---
>
> Key: KAFKA-14133
> URL: https://issues.apache.org/jira/browse/KAFKA-14133
> Project: Kafka
> Issue Type: Sub-task
>Reporter: Christo Lolov
>Assignee: Christo Lolov
>Priority: Major
>
> {color:#de350b}There are tests which use both PowerMock and EasyMock. I have
> put those in https://issues.apache.org/jira/browse/KAFKA-14132. Tests here
> rely solely on EasyMock.{color}
> Unless stated in brackets the tests are in the streams module.
> A list of tests which still require to be moved from EasyMock to Mockito as
> of 2nd of August 2022 which do not have a Jira issue and do not have pull
> requests I am aware of which are opened:
> {color:#ff8b00}In Review{color}
> {color:#00875a}Merged{color}
> # {color:#00875a}WorkerConnectorTest{color} (connect) (owner: [~yash.mayya] )
> # {color:#00875a}WorkerCoordinatorTest{color} (connect) (owner:
> [~yash.mayya] )
> # {color:#00875a}RootResourceTest{color} (connect) (owner: [~yash.mayya] )
> # {color:#00875a}ByteArrayProducerRecordEquals{color} (connect) (owner:
> [~yash.mayya] )
> # {color:#00875a}KStreamFlatTransformTest{color} (owner: Christo)
> # {color:#00875a}KStreamFlatTransformValuesTest{color} (owner: Christo)
> # {color:#00875a}KStreamPrintTest{color} (owner: Christo)
> # {color:#00875a}KStreamRepartitionTest{color} (owner: Christo)
> # {color:#00875a}MaterializedInternalTest{color} (owner: Christo)
> # {color:#00875a}TransformerSupplierAdapterTest{color} (owner: Christo)
> # {color:#00875a}KTableSuppressProcessorMetricsTest{color} (owner: Christo)
> # {color:#00875a}ClientUtilsTest{color} (owner: Christo)
> # {color:#00875a}HighAvailabilityStreamsPartitionAssignorTest{color} (owner:
> Christo)
> # {color:#00875a}TopologyTest{color} (owner: Christo)
> # {color:#00875a}KTableSuppressProcessorTest{color} (owner: Christo)
> # {color:#00875a}ChangeLoggingSessionBytesStoreTest{color} (owner: Christo)
> # {color:#00875a}ChangeLoggingTimestampedWindowBytesStoreTest{color} (owner:
> Christo)
> # {color:#00875a}ChangeLoggingWindowBytesStoreTest{color} (owner: Christo)
> # {color:#00875a}MeteredTimestampedWindowStoreTest{color} (owner: Christo)
> # {color:#00875a}StreamsRebalanceListenerTest{color} (owner: Christo)
> # {color:#00875a}TimestampedKeyValueStoreMaterializerTest{color} (owner:
> Christo)
> # {color:#00875a}CachingInMemoryKeyValueStoreTest{color} (owner: Christo)
> # {color:#00875a}CachingInMemorySessionStoreTest{color} (owner: Christo)
> # {color:#00875a}CachingPersistentSessionStoreTest{color} (owner: Christo)
> # {color:#00875a}CachingPersistentWindowStoreTest{color} (owner: Christo)
> # {color:#00875a}ChangeLoggingKeyValueBytesStoreTest{color} (owner: Christo)
> # {color:#00875a}ChangeLoggingTimestampedKeyValueBytesStoreTest{color}
> (owner: Christo)
> # {color:#00875a}CompositeReadOnlyWindowStoreTest{color} (owner: Christo)
> # {color:#00875a}KeyValueStoreBuilderTest{color} (owner: Christo)
> # {color:#00875a}RocksDBStoreTest{color} (owner: Christo)
> # {color:#00875a}StreamThreadStateStoreProviderTest{color} (owner: Christo)
> # {color:#ff8b00}TaskManagerTest{color} (owner: Christo)
> # {color:#00875a}InternalTopicManagerTest{color} (owner: Christo)
> # {color:#00875a}ProcessorContextImplTest{color} (owner: Christo)
> # {color:#00875a}WriteConsistencyVectorTest{color} (owner: Christo)
> # {color:#00875a}StreamsAssignmentScaleTest{color} (owner: Christo)
> # {color:#00875a}StreamsPartitionAssignorTest{color} (owner: Christo)
> # {color:#00875a}AssignmentTestUtils{color} (owner: Christo)
> # {color:#ff8b00}ProcessorStateManagerTest{color} (owner: Matthew)
> (takeover: Christo)
> # {color:#ff8b00}StandbyTaskTest{color} (owner: Matthew)
> # {color:#ff8b00}StoreChangelogReaderTest{color} (owner: Matthew)
> # {color:#ff8b00}StreamTaskTest{color} (owner: Matthew)
> # {color:#ff8b00}StreamThreadTest{color} (owner: Matthew) (takeover: Christo)
> # {color:#ff8b00}StreamsMetricsImplTest{color} (owner: Dalibor) (Captured in
> https://issues.apache.org/jira/browse/KAFKA-12947) (takeover: Christo)
> # {color:#00875a}TimeOrderedCachingPersistentWindowStoreTest{color} (owner:
> [~shekharrajak])
> # {color:#00875a}TimeOrderedWindowStoreTest{color} (owner: [~shekharrajak])
> [https://github.com/apache/kafka/pull/12777]
> # {color:#00875a}AbstractStreamTest{color} (owner: Christo)
> # {color:#00875a}KStreamTransformValuesTest{color} (owner: Christo)
> # {color:#00875a}KTableImplTest{color} (owner:
[jira] [Comment Edited] (KAFKA-15653) NPE in ChunkedByteStream
[
https://issues.apache.org/jira/browse/KAFKA-15653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17778655#comment-17778655
]
Ismael Juma edited comment on KAFKA-15653 at 10/23/23 1:09 PM:
---
Good catch. I think we should review the design of this change in closer detail
to understand the full implications - is the buffer supplier the only issue or
are there others? The assumption is that anything that is executed in a
different thread would have to ensure thread-safety, etc.
was (Author: ijuma):
Good catch. I think we should review the design of this change in closer detail
to understand the full implications - is the buffer supplier the only issue or
are there others?
> NPE in ChunkedByteStream
>
>
> Key: KAFKA-15653
> URL: https://issues.apache.org/jira/browse/KAFKA-15653
> Project: Kafka
> Issue Type: Bug
> Components: producer
>Affects Versions: 3.6.0
> Environment: Docker container on a Linux laptop, using the latest
> release.
>Reporter: Travis Bischel
>Assignee: Divij Vaidya
>Priority: Major
> Attachments: repro.sh
>
>
> When looping franz-go integration tests, I received an UNKNOWN_SERVER_ERROR
> from producing. The broker logs for the failing request:
>
> {noformat}
> [2023-10-19 22:29:58,160] ERROR [ReplicaManager broker=2] Error processing
> append operation on partition
> 2fa8995d8002fbfe68a96d783f26aa2c5efc15368bf44ed8f2ab7e24b41b9879-24
> (kafka.server.ReplicaManager)
> java.lang.NullPointerException
> at
> org.apache.kafka.common.utils.ChunkedBytesStream.(ChunkedBytesStream.java:89)
> at
> org.apache.kafka.common.record.CompressionType$3.wrapForInput(CompressionType.java:105)
> at
> org.apache.kafka.common.record.DefaultRecordBatch.recordInputStream(DefaultRecordBatch.java:273)
> at
> org.apache.kafka.common.record.DefaultRecordBatch.compressedIterator(DefaultRecordBatch.java:277)
> at
> org.apache.kafka.common.record.DefaultRecordBatch.skipKeyValueIterator(DefaultRecordBatch.java:352)
> at
> org.apache.kafka.storage.internals.log.LogValidator.validateMessagesAndAssignOffsetsCompressed(LogValidator.java:358)
> at
> org.apache.kafka.storage.internals.log.LogValidator.validateMessagesAndAssignOffsets(LogValidator.java:165)
> at kafka.log.UnifiedLog.append(UnifiedLog.scala:805)
> at kafka.log.UnifiedLog.appendAsLeader(UnifiedLog.scala:719)
> at
> kafka.cluster.Partition.$anonfun$appendRecordsToLeader$1(Partition.scala:1313)
> at kafka.cluster.Partition.appendRecordsToLeader(Partition.scala:1301)
> at
> kafka.server.ReplicaManager.$anonfun$appendToLocalLog$6(ReplicaManager.scala:1210)
> at
> scala.collection.StrictOptimizedMapOps.map(StrictOptimizedMapOps.scala:28)
> at
> scala.collection.StrictOptimizedMapOps.map$(StrictOptimizedMapOps.scala:27)
> at scala.collection.mutable.HashMap.map(HashMap.scala:35)
> at
> kafka.server.ReplicaManager.appendToLocalLog(ReplicaManager.scala:1198)
> at kafka.server.ReplicaManager.appendEntries$1(ReplicaManager.scala:754)
> at
> kafka.server.ReplicaManager.$anonfun$appendRecords$18(ReplicaManager.scala:874)
> at
> kafka.server.ReplicaManager.$anonfun$appendRecords$18$adapted(ReplicaManager.scala:874)
> at
> kafka.server.KafkaRequestHandler$.$anonfun$wrap$3(KafkaRequestHandler.scala:73)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:130)
> at java.base/java.lang.Thread.run(Unknown Source)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-15653) NPE in ChunkedByteStream
[
https://issues.apache.org/jira/browse/KAFKA-15653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17778655#comment-17778655
]
Ismael Juma commented on KAFKA-15653:
-
Good catch. I think we should review the design of this change in closer detail
to understand the full implications - is the buffer supplier the only issue or
are there others?
> NPE in ChunkedByteStream
>
>
> Key: KAFKA-15653
> URL: https://issues.apache.org/jira/browse/KAFKA-15653
> Project: Kafka
> Issue Type: Bug
> Components: producer
>Affects Versions: 3.6.0
> Environment: Docker container on a Linux laptop, using the latest
> release.
>Reporter: Travis Bischel
>Assignee: Divij Vaidya
>Priority: Major
> Attachments: repro.sh
>
>
> When looping franz-go integration tests, I received an UNKNOWN_SERVER_ERROR
> from producing. The broker logs for the failing request:
>
> {noformat}
> [2023-10-19 22:29:58,160] ERROR [ReplicaManager broker=2] Error processing
> append operation on partition
> 2fa8995d8002fbfe68a96d783f26aa2c5efc15368bf44ed8f2ab7e24b41b9879-24
> (kafka.server.ReplicaManager)
> java.lang.NullPointerException
> at
> org.apache.kafka.common.utils.ChunkedBytesStream.(ChunkedBytesStream.java:89)
> at
> org.apache.kafka.common.record.CompressionType$3.wrapForInput(CompressionType.java:105)
> at
> org.apache.kafka.common.record.DefaultRecordBatch.recordInputStream(DefaultRecordBatch.java:273)
> at
> org.apache.kafka.common.record.DefaultRecordBatch.compressedIterator(DefaultRecordBatch.java:277)
> at
> org.apache.kafka.common.record.DefaultRecordBatch.skipKeyValueIterator(DefaultRecordBatch.java:352)
> at
> org.apache.kafka.storage.internals.log.LogValidator.validateMessagesAndAssignOffsetsCompressed(LogValidator.java:358)
> at
> org.apache.kafka.storage.internals.log.LogValidator.validateMessagesAndAssignOffsets(LogValidator.java:165)
> at kafka.log.UnifiedLog.append(UnifiedLog.scala:805)
> at kafka.log.UnifiedLog.appendAsLeader(UnifiedLog.scala:719)
> at
> kafka.cluster.Partition.$anonfun$appendRecordsToLeader$1(Partition.scala:1313)
> at kafka.cluster.Partition.appendRecordsToLeader(Partition.scala:1301)
> at
> kafka.server.ReplicaManager.$anonfun$appendToLocalLog$6(ReplicaManager.scala:1210)
> at
> scala.collection.StrictOptimizedMapOps.map(StrictOptimizedMapOps.scala:28)
> at
> scala.collection.StrictOptimizedMapOps.map$(StrictOptimizedMapOps.scala:27)
> at scala.collection.mutable.HashMap.map(HashMap.scala:35)
> at
> kafka.server.ReplicaManager.appendToLocalLog(ReplicaManager.scala:1198)
> at kafka.server.ReplicaManager.appendEntries$1(ReplicaManager.scala:754)
> at
> kafka.server.ReplicaManager.$anonfun$appendRecords$18(ReplicaManager.scala:874)
> at
> kafka.server.ReplicaManager.$anonfun$appendRecords$18$adapted(ReplicaManager.scala:874)
> at
> kafka.server.KafkaRequestHandler$.$anonfun$wrap$3(KafkaRequestHandler.scala:73)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:130)
> at java.base/java.lang.Thread.run(Unknown Source)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-15657) Unexpected errors when producing transactionally in 3.6
[ https://issues.apache.org/jira/browse/KAFKA-15657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1567#comment-1567 ] Ismael Juma commented on KAFKA-15657: - I was wondering the same. We should fix KAFKA-15653 and see if it's the source of the issues you have been seeing. I am not aware of any other change that would result in that sort of problem. > Unexpected errors when producing transactionally in 3.6 > --- > > Key: KAFKA-15657 > URL: https://issues.apache.org/jira/browse/KAFKA-15657 > Project: Kafka > Issue Type: Bug > Components: producer >Affects Versions: 3.6.0 >Reporter: Travis Bischel >Priority: Major > > In loop-testing the franz-go client, I am frequently receiving INVALID_RECORD > (which I created a separate issue for), and INVALID_TXN_STATE and > UNKNOWN_SERVER_ERROR. > INVALID_TXN_STATE is being returned even though the partitions have been > added to the transaction (AddPartitionsToTxn). Nothing about the code has > changed between 3.5 and 3.6, and I have loop-integration-tested this code > against 3.5 thousands of times. 3.6 is newly - and always - returning > INVALID_TXN_STATE. If I change the code to retry on INVALID_TXN_STATE, I > eventually quickly (always) receive UNKNOWN_SERVER_ERROR. In looking at the > broker logs, the broker indicates that sequence numbers are out of order - > but (a) I am repeating requests that were in order (so something on the > broker got a little haywire maybe? or maybe this is due to me ignoring > invalid_txn_state?), _and_ I am not receiving OUT_OF_ORDER_SEQUENCE_NUMBER, I > am receiving UNKNOWN_SERVER_ERROR. > I think the main problem is the client unexpectedly receiving > INVALID_TXN_STATE, but a second problem here is that OOOSN is being mapped to > USE on return for some reason. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-15653) NPE in ChunkedByteStream
[
https://issues.apache.org/jira/browse/KAFKA-15653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1549#comment-1549
]
Ismael Juma commented on KAFKA-15653:
-
cc [~divijvaidya]
> NPE in ChunkedByteStream
>
>
> Key: KAFKA-15653
> URL: https://issues.apache.org/jira/browse/KAFKA-15653
> Project: Kafka
> Issue Type: Bug
> Components: producer
>Affects Versions: 3.6.0
> Environment: Docker container on a Linux laptop, using the latest
> release.
>Reporter: Travis Bischel
>Priority: Major
>
> When looping franz-go integration tests, I received an UNKNOWN_SERVER_ERROR
> from producing. The broker logs for the failing request:
>
> {noformat}
> [2023-10-19 22:29:58,160] ERROR [ReplicaManager broker=2] Error processing
> append operation on partition
> 2fa8995d8002fbfe68a96d783f26aa2c5efc15368bf44ed8f2ab7e24b41b9879-24
> (kafka.server.ReplicaManager)
> java.lang.NullPointerException
> at
> org.apache.kafka.common.utils.ChunkedBytesStream.(ChunkedBytesStream.java:89)
> at
> org.apache.kafka.common.record.CompressionType$3.wrapForInput(CompressionType.java:105)
> at
> org.apache.kafka.common.record.DefaultRecordBatch.recordInputStream(DefaultRecordBatch.java:273)
> at
> org.apache.kafka.common.record.DefaultRecordBatch.compressedIterator(DefaultRecordBatch.java:277)
> at
> org.apache.kafka.common.record.DefaultRecordBatch.skipKeyValueIterator(DefaultRecordBatch.java:352)
> at
> org.apache.kafka.storage.internals.log.LogValidator.validateMessagesAndAssignOffsetsCompressed(LogValidator.java:358)
> at
> org.apache.kafka.storage.internals.log.LogValidator.validateMessagesAndAssignOffsets(LogValidator.java:165)
> at kafka.log.UnifiedLog.append(UnifiedLog.scala:805)
> at kafka.log.UnifiedLog.appendAsLeader(UnifiedLog.scala:719)
> at
> kafka.cluster.Partition.$anonfun$appendRecordsToLeader$1(Partition.scala:1313)
> at kafka.cluster.Partition.appendRecordsToLeader(Partition.scala:1301)
> at
> kafka.server.ReplicaManager.$anonfun$appendToLocalLog$6(ReplicaManager.scala:1210)
> at
> scala.collection.StrictOptimizedMapOps.map(StrictOptimizedMapOps.scala:28)
> at
> scala.collection.StrictOptimizedMapOps.map$(StrictOptimizedMapOps.scala:27)
> at scala.collection.mutable.HashMap.map(HashMap.scala:35)
> at
> kafka.server.ReplicaManager.appendToLocalLog(ReplicaManager.scala:1198)
> at kafka.server.ReplicaManager.appendEntries$1(ReplicaManager.scala:754)
> at
> kafka.server.ReplicaManager.$anonfun$appendRecords$18(ReplicaManager.scala:874)
> at
> kafka.server.ReplicaManager.$anonfun$appendRecords$18$adapted(ReplicaManager.scala:874)
> at
> kafka.server.KafkaRequestHandler$.$anonfun$wrap$3(KafkaRequestHandler.scala:73)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:130)
> at java.base/java.lang.Thread.run(Unknown Source)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-15610) Fix `CoreUtils.swallow()` test gaps
[ https://issues.apache.org/jira/browse/KAFKA-15610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1044#comment-1044 ] Ismael Juma commented on KAFKA-15610: - Yes. > Fix `CoreUtils.swallow()` test gaps > --- > > Key: KAFKA-15610 > URL: https://issues.apache.org/jira/browse/KAFKA-15610 > Project: Kafka > Issue Type: Bug >Reporter: Ismael Juma >Priority: Minor > Labels: newbie > > For example, it should verify that the passed in `logging` is used in case of > an exception. We found that there is no test for this in > https://github.com/apache/kafka/pull/14529#discussion_r1355277747. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14683) Replace EasyMock and PowerMock with Mockito in WorkerSinkTaskTest
[ https://issues.apache.org/jira/browse/KAFKA-14683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-14683: Fix Version/s: 3.7.0 > Replace EasyMock and PowerMock with Mockito in WorkerSinkTaskTest > - > > Key: KAFKA-14683 > URL: https://issues.apache.org/jira/browse/KAFKA-14683 > Project: Kafka > Issue Type: Sub-task > Components: KafkaConnect >Reporter: Hector Geraldino >Assignee: Hector Geraldino >Priority: Minor > Fix For: 3.7.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-12199) Migrate connect:runtime module to JUnit 5
[
https://issues.apache.org/jira/browse/KAFKA-12199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismael Juma updated KAFKA-12199:
Description:
As part of the conversion we need to remove the following from `build.gradle`
and all the conditional code that calls it.
{code:java}
def shouldUseJUnit5 = !(["runtime"].contains(it.project.name)) {code}
Additionally, we can adjust the following snippet to include `connect:runtime`
if we decide to migrate tests incrementally. If we do the complete migration in
one go, then this is not required.
{code:java}
if (project.name == 'streams') {
useJUnitPlatform {
includeTags "integration"
includeTags "org.apache.kafka.test.IntegrationTest"
// Both engines are needed to run JUnit 4 tests alongside JUnit 5 tests.
// junit-vintage (JUnit 4) can be removed once the JUnit 4 migration is
complete.
includeEngines "junit-vintage", "junit-jupiter"
}
} {code}
was:
As part of the conversion we need to remove the following from `build.gradle`:
{code:java}
def shouldUseJUnit5 = !(["runtime"].contains(it.project.name)) {code}
Additionally, we can adjust the following snippet to include `connect:runtime`
if we decide to migrate tests incrementally. If we do the complete migration in
one go, then this is not required.
{code:java}
if (project.name == 'streams') {
useJUnitPlatform {
includeTags "integration"
includeTags "org.apache.kafka.test.IntegrationTest"
// Both engines are needed to run JUnit 4 tests alongside JUnit 5 tests.
// junit-vintage (JUnit 4) can be removed once the JUnit 4 migration is
complete.
includeEngines "junit-vintage", "junit-jupiter"
}
} {code}
> Migrate connect:runtime module to JUnit 5
> -
>
> Key: KAFKA-12199
> URL: https://issues.apache.org/jira/browse/KAFKA-12199
> Project: Kafka
> Issue Type: Sub-task
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Major
>
> As part of the conversion we need to remove the following from `build.gradle`
> and all the conditional code that calls it.
> {code:java}
> def shouldUseJUnit5 = !(["runtime"].contains(it.project.name)) {code}
> Additionally, we can adjust the following snippet to include
> `connect:runtime` if we decide to migrate tests incrementally. If we do the
> complete migration in one go, then this is not required.
> {code:java}
> if (project.name == 'streams') {
> useJUnitPlatform {
>includeTags "integration"
>includeTags "org.apache.kafka.test.IntegrationTest"
> // Both engines are needed to run JUnit 4 tests alongside JUnit 5 tests.
>// junit-vintage (JUnit 4) can be removed once the JUnit 4 migration
> is complete.
>includeEngines "junit-vintage", "junit-jupiter"
> }
>} {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-12199) Migrate connect:runtime module to JUnit 5
[
https://issues.apache.org/jira/browse/KAFKA-12199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismael Juma updated KAFKA-12199:
Description:
As part of the conversion we need to remove the following from `build.gradle`:
{code:java}
def shouldUseJUnit5 = !(["runtime"].contains(it.project.name)) {code}
Additionally, we can adjust the following snippet to include `connect:runtime`
if we decide to migrate tests incrementally. If we do the complete migration in
one go, then this is not required.
{code:java}
if (project.name == 'streams') {
useJUnitPlatform {
includeTags "integration"
includeTags "org.apache.kafka.test.IntegrationTest"
// Both engines are needed to run JUnit 4 tests alongside JUnit 5 tests.
// junit-vintage (JUnit 4) can be removed once the JUnit 4 migration is
complete.
includeEngines "junit-vintage", "junit-jupiter"
}
} {code}
> Migrate connect:runtime module to JUnit 5
> -
>
> Key: KAFKA-12199
> URL: https://issues.apache.org/jira/browse/KAFKA-12199
> Project: Kafka
> Issue Type: Sub-task
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Major
>
> As part of the conversion we need to remove the following from `build.gradle`:
> {code:java}
> def shouldUseJUnit5 = !(["runtime"].contains(it.project.name)) {code}
> Additionally, we can adjust the following snippet to include
> `connect:runtime` if we decide to migrate tests incrementally. If we do the
> complete migration in one go, then this is not required.
>
> {code:java}
> if (project.name == 'streams') {
> useJUnitPlatform {
>includeTags "integration"
>includeTags "org.apache.kafka.test.IntegrationTest"
> // Both engines are needed to run JUnit 4 tests alongside JUnit 5 tests.
>// junit-vintage (JUnit 4) can be removed once the JUnit 4 migration
> is complete.
>includeEngines "junit-vintage", "junit-jupiter"
> }
>} {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-12199) Migrate connect:runtime module to JUnit 5
[
https://issues.apache.org/jira/browse/KAFKA-12199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismael Juma updated KAFKA-12199:
Description:
As part of the conversion we need to remove the following from `build.gradle`:
{code:java}
def shouldUseJUnit5 = !(["runtime"].contains(it.project.name)) {code}
Additionally, we can adjust the following snippet to include `connect:runtime`
if we decide to migrate tests incrementally. If we do the complete migration in
one go, then this is not required.
{code:java}
if (project.name == 'streams') {
useJUnitPlatform {
includeTags "integration"
includeTags "org.apache.kafka.test.IntegrationTest"
// Both engines are needed to run JUnit 4 tests alongside JUnit 5 tests.
// junit-vintage (JUnit 4) can be removed once the JUnit 4 migration is
complete.
includeEngines "junit-vintage", "junit-jupiter"
}
} {code}
was:
As part of the conversion we need to remove the following from `build.gradle`:
{code:java}
def shouldUseJUnit5 = !(["runtime"].contains(it.project.name)) {code}
Additionally, we can adjust the following snippet to include `connect:runtime`
if we decide to migrate tests incrementally. If we do the complete migration in
one go, then this is not required.
{code:java}
if (project.name == 'streams') {
useJUnitPlatform {
includeTags "integration"
includeTags "org.apache.kafka.test.IntegrationTest"
// Both engines are needed to run JUnit 4 tests alongside JUnit 5 tests.
// junit-vintage (JUnit 4) can be removed once the JUnit 4 migration is
complete.
includeEngines "junit-vintage", "junit-jupiter"
}
} {code}
> Migrate connect:runtime module to JUnit 5
> -
>
> Key: KAFKA-12199
> URL: https://issues.apache.org/jira/browse/KAFKA-12199
> Project: Kafka
> Issue Type: Sub-task
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Major
>
> As part of the conversion we need to remove the following from `build.gradle`:
> {code:java}
> def shouldUseJUnit5 = !(["runtime"].contains(it.project.name)) {code}
> Additionally, we can adjust the following snippet to include
> `connect:runtime` if we decide to migrate tests incrementally. If we do the
> complete migration in one go, then this is not required.
> {code:java}
> if (project.name == 'streams') {
> useJUnitPlatform {
>includeTags "integration"
>includeTags "org.apache.kafka.test.IntegrationTest"
> // Both engines are needed to run JUnit 4 tests alongside JUnit 5 tests.
>// junit-vintage (JUnit 4) can be removed once the JUnit 4 migration
> is complete.
>includeEngines "junit-vintage", "junit-jupiter"
> }
>} {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-15623) Migrate remaining tests in streams module to JUnit 5
[
https://issues.apache.org/jira/browse/KAFKA-15623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismael Juma updated KAFKA-15623:
Description:
The following special case from `build.gradle` can be removed once this is
completed:
{code:java}
if (project.name == 'streams') {
useJUnitPlatform {
includeTags "integration"
includeTags "org.apache.kafka.test.IntegrationTest"
// Both engines are needed to run JUnit 4 tests alongside JUnit 5 tests.
// junit-vintage (JUnit 4) can be removed once the JUnit 4 migration is
complete.
includeEngines "junit-vintage", "junit-jupiter"
}
} {code}
> Migrate remaining tests in streams module to JUnit 5
>
>
> Key: KAFKA-15623
> URL: https://issues.apache.org/jira/browse/KAFKA-15623
> Project: Kafka
> Issue Type: Sub-task
>Reporter: Ismael Juma
>Priority: Major
>
> The following special case from `build.gradle` can be removed once this is
> completed:
>
> {code:java}
> if (project.name == 'streams') {
> useJUnitPlatform {
>includeTags "integration"
>includeTags "org.apache.kafka.test.IntegrationTest"
> // Both engines are needed to run JUnit 4 tests alongside JUnit 5 tests.
>// junit-vintage (JUnit 4) can be removed once the JUnit 4 migration
> is complete.
>includeEngines "junit-vintage", "junit-jupiter"
> }
>} {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-15623) Migrate remaining tests in streams module to JUnit 5
[
https://issues.apache.org/jira/browse/KAFKA-15623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismael Juma updated KAFKA-15623:
Description:
The following special case from `build.gradle` can be removed once this is
completed:
{code:java}
if (project.name == 'streams') {
useJUnitPlatform {
includeTags "integration"
includeTags "org.apache.kafka.test.IntegrationTest"
// Both engines are needed to run JUnit 4 tests alongside JUnit 5 tests.
// junit-vintage (JUnit 4) can be removed once the JUnit 4 migration is
complete.
includeEngines "junit-vintage", "junit-jupiter"
}
} {code}
was:
The following special case from `build.gradle` can be removed once this is
completed:
{code:java}
if (project.name == 'streams') {
useJUnitPlatform {
includeTags "integration"
includeTags "org.apache.kafka.test.IntegrationTest"
// Both engines are needed to run JUnit 4 tests alongside JUnit 5 tests.
// junit-vintage (JUnit 4) can be removed once the JUnit 4 migration is
complete.
includeEngines "junit-vintage", "junit-jupiter"
}
} {code}
> Migrate remaining tests in streams module to JUnit 5
>
>
> Key: KAFKA-15623
> URL: https://issues.apache.org/jira/browse/KAFKA-15623
> Project: Kafka
> Issue Type: Sub-task
>Reporter: Ismael Juma
>Priority: Major
>
> The following special case from `build.gradle` can be removed once this is
> completed:
> {code:java}
> if (project.name == 'streams') {
> useJUnitPlatform {
>includeTags "integration"
>includeTags "org.apache.kafka.test.IntegrationTest"
> // Both engines are needed to run JUnit 4 tests alongside JUnit 5 tests.
>// junit-vintage (JUnit 4) can be removed once the JUnit 4 migration
> is complete.
>includeEngines "junit-vintage", "junit-jupiter"
> }
>} {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-12199) Migrate connect:runtime module to JUnit 5
[ https://issues.apache.org/jira/browse/KAFKA-12199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17776230#comment-17776230 ] Ismael Juma commented on KAFKA-12199: - [~jackwangcs] You can pick this up if you like. Note that it is dependent on the migration away from PowerMock. > Migrate connect:runtime module to JUnit 5 > - > > Key: KAFKA-12199 > URL: https://issues.apache.org/jira/browse/KAFKA-12199 > Project: Kafka > Issue Type: Sub-task >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-15623) Migrate remaining tests in streams module to JUnit 5
Ismael Juma created KAFKA-15623: --- Summary: Migrate remaining tests in streams module to JUnit 5 Key: KAFKA-15623 URL: https://issues.apache.org/jira/browse/KAFKA-15623 Project: Kafka Issue Type: Sub-task Reporter: Ismael Juma -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-12220) Replace PowerMock by Mockito (partial)
[ https://issues.apache.org/jira/browse/KAFKA-12220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-12220: Summary: Replace PowerMock by Mockito (partial) (was: Replace PowerMock by Mockito) > Replace PowerMock by Mockito (partial) > -- > > Key: KAFKA-12220 > URL: https://issues.apache.org/jira/browse/KAFKA-12220 > Project: Kafka > Issue Type: Improvement >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Major > > We are migrating project from junit 4 to junit 5 (KAFKA-7339). PowerMock, > however, does not support junit 5 totally > (https://github.com/powermock/powermock/issues/830). Hence, we ought to > replace PowerMock by Mockito before migrating to junit 5 since rewriting all > tests which are depending on PowerMock can bring a bunch of changes. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-12220) Replace PowerMock by Mockito
[ https://issues.apache.org/jira/browse/KAFKA-12220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-12220. - Resolution: Fixed > Replace PowerMock by Mockito > > > Key: KAFKA-12220 > URL: https://issues.apache.org/jira/browse/KAFKA-12220 > Project: Kafka > Issue Type: Improvement >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Major > > We are migrating project from junit 4 to junit 5 (KAFKA-7339). PowerMock, > however, does not support junit 5 totally > (https://github.com/powermock/powermock/issues/830). Hence, we ought to > replace PowerMock by Mockito before migrating to junit 5 since rewriting all > tests which are depending on PowerMock can bring a bunch of changes. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (KAFKA-12220) Replace PowerMock by Mockito
[ https://issues.apache.org/jira/browse/KAFKA-12220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma reopened KAFKA-12220: - > Replace PowerMock by Mockito > > > Key: KAFKA-12220 > URL: https://issues.apache.org/jira/browse/KAFKA-12220 > Project: Kafka > Issue Type: Improvement >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Major > > We are migrating project from junit 4 to junit 5 (KAFKA-7339). PowerMock, > however, does not support junit 5 totally > (https://github.com/powermock/powermock/issues/830). Hence, we ought to > replace PowerMock by Mockito before migrating to junit 5 since rewriting all > tests which are depending on PowerMock can bring a bunch of changes. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-12220) Replace PowerMock by Mockito
[ https://issues.apache.org/jira/browse/KAFKA-12220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-12220. - Resolution: Duplicate The remaining connect tests that need to be migrated are being tracked via https://issues.apache.org/jira/browse/KAFKA-14132. > Replace PowerMock by Mockito > > > Key: KAFKA-12220 > URL: https://issues.apache.org/jira/browse/KAFKA-12220 > Project: Kafka > Issue Type: Improvement >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Major > > We are migrating project from junit 4 to junit 5 (KAFKA-7339). PowerMock, > however, does not support junit 5 totally > (https://github.com/powermock/powermock/issues/830). Hence, we ought to > replace PowerMock by Mockito before migrating to junit 5 since rewriting all > tests which are depending on PowerMock can bring a bunch of changes. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-13414) Replace Powermock/EasyMock by Mockito in connect.storage package
[ https://issues.apache.org/jira/browse/KAFKA-13414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-13414. - Resolution: Fixed KafkaConfigBackingStoreTest is being tracked via KAFKA-14132, marking this as fixed. > Replace Powermock/EasyMock by Mockito in connect.storage package > > > Key: KAFKA-13414 > URL: https://issues.apache.org/jira/browse/KAFKA-13414 > Project: Kafka > Issue Type: Sub-task > Components: KafkaConnect >Reporter: Mickael Maison >Assignee: Christo Lolov >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-14132) Remaining PowerMock to Mockito tests
[
https://issues.apache.org/jira/browse/KAFKA-14132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17776218#comment-17776218
]
Ismael Juma commented on KAFKA-14132:
-
We're very close, so I set a target of 3.7.0 to complete these. :)
> Remaining PowerMock to Mockito tests
>
>
> Key: KAFKA-14132
> URL: https://issues.apache.org/jira/browse/KAFKA-14132
> Project: Kafka
> Issue Type: Sub-task
>Reporter: Christo Lolov
>Assignee: Christo Lolov
>Priority: Major
> Fix For: 3.7.0
>
>
> {color:#de350b}Some of the tests below use EasyMock as well. For those
> migrate both PowerMock and EasyMock to Mockito.{color}
> Unless stated in brackets the tests are in the connect module.
> A list of tests which still require to be moved from PowerMock to Mockito as
> of 2nd of August 2022 which do not have a Jira issue and do not have pull
> requests I am aware of which are opened:
> {color:#ff8b00}InReview{color}
> {color:#00875a}Merged{color}
> # {color:#00875a}ErrorHandlingTaskTest{color} (owner: [~shekharrajak])
> # {color:#00875a}SourceTaskOffsetCommiterTest{color} (owner: Christo)
> # {color:#00875a}WorkerMetricsGroupTest{color} (owner: Divij)
> # {color:#00875a}WorkerTaskTest{color} (owner: [~yash.mayya])
> # {color:#00875a}ErrorReporterTest{color} (owner: [~yash.mayya])
> # {color:#00875a}RetryWithToleranceOperatorTest{color} (owner: [~yash.mayya])
> # {color:#00875a}WorkerErrantRecordReporterTest{color} (owner: [~yash.mayya])
> # {color:#00875a}ConnectorsResourceTest{color} (owner: [~mdedetrich-aiven])
> # {color:#ff8b00}StandaloneHerderTest{color} (owner: [~mdedetrich-aiven])
> ([https://github.com/apache/kafka/pull/12728])
> # KafkaConfigBackingStoreTest (owner: [~mdedetrich-aiven])
> # {color:#00875a}KafkaOffsetBackingStoreTest{color} (owner: Christo)
> ([https://github.com/apache/kafka/pull/12418])
> # {color:#00875a}KafkaBasedLogTest{color} (owner: @bachmanity ])
> # {color:#00875a}RetryUtilTest{color} (owner: [~yash.mayya])
> # {color:#00875a}RepartitionTopicTest{color} (streams) (owner: Christo)
> # {color:#00875a}StateManagerUtilTest{color} (streams) (owner: Christo)
> *The coverage report for the above tests after the change should be >= to
> what the coverage is now.*
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-14132) Remaining PowerMock to Mockito tests
[
https://issues.apache.org/jira/browse/KAFKA-14132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismael Juma updated KAFKA-14132:
Fix Version/s: 3.7.0
> Remaining PowerMock to Mockito tests
>
>
> Key: KAFKA-14132
> URL: https://issues.apache.org/jira/browse/KAFKA-14132
> Project: Kafka
> Issue Type: Sub-task
>Reporter: Christo Lolov
>Assignee: Christo Lolov
>Priority: Major
> Fix For: 3.7.0
>
>
> {color:#de350b}Some of the tests below use EasyMock as well. For those
> migrate both PowerMock and EasyMock to Mockito.{color}
> Unless stated in brackets the tests are in the connect module.
> A list of tests which still require to be moved from PowerMock to Mockito as
> of 2nd of August 2022 which do not have a Jira issue and do not have pull
> requests I am aware of which are opened:
> {color:#ff8b00}InReview{color}
> {color:#00875a}Merged{color}
> # {color:#00875a}ErrorHandlingTaskTest{color} (owner: [~shekharrajak])
> # {color:#00875a}SourceTaskOffsetCommiterTest{color} (owner: Christo)
> # {color:#00875a}WorkerMetricsGroupTest{color} (owner: Divij)
> # {color:#00875a}WorkerTaskTest{color} (owner: [~yash.mayya])
> # {color:#00875a}ErrorReporterTest{color} (owner: [~yash.mayya])
> # {color:#00875a}RetryWithToleranceOperatorTest{color} (owner: [~yash.mayya])
> # {color:#00875a}WorkerErrantRecordReporterTest{color} (owner: [~yash.mayya])
> # {color:#00875a}ConnectorsResourceTest{color} (owner: [~mdedetrich-aiven])
> # {color:#ff8b00}StandaloneHerderTest{color} (owner: [~mdedetrich-aiven])
> ([https://github.com/apache/kafka/pull/12728])
> # KafkaConfigBackingStoreTest (owner: [~mdedetrich-aiven])
> # {color:#00875a}KafkaOffsetBackingStoreTest{color} (owner: Christo)
> ([https://github.com/apache/kafka/pull/12418])
> # {color:#00875a}KafkaBasedLogTest{color} (owner: @bachmanity ])
> # {color:#00875a}RetryUtilTest{color} (owner: [~yash.mayya])
> # {color:#00875a}RepartitionTopicTest{color} (streams) (owner: Christo)
> # {color:#00875a}StateManagerUtilTest{color} (streams) (owner: Christo)
> *The coverage report for the above tests after the change should be >= to
> what the coverage is now.*
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-14132) Remaining PowerMock to Mockito tests
[
https://issues.apache.org/jira/browse/KAFKA-14132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17776217#comment-17776217
]
Ismael Juma commented on KAFKA-14132:
-
[~mdedetrich] Did you see the question above from [~bachmanity1] ?
> Remaining PowerMock to Mockito tests
>
>
> Key: KAFKA-14132
> URL: https://issues.apache.org/jira/browse/KAFKA-14132
> Project: Kafka
> Issue Type: Sub-task
>Reporter: Christo Lolov
>Assignee: Christo Lolov
>Priority: Major
>
> {color:#de350b}Some of the tests below use EasyMock as well. For those
> migrate both PowerMock and EasyMock to Mockito.{color}
> Unless stated in brackets the tests are in the connect module.
> A list of tests which still require to be moved from PowerMock to Mockito as
> of 2nd of August 2022 which do not have a Jira issue and do not have pull
> requests I am aware of which are opened:
> {color:#ff8b00}InReview{color}
> {color:#00875a}Merged{color}
> # {color:#00875a}ErrorHandlingTaskTest{color} (owner: [~shekharrajak])
> # {color:#00875a}SourceTaskOffsetCommiterTest{color} (owner: Christo)
> # {color:#00875a}WorkerMetricsGroupTest{color} (owner: Divij)
> # {color:#00875a}WorkerTaskTest{color} (owner: [~yash.mayya])
> # {color:#00875a}ErrorReporterTest{color} (owner: [~yash.mayya])
> # {color:#00875a}RetryWithToleranceOperatorTest{color} (owner: [~yash.mayya])
> # {color:#00875a}WorkerErrantRecordReporterTest{color} (owner: [~yash.mayya])
> # {color:#00875a}ConnectorsResourceTest{color} (owner: [~mdedetrich-aiven])
> # {color:#ff8b00}StandaloneHerderTest{color} (owner: [~mdedetrich-aiven])
> ([https://github.com/apache/kafka/pull/12728])
> # KafkaConfigBackingStoreTest (owner: [~mdedetrich-aiven])
> # {color:#00875a}KafkaOffsetBackingStoreTest{color} (owner: Christo)
> ([https://github.com/apache/kafka/pull/12418])
> # {color:#00875a}KafkaBasedLogTest{color} (owner: @bachmanity ])
> # {color:#00875a}RetryUtilTest{color} (owner: [~yash.mayya])
> # {color:#00875a}RepartitionTopicTest{color} (streams) (owner: Christo)
> # {color:#00875a}StateManagerUtilTest{color} (streams) (owner: Christo)
> *The coverage report for the above tests after the change should be >= to
> what the coverage is now.*
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KAFKA-7686) Remove PowerMock from Connect Tests
[ https://issues.apache.org/jira/browse/KAFKA-7686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-7686. Resolution: Duplicate > Remove PowerMock from Connect Tests > --- > > Key: KAFKA-7686 > URL: https://issues.apache.org/jira/browse/KAFKA-7686 > Project: Kafka > Issue Type: Task > Components: KafkaConnect >Reporter: Magesh kumar Nandakumar >Assignee: Magesh kumar Nandakumar >Priority: Minor > > Remove PowerMock from Connect Tests -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-15619) Deleted topics will come back again
[ https://issues.apache.org/jira/browse/KAFKA-15619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17775991#comment-17775991 ] Ismael Juma commented on KAFKA-15619: - auto.create.topics.enable means that _metadata requests_ cause topics to be created. It looks like the stress test is likely to be flaky with auto topic creation independently of the change in 3.6 (but it's probably worse with the change in 3.6). > Deleted topics will come back again > --- > > Key: KAFKA-15619 > URL: https://issues.apache.org/jira/browse/KAFKA-15619 > Project: Kafka > Issue Type: Improvement >Affects Versions: 3.5.0, 3.5.1 >Reporter: Deng Ziming >Priority: Major > > Deleted topics will come back again in Spark structured streaming test after > upgrade Kafka from 3.4.0 to 3.5.0, related ticket is: > https://issues.apache.org/jira/browse/SPARK-45529 > > I have basically inferred that this bug comes from > https://github.com/apache/kafka/pull/12590 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-15611) Use virtual threads in the Connect framework
[ https://issues.apache.org/jira/browse/KAFKA-15611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17775865#comment-17775865 ] Ismael Juma commented on KAFKA-15611: - I think we should also discuss the benefit of using virtual threads. > Use virtual threads in the Connect framework > > > Key: KAFKA-15611 > URL: https://issues.apache.org/jira/browse/KAFKA-15611 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Reporter: Greg Harris >Priority: Major > > Virtual threads have been finalized in JDK21, so we may include optional > support for them in Connect. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-15610) Fix `CoreUtils.swallow()` test gaps
[ https://issues.apache.org/jira/browse/KAFKA-15610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-15610: Priority: Minor (was: Major) > Fix `CoreUtils.swallow()` test gaps > --- > > Key: KAFKA-15610 > URL: https://issues.apache.org/jira/browse/KAFKA-15610 > Project: Kafka > Issue Type: Bug >Reporter: Ismael Juma >Priority: Minor > > For example, it should verify that the passed in `logging` is used in case of > an exception. We found that there is no test for this in > https://github.com/apache/kafka/pull/14529#discussion_r1355277747. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-15610) Fix `CoreUtils.swallow()` test gaps
[ https://issues.apache.org/jira/browse/KAFKA-15610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-15610: Labels: newbie (was: ) > Fix `CoreUtils.swallow()` test gaps > --- > > Key: KAFKA-15610 > URL: https://issues.apache.org/jira/browse/KAFKA-15610 > Project: Kafka > Issue Type: Bug >Reporter: Ismael Juma >Priority: Minor > Labels: newbie > > For example, it should verify that the passed in `logging` is used in case of > an exception. We found that there is no test for this in > https://github.com/apache/kafka/pull/14529#discussion_r1355277747. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-15610) Fix `CoreUtils.swallow()` test gaps
Ismael Juma created KAFKA-15610: --- Summary: Fix `CoreUtils.swallow()` test gaps Key: KAFKA-15610 URL: https://issues.apache.org/jira/browse/KAFKA-15610 Project: Kafka Issue Type: Bug Reporter: Ismael Juma For example, it should verify that the passed in `logging` is used in case of an exception. We found that there is no test for this in https://github.com/apache/kafka/pull/14529#discussion_r1355277747. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-15609) Corrupted index uploaded to remote tier
[ https://issues.apache.org/jira/browse/KAFKA-15609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17775761#comment-17775761 ] Ismael Juma commented on KAFKA-15609: - If you use buffered io (not direct io), you still go via the page cache. But if you do not fsync, it's possible for the disk stage to diverge from the uploaded state after a crash (for example). > Corrupted index uploaded to remote tier > --- > > Key: KAFKA-15609 > URL: https://issues.apache.org/jira/browse/KAFKA-15609 > Project: Kafka > Issue Type: Bug > Components: Tiered-Storage >Affects Versions: 3.6.0 >Reporter: Divij Vaidya >Priority: Minor > > While testing Tiered Storage, we have observed corrupt indexes being present > in remote tier. One such situation is covered here at > https://issues.apache.org/jira/browse/KAFKA-15401. This Jira presents another > such possible case of corruption. > Potential cause of index corruption: > We want to ensure that the file we are passing to RSM plugin contains all the > data which is present in MemoryByteBuffer i.e. we should have flushed the > MemoryByteBuffer to the file using force(). In Kafka, when we close a > segment, indexes are flushed asynchronously [1]. Hence, it might be possible > that when we are passing the file to RSM, the file doesn't contain flushed > data. Hence, we may end up uploading indexes which haven't been flushed yet. > Ideally, the contract should enforce that we force flush the content of > MemoryByteBuffer before we give the file for RSM. This will ensure that > indexes are not corrupted/incomplete. > [1] > [https://github.com/apache/kafka/blob/4150595b0a2e0f45f2827cebc60bcb6f6558745d/core/src/main/scala/kafka/log/UnifiedLog.scala#L1613] > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-14482) Move LogLoader to storage module
[ https://issues.apache.org/jira/browse/KAFKA-14482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17774988#comment-17774988 ] Ismael Juma commented on KAFKA-14482: - No worries, thanks for the quick response. :) > Move LogLoader to storage module > > > Key: KAFKA-14482 > URL: https://issues.apache.org/jira/browse/KAFKA-14482 > Project: Kafka > Issue Type: Sub-task >Reporter: Ismael Juma >Assignee: Ismael Juma >Priority: Major > Fix For: 3.7.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-14482) Move LogLoader to storage module
[ https://issues.apache.org/jira/browse/KAFKA-14482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma reassigned KAFKA-14482: --- Assignee: Ismael Juma > Move LogLoader to storage module > > > Key: KAFKA-14482 > URL: https://issues.apache.org/jira/browse/KAFKA-14482 > Project: Kafka > Issue Type: Sub-task >Reporter: Ismael Juma >Assignee: Ismael Juma >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
