Re: Request for Permission

[Luke Chen]

Hi Jeff.

Your accounts are all set.

Thank you.
Luke

On Wed, Mar 15, 2023 at 7:12 AM Jeff Kim 
wrote:

> Hi,
>
> I would like to request permission to contribute to the Apache Kafka wiki.
> For
>
> JIRA ID: jeffkbkim
> Wiki ID: jeff.kim
>
> Thanks,
> Jeff
>

[jira] [Created] (KAFKA-14810) Refactor FileRawSansphotWriter to not reference ReplicateLog

[Jira]

José Armando García Sancio created KAFKA-14810:
--

 Summary: Refactor FileRawSansphotWriter to not reference 
ReplicateLog
 Key: KAFKA-14810
 URL: https://issues.apache.org/jira/browse/KAFKA-14810
 Project: Kafka
  Issue Type: Task
Reporter: José Armando García Sancio
Assignee: José Armando García Sancio


The current implementation of FileRawSnapshotWriter uses an 
Optional to propagate when a new snapshot has been created by 
the state machine. This abstraction is too strict when writing tests and should 
be changed to something like Consumer.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Release notes bug

[Maruthi Vemuri (mavemuri)]

Hello,

Can the zk-upgrade bug 
KAFKA-14206 be removed from 
release notes for 
3.4.0? Looks like 
it’s been superseded by KIP-902 


Thanks,
Maruthi

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.4 #97

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 523949 lines...]
[2023-03-14T23:53:37.262Z] 
[2023-03-14T23:53:37.262Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 169 > KafkaZkClientTest > testBrokerSequenceIdMethods() STARTED
[2023-03-14T23:53:37.262Z] 
[2023-03-14T23:53:37.262Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 169 > KafkaZkClientTest > testBrokerSequenceIdMethods() PASSED
[2023-03-14T23:53:37.262Z] 
[2023-03-14T23:53:37.262Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 169 > KafkaZkClientTest > testAclMethods() STARTED
[2023-03-14T23:53:38.391Z] 
[2023-03-14T23:53:38.391Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 169 > KafkaZkClientTest > testAclMethods() PASSED
[2023-03-14T23:53:38.391Z] 
[2023-03-14T23:53:38.391Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 169 > KafkaZkClientTest > testCreateSequentialPersistentPath() STARTED
[2023-03-14T23:53:38.391Z] 
[2023-03-14T23:53:38.391Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 169 > KafkaZkClientTest > testCreateSequentialPersistentPath() PASSED
[2023-03-14T23:53:38.391Z] 
[2023-03-14T23:53:38.391Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 169 > KafkaZkClientTest > testConditionalUpdatePath() STARTED
[2023-03-14T23:53:38.391Z] 
[2023-03-14T23:53:38.391Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 169 > KafkaZkClientTest > testConditionalUpdatePath() PASSED
[2023-03-14T23:53:38.391Z] 
[2023-03-14T23:53:38.391Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 169 > KafkaZkClientTest > testGetAllTopicsInClusterTriggersWatch() 
STARTED
[2023-03-14T23:53:39.238Z] 
[2023-03-14T23:53:39.238Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 169 > KafkaZkClientTest > testGetAllTopicsInClusterTriggersWatch() 
PASSED
[2023-03-14T23:53:39.238Z] 
[2023-03-14T23:53:39.238Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 169 > KafkaZkClientTest > testDeleteTopicZNode() STARTED
[2023-03-14T23:53:39.238Z] 
[2023-03-14T23:53:39.238Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 169 > KafkaZkClientTest > testDeleteTopicZNode() PASSED
[2023-03-14T23:53:39.238Z] 
[2023-03-14T23:53:39.238Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 169 > KafkaZkClientTest > testDeletePath() STARTED
[2023-03-14T23:53:39.238Z] 
[2023-03-14T23:53:39.238Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 169 > KafkaZkClientTest > testDeletePath() PASSED
[2023-03-14T23:53:39.238Z] 
[2023-03-14T23:53:39.238Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 169 > KafkaZkClientTest > testGetBrokerMethods() STARTED
[2023-03-14T23:53:40.335Z] 
[2023-03-14T23:53:40.335Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 169 > KafkaZkClientTest > testGetBrokerMethods() PASSED
[2023-03-14T23:53:40.335Z] 
[2023-03-14T23:53:40.335Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 169 > KafkaZkClientTest > testJuteMaxBufffer() STARTED
[2023-03-14T23:53:40.336Z] 
[2023-03-14T23:53:40.336Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 169 > KafkaZkClientTest > testJuteMaxBufffer() PASSED
[2023-03-14T23:53:40.336Z] 
[2023-03-14T23:53:40.336Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 169 > KafkaZkClientTest > testCreateTokenChangeNotification() STARTED
[2023-03-14T23:53:40.336Z] 
[2023-03-14T23:53:40.336Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 169 > KafkaZkClientTest > testCreateTokenChangeNotification() PASSED
[2023-03-14T23:53:40.336Z] 
[2023-03-14T23:53:40.336Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 169 > KafkaZkClientTest > testGetTopicsAndPartitions() STARTED
[2023-03-14T23:53:41.638Z] 
[2023-03-14T23:53:41.638Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 169 > KafkaZkClientTest > testGetTopicsAndPartitions() PASSED
[2023-03-14T23:53:41.638Z] 
[2023-03-14T23:53:41.638Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 169 > KafkaZkClientTest > testChroot(boolean) > 
kafka.zk.KafkaZkClientTest.testChroot(boolean)[1] STARTED
[2023-03-14T23:53:41.638Z] 
[2023-03-14T23:53:41.638Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 169 > KafkaZkClientTest > testChroot(boolean) > 
kafka.zk.KafkaZkClientTest.testChroot(boolean)[1] PASSED
[2023-03-14T23:53:41.638Z] 
[2023-03-14T23:53:41.638Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 169 > KafkaZkClientTest > testChroot(boolean) > 
kafka.zk.KafkaZkClientTest.testChroot(boolean)[2] STARTED
[2023-03-14T23:53:42.734Z] 
[2023-03-14T23:53:42.734Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 169 > KafkaZkClientTest > testChroot(boolean) > 
kafka.zk.KafkaZkClientTest.testChroot(boolean)[2] PASSED

Request for Permission

[Jeff Kim]

Hi,

I would like to request permission to contribute to the Apache Kafka wiki.
For

JIRA ID: jeffkbkim
Wiki ID: jeff.kim

Thanks,
Jeff

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1678

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 362079 lines...]
[2023-03-14T22:45:35.521Z] > Task :group-coordinator:compileJava UP-TO-DATE
[2023-03-14T22:45:35.521Z] > Task :group-coordinator:classes UP-TO-DATE
[2023-03-14T22:45:35.521Z] > Task :core:compileJava NO-SOURCE
[2023-03-14T22:45:35.521Z] > Task :server-common:compileTestJava UP-TO-DATE
[2023-03-14T22:45:35.521Z] > Task :server-common:testClasses UP-TO-DATE
[2023-03-14T22:45:35.521Z] > Task :storage:api:compileTestJava UP-TO-DATE
[2023-03-14T22:45:35.521Z] > Task :storage:api:testClasses UP-TO-DATE
[2023-03-14T22:45:35.521Z] > Task :connect:json:compileTestJava UP-TO-DATE
[2023-03-14T22:45:35.521Z] > Task :raft:compileTestJava UP-TO-DATE
[2023-03-14T22:45:35.521Z] > Task :connect:json:testClasses UP-TO-DATE
[2023-03-14T22:45:35.521Z] > Task :raft:testClasses UP-TO-DATE
[2023-03-14T22:45:35.521Z] > Task :connect:json:testJar
[2023-03-14T22:45:35.521Z] > Task :group-coordinator:compileTestJava UP-TO-DATE
[2023-03-14T22:45:35.521Z] > Task :group-coordinator:testClasses UP-TO-DATE
[2023-03-14T22:45:35.521Z] > Task :connect:json:testSrcJar
[2023-03-14T22:45:35.521Z] > Task :metadata:compileTestJava UP-TO-DATE
[2023-03-14T22:45:35.521Z] > Task :metadata:testClasses UP-TO-DATE
[2023-03-14T22:45:35.521Z] > Task 
:clients:generateMetadataFileForMavenJavaPublication
[2023-03-14T22:45:37.518Z] 
[2023-03-14T22:45:37.518Z] > Task :connect:api:javadoc
[2023-03-14T22:45:37.518Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/connect/api/src/main/java/org/apache/kafka/connect/source/SourceRecord.java:44:
 warning - Tag @link: reference not found: org.apache.kafka.connect.data
[2023-03-14T22:45:39.537Z] 1 warning
[2023-03-14T22:45:39.537Z] 
[2023-03-14T22:45:39.537Z] > Task :connect:api:copyDependantLibs UP-TO-DATE
[2023-03-14T22:45:39.537Z] > Task :connect:api:jar UP-TO-DATE
[2023-03-14T22:45:39.537Z] > Task 
:connect:api:generateMetadataFileForMavenJavaPublication
[2023-03-14T22:45:39.537Z] > Task :connect:json:copyDependantLibs UP-TO-DATE
[2023-03-14T22:45:39.537Z] > Task :connect:json:jar UP-TO-DATE
[2023-03-14T22:45:39.537Z] > Task 
:connect:json:generateMetadataFileForMavenJavaPublication
[2023-03-14T22:45:40.711Z] > Task :connect:api:javadocJar
[2023-03-14T22:45:40.711Z] > Task :connect:api:compileTestJava UP-TO-DATE
[2023-03-14T22:45:40.711Z] > Task :connect:api:testClasses UP-TO-DATE
[2023-03-14T22:45:40.711Z] > Task :connect:api:testJar
[2023-03-14T22:45:40.711Z] > Task :connect:api:testSrcJar
[2023-03-14T22:45:40.711Z] > Task 
:connect:json:publishMavenJavaPublicationToMavenLocal
[2023-03-14T22:45:40.711Z] > Task :connect:json:publishToMavenLocal
[2023-03-14T22:45:40.711Z] > Task 
:connect:api:publishMavenJavaPublicationToMavenLocal
[2023-03-14T22:45:40.711Z] > Task :connect:api:publishToMavenLocal
[2023-03-14T22:45:41.710Z] > Task :streams:javadoc
[2023-03-14T22:45:42.707Z] > Task :streams:javadocJar
[2023-03-14T22:45:43.920Z] 
[2023-03-14T22:45:43.920Z] > Task :clients:javadoc
[2023-03-14T22:45:43.920Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/secured/package-info.java:21:
 warning - Tag @link: reference not found: 
org.apache.kafka.common.security.oauthbearer
[2023-03-14T22:45:43.920Z] 1 warning
[2023-03-14T22:45:44.915Z] 
[2023-03-14T22:45:44.915Z] > Task :clients:javadocJar
[2023-03-14T22:45:45.914Z] > Task :clients:srcJar
[2023-03-14T22:45:45.914Z] > Task :clients:testJar
[2023-03-14T22:45:45.914Z] > Task :clients:testSrcJar
[2023-03-14T22:45:46.914Z] > Task 
:clients:publishMavenJavaPublicationToMavenLocal
[2023-03-14T22:45:46.914Z] > Task :clients:publishToMavenLocal
[2023-03-14T22:46:04.831Z] > Task :core:compileScala
[2023-03-14T22:47:07.062Z] > Task :core:classes
[2023-03-14T22:47:07.062Z] > Task :core:compileTestJava NO-SOURCE
[2023-03-14T22:47:32.759Z] > Task :core:compileTestScala
[2023-03-14T22:48:19.184Z] > Task :core:testClasses
[2023-03-14T22:48:19.184Z] > Task :streams:compileTestJava UP-TO-DATE
[2023-03-14T22:48:19.184Z] > Task :streams:testClasses UP-TO-DATE
[2023-03-14T22:48:20.452Z] > Task :streams:testJar
[2023-03-14T22:48:20.452Z] > Task :streams:testSrcJar
[2023-03-14T22:48:20.452Z] > Task 
:streams:publishMavenJavaPublicationToMavenLocal
[2023-03-14T22:48:20.452Z] > Task :streams:publishToMavenLocal
[2023-03-14T22:48:20.452Z] 
[2023-03-14T22:48:20.452Z] Deprecated Gradle features were used in this build, 
making it incompatible with Gradle 9.0.
[2023-03-14T22:48:20.452Z] 
[2023-03-14T22:48:20.452Z] You can use '--warning-mode all' to show the 
individual deprecation warnings and determine if they come from your own 
scripts or plugins.
[2023-03-14T22:48:20.452Z] 
[2023-03-14T22:48:20.452Z] See 
https://docs.gradle.org/8.0.2/userguide/command_line_interface.html#sec:command_line_warnings
[2023-03-14T22:48:20.452Z] 

Re: [DISCUSSION] MirrorMaker2 offset translation for compacted, filtered, and transactional source topics

[Greg Harris]

Hey all!

I realized that the information above is a bit in-the-weeds, and I think a
re-framing of the situation might be necessary.

Since the release of MM2, offsets translation has been limited to only
performing translation ahead of the most recent offset sync. This
limitation appears to have worked for existing use-cases where offset syncs
are infrequent.
For topics which emit offset syncs frequently, the window for offset
translation becomes shorter, and may become unusable. In those unusable
cases, offset translation may stop completely for an otherwise
fully-functional steady-state MM2 instance.
Recently, we have been interested in improving the correctness of offset
translation to address data loss, and those fixes end up causing more
offset syncs to be emitted, making the translation window smaller than
before, and often unusable.

Q1. Would an improvement to allow translation from earlier in the topic be
reasonable to propose in a KIP?
Q2. Is anyone relying on the current poor correctness and high availability
translation, such that making the availability worse is a
backwards-incompatible regression?
Q3. Should we prioritize correctness, even if it hurts availability?
Q4. Should we address correctness and availability of this feature in a
patch or only minor releases?
Q5. Is there some tactical improvement to availability we can make which
does not count as backwards-incompatible, allowing us to land the
correctness fix without a KIP?
Q6. Do you have any suggestions on how to improve availability of offset
translation?

I'm interested in finding a tactical solution that we can backport, and a
holistic solution for more future use-cases.
I hope that the above is more clear.

Thanks!
Greg

On Fri, Mar 10, 2023 at 12:16 PM Greg Harris  wrote:

> Hi all,
>
> Recently, we've been experimenting with using MM2 to mirror topics that
> were populated by transactional producers. We've noticed that MM2
> replicates records but not transaction markers, causing certain offsets to
> appear in the source topic but not destination topic. These behaviors can
> also be seen when using Filter SMTs, or when replicating topics which have
> undergone compaction, which cause the same concentration of offsets in the
> target topic.
>
> This has the following negative effects with offset translation:
> P1. When starting replication on an existing topic with existing consumer
> groups, offsets are translated beyond the end of the topic, leading to
> "negative lag" for the downstream consumer group
> P2. When in a "negative lag" situation, and a consumer fail-over from
> source to is triggered, downstream consumption will stall until the
> downstream offsets exceed the "negative lag" offsets.
> P3. When failing over from source to target, certain records may have been
> ahead of the upstream consumer group and behind the downstream consumer
> group, leading to records not being delivered at least once.
>
> We merged a solution the above by making a change to the translation logic
> in https://issues.apache.org/jira/browse/KAFKA-12468 , and settled on a
> strategy to make offset translation more conservative, effectively making
> it such that the MirrorCheckpointTask only emits offsets at or immediately
> after the latest offset sync. This has the effect that offsets are more
> correct than previously, but that did not come without costs:
>
> P4. More offset syncs must be emitted to the offset syncs topic to enforce
> the `offset.lag.max` config property, once per `offset.max.lag` records
> (regression in the original PR, addressed by
> https://issues.apache.org/jira/browse/KAFKA-14797)
> P5. More recent offset syncs narrow the window in which translation can
> take place, leading to some translated offsets becoming excessively stale.
> This limitation is captured in
> https://issues.apache.org/jira/browse/KAFKA-14666 .
> P6. Even with the above fixes, offset translation won't be able to
> translate ahead the latest offset sync, and offsets may not converge
> exactly to the end of the topic.
>
> Fixing KAFKA-14797 appears possible without a KIP, but it is unclear
> whether KAFKA-14666 requires a KIP to resolve.
>
> To summarize:
> * Released versions of Kafka have reasonable behavior for normal topics,
> and correctness problems for compacted, filtered, and transactional topics.
> * KAFKA-12468 fixes correctness for compacted, filtered, and transactional
> topics, and regresses availability for all topics
> * KAFKA-14797 makes availability better for normal topics, but still worse
> than release.
> * KAFKA-14666 makes availability better for all topics, but still worse
> than release.
>
> Questions:
> Q1. Does KAFKA-14666 require a KIP to resolve?
> Q2. Is the increased likelihood of KAFKA-14666 caused by KAFKA-14797 a
> regression in behavior?
> Q3. Is the KAFKA-12468 correctness fix worth the general availability loss
> (P6) that is bounded by offset.lag.max?
> Q4. Is some or all of the above eligible for release in a 

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.3 #161

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 421497 lines...]
[2023-03-14T19:54:23.170Z] 
[2023-03-14T19:54:23.170Z] 
org.apache.kafka.streams.integration.StoreQueryIntegrationTest > 
shouldQueryOnlyActivePartitionStoresByDefault PASSED
[2023-03-14T19:54:23.170Z] 
[2023-03-14T19:54:23.170Z] 
org.apache.kafka.streams.integration.StoreQueryIntegrationTest > 
shouldQueryStoresAfterAddingAndRemovingStreamThread STARTED
[2023-03-14T19:54:39.450Z] 
[2023-03-14T19:54:39.450Z] 
org.apache.kafka.streams.integration.StoreQueryIntegrationTest > 
shouldQueryStoresAfterAddingAndRemovingStreamThread PASSED
[2023-03-14T19:54:39.450Z] 
[2023-03-14T19:54:39.450Z] 
org.apache.kafka.streams.integration.StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStoresMultiStreamThreadsNamedTopology STARTED
[2023-03-14T19:54:48.936Z] 
[2023-03-14T19:54:48.936Z] 
org.apache.kafka.streams.integration.StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStoresMultiStreamThreadsNamedTopology PASSED
[2023-03-14T19:54:48.936Z] 
[2023-03-14T19:54:48.936Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testInnerRepartitioned[caching enabled = true] STARTED
[2023-03-14T19:55:11.260Z] 
[2023-03-14T19:55:11.260Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testInnerRepartitioned[caching enabled = true] PASSED
[2023-03-14T19:55:11.260Z] 
[2023-03-14T19:55:11.260Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testOuterRepartitioned[caching enabled = true] STARTED
[2023-03-14T19:55:36.825Z] 
[2023-03-14T19:55:36.825Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testOuterRepartitioned[caching enabled = true] PASSED
[2023-03-14T19:55:36.825Z] 
[2023-03-14T19:55:36.825Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testInner[caching enabled = true] STARTED
[2023-03-14T19:55:52.956Z] 
[2023-03-14T19:55:52.956Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testInner[caching enabled = true] PASSED
[2023-03-14T19:55:52.956Z] 
[2023-03-14T19:55:52.956Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testOuter[caching enabled = true] STARTED
[2023-03-14T19:56:08.679Z] 
[2023-03-14T19:56:08.680Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testOuter[caching enabled = true] PASSED
[2023-03-14T19:56:08.680Z] 
[2023-03-14T19:56:08.680Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testLeft[caching enabled = true] STARTED
[2023-03-14T19:56:27.588Z] 
[2023-03-14T19:56:27.588Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testLeft[caching enabled = true] PASSED
[2023-03-14T19:56:27.588Z] 
[2023-03-14T19:56:27.588Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testMultiInner[caching enabled = true] STARTED
[2023-03-14T19:56:52.611Z] 
[2023-03-14T19:56:52.611Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testMultiInner[caching enabled = true] PASSED
[2023-03-14T19:56:52.611Z] 
[2023-03-14T19:56:52.611Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testLeftRepartitioned[caching enabled = true] STARTED
[2023-03-14T19:57:22.114Z] 
[2023-03-14T19:57:22.114Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testLeftRepartitioned[caching enabled = true] PASSED
[2023-03-14T19:57:22.114Z] 
[2023-03-14T19:57:22.114Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testSelfJoin[caching enabled = true] STARTED
[2023-03-14T19:57:27.617Z] 
[2023-03-14T19:57:27.617Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testSelfJoin[caching enabled = true] PASSED
[2023-03-14T19:57:27.617Z] 
[2023-03-14T19:57:27.617Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testInnerRepartitioned[caching enabled = false] STARTED
[2023-03-14T19:57:49.750Z] 
[2023-03-14T19:57:49.750Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testInnerRepartitioned[caching enabled = false] PASSED
[2023-03-14T19:57:49.750Z] 
[2023-03-14T19:57:49.750Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testOuterRepartitioned[caching enabled = false] STARTED
[2023-03-14T19:58:21.224Z] 
[2023-03-14T19:58:21.224Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testOuterRepartitioned[caching enabled = false] PASSED
[2023-03-14T19:58:21.224Z] 
[2023-03-14T19:58:21.224Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testInner[caching enabled = false] STARTED
[2023-03-14T19:58:31.636Z] 
[2023-03-14T19:58:31.636Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testInner[caching enabled = false] PASSED

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.4 #96

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 524457 lines...]
[2023-03-14T19:56:38.136Z] > Task :streams:javadoc
[2023-03-14T19:56:38.136Z] 
/home/jenkins/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:890:
 warning - Tag @link: reference not found: DefaultPartitioner
[2023-03-14T19:56:38.136Z] 
/home/jenkins/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:919:
 warning - Tag @link: reference not found: DefaultPartitioner
[2023-03-14T19:56:38.136Z] 
/home/jenkins/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:939:
 warning - Tag @link: reference not found: DefaultPartitioner
[2023-03-14T19:56:38.136Z] 
/home/jenkins/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:854:
 warning - Tag @link: reference not found: DefaultPartitioner
[2023-03-14T19:56:38.136Z] 
/home/jenkins/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:890:
 warning - Tag @link: reference not found: DefaultPartitioner
[2023-03-14T19:56:38.136Z] 
/home/jenkins/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:919:
 warning - Tag @link: reference not found: DefaultPartitioner
[2023-03-14T19:56:38.136Z] 
/home/jenkins/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:939:
 warning - Tag @link: reference not found: DefaultPartitioner
[2023-03-14T19:56:38.136Z] 
/home/jenkins/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/kstream/Produced.java:84:
 warning - Tag @link: reference not found: DefaultPartitioner
[2023-03-14T19:56:38.136Z] 
/home/jenkins/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/kstream/Produced.java:136:
 warning - Tag @link: reference not found: DefaultPartitioner
[2023-03-14T19:56:38.136Z] 
/home/jenkins/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/kstream/Produced.java:147:
 warning - Tag @link: reference not found: DefaultPartitioner
[2023-03-14T19:56:38.136Z] 
/home/jenkins/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/kstream/Repartitioned.java:101:
 warning - Tag @link: reference not found: DefaultPartitioner
[2023-03-14T19:56:38.136Z] 
/home/jenkins/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/kstream/Repartitioned.java:167:
 warning - Tag @link: reference not found: DefaultPartitioner
[2023-03-14T19:56:38.136Z] 
/home/jenkins/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/TopologyConfig.java:62:
 warning - Tag @link: missing '#': "org.apache.kafka.streams.StreamsBuilder()"
[2023-03-14T19:56:38.136Z] 
/home/jenkins/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/TopologyConfig.java:62:
 warning - Tag @link: can't find org.apache.kafka.streams.StreamsBuilder() in 
org.apache.kafka.streams.TopologyConfig
[2023-03-14T19:56:38.136Z] 
/home/jenkins/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/TopologyDescription.java:38:
 warning - Tag @link: reference not found: ProcessorContext#forward(Object, 
Object) forwards
[2023-03-14T19:56:38.136Z] 
/home/jenkins/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/query/Position.java:44:
 warning - Tag @link: can't find query(Query,
[2023-03-14T19:56:38.136Z]  PositionBound, boolean) in 
org.apache.kafka.streams.processor.StateStore
[2023-03-14T19:56:38.136Z] 
/home/jenkins/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:109:
 warning - Tag @link: reference not found: this#getResult()
[2023-03-14T19:56:38.136Z] 
/home/jenkins/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:116:
 warning - Tag @link: reference not found: this#getFailureReason()
[2023-03-14T19:56:38.136Z] 
/home/jenkins/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:116:
 warning - Tag @link: reference not found: this#getFailureMessage()
[2023-03-14T19:56:38.136Z] 
/home/jenkins/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:154:
 warning - Tag @link: reference not found: this#isSuccess()
[2023-03-14T19:56:38.136Z] 
/home/jenkins/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:154:
 warning - Tag @link: reference not found: this#isFailure()
[2023-03-14T19:56:39.327Z] 
/home/jenkins/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:890:
 warning - Tag @link: reference not found: DefaultPartitioner
[2023-03-14T19:56:39.327Z] 

[jira] [Created] (KAFKA-14809) Fix logging conditional on WorkerSourceTask

[Hector Geraldino (Jira)]

Hector Geraldino created KAFKA-14809:


 Summary: Fix logging conditional on WorkerSourceTask
 Key: KAFKA-14809
 URL: https://issues.apache.org/jira/browse/KAFKA-14809
 Project: Kafka
  Issue Type: Bug
  Components: KafkaConnect
Reporter: Hector Geraldino
Assignee: Hector Geraldino


There's an *{{if}}* condition when [committing 
offsets|https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSourceTask.java#L219]
 that is referencing the wrong variable, so the statement always evaluates to 
{*}true{*}.

It's a subtle bug, which went undetected probably because its only used to log 
information about pending committable offsets.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1677

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 451725 lines...]
[2023-03-14T19:21:54.971Z] > Task :storage:api:testClasses UP-TO-DATE
[2023-03-14T19:21:54.971Z] > Task :connect:json:compileTestJava UP-TO-DATE
[2023-03-14T19:21:54.971Z] > Task :raft:compileTestJava UP-TO-DATE
[2023-03-14T19:21:54.971Z] > Task :connect:json:testClasses UP-TO-DATE
[2023-03-14T19:21:54.971Z] > Task :raft:testClasses UP-TO-DATE
[2023-03-14T19:21:54.971Z] > Task :group-coordinator:compileTestJava UP-TO-DATE
[2023-03-14T19:21:54.971Z] > Task :group-coordinator:testClasses UP-TO-DATE
[2023-03-14T19:21:54.971Z] > Task :connect:json:testJar
[2023-03-14T19:21:54.971Z] > Task :connect:json:testSrcJar
[2023-03-14T19:21:54.971Z] > Task :metadata:compileTestJava UP-TO-DATE
[2023-03-14T19:21:54.971Z] > Task :metadata:testClasses UP-TO-DATE
[2023-03-14T19:21:54.971Z] > Task 
:clients:generateMetadataFileForMavenJavaPublication
[2023-03-14T19:21:56.224Z] > Task :streams:copyDependantLibs
[2023-03-14T19:21:56.224Z] > Task :streams:jar UP-TO-DATE
[2023-03-14T19:21:56.224Z] > Task 
:streams:generateMetadataFileForMavenJavaPublication
[2023-03-14T19:21:58.444Z] 
[2023-03-14T19:21:58.444Z] > Task :connect:api:javadoc
[2023-03-14T19:21:58.444Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk_2/connect/api/src/main/java/org/apache/kafka/connect/source/SourceRecord.java:44:
 warning - Tag @link: reference not found: org.apache.kafka.connect.data
[2023-03-14T19:21:59.576Z] 1 warning
[2023-03-14T19:22:00.673Z] 
[2023-03-14T19:22:00.673Z] > Task :connect:api:copyDependantLibs UP-TO-DATE
[2023-03-14T19:22:00.673Z] > Task :connect:api:jar UP-TO-DATE
[2023-03-14T19:22:00.673Z] > Task 
:connect:api:generateMetadataFileForMavenJavaPublication
[2023-03-14T19:22:00.673Z] > Task :connect:json:copyDependantLibs UP-TO-DATE
[2023-03-14T19:22:00.673Z] > Task :connect:json:jar UP-TO-DATE
[2023-03-14T19:22:00.673Z] > Task 
:connect:json:generateMetadataFileForMavenJavaPublication
[2023-03-14T19:22:00.673Z] > Task :connect:api:javadocJar
[2023-03-14T19:22:00.673Z] > Task :connect:api:compileTestJava UP-TO-DATE
[2023-03-14T19:22:00.673Z] > Task :connect:api:testClasses UP-TO-DATE
[2023-03-14T19:22:00.673Z] > Task 
:connect:json:publishMavenJavaPublicationToMavenLocal
[2023-03-14T19:22:00.673Z] > Task :connect:json:publishToMavenLocal
[2023-03-14T19:22:00.673Z] > Task :connect:api:testJar
[2023-03-14T19:22:00.673Z] > Task :connect:api:testSrcJar
[2023-03-14T19:22:00.673Z] > Task 
:connect:api:publishMavenJavaPublicationToMavenLocal
[2023-03-14T19:22:00.673Z] > Task :connect:api:publishToMavenLocal
[2023-03-14T19:22:04.169Z] > Task :streams:javadoc
[2023-03-14T19:22:04.169Z] > Task :streams:javadocJar
[2023-03-14T19:22:04.169Z] > Task :streams:processTestResources UP-TO-DATE
[2023-03-14T19:22:06.297Z] 
[2023-03-14T19:22:06.297Z] > Task :clients:javadoc
[2023-03-14T19:22:06.297Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk_2/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/secured/package-info.java:21:
 warning - Tag @link: reference not found: 
org.apache.kafka.common.security.oauthbearer
[2023-03-14T19:22:06.297Z] 1 warning
[2023-03-14T19:22:07.483Z] 
[2023-03-14T19:22:07.483Z] > Task :clients:javadocJar
[2023-03-14T19:22:08.502Z] > Task :clients:srcJar
[2023-03-14T19:22:09.520Z] > Task :clients:testJar
[2023-03-14T19:22:09.520Z] > Task :clients:testSrcJar
[2023-03-14T19:22:09.520Z] > Task 
:clients:publishMavenJavaPublicationToMavenLocal
[2023-03-14T19:22:09.520Z] > Task :clients:publishToMavenLocal
[2023-03-14T19:22:25.292Z] > Task :core:compileScala
[2023-03-14T19:24:19.015Z] > Task :core:classes
[2023-03-14T19:24:19.015Z] > Task :core:compileTestJava NO-SOURCE
[2023-03-14T19:24:37.292Z] > Task :core:compileTestScala
[2023-03-14T19:26:16.348Z] > Task :core:testClasses
[2023-03-14T19:26:35.409Z] > Task :streams:compileTestJava
[2023-03-14T19:26:35.409Z] > Task :streams:testClasses
[2023-03-14T19:26:35.409Z] > Task :streams:testJar
[2023-03-14T19:26:35.409Z] > Task :streams:testSrcJar
[2023-03-14T19:26:35.409Z] > Task 
:streams:publishMavenJavaPublicationToMavenLocal
[2023-03-14T19:26:35.409Z] > Task :streams:publishToMavenLocal
[2023-03-14T19:26:35.409Z] 
[2023-03-14T19:26:35.409Z] Deprecated Gradle features were used in this build, 
making it incompatible with Gradle 9.0.
[2023-03-14T19:26:35.409Z] 
[2023-03-14T19:26:35.409Z] You can use '--warning-mode all' to show the 
individual deprecation warnings and determine if they come from your own 
scripts or plugins.
[2023-03-14T19:26:35.409Z] 
[2023-03-14T19:26:35.409Z] See 
https://docs.gradle.org/8.0.2/userguide/command_line_interface.html#sec:command_line_warnings
[2023-03-14T19:26:35.409Z] 
[2023-03-14T19:26:35.409Z] BUILD SUCCESSFUL in 5m 8s
[2023-03-14T19:26:35.409Z] 86 actionable tasks: 35 executed, 51 up-to-date
[Pipeline] sh
[2023-03-14T19:26:38.688Z] + grep 

Re: [DISCUSS] KIP-641 An new java interface to replace 'kafka.common.MessageReader'

[Chia-Ping Tsai]

Hi juma

> Ismael Juma  於 2023年3月15日 上午3:04 寫道：
> 
> Hi Chia,
> 
> Regarding `org.apache.kafka.clients.tool`, a few comments:
> 
> 1. Why is it in `clients`? We don't generally consider tools to be a client.
> 2. Why is it `tool`? We have a package `org.apache.kafka.tools`, so it's a
> bit odd that this one uses singular instead of plural.

`org.apache.kafka.tools` is a good package. However, it is used by tool module 
already. Not sure whether it is good idea to use same package in different 
module. Maybe it is fine since we don’t expect user has to import both tool jar 
and client jar for developing plugins.

> 3. Also, should we follow the `storage-api` example and have a module for
> all extensible interfaces used by tools?

That seems good to me. However, it is a bit overkill if only few pluggable 
interfaces comes from tools. The rule I followed is server-side plugins (for 
example, AlterConfigPolicy, Authorizer, CreateTopicPolicy, and so on.

In short, I considered `org.apache.kafka.tools` before, but it is used by tool 
module already. If different modules has same package does not store up trouble 
for the future, I prefer the package ``org.apache.kafka.tools``

 —
Chia-ping


> 
> Ismael
> 
> On Tue, Mar 14, 2023 at 9:23 AM Chia-Ping Tsai  wrote:
> 
>> 
>> 
>>> Chris Egerton  於 2023年3月15日 上午12:04 寫道：
>>> 
>>> Hi Chia-Ping,
>>> 
>>> Thanks for the KIP. I find the interface definition really polished and
>>> intuitive! One small question--I noticed the change of the package to
>>> "org.apache.kafka.clients.tool". It doesn't look like there's any
>> precedent
>>> for using that package. We also use the "org.apache.kafka.common" package
>>> for the "MessageFormatter" interface, which is in some ways the
>> equivalent
>>> pluggable interface for the console consumer.
>> 
>> It seems to me those pluggable interfaces (MessageFormatter and
>> RecordReader) should not be a part of “common” package. They are used by
>> specify tools only. `Configurable`, by contrast, is good to be located at
>> `common` package since it is used widely in our code base.
>> 
>> 
>>> 
>>> Do we know if it's necessary to preserve the Checkstyle import
>> limitations
>>> (which I'm assuming are what motivated the shift in package name)? It
>> seems
>>> like it might be better to just relax that constraint in order to
>> colocate
>>> the pluggable interfaces for our console producer/consumer.
>> 
>> I love checkstyle import, and that is one of reason the KIP isolates the
>> new interface to a separate package. We have to add `allowed rule` one by
>> one if those dedicated interfaces are using the `common` package. The
>> constraint of new package can be relax to colocate the pluggable interfaces
>> (used by tools), and the `relax` won’t impact other existent packages.
>> 
>> 
>>> 
>>> Cheers,
>>> 
>>> Chris
>>> 
>>> On Tue, Mar 7, 2023 at 6:30 AM Chia-Ping Tsai 
>> wrote:
>>> 
 hi Mickael
 
> ?> configs) in the Compatibility, Deprecation, and Migration Plan, I
> guess these can be removed now.
 
 Done! thanks for feedback
 
> Mickael Maison  於 2023年3月7日 下午7:13 寫道：
> 
> Hi Chia-Ping,
> 
> The new API looks good.
> I still see mentions to configure(InputStream inputStream, Map ?> configs) in the Compatibility, Deprecation, and Migration Plan, I
> guess these can be removed now.
> 
> Thanks,
> Mickael
> 
> On Fri, Mar 3, 2023 at 2:37 PM Chia-Ping Tsai 
 wrote:
>> 
>> Dear all,
>> 
>> there are some changes for KIP-614
>> 
>> 
 
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158866569
>> 
>> 1) the interface RecordReader extends Configurable.
>> 2) the input stream is removed from RecordReader#configure method
>> 3) RecordReader#readRecords accept InputStream as argument, and the
 returned type is changed from single ProducerRecord to
 Iterator
>> 
>> Please take a look and then start to vote if you have free time.
>> thanks.
>> 
>> vote:
>> https://lists.apache.org/thread/kjdtyfg5xytn60q0qvxhfopzmfp9tsxr
 
 
>> 
>> 

[jira] [Resolved] (KAFKA-14794) Unable to deserialize base64 JSON strings

[Jira]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

José Armando García Sancio resolved KAFKA-14794.

Fix Version/s: 3.5.0
   Resolution: Fixed

> Unable to deserialize base64 JSON strings 
> --
>
> Key: KAFKA-14794
> URL: https://issues.apache.org/jira/browse/KAFKA-14794
> Project: Kafka
>  Issue Type: Bug
>Reporter: José Armando García Sancio
>Assignee: José Armando García Sancio
>Priority: Major
> Fix For: 3.5.0
>
>
> h1. Problem
> The following test fails:
> {code:java}
> @Test
> public void testBinaryNode() throws IOException {
>     byte[] expected = new byte[] {5, 2, 9, 4, 1, 8, 7, 0, 3, 6};
>     StringWriter writer = new StringWriter();
>     ObjectMapper mapper = new ObjectMapper();
>     mapper.writeTree(mapper.createGenerator(writer), new 
> BinaryNode(expected));
>     JsonNode binaryNode = mapper.readTree(writer.toString());
>     assertTrue(binaryNode.isTextual(), binaryNode.toString());
>     byte[] actual = MessageUtil.jsonNodeToBinary(binaryNode, "Test base64 
> JSON string");
>     assertEquals(expected, actual);
> }
> {code}
> with the following error:
> {code:java}
>  Gradle Test Run :clients:test > Gradle Test Executor 20 > MessageUtilTest > 
> testBinaryNode() FAILED
>     java.lang.RuntimeException: Test base64 JSON string: expected 
> Base64-encoded binary data.
>         at 
> org.apache.kafka.common.protocol.MessageUtil.jsonNodeToBinary(MessageUtil.java:165)
>         at 
> org.apache.kafka.common.protocol.MessageUtilTest.testBinaryNode(MessageUtilTest.java:102)
> {code}
> The reason for the failure is because FasterXML Jackson deserializes base64 
> JSON strings to a TextNode not to a BinaryNode.
> h1. Solution
> The method {{MessageUtil::jsonNodeToBinary}} should not assume that the input 
> {{JsonNode}} is always a {{{}BinaryNode{}}}. It should also support decoding 
> {{{}TextNode{}}}.
> {{JsonNode::binaryValue}} is supported by both {{BinaryNode}} and 
> {{{}TextNode{}}}.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-641 An new java interface to replace 'kafka.common.MessageReader'

[Ismael Juma]

Hi Chia,

Regarding `org.apache.kafka.clients.tool`, a few comments:

1. Why is it in `clients`? We don't generally consider tools to be a client.
2. Why is it `tool`? We have a package `org.apache.kafka.tools`, so it's a
bit odd that this one uses singular instead of plural.
3. Also, should we follow the `storage-api` example and have a module for
all extensible interfaces used by tools?

Ismael

On Tue, Mar 14, 2023 at 9:23 AM Chia-Ping Tsai  wrote:

>
>
> > Chris Egerton  於 2023年3月15日 上午12:04 寫道：
> >
> > Hi Chia-Ping,
> >
> > Thanks for the KIP. I find the interface definition really polished and
> > intuitive! One small question--I noticed the change of the package to
> > "org.apache.kafka.clients.tool". It doesn't look like there's any
> precedent
> > for using that package. We also use the "org.apache.kafka.common" package
> > for the "MessageFormatter" interface, which is in some ways the
> equivalent
> > pluggable interface for the console consumer.
>
> It seems to me those pluggable interfaces (MessageFormatter and
> RecordReader) should not be a part of “common” package. They are used by
> specify tools only. `Configurable`, by contrast, is good to be located at
> `common` package since it is used widely in our code base.
>
>
> >
> > Do we know if it's necessary to preserve the Checkstyle import
> limitations
> > (which I'm assuming are what motivated the shift in package name)? It
> seems
> > like it might be better to just relax that constraint in order to
> colocate
> > the pluggable interfaces for our console producer/consumer.
>
> I love checkstyle import, and that is one of reason the KIP isolates the
> new interface to a separate package. We have to add `allowed rule` one by
> one if those dedicated interfaces are using the `common` package. The
> constraint of new package can be relax to colocate the pluggable interfaces
> (used by tools), and the `relax` won’t impact other existent packages.
>
>
> >
> > Cheers,
> >
> > Chris
> >
> > On Tue, Mar 7, 2023 at 6:30 AM Chia-Ping Tsai 
> wrote:
> >
> >> hi Mickael
> >>
> >>> ?> configs) in the Compatibility, Deprecation, and Migration Plan, I
> >>> guess these can be removed now.
> >>
> >> Done! thanks for feedback
> >>
> >>> Mickael Maison  於 2023年3月7日 下午7:13 寫道：
> >>>
> >>> Hi Chia-Ping,
> >>>
> >>> The new API looks good.
> >>> I still see mentions to configure(InputStream inputStream, Map >>> ?> configs) in the Compatibility, Deprecation, and Migration Plan, I
> >>> guess these can be removed now.
> >>>
> >>> Thanks,
> >>> Mickael
> >>>
> >>> On Fri, Mar 3, 2023 at 2:37 PM Chia-Ping Tsai 
> >> wrote:
> 
>  Dear all,
> 
>  there are some changes for KIP-614
> 
> 
> >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158866569
> 
>  1) the interface RecordReader extends Configurable.
>  2) the input stream is removed from RecordReader#configure method
>  3) RecordReader#readRecords accept InputStream as argument, and the
> >> returned type is changed from single ProducerRecord to
> >> Iterator
> 
>  Please take a look and then start to vote if you have free time.
> thanks.
> 
>  vote:
> https://lists.apache.org/thread/kjdtyfg5xytn60q0qvxhfopzmfp9tsxr
> >>
> >>
>
>

Re: Regarding Jira Account Creation

[Guozhang Wang]

Hi Rohit,

You can request the account creation in this url:
https://selfserve.apache.org/jira-account.html

On Tue, Mar 14, 2023 at 7:19 AM Rohit -  wrote:
>
> Hi, I am Rohit, working as a Software Engineer at Confluent. Recently I
> started working on https://issues.apache.org/jira/browse/KAFKA-14401. And
> raised a PR for the same here(https://github.com/apache/kafka/pull/13361).
> I wanted to have a JIRA account to assign the issue to myself and work on
> KAFKA-14401. Can you please help me with the creation of a JIRA account
> with username "rohits64" and email as either rohitsanja...@gmail.com or
> roh...@confluent.io.
>
> Thanks and Regards
> Rohit
> Software Engineer
> Confluent

Re: [VOTE] KIP-641 An new java interface to replace 'kafka.common.MessageReader'

[Chia-Ping Tsai]

the vote for KIP-641 is completed

+3 (binding) Luke Chen , Mickael Maison, Chris Egerton
+2 (non-binding) Alexandre Dupriez , Federico Valeri

thanks for all feedback and suggestions
--
chia-ping

On 2023/02/18 08:36:50 Chia-Ping Tsai wrote:
> Hi,
> 
> I'd like to start the vote on KIP-614: An new java interface to replace 
> 'kafka.common.MessageReader'
> 
> KIP-614: 
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158866569
> 
> thread: 
> https://lists.apache.org/thread.html/r6db6708f64345bb8fe0d573e05014fb790e69d501f21f855ca65619a%40%3Cdev.kafka.apache.org%3E
> 
> Cheers,
> Chia-Ping

Re: [VOTE] KIP-641 An new java interface to replace 'kafka.common.MessageReader'

[Chris Egerton]

+1 (binding). Thanks!

On Mon, Mar 13, 2023 at 11:39 AM Chia-Ping Tsai  wrote:

> ping for more voting :_
>
> On 2023/02/18 08:36:50 Chia-Ping Tsai wrote:
> > Hi,
> >
> > I'd like to start the vote on KIP-614: An new java interface to replace
> 'kafka.common.MessageReader'
> >
> > KIP-614:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158866569
> >
> > thread:
> https://lists.apache.org/thread.html/r6db6708f64345bb8fe0d573e05014fb790e69d501f21f855ca65619a%40%3Cdev.kafka.apache.org%3E
> >
> > Cheers,
> > Chia-Ping
>

Re: [DISCUSS] KIP-641 An new java interface to replace 'kafka.common.MessageReader'

[Chris Egerton]

Fair enough 

On Tue, Mar 14, 2023 at 12:23 PM Chia-Ping Tsai  wrote:

>
>
> > Chris Egerton  於 2023年3月15日 上午12:04 寫道：
> >
> > Hi Chia-Ping,
> >
> > Thanks for the KIP. I find the interface definition really polished and
> > intuitive! One small question--I noticed the change of the package to
> > "org.apache.kafka.clients.tool". It doesn't look like there's any
> precedent
> > for using that package. We also use the "org.apache.kafka.common" package
> > for the "MessageFormatter" interface, which is in some ways the
> equivalent
> > pluggable interface for the console consumer.
>
> It seems to me those pluggable interfaces (MessageFormatter and
> RecordReader) should not be a part of “common” package. They are used by
> specify tools only. `Configurable`, by contrast, is good to be located at
> `common` package since it is used widely in our code base.
>
>
> >
> > Do we know if it's necessary to preserve the Checkstyle import
> limitations
> > (which I'm assuming are what motivated the shift in package name)? It
> seems
> > like it might be better to just relax that constraint in order to
> colocate
> > the pluggable interfaces for our console producer/consumer.
>
> I love checkstyle import, and that is one of reason the KIP isolates the
> new interface to a separate package. We have to add `allowed rule` one by
> one if those dedicated interfaces are using the `common` package. The
> constraint of new package can be relax to colocate the pluggable interfaces
> (used by tools), and the `relax` won’t impact other existent packages.
>
>
> >
> > Cheers,
> >
> > Chris
> >
> > On Tue, Mar 7, 2023 at 6:30 AM Chia-Ping Tsai 
> wrote:
> >
> >> hi Mickael
> >>
> >>> ?> configs) in the Compatibility, Deprecation, and Migration Plan, I
> >>> guess these can be removed now.
> >>
> >> Done! thanks for feedback
> >>
> >>> Mickael Maison  於 2023年3月7日 下午7:13 寫道：
> >>>
> >>> Hi Chia-Ping,
> >>>
> >>> The new API looks good.
> >>> I still see mentions to configure(InputStream inputStream, Map >>> ?> configs) in the Compatibility, Deprecation, and Migration Plan, I
> >>> guess these can be removed now.
> >>>
> >>> Thanks,
> >>> Mickael
> >>>
> >>> On Fri, Mar 3, 2023 at 2:37 PM Chia-Ping Tsai 
> >> wrote:
> 
>  Dear all,
> 
>  there are some changes for KIP-614
> 
> 
> >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158866569
> 
>  1) the interface RecordReader extends Configurable.
>  2) the input stream is removed from RecordReader#configure method
>  3) RecordReader#readRecords accept InputStream as argument, and the
> >> returned type is changed from single ProducerRecord to
> >> Iterator
> 
>  Please take a look and then start to vote if you have free time.
> thanks.
> 
>  vote:
> https://lists.apache.org/thread/kjdtyfg5xytn60q0qvxhfopzmfp9tsxr
> >>
> >>
>
>

Re: [DISCUSS] KIP-641 An new java interface to replace 'kafka.common.MessageReader'

[Chia-Ping Tsai]

> Chris Egerton  於 2023年3月15日 上午12:04 寫道：
> 
> Hi Chia-Ping,
> 
> Thanks for the KIP. I find the interface definition really polished and
> intuitive! One small question--I noticed the change of the package to
> "org.apache.kafka.clients.tool". It doesn't look like there's any precedent
> for using that package. We also use the "org.apache.kafka.common" package
> for the "MessageFormatter" interface, which is in some ways the equivalent
> pluggable interface for the console consumer.

It seems to me those pluggable interfaces (MessageFormatter and RecordReader) 
should not be a part of “common” package. They are used by specify tools only. 
`Configurable`, by contrast, is good to be located at `common` package since it 
is used widely in our code base.


> 
> Do we know if it's necessary to preserve the Checkstyle import limitations
> (which I'm assuming are what motivated the shift in package name)? It seems
> like it might be better to just relax that constraint in order to colocate
> the pluggable interfaces for our console producer/consumer.

I love checkstyle import, and that is one of reason the KIP isolates the new 
interface to a separate package. We have to add `allowed rule` one by one if 
those dedicated interfaces are using the `common` package. The constraint of 
new package can be relax to colocate the pluggable interfaces (used by tools), 
and the `relax` won’t impact other existent packages.


> 
> Cheers,
> 
> Chris
> 
> On Tue, Mar 7, 2023 at 6:30 AM Chia-Ping Tsai  wrote:
> 
>> hi Mickael
>> 
>>> ?> configs) in the Compatibility, Deprecation, and Migration Plan, I
>>> guess these can be removed now.
>> 
>> Done! thanks for feedback
>> 
>>> Mickael Maison  於 2023年3月7日 下午7:13 寫道：
>>> 
>>> Hi Chia-Ping,
>>> 
>>> The new API looks good.
>>> I still see mentions to configure(InputStream inputStream, Map>> ?> configs) in the Compatibility, Deprecation, and Migration Plan, I
>>> guess these can be removed now.
>>> 
>>> Thanks,
>>> Mickael
>>> 
>>> On Fri, Mar 3, 2023 at 2:37 PM Chia-Ping Tsai 
>> wrote:
 
 Dear all,
 
 there are some changes for KIP-614
 
 
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158866569
 
 1) the interface RecordReader extends Configurable.
 2) the input stream is removed from RecordReader#configure method
 3) RecordReader#readRecords accept InputStream as argument, and the
>> returned type is changed from single ProducerRecord to
>> Iterator
 
 Please take a look and then start to vote if you have free time. thanks.
 
 vote: https://lists.apache.org/thread/kjdtyfg5xytn60q0qvxhfopzmfp9tsxr
>> 
>> 

Re: [DISCUSS] KIP-641 An new java interface to replace 'kafka.common.MessageReader'

[Chris Egerton]

Hi Chia-Ping,

Thanks for the KIP. I find the interface definition really polished and
intuitive! One small question--I noticed the change of the package to
"org.apache.kafka.clients.tool". It doesn't look like there's any precedent
for using that package. We also use the "org.apache.kafka.common" package
for the "MessageFormatter" interface, which is in some ways the equivalent
pluggable interface for the console consumer.

Do we know if it's necessary to preserve the Checkstyle import limitations
(which I'm assuming are what motivated the shift in package name)? It seems
like it might be better to just relax that constraint in order to colocate
the pluggable interfaces for our console producer/consumer.

Cheers,

Chris

On Tue, Mar 7, 2023 at 6:30 AM Chia-Ping Tsai  wrote:

> hi Mickael
>
> > ?> configs) in the Compatibility, Deprecation, and Migration Plan, I
> > guess these can be removed now.
>
> Done! thanks for feedback
>
> > Mickael Maison  於 2023年3月7日 下午7:13 寫道：
> >
> > Hi Chia-Ping,
> >
> > The new API looks good.
> > I still see mentions to configure(InputStream inputStream, Map > ?> configs) in the Compatibility, Deprecation, and Migration Plan, I
> > guess these can be removed now.
> >
> > Thanks,
> > Mickael
> >
> > On Fri, Mar 3, 2023 at 2:37 PM Chia-Ping Tsai 
> wrote:
> >>
> >> Dear all,
> >>
> >> there are some changes for KIP-614
> >>
> >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158866569
> >>
> >> 1) the interface RecordReader extends Configurable.
> >> 2) the input stream is removed from RecordReader#configure method
> >> 3) RecordReader#readRecords accept InputStream as argument, and the
> returned type is changed from single ProducerRecord to
> Iterator
> >>
> >> Please take a look and then start to vote if you have free time. thanks.
> >>
> >> vote: https://lists.apache.org/thread/kjdtyfg5xytn60q0qvxhfopzmfp9tsxr
>
>

[jira] [Resolved] (KAFKA-10228) producer: NETWORK_EXCEPTION is thrown instead of a request timeout

[Kirk True (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-10228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kirk True resolved KAFKA-10228.
---
Fix Version/s: 3.5.0
   Resolution: Duplicate

> producer: NETWORK_EXCEPTION is thrown instead of a request timeout
> --
>
> Key: KAFKA-10228
> URL: https://issues.apache.org/jira/browse/KAFKA-10228
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Affects Versions: 2.3.1
>Reporter: Christian Becker
>Assignee: Kirk True
>Priority: Major
> Fix For: 3.5.0
>
>
> We're currently seeing an issue with the java client (producer), when message 
> producing runs into a timeout. Namely a NETWORK_EXCEPTION is thrown instead 
> of a timeout exception.
> *Situation and relevant code:*
> Config
> {code:java}
> request.timeout.ms: 200
> retries: 3
> acks: all{code}
> {code:java}
> for (UnpublishedEvent event : unpublishedEvents) {
> ListenableFuture> future;
> future = kafkaTemplate.send(new ProducerRecord<>(event.getTopic(), 
> event.getKafkaKey(), event.getPayload()));
> futures.add(future.completable());
> }
> CompletableFuture.allOf(futures.stream().toArray(CompletableFuture[]::new)).join();{code}
> We're using the KafkaTemplate from SpringBoot here, but it shouldn't matter, 
> as it's merely a wrapper. There we put in batches of messages to be sent.
> 200ms later, we can see the following in the logs: (not sure about the order, 
> they've arrived in the same ms, so our logging system might not display them 
> in the right order)
> {code:java}
> [Producer clientId=producer-1] Received invalid metadata error in produce 
> request on partition events-6 due to 
> org.apache.kafka.common.errors.NetworkException: The server disconnected 
> before a response was received.. Going to request metadata update now
> [Producer clientId=producer-1] Got error produce response with correlation id 
> 3094 on topic-partition events-6, retrying (2 attempts left). Error: 
> NETWORK_EXCEPTION {code}
> There is also a corresponding error on the broker (within a few ms):
> {code:java}
> Attempting to send response via channel for which there is no open 
> connection, connection id XXX (kafka.network.Processor) {code}
> This was somewhat unexpected and sent us for a hunt across the infrastructure 
> for possible connection issues, but we've found none.
> Side note: In some cases the retries worked and the messages were 
> successfully produced.
> Only after many hours of heavy debugging, we've noticed, that the error might 
> be related to the low timeout setting. We've removed that setting now, as it 
> was a remnant from the past and no longer valid for our use-case. However in 
> order to avoid other people having that issue again and to simplify future 
> debugging, some form of timeout exception should be thrown.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Regarding Jira Account Creation

[Rohit -]

Hi, I am Rohit, working as a Software Engineer at Confluent. Recently I
started working on https://issues.apache.org/jira/browse/KAFKA-14401. And
raised a PR for the same here(https://github.com/apache/kafka/pull/13361).
I wanted to have a JIRA account to assign the issue to myself and work on
KAFKA-14401. Can you please help me with the creation of a JIRA account
with username "rohits64" and email as either rohitsanja...@gmail.com or
roh...@confluent.io.

Thanks and Regards
Rohit
Software Engineer
Confluent

[jira] [Resolved] (KAFKA-14803) topic deletion bug

[Chris Egerton (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Egerton resolved KAFKA-14803.
---
Resolution: Duplicate

> topic deletion bug
> --
>
> Key: KAFKA-14803
> URL: https://issues.apache.org/jira/browse/KAFKA-14803
> Project: Kafka
>  Issue Type: Bug
>  Components: controller, replication
>Affects Versions: 3.3.2
> Environment: AWS m5.xlarge EC2 instance
>Reporter: Behavox
>Priority: Major
> Attachments: server.properties
>
>
> topic deletion doesn't work as expected when attempting to delete topic(s), 
> after successful deletion topic is recreated in a multi-controller 
> environment with 3 controllers and ReplicationFactor: 2
> How to reproduce - attempt to delete topic. Topic is removed successfully and 
> recreated right after removal. Example below shows a single topic named 
> example-topic. We have a total count of 17000 topics in the affected cluster. 
>  
> Our config is attached. 
> Run 1
> [2023-03-10 16:16:45,625] INFO [Controller 1] Removed topic example-topic 
> with ID fh_aQcc3Sf2yVBTMrltBlQ. 
> (org.apache.kafka.controller.ReplicationControlManager)
> [2023-03-10 16:19:04,722] INFO [Controller 1] Created topic example-topic 
> with topic ID a-7OZG_XQhiCatOBft-9-g. 
> (org.apache.kafka.controller.ReplicationControlManager)
> [2023-03-10 16:16:45,730] INFO [Controller 2] Removed topic example-topic 
> with ID fh_aQcc3Sf2yVBTMrltBlQ. 
> (org.apache.kafka.controller.ReplicationControlManager)
> [2023-03-10 16:19:04,851] INFO [Controller 2] Created topic example-topic 
> with topic ID a-7OZG_XQhiCatOBft-9-g. 
> (org.apache.kafka.controller.ReplicationControlManager)
> [2023-03-10 16:16:45,837] INFO [Controller 3] Removed topic example-topic 
> with ID fh_aQcc3Sf2yVBTMrltBlQ. 
> (org.apache.kafka.controller.ReplicationControlManager)
> [2023-03-10 16:19:04,833] INFO [Controller 3] Created topic example-topic 
> with topic ID a-7OZG_XQhiCatOBft-9-g. 
> (org.apache.kafka.controller.ReplicationControlManager)
> Run 2
> [2023-03-10 16:20:22,469] INFO [Controller 1] Removed topic example-topic 
> with ID a-7OZG_XQhiCatOBft-9-g. 
> (org.apache.kafka.controller.ReplicationControlManager)
> [2023-03-10 16:22:19,711] INFO [Controller 1] Created topic example-topic 
> with topic ID xxlJlIe_SvqQHtfgbX2eLA. 
> (org.apache.kafka.controller.ReplicationControlManager)
> [2023-03-10 16:20:22,674] INFO [Controller 2] Removed topic example-topic 
> with ID a-7OZG_XQhiCatOBft-9-g. 
> (org.apache.kafka.controller.ReplicationControlManager)
> [2023-03-10 16:22:20,022] INFO [Controller 2] Created topic example-topic 
> with topic ID xxlJlIe_SvqQHtfgbX2eLA. 
> (org.apache.kafka.controller.ReplicationControlManager)
> [2023-03-10 16:20:22,674] INFO [Controller 3] Removed topic example-topic 
> with ID a-7OZG_XQhiCatOBft-9-g. 
> (org.apache.kafka.controller.ReplicationControlManager)
> [2023-03-10 16:22:20,020] INFO [Controller 3] Created topic example-topic 
> with topic ID xxlJlIe_SvqQHtfgbX2eLA. 
> (org.apache.kafka.controller.ReplicationControlManager)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14808) Partition becomes leaderless when new partition reassignment removes the adding replica

[Shenglong Zhang (Jira)]

Shenglong Zhang created KAFKA-14808:
---

 Summary: Partition becomes leaderless when new partition 
reassignment removes the adding replica
 Key: KAFKA-14808
 URL: https://issues.apache.org/jira/browse/KAFKA-14808
 Project: Kafka
  Issue Type: Bug
  Components: controller
Affects Versions: 3.4.0
Reporter: Shenglong Zhang


If there is ongoing partition reassignment and any adding replica has been 
elected as leader (due to preferred leader election or other reason), the 
partition will immediately becomes leaderless on receiving a new partition 
reassignment which removes that adding replica.

1) partition-0 has replicas [0, 2]

2) partition-0 is being reassigned to [1, 0, 2], and somehow this reassignment 
is stuck (e.g. broker 2 is down).

3) Preferred leader election is triggered, and broker 1 is elected as leader.

4) When submitting a new partition reassignment to [2, 0, 3], which remove 
broker 1 and add broker 3, partition will become leaderless.

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-14804) Connect docs fail to build with Gradle Swagger plugin 2.2.8

[Mickael Maison (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mickael Maison resolved KAFKA-14804.

Fix Version/s: 3.5.0
   Resolution: Fixed

> Connect docs fail to build with Gradle Swagger plugin 2.2.8
> ---
>
> Key: KAFKA-14804
> URL: https://issues.apache.org/jira/browse/KAFKA-14804
> Project: Kafka
>  Issue Type: Bug
>Reporter: David Arthur
>Assignee: Mickael Maison
>Priority: Minor
> Fix For: 3.5.0
>
>
> There is an incompatibility somewhere between versions 2.2.0 and 2.2.8 that 
> cause the following error when building the connect docs:
> {code}
> Caused by: org.gradle.api.GradleException: 
> io.swagger.v3.jaxrs2.integration.SwaggerLoader.setOpenAPI31(java.lang.Boolean)
> at 
> io.swagger.v3.plugins.gradle.tasks.ResolveTask.resolve(ResolveTask.java:458)
> at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
> at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:125)
> at 
> org.gradle.api.internal.project.taskfactory.StandardTaskAction.doExecute(StandardTaskAction.java:58)
> at 
> org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:51)
> at 
> org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:29)
> at 
> org.gradle.api.internal.tasks.execution.TaskExecution$3.run(TaskExecution.java:242)
> at 
> org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:29)
> at 
> org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:26)
> at 
> org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
> at 
> org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
> at 
> org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
> at 
> org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
> at 
> org.gradle.internal.operations.DefaultBuildOperationRunner.run(DefaultBuildOperationRunner.java:47)
> at 
> org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:68)
> at 
> org.gradle.api.internal.tasks.execution.TaskExecution.executeAction(TaskExecution.java:227)
> at 
> org.gradle.api.internal.tasks.execution.TaskExecution.executeActions(TaskExecution.java:210)
> at 
> org.gradle.api.internal.tasks.execution.TaskExecution.executeWithPreviousOutputFiles(TaskExecution.java:193)
> at 
> org.gradle.api.internal.tasks.execution.TaskExecution.execute(TaskExecution.java:166)
> at 
> org.gradle.internal.execution.steps.ExecuteStep.executeInternal(ExecuteStep.java:93)
> at 
> org.gradle.internal.execution.steps.ExecuteStep.access$000(ExecuteStep.java:44)
> at 
> org.gradle.internal.execution.steps.ExecuteStep$1.call(ExecuteStep.java:57)
> at 
> org.gradle.internal.execution.steps.ExecuteStep$1.call(ExecuteStep.java:54)
> at 
> org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:204)
> at 
> org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:199)
> at 
> org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
> at 
> org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
> at 
> org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
> at 
> org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
> at 
> org.gradle.internal.operations.DefaultBuildOperationRunner.call(DefaultBuildOperationRunner.java:53)
> at 
> org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:73)
> at 
> org.gradle.internal.execution.steps.ExecuteStep.execute(ExecuteStep.java:54)
> at 
> org.gradle.internal.execution.steps.ExecuteStep.execute(ExecuteStep.java:44)
> at 
> 

[jira] [Created] (KAFKA-14807) MirrorMaker2 config source.consumer.auto.offset.reset=latest leading to the pause of replication of consumer groups

[Zhaoli (Jira)]

Zhaoli created KAFKA-14807:
--

 Summary: MirrorMaker2 config 
source.consumer.auto.offset.reset=latest leading to the pause of replication of 
consumer groups
 Key: KAFKA-14807
 URL: https://issues.apache.org/jira/browse/KAFKA-14807
 Project: Kafka
  Issue Type: Bug
  Components: mirrormaker
Affects Versions: 3.3.2, 3.3.1, 3.4.0
 Environment: centos7
Reporter: Zhaoli


We use MirrorMaker2 to replicate messages and consumergroup offsets from kafka 
cluster `source` to cluster `target`.
In order to reduce the load on the source cluster, we add this configuration to 
mm2 to avoid replicate the whole history messages:
{code:java}
source.consumer.auto.offset.reset=latest {code}
After that, we found part of the consumergroup offsets have stopped replicating.

The common characteristic of these consumergroups is  their EMPTY status，which 
means they have no active members at that monent. All the active 
consumergroups‘ offset replication work as normal.

After researching the source code，we found this is because the configuration 
above also affect the consumption of topic `mm2-offset-syncs`, therefore the 
map `
offsetSyncs` dosen't hold the whole topicPartitions, and the lost 
topicPartitions lead to the pause of replication of the EMPTY consumer groups.
{code:java}
private final Map offsetSyncs = new HashMap<>(); 
{code}
 

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14806) Add connection timeout in PlaintextSender used by SelectorTests

[Alexandre Dupriez (Jira)]

Alexandre Dupriez created KAFKA-14806:
-

 Summary: Add connection timeout in PlaintextSender used by 
SelectorTests
 Key: KAFKA-14806
 URL: https://issues.apache.org/jira/browse/KAFKA-14806
 Project: Kafka
  Issue Type: Test
Reporter: Alexandre Dupriez


Tests in `SelectorTest` can fail due to spurious connection timeouts. One 
example can be found in [this 
build|https://github.com/apache/kafka/pull/13378/checks?check_run_id=11970595528]
 where the client connection the `PlaintextSender` tried to open could not be 
established before the test timed out.

It may be worth enforcing connection timeout and retries if this can add to the 
selector tests resiliency. Note that `PlaintextSender` is only used by the 
`SelectorTest` so the scope of the change would remain local.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Kafka Stream Metrics (3.3 / KAFKA-13945) and Documentation

[Bruno Cadonna]

Hi Neil,

I think you are right that the type is wrong. Thank you for spotting the 
mistake!


Since there is a MBean name column, I do not think we need to introduce 
a new section. We just need to change 
"type=stream-processor-node-metrics" to "type=stream-topic-metrics" for 
metrics


bytes-consumed-total
bytes-produced-total
records-consumed-total
records-produced-total

Those metrics have already the "topic" property added to the MBean name. 
All other metrics in the table do not have that property.


What do you think?

Best,
Bruno

On 13.03.23 22:59, Neil Buesing wrote:

I have been looking at the new metrics in KAFKA-13945 (KIP-846). The
documentation states that this metrics are procesor-node metrics

https://kafka.apache.org/documentation/#kafka_streams_node_monitoring


*kafka.streams:type=stream-processor-node-metrics,thread-id=([-.\w]+),task-id=([-.\w]+),processor-node-id=([-.\w]+),topic=([-.\w]+)*

My observations is that the documentation should have a
"stream-topic-metrics" section


*kafka.streams:type=stream-topic-metrics,thread-id=([-.\w]+),task-id=([-.\w]+),processor-node-id=([-.\w]+),topic=([-.\w]+)*

It looks like they are in TopicMetrics.java which was created for this
effort.

I believe this is just a minor document change adding a new section "Task
Metrics" and placing these 4 metrics in this section with the given type.

Thanks,

-Neil