Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1706

2023-03-24 Thread Apache Jenkins Server 
See 


Changes:


--
[...truncated 3160908 lines...]
[2023-03-25T02:09:13.185Z] at 
org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:385)
[2023-03-25T02:09:13.185Z] ... 5 more
[2023-03-25T02:09:13.185Z] 
[2023-03-25T02:09:13.185Z] Caused by:
[2023-03-25T02:09:13.185Z] java.lang.NullPointerException
[2023-03-25T02:09:13.185Z] 
[2023-03-25T02:09:13.185Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 178 > StoreQueryIntegrationTest > 
shouldQuerySpecificActivePartitionStores() STARTED
[2023-03-25T02:09:14.121Z] 
[2023-03-25T02:09:14.121Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 178 > StoreQueryIntegrationTest > 
shouldQuerySpecificActivePartitionStores() PASSED
[2023-03-25T02:09:14.121Z] 
[2023-03-25T02:09:14.121Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 178 > StoreQueryIntegrationTest > 
shouldFailWithIllegalArgumentExceptionWhenIQPartitionerReturnsMultiplePartitions()
 STARTED
[2023-03-25T02:09:16.918Z] 
[2023-03-25T02:09:16.918Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 178 > StoreQueryIntegrationTest > 
shouldFailWithIllegalArgumentExceptionWhenIQPartitionerReturnsMultiplePartitions()
 PASSED
[2023-03-25T02:09:16.918Z] 
[2023-03-25T02:09:16.918Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 178 > StoreQueryIntegrationTest > 
shouldQueryAllStalePartitionStores() STARTED
[2023-03-25T02:09:20.496Z] 
[2023-03-25T02:09:20.496Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 178 > StoreQueryIntegrationTest > 
shouldQueryAllStalePartitionStores() PASSED
[2023-03-25T02:09:20.496Z] 
[2023-03-25T02:09:20.496Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 178 > StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStoresMultiStreamThreads() STARTED
[2023-03-25T02:09:24.073Z] 
[2023-03-25T02:09:24.073Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 178 > StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStoresMultiStreamThreads() PASSED
[2023-03-25T02:09:24.073Z] 
[2023-03-25T02:09:24.073Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 178 > StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStores() STARTED
[2023-03-25T02:09:28.682Z] 
[2023-03-25T02:09:28.682Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 178 > StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStores() PASSED
[2023-03-25T02:09:28.682Z] 
[2023-03-25T02:09:28.682Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 178 > StoreQueryIntegrationTest > 
shouldQueryOnlyActivePartitionStoresByDefault() STARTED
[2023-03-25T02:09:32.259Z] 
[2023-03-25T02:09:32.259Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 178 > StoreQueryIntegrationTest > 
shouldQueryOnlyActivePartitionStoresByDefault() PASSED
[2023-03-25T02:09:32.259Z] 
[2023-03-25T02:09:32.259Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 178 > StoreQueryIntegrationTest > 
shouldQueryStoresAfterAddingAndRemovingStreamThread() STARTED
[2023-03-25T02:09:38.177Z] 
[2023-03-25T02:09:38.177Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 178 > StoreQueryIntegrationTest > 
shouldQueryStoresAfterAddingAndRemovingStreamThread() PASSED
[2023-03-25T02:09:38.177Z] 
[2023-03-25T02:09:38.177Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 178 > StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStoresMultiStreamThreadsNamedTopology() STARTED
[2023-03-25T02:09:41.819Z] 
[2023-03-25T02:09:41.819Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 178 > StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStoresMultiStreamThreadsNamedTopology() PASSED
[2023-03-25T02:09:42.757Z] 
[2023-03-25T02:09:42.757Z] 619 tests completed, 151 failed
[2023-03-25T02:09:44.547Z] 
org.apache.kafka.streams.integration.PauseResumeIntegrationTest.shouldPauseAndResumeAllKafkaStreamsWithNamedTopologies(boolean)[2]
 failed, log available in 
/home/jenkins/workspace/Kafka_kafka_trunk/streams/build/reports/testOutput/org.apache.kafka.streams.integration.PauseResumeIntegrationTest.shouldPauseAndResumeAllKafkaStreamsWithNamedTopologies(boolean)[2].test.stdout
[2023-03-25T02:09:44.547Z] 
[2023-03-25T02:09:44.547Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 179 > PauseResumeIntegrationTest > 
shouldPauseAndResumeAllKafkaStreamsWithNamedTopologies(boolean) > 
org.apache.kafka.streams.integration.PauseResumeIntegrationTest.shouldPauseAndResumeAllKafkaStreamsWithNamedTopologies(boolean)[2]
 FAILED
[2023-03-25T02:09:44.547Z] java.lang.AssertionError: Assertion failed with 
an exception after 6 ms
[2023-03-25T02:09:44.547Z] at

[jira] [Resolved] (KAFKA-14365) Extract common logic from Fetcher

2023-03-24 Thread Kirk True (Jira) 


 [ 
https://issues.apache.org/jira/browse/KAFKA-14365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kirk True resolved KAFKA-14365.
---
Fix Version/s: 3.5.0
 Reviewer: Guozhang Wang
   Resolution: Fixed

> Extract common logic from Fetcher
> -
>
> Key: KAFKA-14365
> URL: https://issues.apache.org/jira/browse/KAFKA-14365
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients, consumer
>Reporter: Kirk True
>Assignee: Kirk True
>Priority: Major
> Fix For: 3.5.0
>
>
> The {{Fetcher}} class is used internally by the {{KafkaConsumer}} to fetch 
> records from the brokers. There is ongoing work to create a new consumer 
> implementation with a significantly refactored threading model. The threading 
> refactor work requires a similarly refactored {{{}Fetcher{}}}.
> This task includes refactoring {{Fetcher}} by extracting out some common 
> logic to allow forthcoming implementations to leverage it.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14847) Separate the callers of commitAllTasks v.s. commitTasks for EOS(-v2) and ALOS

2023-03-24 Thread Guozhang Wang (Jira) 
Guozhang Wang created KAFKA-14847:
-

 Summary: Separate the callers of commitAllTasks v.s. commitTasks 
for EOS(-v2) and ALOS
 Key: KAFKA-14847
 URL: https://issues.apache.org/jira/browse/KAFKA-14847
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: Guozhang Wang


Today, EOS-v2/v1 and ALOS shares the same internal callpath inside 
TaskManager/TaskExecutor for committing tasks from various scenarios, the call 
path {{commitTasksAndMaybeUpdateCommitableOffsets}} -> 
{{commitOffsetsOrTransaction}} takes in a list of tasks as its input, which can 
be a subset of the tasks that thread / task manager owns. For EOS-v1 / ALOS, 
this is fine to commit just a subset of the tasks; however for EOS-v1, since 
all tasks participate in the same txn it could lead to dangerous violations, 
and today we are relying on all the callers of the commit function to make sure 
that the list of tasks they passed in, under EOS-v2, would still not violate 
the semantics. As summarized today (thanks to Matthias), today that callee 
could be triggered in the following cases:

1) Inside handleRevocation() -- this is a clean path, an we add all non-revoked 
tasks with commitNeeded() flag set to the commit -- so this seems to be fine.
2) tryCloseCleanAllActiveTasks() -- here we only call it, if 
tasksToCloseDirty.isEmpty() -- so it seems fine, too.
3) commit() with a list of task handed in -- we call commit() inside the TM 
three time
3.a) inside commitAll() as commit(tasks.values()) (passing in all tasks)
3.b) inside maybeCommitActiveTasksPerUserRequested as 
commit(activeTaskIterable()); (passing in all tasks)
3.c) inside handleCorruption() -- here, we only consider RUNNING and RESTORING 
tasks, which are not corrupted -- note we only throw a TaskCorruptedException 
during restore state initialization, thus, corrupted tasks did not process 
anything yet, and all other tasks should be clean to be committed.
3.d) commitSuccessfullyProcessedTasks() -- under EOS-v2, as we just commit a 
subset of tasks' source offsets while at the same time we still commit those 
unsuccessful task's outgoing records if there are any.

Just going through this list of callers itself, as demonstrated above, is 
already pretty complex, and very vulnerable to bugs. It's better to not rely on 
the callers, but the callees to make sure that's the case. More concretely, I 
think we can introduce a new function called {{commitAllTasks}} such that under 
EOS-v2, the caller always call {{commitAllTasks}} instead, and if there are 
some tasks that should not be committed because we know they have not processed 
any data, the {{commitAllTasks}} callee itself would do some clever filtering 
internally.

Given its scope, I think it's better to do this refactoring after EOS-v1 is 
removed.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1705

2023-03-24 Thread Apache Jenkins Server 
See 


Changes:


--
[...truncated 543530 lines...]
[2023-03-24T21:45:44.264Z] > Task :group-coordinator:classes UP-TO-DATE
[2023-03-24T21:45:44.264Z] > Task :server-common:compileTestJava UP-TO-DATE
[2023-03-24T21:45:44.264Z] > Task :server-common:testClasses UP-TO-DATE
[2023-03-24T21:45:44.264Z] > Task :connect:json:compileTestJava UP-TO-DATE
[2023-03-24T21:45:44.264Z] > Task :connect:json:testClasses UP-TO-DATE
[2023-03-24T21:45:44.264Z] > Task :raft:compileTestJava UP-TO-DATE
[2023-03-24T21:45:44.264Z] > Task :raft:testClasses UP-TO-DATE
[2023-03-24T21:45:44.264Z] > Task :core:compileJava NO-SOURCE
[2023-03-24T21:45:45.507Z] > Task :connect:json:testJar
[2023-03-24T21:45:45.507Z] > Task :storage:api:compileTestJava UP-TO-DATE
[2023-03-24T21:45:45.507Z] > Task :storage:api:testClasses UP-TO-DATE
[2023-03-24T21:45:45.507Z] > Task :group-coordinator:compileTestJava UP-TO-DATE
[2023-03-24T21:45:45.507Z] > Task :group-coordinator:testClasses UP-TO-DATE
[2023-03-24T21:45:45.507Z] > Task :connect:json:testSrcJar
[2023-03-24T21:45:45.507Z] > Task :metadata:compileTestJava UP-TO-DATE
[2023-03-24T21:45:45.507Z] > Task :metadata:testClasses UP-TO-DATE
[2023-03-24T21:45:45.507Z] > Task 
:streams:generateMetadataFileForMavenJavaPublication
[2023-03-24T21:45:45.507Z] > Task 
:clients:generateMetadataFileForMavenJavaPublication
[2023-03-24T21:45:50.138Z] 
[2023-03-24T21:45:50.138Z] > Task :connect:api:javadoc
[2023-03-24T21:45:50.138Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk_2/connect/api/src/main/java/org/apache/kafka/connect/source/SourceRecord.java:44:
 warning - Tag @link: reference not found: org.apache.kafka.connect.data
[2023-03-24T21:45:52.158Z] 1 warning
[2023-03-24T21:45:52.158Z] 
[2023-03-24T21:45:52.158Z] > Task :connect:api:copyDependantLibs UP-TO-DATE
[2023-03-24T21:45:52.158Z] > Task :connect:api:jar UP-TO-DATE
[2023-03-24T21:45:52.158Z] > Task 
:connect:api:generateMetadataFileForMavenJavaPublication
[2023-03-24T21:45:52.158Z] > Task :connect:json:copyDependantLibs UP-TO-DATE
[2023-03-24T21:45:52.158Z] > Task :connect:json:jar UP-TO-DATE
[2023-03-24T21:45:52.158Z] > Task 
:connect:json:generateMetadataFileForMavenJavaPublication
[2023-03-24T21:45:52.158Z] > Task :connect:api:javadocJar
[2023-03-24T21:45:52.158Z] > Task 
:connect:json:publishMavenJavaPublicationToMavenLocal
[2023-03-24T21:45:52.158Z] > Task :connect:json:publishToMavenLocal
[2023-03-24T21:45:52.158Z] > Task :connect:api:compileTestJava UP-TO-DATE
[2023-03-24T21:45:52.158Z] > Task :connect:api:testClasses UP-TO-DATE
[2023-03-24T21:45:52.158Z] > Task :connect:api:testJar
[2023-03-24T21:45:52.158Z] > Task :connect:api:testSrcJar
[2023-03-24T21:45:52.158Z] > Task 
:connect:api:publishMavenJavaPublicationToMavenLocal
[2023-03-24T21:45:52.158Z] > Task :connect:api:publishToMavenLocal
[2023-03-24T21:45:56.111Z] > Task :streams:javadoc
[2023-03-24T21:45:56.111Z] > Task :streams:javadocJar
[2023-03-24T21:45:57.882Z] 
[2023-03-24T21:45:57.882Z] > Task :clients:javadoc
[2023-03-24T21:45:57.882Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk_2/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/secured/package-info.java:21:
 warning - Tag @link: reference not found: 
org.apache.kafka.common.security.oauthbearer
[2023-03-24T21:45:58.829Z] 1 warning
[2023-03-24T21:45:59.886Z] 
[2023-03-24T21:45:59.886Z] > Task :clients:javadocJar
[2023-03-24T21:46:00.891Z] > Task :clients:srcJar
[2023-03-24T21:46:01.838Z] > Task :clients:testJar
[2023-03-24T21:46:01.838Z] > Task :clients:testSrcJar
[2023-03-24T21:46:01.838Z] > Task 
:clients:publishMavenJavaPublicationToMavenLocal
[2023-03-24T21:46:01.838Z] > Task :clients:publishToMavenLocal
[2023-03-24T21:46:15.967Z] > Task :core:compileScala
[2023-03-24T21:47:51.835Z] > Task :core:classes
[2023-03-24T21:47:51.835Z] > Task :core:compileTestJava NO-SOURCE
[2023-03-24T21:48:19.787Z] > Task :core:compileTestScala
[2023-03-24T21:49:54.357Z] > Task :core:testClasses
[2023-03-24T21:49:54.357Z] > Task :streams:compileTestJava UP-TO-DATE
[2023-03-24T21:49:54.357Z] > Task :streams:testClasses UP-TO-DATE
[2023-03-24T21:49:54.357Z] > Task :streams:testJar
[2023-03-24T21:49:54.357Z] > Task :streams:testSrcJar
[2023-03-24T21:49:54.357Z] > Task 
:streams:publishMavenJavaPublicationToMavenLocal
[2023-03-24T21:49:54.357Z] > Task :streams:publishToMavenLocal
[2023-03-24T21:49:54.357Z] 
[2023-03-24T21:49:54.357Z] Deprecated Gradle features were used in this build, 
making it incompatible with Gradle 9.0.
[2023-03-24T21:49:54.357Z] 
[2023-03-24T21:49:54.357Z] You can use '--warning-mode all' to show the 
individual deprecation warnings and determine if they come from your own 
scripts or plugins.
[2023-03-24T21:49:54.357Z] 
[2023-03-24T21:49:54.357Z] See 
https://docs.gradle.org/8.0.2/userguide/command_line_interface.html#sec:command_line_warnings

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1704

2023-03-24 Thread Apache Jenkins Server 
See

[jira] [Resolved] (KAFKA-14436) Initialize KRaft with arbitrary epoch

2023-03-24 Thread Colin McCabe (Jira) 


 [ 
https://issues.apache.org/jira/browse/KAFKA-14436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin McCabe resolved KAFKA-14436.
--
Fix Version/s: 3.4.0
   Resolution: Won't Fix

> Initialize KRaft with arbitrary epoch
> -
>
> Key: KAFKA-14436
> URL: https://issues.apache.org/jira/browse/KAFKA-14436
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: David Arthur
>Assignee: Alyssa Huang
>Priority: Major
> Fix For: 3.4.0
>
>
> For the ZK migration, we need to be able to initialize Raft with an 
> arbitrarily high epoch (within the size limit). This is because during the 
> migration, we want to write the Raft epoch as the controller epoch in ZK. We 
> require that epochs in /controller_epoch are monotonic in order for brokers 
> to behave normally. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[GitHub] [kafka-site] anlance closed pull request #493: Fixed typos in introduction documentation

2023-03-24 Thread via GitHub 


anlance closed pull request #493: Fixed typos in introduction documentation
URL: https://github.com/apache/kafka-site/pull/493


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Created] (KAFKA-14846) Fix overly large record batches in ZkMigrationClient

2023-03-24 Thread Colin McCabe (Jira) 
Colin McCabe created KAFKA-14846:


 Summary: Fix overly large record batches in ZkMigrationClient
 Key: KAFKA-14846
 URL: https://issues.apache.org/jira/browse/KAFKA-14846
 Project: Kafka
  Issue Type: Sub-task
Affects Versions: 3.4.0
Reporter: Colin McCabe


ZkMigrationClient should not create overly large record batches



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-14493) Zk to KRaft migration state machine in KRaft controller

2023-03-24 Thread Colin McCabe (Jira) 


 [ 
https://issues.apache.org/jira/browse/KAFKA-14493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin McCabe resolved KAFKA-14493.
--
Fix Version/s: 3.4.0
   Resolution: Fixed

> Zk to KRaft migration state machine in KRaft controller
> ---
>
> Key: KAFKA-14493
> URL: https://issues.apache.org/jira/browse/KAFKA-14493
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Akhilesh Chaganti
>Assignee: Akhilesh Chaganti
>Priority: Major
> Fix For: 3.4.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-14458) RPC Handler to ZkBrokers from KRaft Controller

2023-03-24 Thread Colin McCabe (Jira) 


 [ 
https://issues.apache.org/jira/browse/KAFKA-14458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin McCabe resolved KAFKA-14458.
--
Fix Version/s: 3.4.0
   Resolution: Fixed

> RPC Handler to ZkBrokers from KRaft Controller
> --
>
> Key: KAFKA-14458
> URL: https://issues.apache.org/jira/browse/KAFKA-14458
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Akhilesh Chaganti
>Assignee: Akhilesh Chaganti
>Priority: Major
> Fix For: 3.4.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-14446) API forwarding support in ZkBrokers

2023-03-24 Thread Colin McCabe (Jira) 


 [ 
https://issues.apache.org/jira/browse/KAFKA-14446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin McCabe resolved KAFKA-14446.
--
Fix Version/s: 3.4.0
   Resolution: Fixed

> API forwarding support in ZkBrokers
> ---
>
> Key: KAFKA-14446
> URL: https://issues.apache.org/jira/browse/KAFKA-14446
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Akhilesh Chaganti
>Assignee: Akhilesh Chaganti
>Priority: Major
> Fix For: 3.4.0
>
>
> To support migration, zkBrokers should be able to forward API requests to the 
> Controller, whether it is zkController or kraftController. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-14447) Controlled shutdown for ZK brokers during migration

2023-03-24 Thread Colin McCabe (Jira) 


 [ 
https://issues.apache.org/jira/browse/KAFKA-14447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin McCabe resolved KAFKA-14447.
--
Fix Version/s: 3.4.0
   (was: 3.4.1)
   Resolution: Fixed

> Controlled shutdown for ZK brokers during migration
> ---
>
> Key: KAFKA-14447
> URL: https://issues.apache.org/jira/browse/KAFKA-14447
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: David Arthur
>Assignee: Luke Chen
>Priority: Major
> Fix For: 3.4.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14845) Broker ZNode creation can fail due to a session ID unknown to the broker

2023-03-24 Thread Alexandre Dupriez (Jira) 
Alexandre Dupriez created KAFKA-14845:
-

 Summary: Broker ZNode creation can fail due to a session ID 
unknown to the broker
 Key: KAFKA-14845
 URL: https://issues.apache.org/jira/browse/KAFKA-14845
 Project: Kafka
  Issue Type: Bug
Reporter: Alexandre Dupriez
 Attachments: broker-registration.drawio

Our production environment faced a use case where registration of a broker 
failed due to the presence of a "conflicting" broker znode in Zookeeper. This 
case is not without familiarity to that fixed by KAFKA-6584 and induced by the 
Zookeeper bug (or feature) tracked in ZOOKEEPER-2985 opened as of today.

A network partition disturbed communication channels between the Kafka and 
Zookeeper clusters for about 20% of the brokers in the cluster. One of this 
broker was not able to re-register with Zookeeper and was excluded from the 
cluster until it was restarted. Broker logs show the failed registration due to 
a "conflicting" znode write which in this case is not covered by KAFKA-6584. 
The broker did not restart and was not unhealthy. In the following logs, the 
broker IP is 1.2.3.4.

The sequence of logs on the broker is as follows.

First, a connection is established with the Zookeeper node 3.

 
{code:java}
[2023-03-05 16:01:55,342] INFO Socket connection established, initiating 
session, client: /1.2.3.4:40200, server: zk.3/5.6.7.8:2182 
(org.apache.zookeeper.ClientCnxn)
[2023-03-05 16:01:55,342] INFO channel is connected: [id: 0x2b45ae40, 
L:/1.1.3.4:40200 - R:zk.3/5.6.7.8:2182] 
(org.apache.zookeeper.ClientCnxnSocketNetty){code}
 

An existing Zookeeper session was expired, and upon reconnection, the Zookeeper 
state change handler was invoked. The creation of the ephemeral znode 
/brokers/ids/18 started on the controller thread.

 
{code:java}
[2023-03-05 16:01:55,345] INFO Creating /brokers/ids/18 (is it secure? false) 
(kafka.zk.KafkaZkClient){code}
 

The client "session" timed out after 6 seconds. Note the session is 0x0 and the 
absence of "{_}Session establishment complete{_}" log: the broker appears to 
have never received or processed the response from the Zookeeper node.

 
{code:java}
[2023-03-05 16:02:01,343] INFO Client session timed out, have not heard from 
server in 6000ms for sessionid 0x0, closing socket connection and attempting 
reconnect (org.apache.zookeeper.ClientCnxn)
[2023-03-05 16:02:01,343] INFO channel is disconnected: [id: 0x2b45ae40, 
L:/1.2.3.4:40200 ! R:zk.3/5.6.7.8:2182] 
(org.apache.zookeeper.ClientCnxnSocketNetty){code}
 

Pending requests were aborted with a {{CONNECTIONLOSS}} error and the client 
started waiting on a new connection notification.

 
{code:java}
[2023-03-05 16:02:01,343] INFO [ZooKeeperClient Kafka server] Waiting until 
connected. (kafka.zookeeper.ZooKeeperClient){code}
 

A new connection was created with the Zookeeper node 1. Note that a valid (new) 
session ({{{}0x1006c6e0b830001{}}}) was reported by Kafka this time.

 
{code:java}
[2023-03-05 16:02:02,037] INFO Socket connection established, initiating 
session, client: /1.2.3.4:58080, server: zk.1/9.10.11.12:2182 
(org.apache.zookeeper.ClientCnxn)
[2023-03-05 16:02:02,037] INFO channel is connected: [id: 0x68fba106, 
L:/1.2.3.4:58080 - R:zk.1/9.10.11.12:2182] 
(org.apache.zookeeper.ClientCnxnSocketNetty)
[2023-03-05 16:02:03,054] INFO Session establishment complete on server 
zk.1/9.10.11.12:2182, sessionid = 0x1006c6e0b830001, negotiated timeout = 18000 
(org.apache.zookeeper.ClientCnxn){code}
 

The Kafka ZK client is notified of the connection.

 
{code:java}
[2023-03-05 16:02:03,054] INFO [ZooKeeperClient Kafka server] Connected. 
(kafka.zookeeper.ZooKeeperClient){code}
 

The broker sends the request to create the znode {{/brokers/ids/18}} which 
already exists. The error path implemented for KAFKA-6584 is then followed. 
However, in this case, the session owning the ephemeral node 
{{0x30043230ac1}} ({{{}216172783240153793{}}}) is different from the last 
active Zookeeper session which the broker has recorded. And it is also 
different from the current session {{0x1006c6e0b830001}} 
({{{}72176813933264897{}}}), hence the recreation of the broker znode is not 
attempted.

 
{code:java}
[2023-03-05 16:02:04,466] ERROR Error while creating ephemeral at 
/brokers/ids/18, node already exists and owner '216172783240153793' does not 
match current session '72176813933264897' 
(kafka.zk.KafkaZkClient$CheckedEphemeral)
org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = 
NodeExists
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:126)
        at 
kafka.zk.KafkaZkClient$CheckedEphemeral.getAfterNodeExists(KafkaZkClient.scala:1821)
        at 
kafka.zk.KafkaZkClient$CheckedEphemeral.create(KafkaZkClient.scala:1759)
        at 
kafka.zk.KafkaZkClient.checkedEphemeralCreate(KafkaZkClient.scala:1726)
        at

[jira] [Resolved] (KAFKA-14751) Official website CONTACT page IRC channel link change

2023-03-24 Thread Mickael Maison (Jira) 


 [ 
https://issues.apache.org/jira/browse/KAFKA-14751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mickael Maison resolved KAFKA-14751.

Resolution: Fixed

> Official website CONTACT page IRC channel link change 
> --
>
> Key: KAFKA-14751
> URL: https://issues.apache.org/jira/browse/KAFKA-14751
> Project: Kafka
>  Issue Type: Improvement
>  Components: documentation
>Reporter: Koma Zhang
>Assignee: Koma Zhang
>Priority: Minor
>  Labels: documentation, newbie
>
> "chat.freenode.net" this link should be change to "webchat.freenode.net"



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[GitHub] [kafka-site] mimaison merged pull request #494: KAFKA-14751: Official website CONTACT page IRC channel link change

2023-03-24 Thread via GitHub 


mimaison merged PR #494:
URL: https://github.com/apache/kafka-site/pull/494


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] mimaison commented on pull request #493: Fixed typos in introduction documentation

2023-03-24 Thread via GitHub 


mimaison commented on PR #493:
URL: https://github.com/apache/kafka-site/pull/493#issuecomment-1482918407

   I believe we really mean `active/active` here. This means but clusters are 
actively used, compared to active/passive where only 1 of the clusters is used.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [VOTE] KIP-906 Tools migration guidelines

2023-03-24 Thread John Roesler 
Thanks Federico,

I'm +1 (binding)

-John

On Fri, Mar 24, 2023, at 01:11, Federico Valeri wrote:
> Bumping this thread for more votes.
>
> Thanks.
>
> On Wed, Mar 15, 2023, 9:57 AM Alexandre Dupriez 
> wrote:
>
>> Hi, Frederico,
>>
>> Thanks for the KIP.
>>
>> Non-binding +1.
>>
>> Thanks,
>> Alexandre
>>
>> Le mer. 15 mars 2023 à 08:28, Luke Chen  a écrit :
>> >
>> > +1 from me.
>> >
>> > Luke
>> >
>> > On Wed, Mar 15, 2023 at 4:11 PM Federico Valeri 
>> > wrote:
>> >
>> > > Hi everyone,
>> > >
>> > > I'd like to start the vote on KIP-906 Tools migration guidelines.
>> > >
>> > >
>> > >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-906%3A+Tools+migration+guidelines
>> > >
>> > > Discussion thread:
>> > > https://lists.apache.org/thread/o2ytmjj2tyc2xcy6xh5tco31yyjzvz8p
>> > >
>> > > Thanks
>> > > Fede
>> > >
>>

Re: [DISCUSS] KIP-910: Update Source offsets for Source Connectors without producing records

2023-03-24 Thread John Roesler 
Thanks for the KIP, Sagar!

At first glance, this seems like a very useful feature.

A common pain point in Streams is when upstream producers don't send regular 
updates and stream time cannot advance. This causes stream-time-driven 
operations to appear to hang, like time windows not closing, suppressions not 
firing, etc.

>From your KIP, I have a good idea of how the feature would be integrated into 
>connect, and it sounds good to me. I don't quite see how downstream clients, 
>such as a downstream Streams or Flink application, or users of the Consumer 
>would make use of this feature. Could you add some examples of that nature?

Thank you,
-John

On Fri, Mar 24, 2023, at 05:23, Sagar wrote:
> Hi All,
>
> Bumping the thread again.
>
> Sagar.
>
>
> On Fri, Mar 10, 2023 at 4:42 PM Sagar  wrote:
>
>> Hi All,
>>
>> Bumping this discussion thread again.
>>
>> Thanks!
>> Sagar.
>>
>> On Thu, Mar 2, 2023 at 3:44 PM Sagar  wrote:
>>
>>> Hi All,
>>>
>>> I wanted to create a discussion thread for KIP-910:
>>>
>>>
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-910%3A+Update+Source+offsets+for+Source+Connectors+without+producing+records
>>>
>>> Thanks!
>>> Sagar.
>>>
>>

Re: [DISCUSS] KIP-910: Update Source offsets for Source Connectors without producing records

2023-03-24 Thread Sagar 
Hi All,

Bumping the thread again.

Sagar.


On Fri, Mar 10, 2023 at 4:42 PM Sagar  wrote:

> Hi All,
>
> Bumping this discussion thread again.
>
> Thanks!
> Sagar.
>
> On Thu, Mar 2, 2023 at 3:44 PM Sagar  wrote:
>
>> Hi All,
>>
>> I wanted to create a discussion thread for KIP-910:
>>
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-910%3A+Update+Source+offsets+for+Source+Connectors+without+producing+records
>>
>> Thanks!
>> Sagar.
>>
>

Re: [DISCUSS] KIP-908: Add description field to connector configuration

2023-03-24 Thread Mickael Maison 
Thanks Chris for the discussion.

Since nobody else expressed interest in this KIP, I'm going to discard it.

On Wed, Mar 8, 2023 at 8:41 PM Chris Egerton  wrote:
>
> Hi Mickael,
>
> I do sympathize with the desire for a "quick fix". I think your point about
> these being problematic to support sums up my hesitation here pretty well,
> both with respect to the potential footgun of unintended rebalances (should
> users try to do more with this field than we expect), and with making other
> similar improvements (i.e., expanded metadata support in Kafka Connect)
> more difficult to design correctly for us and to use effectively for users
> (especially if we were to deprecate/remove the description field at a later
> date).
>
> It's also worth noting that users can already add a "description" field to
> the JSON config for any connector they create; the benefit provided by this
> KIP is that they're automatically prompted to do so if they're using a
> UI/CLI/etc. that builds on top of the various /connector-plugins endpoints
> to find the set of configuration properties that a user can/should populate
> before creating a connector.
>
> I'd welcome thoughts from you and others on whether the tradeoffs here are
> worth it; right now I'm still not sure about this one.
>
> Cheers,
>
> Chris
>
> On Tue, Mar 7, 2023 at 6:42 AM Mickael Maison 
> wrote:
>
> > H Chris,
> >
> > Thanks for taking a look.
> >
> > 1. Yes updating the description can potentially trigger a rebalance. I
> > don't expect users to frequently update the description so I thought
> > this was acceptable. I've added a note to the KIP to mention it.
> >
> > 2. The tags model you described could be interesting but it looks
> > pretty involved with multiple new endpoints and brand new concepts.
> >
> > With this KIP, I really took the most basic approach and proposed the
> > simplest set of changes that could get in the next release and, I
> > think, immediately bring benefits. However sometimes this is not the
> > best approach as a "quick fix" could end up problematic to support in
> > the future. The drawbacks (new connector config field + causing a
> > rebalance on update) look relatively benign to me so I thought this
> > could be an acceptable proposal.
> >
> > Thanks,
> > Mickael
> >
> >
> >
> > On Thu, Feb 23, 2023 at 2:45 PM Chris Egerton 
> > wrote:
> > >
> > > Actually, I misspoke--a rebalance isn't triggered when an existing
> > > connector's config is updated. Assuming the set of workers remains
> > stable,
> > > a rebalance is only necessary when a new connector is created, an
> > existing
> > > one is deleted, or a new set of task configs is generated.
> > >
> > > This weakens the point about unnecessary rebalances when a connector's
> > > description is updated, but doesn't entirely address it. Spurious
> > > rebalances may still be triggered if a new set of task configs is
> > > generated, which for reasons outlined above, is fairly likely.
> > >
> > > On Thu, Feb 23, 2023 at 7:41 AM Chris Egerton  wrote:
> > >
> > > > Hi Mickael,
> > > >
> > > > Thanks for the KIP!
> > > >
> > > > While it's tempting to add this field to the out-of-the-box connector
> > > > config def, I'm a little hesitant, for two reasons.
> > > >
> > > > 1. Adding this directly to the connector config could have unintended
> > > > consequences on the actual data processing by the connector. Any time a
> > > > connector's config is modified, the Connector object running for it is
> > > > restarted with that new config. In most cases this is a trivial
> > operation
> > > > since we have incremental rebalancing enabled by default, the
> > connector can
> > > > (and probably should!) generate task configs that are functionally
> > > > identical to the ones it last generated, and most (though not all)
> > > > Connector classes are fairly lightweight and leave the real work to
> > their
> > > > Task class. However, due to KAFKA-9228 [1], it's not just common
> > practice
> > > > for connectors to perform transparent passthrough of most of their
> > configs
> > > > when generating task configs, it's actually necessary to work around a
> > bug
> > > > in the runtime. As a result, tweaking the description of a connector
> > would
> > > > be fairly likely to result in a full restart of all of its tasks, in
> > > > addition to triggering two rebalances (which may not be so lightweight
> > if
> > > > users are still running with eager rebalancing... which, sadly, I've
> > heard
> > > > is still happening today).
> > > >
> > > > 2. The motivation section mentions some information that might go into
> > the
> > > > description field, such as the team that owns the connector and
> > emergency
> > > > contact info. It seems like this info might benefit from a little more
> > > > structure if we're trying to design for programmatic access by GUIs and
> > > > CLIs (which I'm assuming is the case, since I can't imagine a human
> > being
> > > > getting much use out of the raw

[jira] [Created] (KAFKA-14844) Kafka Connect's OffsetBackingStore interface should handle (de)serialization and connector namespacing

2023-03-24 Thread Yash Mayya (Jira) 
Yash Mayya created KAFKA-14844:
--

 Summary: Kafka Connect's OffsetBackingStore interface should 
handle (de)serialization and connector namespacing
 Key: KAFKA-14844
 URL: https://issues.apache.org/jira/browse/KAFKA-14844
 Project: Kafka
  Issue Type: Task
  Components: KafkaConnect
Reporter: Yash Mayya
Assignee: Yash Mayya


Relevant discussion here - 
[https://github.com/apache/kafka/pull/13434/files#r114972]

 

TLDR - we should move serialization / deserialization and key construction 
(connector namespacing) for source connector offsets from the 
OffsetStorageWriter / OffsetStorageReader interfaces into the 
OffsetBackingStore interface. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Need Help on Deploying Apache Kafka to Azure Kubernetes

2023-03-24 Thread Prabhu A 
Hi Kafka Team,

I was trying to deploy apache kafka to azure kubernetes. Could anyone
please send us the Step by Step procedure to deploy Apache Kafka to azure
kubernetes and steps needed to test the kafka after the kubernetes
deployment.

Hoping for the reply!!



*Regards,*
*Prabhu A*

[jira] [Created] (KAFKA-14843) Connector plugins config endpoint does not include Common configs

2023-03-24 Thread Jorge Esteban Quilcate Otoya (Jira) 
Jorge Esteban Quilcate Otoya created KAFKA-14843:


 Summary: Connector plugins config endpoint does not include Common 
configs
 Key: KAFKA-14843
 URL: https://issues.apache.org/jira/browse/KAFKA-14843
 Project: Kafka
  Issue Type: Bug
  Components: KafkaConnect
Affects Versions: 3.2.0
Reporter: Jorge Esteban Quilcate Otoya


Connector plugins GET config endpoint introduced in 
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+APIs+to+list+all+connector+plugins+and+retrieve+their+configuration+definitions|https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+APIs+to+list+all+connector+plugins+and+retrieve+their+configuration+definitions,]
 allows to get plugin configuration from the rest endpoint.

This configuration only includes the plugin configuration, but not the base 
configuration of the Sink/Source Connector.

For instance, when validating the configuration of a plugin, _all_ configs are 
returned:

```

curl -s 
$CONNECT_URL/connector-plugins/io.aiven.kafka.connect.http.HttpSinkConnector/config
 | jq -r '.[].name' | sort -u | wc -l     
21

curl -s 
$CONNECT_URL/connector-plugins/io.aiven.kafka.connect.http.HttpSinkConnector/config/validate
 -XPUT -H 'Content-type: application/json' --data "\{\"connector.class\": 
\"io.aiven.kafka.connect.http.HttpSinkConnector\", \"topics\": 
\"example-topic-name\"}" | jq -r '.configs[].definition.name' | sort -u | wc -l
39

```

and the missing configs are all from base config:

```

diff validate.txt config.txt                                                    
                                                
6,14d5
< config.action.reload
< connector.class
< errors.deadletterqueue.context.headers.enable
< errors.deadletterqueue.topic.name
< errors.deadletterqueue.topic.replication.factor
< errors.log.enable
< errors.log.include.messages
< errors.retry.delay.max.ms
< errors.retry.timeout
16d6
< header.converter
24d13
< key.converter
26d14
< name
33d20
< predicates
35,39d21
< tasks.max
< topics
< topics.regex
< transforms
< value.converter

```

Would be great to get the base configs from the same endpoint as well, so we 
could rely on it instead of using the validate endpoint to get all configs.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-906 Tools migration guidelines

2023-03-24 Thread Federico Valeri 
Bumping this thread for more votes.

Thanks.

On Wed, Mar 15, 2023, 9:57 AM Alexandre Dupriez 
wrote:

> Hi, Frederico,
>
> Thanks for the KIP.
>
> Non-binding +1.
>
> Thanks,
> Alexandre
>
> Le mer. 15 mars 2023 à 08:28, Luke Chen  a écrit :
> >
> > +1 from me.
> >
> > Luke
> >
> > On Wed, Mar 15, 2023 at 4:11 PM Federico Valeri 
> > wrote:
> >
> > > Hi everyone,
> > >
> > > I'd like to start the vote on KIP-906 Tools migration guidelines.
> > >
> > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-906%3A+Tools+migration+guidelines
> > >
> > > Discussion thread:
> > > https://lists.apache.org/thread/o2ytmjj2tyc2xcy6xh5tco31yyjzvz8p
> > >
> > > Thanks
> > > Fede
> > >
>