[GitHub] [kafka] yashmayya commented on pull request #13530: KAFKA-14858: Handle exceptions thrown from Connector::taskConfigs in Connect's standalone mode
yashmayya commented on PR #13530: URL: https://github.com/apache/kafka/pull/13530#issuecomment-1586626234 One thing to note is that all existing connector failure state transitions occur through `WorkerConnector::onFailure` which updates an internal state to `FAILED` as well as the external status via `ConnectorStatus::onFailure`. This allows it to block other state transitions (other than to the stopped state) when its state is failed. We aren't doing that here (updating the `WorkerConnector`'s internal state) but I think it seems reasonable to allow pauses and resumes on a connector which has failed to generate task configs since the task configs generation will be re-attempted on the resume anyway. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] dengziming commented on a diff in pull request #13826: KAFKA-15060: fix the ApiVersionManager interface
dengziming commented on code in PR #13826:
URL: https://github.com/apache/kafka/pull/13826#discussion_r1226040921
##
core/src/main/scala/kafka/server/ApiVersionManager.scala:
##
@@ -141,24 +136,22 @@ class DefaultApiVersionManager(
val enabledApis = ApiKeys.apisForListener(listenerType).asScala
override def apiVersionResponse(throttleTimeMs: Int): ApiVersionsResponse = {
-val supportedFeatures = features.supportedFeatures
+val supportedFeatures = brokerFeatures.supportedFeatures
val finalizedFeatures = metadataCache.features()
val controllerApiVersions =
forwardingManager.flatMap(_.controllerApiVersions)
ApiVersionsResponse.createApiVersionsResponse(
throttleTimeMs,
- metadataCache.metadataVersion().highestSupportedRecordVersion,
+ finalizedFeatures.metadataVersion().highestSupportedRecordVersion,
supportedFeatures,
- finalizedFeatures.features.map(kv => (kv._1,
kv._2.asInstanceOf[java.lang.Short])).asJava,
- finalizedFeatures.epoch,
+ finalizedFeatures.finalizedFeatures(),
+ finalizedFeatures.finalizedFeaturesEpoch(),
controllerApiVersions.orNull,
listenerType,
enableUnstableLastVersion,
zkMigrationEnabled
)
}
- override def apiVersionResponse(throttleTimeMs: Int, finalizedFeatures:
Map[String, java.lang.Short], finalizedFeatureEpoch: Long): ApiVersionsResponse
= {
-throw new UnsupportedOperationException("This method is not supported in
DefaultApiVersionManager, use apiVersionResponse(throttleTimeMs) instead")
- }
+ override def features: Features = metadataCache.features()
Review Comment:
It seems this method is not used.
##
core/src/main/scala/kafka/server/ControllerServer.scala:
##
@@ -326,6 +329,9 @@ class ControllerServer(
// register this instance for dynamic config changes to the KafkaConfig
config.dynamicConfig.addReconfigurables(this)
+ // Set up the metadata version publisher.
Review Comment:
nit: metadata version publisher -> feature publisher
##
core/src/main/scala/kafka/server/ApiVersionManager.scala:
##
@@ -132,7 +127,7 @@ class SimpleApiVersionManager(
class DefaultApiVersionManager(
val listenerType: ListenerType,
forwardingManager: Option[ForwardingManager],
- features: BrokerFeatures,
+ brokerFeatures: BrokerFeatures,
Review Comment:
We should update the class docs to make it consistent with the field name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Resolved] (KAFKA-15036) Kraft leader change fails when invoking getFinalizedFeatures
[
https://issues.apache.org/jira/browse/KAFKA-15036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Deng Ziming resolved KAFKA-15036.
-
Resolution: Fixed
> Kraft leader change fails when invoking getFinalizedFeatures
>
>
> Key: KAFKA-15036
> URL: https://issues.apache.org/jira/browse/KAFKA-15036
> Project: Kafka
> Issue Type: Improvement
>Reporter: Deng Ziming
>Assignee: Deng Ziming
>Priority: Major
>
> When kraft leader changes, we can receiving a error as follows:
>
> {{[2023-05-24 18:00:02,898] WARN [QuorumController id=3002]
> getFinalizedFeatures: failed with unknown server exception RuntimeException
> in 271 us. The controller is already in standby mode.
> (org.apache.kafka.controller.QuorumController)
> java.lang.RuntimeException: No in-memory snapshot for epoch 9. Snapshot
> epochs are:
> at
> org.apache.kafka.timeline.SnapshotRegistry.getSnapshot(SnapshotRegistry.java:173)
> at
> org.apache.kafka.timeline.SnapshotRegistry.iterator(SnapshotRegistry.java:131)
> at org.apache.kafka.timeline.TimelineObject.get(TimelineObject.java:69)
> at
> org.apache.kafka.controller.FeatureControlManager.finalizedFeatures(FeatureControlManager.java:303)
> at
> org.apache.kafka.controller.QuorumController.lambda$finalizedFeatures$16(QuorumController.java:2016)
> at
> org.apache.kafka.controller.QuorumController$ControllerReadEvent.run(QuorumController.java:546)
> at
> org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127)
> at
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
> at
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
> at java.base/java.lang.Thread.run(Thread.java:829)}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [kafka] dengziming merged pull request #13832: KAFKA-15036: Add a test case for controller failover
dengziming merged PR #13832: URL: https://github.com/apache/kafka/pull/13832 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] stevenbooke opened a new pull request, #13842: KAFKA-14995: Automate asf.yaml collaborators refresh
stevenbooke opened a new pull request, #13842: URL: https://github.com/apache/kafka/pull/13842 Create Github Action workflow to run Python script which will automate the asf.yaml collaborators refresh. Tested the workflow locally using https://github.com/nektos/act. ### Committer Checklist (excluded from commit message) - [NA] Verify design and implementation - [X] Verify test coverage and CI build status - [X] Verify documentation (including upgrade notes) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (KAFKA-14995) Automate asf.yaml collaborators refresh
[
https://issues.apache.org/jira/browse/KAFKA-14995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17731410#comment-17731410
]
ASF GitHub Bot commented on KAFKA-14995:
stevenbooke opened a new pull request, #521:
URL: https://github.com/apache/kafka-site/pull/521
Add Kafka Committers github login to be used for automating the refresh of
Kafka collaborators.
> Automate asf.yaml collaborators refresh
> ---
>
> Key: KAFKA-14995
> URL: https://issues.apache.org/jira/browse/KAFKA-14995
> Project: Kafka
> Issue Type: Improvement
>Reporter: John Roesler
>Assignee: Steven Booke
>Priority: Minor
> Labels: newbie
>
> We have added a policy to use the asf.yaml Github Collaborators:
> [https://github.com/apache/kafka-site/pull/510]
> The policy states that we set this list to be the top 20 commit authors who
> are not Kafka committers. Unfortunately, it's not trivial to compute this
> list.
> Here is the process I followed to generate the list the first time (note that
> I generated this list on 2023-04-28, so the lookback is one year:
> 1. List authors by commit volume in the last year:
> {code:java}
> $ git shortlog --email --numbered --summary --since=2022-04-28 | vim {code}
> 2. manually filter out the authors who are committers, based on
> [https://kafka.apache.org/committers]
> 3. truncate the list to 20 authors
> 4. for each author
> 4a. Find a commit in the `git log` that they were the author on:
> {code:java}
> commit 440bed2391338dc10fe4d36ab17dc104b61b85e8
> Author: hudeqi <1217150...@qq.com>
> Date: Fri May 12 14:03:17 2023 +0800
> ...{code}
> 4b. Look up that commit in Github:
> [https://github.com/apache/kafka/commit/440bed2391338dc10fe4d36ab17dc104b61b85e8]
> 4c. Copy their Github username into .asf.yaml under both the PR whitelist and
> the Collaborators lists.
> 5. Send a PR to update .asf.yaml: [https://github.com/apache/kafka/pull/13713]
>
> This is pretty time consuming and is very scriptable. Two complications:
> * To do the filtering, we need to map from Git log "Author" to documented
> Kafka "Committer" that we can use to perform the filter. Suggestion: just
> update the structure of the "Committers" page to include their Git "Author"
> name and email
> ([https://github.com/apache/kafka-site/blob/asf-site/committers.html)]
> * To generate the YAML lists, we need to map from Git log "Author" to Github
> username. There's presumably some way to do this in the Github REST API (the
> mapping is based on the email, IIUC), or we could also just update the
> Committers page to also document each committer's Github username.
>
> Ideally, we would write this script (to be stored in the Apache Kafka repo)
> and create a Github Action to run it every three months.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-15046) Produce performance issue under high disk load
[
https://issues.apache.org/jira/browse/KAFKA-15046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17731375#comment-17731375
]
Jun Rao commented on KAFKA-15046:
-
The potential issue with moving the LeaderEpochFile flushing to an arbitrary
background thread is that it may not be synchronized with the log flushing and
could lead to a situation where the flushed LeaderEpochFile doesn't reflect all
the log data before the recovery point.
> Produce performance issue under high disk load
> --
>
> Key: KAFKA-15046
> URL: https://issues.apache.org/jira/browse/KAFKA-15046
> Project: Kafka
> Issue Type: Improvement
> Components: core
>Affects Versions: 3.3.2
>Reporter: Haruki Okada
>Assignee: Haruki Okada
>Priority: Major
> Labels: performance
> Attachments: image-2023-06-01-12-46-30-058.png,
> image-2023-06-01-12-52-40-959.png, image-2023-06-01-12-54-04-211.png,
> image-2023-06-01-12-56-19-108.png
>
>
> * Phenomenon:
> ** !image-2023-06-01-12-46-30-058.png|width=259,height=236!
> ** Producer response time 99%ile got quite bad when we performed replica
> reassignment on the cluster
> *** RequestQueue scope was significant
> ** Also request-time throttling happened at the incidental time. This caused
> producers to delay sending messages in the mean time.
> ** The disk I/O latency was higher than usual due to the high load for
> replica reassignment.
> *** !image-2023-06-01-12-56-19-108.png|width=255,height=128!
> * Analysis:
> ** The request-handler utilization was much higher than usual.
> *** !image-2023-06-01-12-52-40-959.png|width=278,height=113!
> ** Also, thread time utilization was much higher than usual on almost all
> users
> *** !image-2023-06-01-12-54-04-211.png|width=276,height=110!
> ** From taking jstack several times, for most of them, we found that a
> request-handler was doing fsync for flusing ProducerState and meanwhile other
> request-handlers were waiting Log#lock for appending messages.
> *
> **
> ***
> {code:java}
> "data-plane-kafka-request-handler-14" #166 daemon prio=5 os_prio=0
> cpu=51264789.27ms elapsed=599242.76s tid=0x7efdaeba7770 nid=0x1e704
> runnable [0x7ef9a12e2000]
>java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.FileDispatcherImpl.force0(java.base@11.0.17/Native
> Method)
> at
> sun.nio.ch.FileDispatcherImpl.force(java.base@11.0.17/FileDispatcherImpl.java:82)
> at
> sun.nio.ch.FileChannelImpl.force(java.base@11.0.17/FileChannelImpl.java:461)
> at
> kafka.log.ProducerStateManager$.kafka$log$ProducerStateManager$$writeSnapshot(ProducerStateManager.scala:451)
> at
> kafka.log.ProducerStateManager.takeSnapshot(ProducerStateManager.scala:754)
> at kafka.log.UnifiedLog.roll(UnifiedLog.scala:1544)
> - locked <0x00060d75d820> (a java.lang.Object)
> at kafka.log.UnifiedLog.maybeRoll(UnifiedLog.scala:1523)
> - locked <0x00060d75d820> (a java.lang.Object)
> at kafka.log.UnifiedLog.append(UnifiedLog.scala:919)
> - locked <0x00060d75d820> (a java.lang.Object)
> at kafka.log.UnifiedLog.appendAsLeader(UnifiedLog.scala:760)
> at
> kafka.cluster.Partition.$anonfun$appendRecordsToLeader$1(Partition.scala:1170)
> at kafka.cluster.Partition.appendRecordsToLeader(Partition.scala:1158)
> at
> kafka.server.ReplicaManager.$anonfun$appendToLocalLog$6(ReplicaManager.scala:956)
> at
> kafka.server.ReplicaManager$$Lambda$2379/0x000800b7c040.apply(Unknown
> Source)
> at
> scala.collection.StrictOptimizedMapOps.map(StrictOptimizedMapOps.scala:28)
> at
> scala.collection.StrictOptimizedMapOps.map$(StrictOptimizedMapOps.scala:27)
> at scala.collection.mutable.HashMap.map(HashMap.scala:35)
> at
> kafka.server.ReplicaManager.appendToLocalLog(ReplicaManager.scala:944)
> at kafka.server.ReplicaManager.appendRecords(ReplicaManager.scala:602)
> at kafka.server.KafkaApis.handleProduceRequest(KafkaApis.scala:666)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:175)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:75)
> at java.lang.Thread.run(java.base@11.0.17/Thread.java:829) {code}
> *
> ** Also there were bunch of logs that writing producer snapshots took
> hundreds of milliseconds.
> ***
> {code:java}
> ...
> [2023-05-01 11:08:36,689] INFO [ProducerStateManager partition=xxx-4] Wrote
> producer snapshot at offset 1748817854 with 8 producer ids in 809 ms.
> (kafka.log.ProducerStateManager)
> [2023-05-01 11:08:37,319] INFO [ProducerStateManager partition=yyy-34] Wrote
> producer snapshot at offset 247996937813 with 0 producer ids in 547 ms.
> (kafka.log.ProducerStateManager)
>
[GitHub] [kafka] hgeraldino commented on a diff in pull request #13383: KAFKA-14059 Replace PowerMock with Mockito in WorkerSourceTaskTest
hgeraldino commented on code in PR #13383:
URL: https://github.com/apache/kafka/pull/13383#discussion_r1220625126
##
connect/runtime/src/test/java/org/apache/kafka/connect/runtime/WorkerSourceTaskTest.java:
##
@@ -706,95 +662,72 @@ public void testSourceTaskIgnoresProducerException()
throws Exception {
// and no ConnectException will be thrown
SourceRecord record1 = new SourceRecord(PARTITION, OFFSET, TOPIC, 1,
KEY_SCHEMA, KEY, RECORD_SCHEMA, RECORD);
SourceRecord record2 = new SourceRecord(PARTITION, offset2, TOPIC, 2,
KEY_SCHEMA, KEY, RECORD_SCHEMA, RECORD);
-expectOffsetFlush(true);
-expectSendRecordOnce();
-expectSendRecordProducerCallbackFail();
-sourceTask.commitRecord(EasyMock.anyObject(SourceRecord.class),
EasyMock.isNull());
-//As of KAFKA-14079 all offsets should be committed, even for failed
records (if ignored)
-//Only the last offset will be passed to the method as everything up
to that point is committed
-//Before KAFKA-14079 offset 12 would have been passed and not 13 as it
would have been unacked
-offsetWriter.offset(PARTITION, offset2);
-PowerMock.expectLastCall();
+expectOffsetFlush();
+expectPreliminaryCalls();
-PowerMock.replayAll();
+when(producer.send(any(ProducerRecord.class), any(Callback.class)))
+.thenAnswer(producerSendAnswer(true))
+.thenAnswer(producerSendAnswer(false));
//Send records and then commit offsets and verify both were committed
and no exception
-Whitebox.setInternalState(workerTask, "toSend", Arrays.asList(record1,
record2));
-Whitebox.invokeMethod(workerTask, "sendRecords");
-Whitebox.invokeMethod(workerTask, "updateCommittableOffsets");
+workerTask.toSend = Arrays.asList(record1, record2);
+workerTask.sendRecords();
+workerTask.updateCommittableOffsets();
workerTask.commitOffsets();
-PowerMock.verifyAll();
+//As of KAFKA-14079 all offsets should be committed, even for failed
records (if ignored)
+//Only the last offset will be passed to the method as everything up
to that point is committed
+//Before KAFKA-14079 offset 12 would have been passed and not 13 as it
would have been unacked
+verify(offsetWriter).offset(PARTITION, offset2);
+verify(sourceTask).commitRecord(any(SourceRecord.class), isNull());
//Double check to make sure all submitted records were cleared
-assertEquals(0, ((SubmittedRecords)
Whitebox.getInternalState(workerTask,
-"submittedRecords")).records.size());
+assertEquals(0, workerTask.submittedRecords.records.size());
}
@Test
public void testSlowTaskStart() throws Exception {
final CountDownLatch startupLatch = new CountDownLatch(1);
final CountDownLatch finishStartupLatch = new CountDownLatch(1);
-
createWorkerTask();
-offsetStore.start();
-EasyMock.expectLastCall();
-sourceTask.initialize(EasyMock.anyObject(SourceTaskContext.class));
-EasyMock.expectLastCall();
-sourceTask.start(TASK_PROPS);
-EasyMock.expectLastCall().andAnswer(() -> {
+doAnswer((Answer) invocation -> {
startupLatch.countDown();
-assertTrue(awaitLatch(finishStartupLatch));
+ConcurrencyUtils.awaitLatch(finishStartupLatch, "Timeout waiting
for task to stop");
Review Comment:
Hmm... not really. I mean, the countDownLatch is updated after calling
`workerTask.stop()` on L710, but calling `stop()` only sets an internal flag on
the `WorkerTask` and doesn't really block, so technically no.
I don't understand the purpose of this synchronization, so decided to remove
`finishStartupLatch` altogether
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] cadonna merged pull request #13829: KAFKA-10199: Re-add revived tasks to the state updater after handling
cadonna merged PR #13829: URL: https://github.com/apache/kafka/pull/13829 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] cadonna commented on pull request #13829: KAFKA-10199: Re-add revived tasks to the state updater after handling
cadonna commented on PR #13829: URL: https://github.com/apache/kafka/pull/13829#issuecomment-1586163556 Build failures are unrelated: ``` Build / JDK 11 and Scala 2.13 / org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationExactlyOnceTest.testReplicateSourceDefault() Build / JDK 11 and Scala 2.13 / org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationExactlyOnceTest.testOffsetSyncsTopicsOnTarget() Build / JDK 11 and Scala 2.13 / org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationExactlyOnceTest.testSyncTopicConfigs() Build / JDK 11 and Scala 2.13 / org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationExactlyOnceTest.testOffsetSyncsTopicsOnTarget() Build / JDK 11 and Scala 2.13 / kafka.controller.ControllerIntegrationTest.testTopicIdPersistsThroughControllerRestart() Build / JDK 17 and Scala 2.13 / org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationExactlyOnceTest.testReplicationWithEmptyPartition() Build / JDK 17 and Scala 2.13 / org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationExactlyOnceTest.testOffsetSyncsTopicsOnTarget() Build / JDK 17 and Scala 2.13 / org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationExactlyOnceTest.testReplicationWithEmptyPartition() Build / JDK 17 and Scala 2.13 / kafka.api.TransactionsTest.testBumpTransactionalEpoch(String).quorum=zk Build / JDK 8 and Scala 2.12 / org.apache.kafka.clients.consumer.KafkaConsumerTest.testGracefulClose() Build / JDK 8 and Scala 2.12 / org.apache.kafka.connect.mirror.integration.DedicatedMirrorIntegrationTest.testMultiNodeCluster() Build / JDK 8 and Scala 2.12 / org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationBaseTest.testReplicateSourceDefault() Build / JDK 8 and Scala 2.12 / integration.kafka.server.FetchFromFollowerIntegrationTest.testRackAwareRangeAssignor() Build / JDK 8 and Scala 2.12 / kafka.admin.TopicCommandIntegrationTest.testDescribeUnderMinIsrPartitions(String).quorum=zk ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (KAFKA-15072) Flaky test MirrorConnectorsIntegrationExactlyOnceTest.testReplicationWithEmptyPartition
[
https://issues.apache.org/jira/browse/KAFKA-15072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17731344#comment-17731344
]
Bruno Cadonna commented on KAFKA-15072:
---
I also see the following test that fail with the same error:
{code}
Build / JDK 11 and Scala 2.13 /
org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationExactlyOnceTest.testReplicateSourceDefault()
Build / JDK 11 and Scala 2.13 /
org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationExactlyOnceTest.testOffsetSyncsTopicsOnTarget()
Build / JDK 11 and Scala 2.13 /
org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationExactlyOnceTest.testSyncTopicConfigs()
Build / JDK 11 and Scala 2.13 /
org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationExactlyOnceTest.testOffsetSyncsTopicsOnTarget()
Build / JDK 17 and Scala 2.13 /
org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationExactlyOnceTest.testReplicationWithEmptyPartition()
Build / JDK 17 and Scala 2.13 /
org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationExactlyOnceTest.testOffsetSyncsTopicsOnTarget()
Build / JDK 17 and Scala 2.13 /
org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationExactlyOnceTest.testReplicationWithEmptyPartition()
{code}
Should we open separate tickets for them or can we document them under this? I
would prefer the latter to avoid to open separate tickets that can be probably
resolved by the same PR.
> Flaky test
> MirrorConnectorsIntegrationExactlyOnceTest.testReplicationWithEmptyPartition
> ---
>
> Key: KAFKA-15072
> URL: https://issues.apache.org/jira/browse/KAFKA-15072
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
>Affects Versions: 3.5.0
>Reporter: Josep Prat
>Priority: Major
>
> Test
> MirrorConnectorsIntegrationExactlyOnceTest.testReplicationWithEmptyPartition
> became flaky again, but it's a different error this time.
> Occurrence:
> [https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-13824/1/testReport/junit/org.apache.kafka.connect.mirror.integration/MirrorConnectorsIntegrationExactlyOnceTest/Build___JDK_17_and_Scala_2_13___testReplicationWithEmptyPartition__/]
>
> h3. Error Message
> {code:java}
> java.lang.AssertionError: Connector MirrorHeartbeatConnector tasks did not
> start in time on cluster: backup-connect-cluster{code}
> h3. Stacktrace
> {code:java}
> java.lang.AssertionError: Connector MirrorHeartbeatConnector tasks did not
> start in time on cluster: backup-connect-cluster at
> org.apache.kafka.connect.util.clusters.EmbeddedConnectClusterAssertions.assertConnectorAndAtLeastNumTasksAreRunning(EmbeddedConnectClusterAssertions.java:301)
> at
> org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationBaseTest.waitUntilMirrorMakerIsRunning(MirrorConnectorsIntegrationBaseTest.java:912)
> at
> org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationBaseTest.testReplicationWithEmptyPartition(MirrorConnectorsIntegrationBaseTest.java:415)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method) at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:568) at
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)
> at
> org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
> at
> org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
> at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
> at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86)
> at
> org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
> at
> org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
> at
>
[jira] [Comment Edited] (KAFKA-15072) Flaky test MirrorConnectorsIntegrationExactlyOnceTest.testReplicationWithEmptyPartition
[
https://issues.apache.org/jira/browse/KAFKA-15072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17731344#comment-17731344
]
Bruno Cadonna edited comment on KAFKA-15072 at 6/11/23 1:05 PM:
I also see the following tests that fail with the same error:
{code}
Build / JDK 11 and Scala 2.13 /
org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationExactlyOnceTest.testReplicateSourceDefault()
Build / JDK 11 and Scala 2.13 /
org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationExactlyOnceTest.testOffsetSyncsTopicsOnTarget()
Build / JDK 11 and Scala 2.13 /
org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationExactlyOnceTest.testSyncTopicConfigs()
Build / JDK 11 and Scala 2.13 /
org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationExactlyOnceTest.testOffsetSyncsTopicsOnTarget()
Build / JDK 17 and Scala 2.13 /
org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationExactlyOnceTest.testReplicationWithEmptyPartition()
Build / JDK 17 and Scala 2.13 /
org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationExactlyOnceTest.testOffsetSyncsTopicsOnTarget()
Build / JDK 17 and Scala 2.13 /
org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationExactlyOnceTest.testReplicationWithEmptyPartition()
{code}
Should we open separate tickets for them or can we document them under this? I
would prefer the latter to avoid to open separate tickets that can be probably
resolved by the same PR.
was (Author: cadonna):
I also see the following test that fail with the same error:
{code}
Build / JDK 11 and Scala 2.13 /
org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationExactlyOnceTest.testReplicateSourceDefault()
Build / JDK 11 and Scala 2.13 /
org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationExactlyOnceTest.testOffsetSyncsTopicsOnTarget()
Build / JDK 11 and Scala 2.13 /
org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationExactlyOnceTest.testSyncTopicConfigs()
Build / JDK 11 and Scala 2.13 /
org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationExactlyOnceTest.testOffsetSyncsTopicsOnTarget()
Build / JDK 17 and Scala 2.13 /
org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationExactlyOnceTest.testReplicationWithEmptyPartition()
Build / JDK 17 and Scala 2.13 /
org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationExactlyOnceTest.testOffsetSyncsTopicsOnTarget()
Build / JDK 17 and Scala 2.13 /
org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationExactlyOnceTest.testReplicationWithEmptyPartition()
{code}
Should we open separate tickets for them or can we document them under this? I
would prefer the latter to avoid to open separate tickets that can be probably
resolved by the same PR.
> Flaky test
> MirrorConnectorsIntegrationExactlyOnceTest.testReplicationWithEmptyPartition
> ---
>
> Key: KAFKA-15072
> URL: https://issues.apache.org/jira/browse/KAFKA-15072
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
>Affects Versions: 3.5.0
>Reporter: Josep Prat
>Priority: Major
>
> Test
> MirrorConnectorsIntegrationExactlyOnceTest.testReplicationWithEmptyPartition
> became flaky again, but it's a different error this time.
> Occurrence:
> [https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-13824/1/testReport/junit/org.apache.kafka.connect.mirror.integration/MirrorConnectorsIntegrationExactlyOnceTest/Build___JDK_17_and_Scala_2_13___testReplicationWithEmptyPartition__/]
>
> h3. Error Message
> {code:java}
> java.lang.AssertionError: Connector MirrorHeartbeatConnector tasks did not
> start in time on cluster: backup-connect-cluster{code}
> h3. Stacktrace
> {code:java}
> java.lang.AssertionError: Connector MirrorHeartbeatConnector tasks did not
> start in time on cluster: backup-connect-cluster at
> org.apache.kafka.connect.util.clusters.EmbeddedConnectClusterAssertions.assertConnectorAndAtLeastNumTasksAreRunning(EmbeddedConnectClusterAssertions.java:301)
> at
> org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationBaseTest.waitUntilMirrorMakerIsRunning(MirrorConnectorsIntegrationBaseTest.java:912)
> at
> org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationBaseTest.testReplicationWithEmptyPartition(MirrorConnectorsIntegrationBaseTest.java:415)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method) at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at
[jira] [Resolved] (KAFKA-15077) FileTokenRetriever doesn't trim the token before returning it.
[
https://issues.apache.org/jira/browse/KAFKA-15077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manikumar resolved KAFKA-15077.
---
Resolution: Fixed
> FileTokenRetriever doesn't trim the token before returning it.
> --
>
> Key: KAFKA-15077
> URL: https://issues.apache.org/jira/browse/KAFKA-15077
> Project: Kafka
> Issue Type: Bug
> Components: clients
>Reporter: Sushant Mahajan
>Assignee: Sushant Mahajan
>Priority: Minor
> Fix For: 3.6.0
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The {{FileTokenRetriever}} class is used to read the access_token from a file
> on the clients system and then the info is passed along with jaas config to
> the {{{}OAuthBearerSaslServer{}}}.
> The server uses the class {{OAuthBearerClientInitialResponse}} to validate
> the token format.
> In case the token was sent using {{FileTokenRetriever}} on the client side,
> some EOL character is getting appended to the token, causing authentication
> to fail with the message (in case to topic create):
> {{ERROR org.apache.kafka.common.errors.SaslAuthenticationException:
> Authentication failed during authentication due to invalid credentials with
> SASL mechanism OAUTHBEARER}}
>
> On the server side the following line
> [https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internals/OAuthBearerClientInitialResponse.java#L68]
> with throw an exception failing the request.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [kafka] omkreddy merged pull request #13835: KAFKA-15077: Added code to trim token in FileTokenRetriever, added test.
omkreddy merged PR #13835: URL: https://github.com/apache/kafka/pull/13835 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] omkreddy commented on pull request #13835: KAFKA-15077: Added code to trim token in FileTokenRetriever, added test.
omkreddy commented on PR #13835: URL: https://github.com/apache/kafka/pull/13835#issuecomment-1586036037 Test failures are not related -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
