[
https://issues.apache.org/jira/browse/FLINK-41515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18145567#comment-18145567
]
Sylwester Lachiewicz commented on FLINK-41515:
----------------------------------------------
The JDK 21 failure has its cause in the job log. The job failed because the
{{KafkaWriter}}'s producer in the TaskManager timed out waiting for the topic:
{noformat}
Caused by: org.apache.kafka.common.errors.TimeoutException: Topic
kafka-single-topic-6550890708556829199 not present in metadata after 60000 ms.
{noformat}
The test JVM had created that topic, with all partitions ready, 60 s earlier
(17:06:13). While the writer blocks in {{send()}}, the source is backpressured,
so {{waitExpectedSizeData}} never receives its records. The JDK 11 failure fits
the same mechanism with different timing: enough records reached the collect
sink, then the savepoint could not complete while the writer was blocked. That
job's log has no producer exception, so this part is inferred.
The admin-client warnings for {{localhost:32869}} are unrelated. All 16 clients
were left open by {{KafkaSourceE2ECase}}, which ran earlier in the same JVM
against a broker on that port. They reconnect when their 5-minute metadata
refresh comes due, after that broker has stopped. The sink case's own admin
clients log no connection failures.
Why the TaskManager cannot see the topic is still unknown, because CI drops the
Flink container output: it reaches the test log at INFO through the
{{FlinkContainerTestEnvironment}} logger, which
[tools/ci/log4j.properties|https://github.com/apache/flink-connector-kafka/blob/main/tools/ci/log4j.properties]
sets to WARN. [PR
#346|https://github.com/apache/flink-connector-kafka/pull/346] logs the
JobManager, TaskManager and Kafka container output when an E2E test fails. Of
the last 50 failed CI runs, only the two above show this failure.
> KafkaSinkE2ECase restartFromSavepoint fails before the savepoint with POOLING
> and AT_LEAST_ONCE
> -----------------------------------------------------------------------------------------------
>
> Key: FLINK-41515
> URL: https://issues.apache.org/jira/browse/FLINK-41515
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
> Affects Versions: kafka-5.1.0
> Reporter: Sylwester Lachiewicz
> Priority: Major
> Labels: pull-request-available, test-stability
>
> {{KafkaSinkE2ECase}} fails intermittently in
> {{SinkTestSuiteBase.restartFromSavepoint}} while it stops the first job with
> a savepoint, before any restore. Both failures so far use {{ExternalContext:
> [POOLING], Semantic: [AT_LEAST_ONCE]}}.
> ||Test||Where||Result||Run||
> |{{testScaleUp}}|{{SinkTestSuiteBase.java:288}},
> {{stopWithSavepoint(...).get(30, TimeUnit.SECONDS)}}|{{TimeoutException}}
> after 33.7 s|[Flink 2.2.1, JDK
> 11|https://github.com/apache/flink-connector-kafka/actions/runs/39342878234/job/118368418382],
> 2026-10-20|
> |{{testScaleDown}}|{{SinkTestSuiteBase.java:280}} ->
> {{waitExpectedSizeData:647}}|failure after 60.8 s|[Flink 2.2.1, JDK
> 21|https://github.com/apache/flink-connector-kafka/actions/runs/43735631597/job/132877501370],
> 2026-11-15|
> The other jobs of the same commits passed, or were cancelled by fail-fast.
> Neither run's PR touches the E2E module ([PR
> #314|https://github.com/apache/flink-connector-kafka/pull/314], [PR
> #288|https://github.com/apache/flink-connector-kafka/pull/288]), and
> {{AT_LEAST_ONCE}} writes without transactions.
> What the logs show:
> * The job logs contain no output from the Flink JobManager or TaskManager
> containers, and the runs uploaded no artifacts, so the cause can't be read
> from CI.
> * In the 2026-10-20 run, the test JVM's admin clients failed to connect to
> the broker's mapped port ({{localhost:32869}}) 634 times between 00:06:06 and
> the failure at 00:07:20, while other tests in the class were still passing.
> The JDK 17 job of the same run logs 29 such warnings, all at teardown.
> Earlier {{KafkaSinkE2ECase}} failures,
> [FLINK-26921|https://issues.apache.org/jira/browse/FLINK-26921] and
> [FLINK-30911|https://issues.apache.org/jira/browse/FLINK-30911], were closed
> as Cannot Reproduce. Collecting the Flink container logs when an E2E test
> fails would make the next occurrence diagnosable.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)