DanielLeens opened a new pull request, #12446: URL: https://github.com/apache/seatunnel/pull/12446
### Purpose of this pull request `kudu-connector-it` is being cancelled by its 90-minute `timeout-minutes` on PRs that do not touch Kudu (eight PRs updated in the last two days: #12394, #12137, #11727, #11077, #11545, ... on both JDK legs) and on `dev` push run 35625308814 (JDK 11), while its green runs take about 30 minutes. This is a hang, not growth, and this PR makes the next occurrence diagnosable instead of silent. It does **not** fix the hang. ### What the logs show Both cancellations I inspected hang in `KuduIT.testKuduWholeDatabaseRead`, on different Flink versions (`dev` on Flink 1.18.0, #12394 on Flink 1.15.3, where it was the very first job on a freshly started container): 1. The client submits `kudu_to_assert_with_all_tables.conf`; the JobManager logs `Starting split enumerator`, `switched from state CREATED to RUNNING`, `Deploying Source: Kudu-Source ...` and then **logs nothing else for the job**. 2. The TaskManager receives the task, starts `BlobClient Downloading ... from jobmanager/172.18.0.5:6124`, and ten seconds later logs `Slot offering to JobManager did not finish in time. Retrying the slot offering.` The BLOB download later fails with `GET operation failed: Read timed out`. 3. Two minutes after submission the ResourceManager (same JVM as the JobMaster) logs `The heartbeat of JobManager with id ... timed out` and the TaskManager fails the task with `Disconnect from JobManager responsible for ...`. 4. For the remaining ~80 minutes the TaskManager loops `Could not resolve JobManager address pekko.tcp://flink@jobmanager:6123/user/rpc/jobmanager_N`, the ResourceManager loops `Missing resources: Job ... numberOfRequiredSlots=1`, and the SeaTunnel client inside the JobManager container waits forever, until the workflow cancels the job and the containers are destroyed. So the JobMaster RPC endpoint and the BlobServer both stop responding right after the first deployment, while the ResourceManager in the same JVM keeps running. Nothing SeaTunnel runs on the JobMaster main thread: `FlinkSourceEnumerator.start()`/`addReader()` (and through them `KuduSourceSplitEnumerator.open()`/`run()`, which create the Kudu client and build scan tokens) execute on Flink's SourceCoordinator thread. Which thread is actually stuck can only be answered by a thread dump of the JobManager JVM, and none exists because the cancellation happens first. ### Change (test infrastructure only) `AbstractTestFlinkContainer#executeJob(String, List<String>)`: - runs the job submission on a daemon thread and waits at most 30 minutes for it; - on timeout, sends `SIGQUIT` to every JVM in the JobManager and TaskManager containers (found through `/proc`, since the Flink images ship neither `pgrep` nor `jstack`/`jcmd`), waits five seconds so HotSpot can print the dumps to the containers' stdout, which testcontainers already forwards into the test log, and then fails the test with a message pointing at them; - rethrows `IOException`/`InterruptedException`/`RuntimeException` from the submission unchanged, so every existing caller sees the same exceptions as before. `executeJob(String)` delegates to this method, so both Flink entry points are covered; `savepointJob`, `cancelJob` and `restoreJob` are untouched. Green jobs are unaffected: no Flink E2E job legitimately runs anywhere near 30 minutes (the longest whole IT class in `jdbc-connectors-it-part-1` is 25 minutes across all containers). A hung Kudu job now fails after 30 minutes with thread dumps instead of being cancelled after 90 minutes with nothing. ### Files - `seatunnel-e2e/seatunnel-e2e-common/src/test/java/org/apache/seatunnel/e2e/common/container/flink/AbstractTestFlinkContainer.java` ### How was this patch tested? Test-infrastructure-only change; verified by this PR's GitHub CI (the Flink E2E jobs exercise the new path on every green job). Local checks were limited to `./mvnw spotless:apply -pl seatunnel-e2e/seatunnel-e2e-common`. The timeout branch itself is only exercised by a real hang; the next `kudu-connector-it` cancellation on `dev` will show whether the dumps land in the log. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
