loustler opened a new issue, #11679: URL: https://github.com/apache/seatunnel/issues/11679
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues. ### What happened When a batch job is cancelled because a downstream catalog rejects it on authorization grounds, the cluster reaches a terminal state correctly but the `seatunnel.sh` client process that submitted the job never exits. Anything waiting on that process — a test harness, a shell script, an orchestrator — blocks indefinitely. This was first spotted by @DanielLeens during review of #11648 ([review comment](https://github.com/apache/seatunnel/pull/11648#pullrequestreview-4873922426)), who traced the runtime path in detail and, importantly, declined to attribute it to that PR without proof. I ran that attribution down, and he was right: **it reproduces on pristine `dev` with none of that PR's changes present**, so this is an existing engine behaviour rather than something a dependency upgrade introduced. Filing it separately as he suggested. **Observed sequence** (from `PaimonWithS3IT.privilegeEnabledPaimonSourceUnAuthorized`, which is an existing test on `dev`): 1. `executeJob("/fake_to_paimon_with_s3_with_privilege.conf")` runs and finishes normally, exit code 0. 2. `executeJob("/paimon_to_paimon_with_s3_with_privilege.conf")` is submitted. `PaimonCatalog` rejects the read with `NoPrivilegeException: User paimon doesn't have privilege SELECT on table seatunnel_namespace11.st_test`. 3. `SubPlan.handleCheckpointError()` moves the pipeline `RUNNING -> CANCELING`. The pipeline is resubmitted and fails identically, repeatedly. 4. The coordinator's own periodic status dump then reports `pendingJobCount / runningJobCount / cancellingJobCount / canceledJobCount = 0` — the cluster believes nothing is outstanding. 5. The job id is never mentioned again: not FINISHED, not FAILED, not CANCELED. 6. The client `seatunnel.sh` process never exits. The caller blocks forever. The test asserts `exitCode == 1`, i.e. it expects the CLI to exit promptly and non-zero once the job fails. That assertion is never reached. **It is not deterministic.** On one and the same commit, the two CI legs disagreed: | leg | `PaimonWithS3IT` | |---|---| | JDK 11 | passed — `Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 118.481 s` | | JDK 8 | started, never produced a result line, killed at the 180-minute job timeout | Worth noting the SeaTunnel process itself is on the same JDK in both cases — the E2E harness pins `seatunnelhub/openjdk:8u342` regardless of the matrix leg — so the difference between the legs is timing, not a JDK-specific code path. That points at a race rather than a deterministic deadlock. On the same JDK 8 leg where `PaimonWithS3IT` hung, `PaimonIT` completed normally (`Tests run: 26, Failures: 0, Errors: 0, Time elapsed: 1,206.934 s`), so this is not a slow or overloaded runner. **Why this matters beyond the test.** If a job that a downstream privilege or authorization check legitimately cancels can leave the *client* unable to observe a terminal status, then anything that shells out to `seatunnel.sh` and waits on its exit code can hang indefinitely rather than fail. That is an operability concern independent of the E2E harness that exposed it. **Where the cause probably lives** (not verified — I did not get a thread dump of the blocked client): - `seatunnel-engine-server` — `SubPlan.canRestorePipeline()` / `handleCheckpointError()` / `pipelineMaxRestoreNum` - `seatunnel-engine-client` — whatever keeps the job-status poll from observing a terminal state after a pipeline is cancelled by a checkpoint error ### SeaTunnel Version `dev` (3.0.0-SNAPSHOT). Reproduced on a branch that is `origin/dev` plus a single three-line comment, added only so the `changes` path filter would schedule `paimon-connector-it`. The comment sits above a class declaration and changes no behaviour, and the test methods involved are byte-for-byte identical to `dev`. ``` branch: claude/dev-baseline-paimon-s3 @ 80fe7dc77 CI run: 31100290272 job 92614470554 paimon-connector-it (11, ubuntu-latest) -> PaimonWithS3IT 4/4 in 118.481 s job 92614470569 paimon-connector-it (8, ubuntu-latest) -> PaimonWithS3IT hung, killed at 180 min ``` In the JDK 8 job log, after `PaimonWithS3IT` starts there are 5 occurrences of `/fake_to_paimon_with_s3_with_privilege.conf` and 2 of `/paimon_to_paimon_with_s3_with_privilege.conf`, no `Tests run:` line for the class, and the job ends with `##[error]The operation was canceled.` at the 180-minute mark. ### SeaTunnel Config The second job — the one whose cancellation leaves the client stuck. `user = paimon` has been granted INSERT but not SELECT on the source table, so the read is rejected by the privilege manager. ```conf env { parallelism = 1 job.mode = "BATCH" } source { Paimon { warehouse = "s3a://privilegetest/" database = "seatunnel_namespace11" table = "st_test" user = "paimon" password = "123456" paimon.hadoop.conf = { fs.s3a.access-key = minio fs.s3a.secret-key = miniominio fs.s3a.endpoint = "http://minio:9000"; fs.s3a.path.style.access = true fs.s3a.aws.credentials.provider = org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider } } } sink { Paimon { warehouse = "s3a://test/" database = "seatunnel_namespace13" table = "st_test_sink_priv" paimon.hadoop.conf = { fs.s3a.access-key = minio fs.s3a.secret-key = miniominio fs.s3a.endpoint = "http://minio:9000"; fs.s3a.path.style.access = true fs.s3a.aws.credentials.provider = org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider } } } ``` ### Running Command ```shell # Reproduces via the existing E2E test on an unmodified dev tree. # The hang is in privilegeEnabledPaimonSourceUnAuthorized, which submits two jobs; # the second is rejected by the Paimon privilege check. ./mvnw -B -T 1 verify -DskipUT=true -DskipIT=false \ -D"license.skipAddThirdParty"=true -D"skip.ui"=true --no-snapshot-updates \ -pl :connector-paimon-e2e -am -Pci # It does not reproduce on every run - see the two CI legs above. Expect to need # several attempts. ``` ### Error Exception There is no exception. That is the problem: the job stops being reported and the client never returns. ```log # cluster side - the pipeline is cancelled correctly NoPrivilegeException: User paimon doesn't have privilege SELECT on table seatunnel_namespace11.st_test ... SubPlan.handleCheckpointError() -> pipeline turned from state RUNNING to CANCELING ... the pipeline is resubmitted and fails identically, repeatedly # then the coordinator reports nothing outstanding, every ~10s, for the next ~2 hours pendingJobCount : 0, runningJobCount : 0, cancellingJobCount : 0, canceledJobCount : 0 # the job id is never mentioned again - no FINISHED, no FAILED, no CANCELED # the client seatunnel.sh process never exits, so the caller never unblocks # CI eventually kills the whole job ##[error]The operation was canceled. ``` ### Zeta or Flink or Spark Version Zeta, `dev` (3.0.0-SNAPSHOT). ### Java or Scala Version The SeaTunnel process runs on the E2E image `seatunnelhub/openjdk:8u342` in both cases. The surrounding Maven/test JVM was JDK 8 on the leg that hung and JDK 11 on the leg that passed. ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! Not for the engine fix itself — the suspected area is Zeta's restore/cancel and client status-polling logic, which I do not know well enough to change safely. I am happy to help with reproduction, to test a candidate fix, and to supply more log detail from the runs above on request. A separate, smaller mitigation is being handled in #11648: bounding `executeJob` in the affected test so a recurrence fails in seconds instead of consuming a full 180-minute CI slot. That only limits the blast radius in CI; it does not address the behaviour reported here, and given this reproduces on `dev` the same bound is arguably worth having on `dev` independently of that PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
