nielifeng opened a new issue, #11437:
URL: https://github.com/apache/seatunnel/issues/11437
### Search before asking
- [x] I searched the existing issues. This is related to #10568 and #9807,
but the
failure below is different: it reproduces on current `dev` with #10567
included and
blocks before task deployment.
### What happened
A fixed-slot streaming job cannot recover after the active Zeta Master
process exits while
both Workers remain alive.
Environment:
- 3 dedicated Master nodes and 2 dedicated Worker nodes
- 8 fixed slots per Worker, 16 total
- `job-schedule-strategy: WAIT`, `dynamic-slot: false`
- shared NFS checkpoint storage
- current `dev` commit `d6df7f2f25abface0aa9498a8fc7b3a18572c32e`
- the commit contains merged PR #10567 (`e365ebfa1`)
The test job was FakeSource to Console, streaming mode, parallelism 10. It
used 11 task
groups/slots and completed 7 checkpoints before the fault. Source and sink
QPS were both about
95,760 rows/s.
At `2026-07-15T09:38:34Z`, I sent `SIGKILL` to the active Master's systemd
`MainPID`.
- a replacement Master was elected in about 1 second;
- the failed Master process restarted and the cluster returned to 5 members
in 21 seconds;
- the replacement Master loaded checkpoint 7;
- the job entered `PENDING/RESTORE` and never returned to `RUNNING`;
- after 200 seconds, source and sink QPS were both zero;
- Workers still owned all 11 old slots, leaving only 5 of 16 slots free;
- the pending diagnostic reported all 11 task groups lacking resources with
`RESOURCE_NOT_ENOUGH: Slot request future not created`.
Cancelling the job removed it from the running/pending job counts, but the
11 slots remained
assigned and both Workers still reported the cancelled job ID. Restarting
the Workers was required
to return to 16 free slots.
Unlike #10568, there was no `TaskGroupLocation ... already exists` exception
in any Master or
Worker log. PR #10567 appears to address the idempotent deployment error,
but recovery now remains
blocked in fixed-slot resource allocation before `deployTask()` can run.
Steps to reproduce:
1. Start 3 Zeta Masters and 2 Zeta Workers with fixed slots and `WAIT`
scheduling.
2. Submit the streaming job below and wait for at least two successful
checkpoints.
3. Kill only the active Master Java process. Keep both Workers alive.
4. Wait for a replacement Master and for the failed Master to rejoin.
5. Query `/overview`, `/running-jobs`, and `/pending-jobs`.
6. Cancel the job and query the Worker slot snapshot again.
Expected behavior:
- the job should reconnect to or reuse the Worker-side task groups and
return to `RUNNING`;
- the fixed-slot count should remain 11 assigned / 5 free during recovery;
- cancelling the job should return all 16 slots without restarting Workers.
### SeaTunnel Version
`3.0.0-SNAPSHOT`, commit `d6df7f2f25abface0aa9498a8fc7b3a18572c32e`
Package SHA-256:
`6f59b02eeb52b8317eb3053dd6d8dcb722526beea48a215e94a17f978d46518c`
### SeaTunnel Config
```conf
seatunnel:
engine:
job-schedule-strategy: WAIT
slot-service:
dynamic-slot: false
slot-num: 8
slot-allocate-strategy: SLOT_RATIO
checkpoint:
interval: 30000
timeout: 120000
storage:
type: hdfs
max-retained: 3
plugin-config:
namespace: /shared/seatunnel/checkpoint/
storage.type: hdfs
```
```conf
env {
job.name = "zeta-fixed-slot-master-failover"
job.mode = "STREAMING"
parallelism = 10
checkpoint.interval = 30000
checkpoint.timeout = 120000
}
source {
FakeSource {
plugin_output = "benchmark"
row.num = 72000000
split.num = 3600
split.read-interval = 1
string.length = 128
schema {
fields {
id = "bigint"
bucket = "int"
value = "double"
payload = "string"
active = "boolean"
}
}
}
}
sink {
Console {
plugin_input = "benchmark"
log.print.data = false
}
}
```
### Running Command
```shell
bin/seatunnel.sh --config zeta-fixed-slot-master-failover.conf --async
# After at least two completed checkpoints, on the active Master host:
pid=$(systemctl show -p MainPID --value seatunnel-master.service)
kill -9 "$pid"
```
### Error Exception
```log
begin restore job (...) from master active switch
pipeline(1) start with savePoint on checkPointId(7)
registerWorker: ... assignedSlots=[... ownerJobID=<job-id>, assigned=true
...]
Pre resource application failed for job: <job-id>, ... failed: 6/11,
error: can't apply resource request: ResourceProfile{cpu=CPU{core=0},
heapMemory=Memory{bytes=0}}
Current strategy is WAIT, and resources is not enough, skipping this
schedule,
JobID: <job-id>
```
`GET /pending-jobs` after more than 120 seconds:
```json
{
"queueSummary": {
"size": 1,
"scheduleStrategy": "WAIT",
"lackingTaskGroups": 11
},
"clusterSnapshot": {
"totalSlots": 16,
"freeSlots": 5,
"assignedSlots": 11,
"workerCount": 2
},
"pendingJobs": [
{
"pendingSourceState": "RESTORE",
"jobStatus": "PENDING",
"totalTaskGroups": 11,
"allocatedTaskGroups": 0,
"lackingTaskGroups": 11,
"failureReason": "RESOURCE_NOT_ENOUGH",
"failureMessage": "Slot request future not created"
}
]
}
```
### Zeta or Flink or Spark Version
Zeta `3.0.0-SNAPSHOT`, Hazelcast `5.1`
### Java or Scala Version
OpenJDK `11.0.31`
### Screenshots
Not applicable. REST samples and logs were collected for the full fault
window.
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
