DanielLeens commented on issue #12132:
URL: https://github.com/apache/seatunnel/issues/12132#issuecomment-5556159167
More sightings (now 10 job cancellations across #11503, #11727, #11077,
#11757 and dev pushes today; #11727 run 33994609465 lost both Kudu legs,
00:03->01:34 and 00:05->01:35 UTC), plus a tighter timeline from the Flink 1.18
variant (run 33978895542, job 101340798483), where the JobManager container
kept logging so the JobMaster's last action is visible:
```
18:17:50,669 SourceCoordinator - Starting split enumerator for source
Source: Kudu-Source.
18:17:50,691 ExecutionGraph - Job kudu_to_assert.conf switched from state
CREATED to RUNNING.
18:17:50,733 JobMaster - JobManager successfully registered at
ResourceManager
18:17:50,839 DefaultSlotStatusSyncer - Starting allocation of slot
2553ccd8... from 172.18.0.6:43939 for job a8b499b1...
18:17:50,946 ExecutionGraph - Deploying Source: Kudu-Source ->
MultiTableSink-Sink: Writer (1/1) (attempt #0) with attempt id ...
(nothing from the JobMaster for two minutes)
18:19:50,731 StandaloneResourceManager - The heartbeat of JobManager with id
3d329b58... timed out.
```
So the JobMaster main thread stops answering exactly at `Execution.deploy`
-> `submitTask` for the very first task of the first job on a fresh
JobManager/TaskManager pair, on both 1.15.3 and 1.18.0; the ResourceManager and
Dispatcher in the same JVM keep working. On the TaskManager side the matching
symptom is `BlobClient - Downloading <job>/p-... from jobmanager` followed by
silence (1.15.3) or repeated `Failed to fetch BLOB` (1.18.0), i.e. the
permanent BLOB for the job is never served. The 1.13.6 leg and later jobs on
the same pair never hit it. A JobManager thread dump taken about a minute after
`Deploying ... (attempt #0)` would show what the JobMaster main thread and the
BlobServer are blocked on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
