gudladona commented on issue #20657:
URL: https://github.com/apache/hudi/issues/20657#issuecomment-6860611017
Driver log of the occurrence described above, filtered to this incident
(hosts, partition values and base path redacted). Times are from the driver
log, except rows marked (S3): those are S3 LastModified, server-side and to the
second, so they can be about a second off the driver clock.
| Time (UTC) | Event |
|---|---|
| 12:35:24 | Executors 25 and 34 are decommissioned while stage 659 (the
bulk-insert write stage) is running. |
| 12:36:25 – 12:36:35 | 16 write tasks of stage 659.0 fail with
`FetchFailed` on map output from executors 25/34; the stage is resubmitted.
Task 406.0 (TID 150895, executor 32) needs none of the lost blocks and keeps
running. |
| 12:36:31 (S3) | TID 150895 rolls over: `-0_406-659-150895` (63 MB) is
closed and uploaded; it continues writing `-1_406-659-150895`. |
| 12:36:47 | The write job ends with the resubmitted attempt's results;
commit of `20261112123519782` starts. |
| 12:36:50.270 | Marker reconciliation lists 136 paths, including **both**
of TID 150895's files. |
| 12:36:51 (S3) | `-0` is deleted by the reconciliation (its S3 delete
marker; the delete stage ran 12:36:50.293–.578 by the driver clock). `-1` is
still open, so not on storage yet, and its delete is a no-op. |
| 12:36:50.655 | Instant completed. |
| 12:38:32.948 | TID 150895 is cancelled: `TaskKilled (Stage finished)`, 1 m
42 s after the commit. |
| 12:38:33 (S3) | `-1_406-659-150895` (3.8 MB, partial) appears on S3,
written by `finally { writer.close() }` on the kill path. It is in neither the
commit's write stats nor the marker directory (removed 12:36:52), and a storage
listing counts it as data of the completed instant. |
The two files show the gap: reconciliation deleted the file the zombie had
already closed (`-0`), but could not delete the one it still had open (`-1`),
because that file only reached storage when the task was killed, after the
commit.
S3 object versions of the two files (bucket versioning on; path prefix
redacted; the later delete of `-1` is unrelated to the write):
```
File
LastModified (UTC) Size Kind
e02198ad-58b4-4d6b-85f8-2607c2efecb5-0_406-659-150895_20261112123519782.parquet
12:36:31 63037992 version
e02198ad-58b4-4d6b-85f8-2607c2efecb5-0_406-659-150895_20261112123519782.parquet
12:36:51 - delete marker
e02198ad-58b4-4d6b-85f8-2607c2efecb5-1_406-659-150895_20261112123519782.parquet
12:38:33 3778377 version
e02198ad-58b4-4d6b-85f8-2607c2efecb5-1_406-659-150895_20261112123519782.parquet
+1 day 06:41:04 - delete marker
```
<details><summary>Driver log excerpt</summary>
```
# Driver log, Spark 3.5.5 / Hudi 1.1.1-based, row-writer bulk insert,
instant 20261112123519782 (UTC).
# Filtered to the events of this incident; hosts, partition values and base
path redacted.
12:35:24,382 WARN
org.apache.spark.scheduler.cluster.k8s.KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint:Received
executor 25 decommissioned message
12:35:24,449 WARN
org.apache.spark.scheduler.cluster.k8s.KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint:Received
executor 34 decommissioned message
12:36:25,397 INFO <app SparkListener>:onExecutorRemoved: executorId = "25",
reason = "Executor decommission: Executor 25 is decommissioned."
12:36:25,464 INFO <app SparkListener>:onExecutorRemoved: executorId = "34",
reason = "Executor decommission: Executor 34 is decommissioned."
12:36:25,598 WARN org.apache.spark.scheduler.TaskSetManager:Lost task 427.0
in stage 659.0 (TID 150911) (host-1 executor 7): FetchFailed(BlockManagerId(34,
host-2, 43943, None), shuffleId=164, mapIndex=86, mapId=150118, reduceId=427,
...)
12:36:25,627 WARN org.apache.spark.scheduler.TaskSetManager:Lost task 428.0
in stage 659.0 (TID 150912) (host-1 executor 7): FetchFailed(BlockManagerId(34,
host-2, 43943, None), shuffleId=164, mapIndex=86, mapId=150118, reduceId=428,
...)
12:36:27,042 INFO <app SparkListener>:onExecutorRemoved: executorId = "34",
reason = "The executor with id 34 was deleted by a user or the framework."
12:36:27,042 INFO <app SparkListener>:onExecutorRemoved: executorId = "25",
reason = "The executor with id 25 was deleted by a user or the framework."
12:36:27,892 WARN org.apache.spark.scheduler.TaskSetManager:Lost task 411.0
in stage 659.0 (TID 150898) (host-3 executor 33):
FetchFailed(BlockManagerId(34, host-2, 43943, None), shuffleId=164,
mapIndex=85, mapId=150117, reduceId=411, ...)
12:36:28,157 WARN org.apache.spark.scheduler.TaskSetManager:Lost task 403.0
in stage 659.0 (TID 150892) (host-4 executor 32):
FetchFailed(BlockManagerId(34, host-2, 43943, None), shuffleId=164,
mapIndex=86, mapId=150118, reduceId=403, ...)
12:36:28,650 WARN org.apache.spark.scheduler.TaskSetManager:Lost task 285.0
in stage 659.0 (TID 150814) (host-1 executor 7): FetchFailed(BlockManagerId(25,
host-5, 41631, None), shuffleId=164, mapIndex=47, mapId=150079, reduceId=285,
...)
12:36:28,932 WARN org.apache.spark.scheduler.TaskSetManager:Lost task 261.0
in stage 659.0 (TID 150794) (host-1 executor 7): FetchFailed(BlockManagerId(25,
host-5, 41631, None), shuffleId=164, mapIndex=47, mapId=150079, reduceId=261,
...)
12:36:31,540 WARN org.apache.spark.scheduler.TaskSetManager:Lost task 419.0
in stage 659.0 (TID 150906) (host-3 executor 33):
FetchFailed(BlockManagerId(25, host-5, 41631, None), shuffleId=164,
mapIndex=47, mapId=150079, reduceId=419, ...)
12:36:31,987 WARN org.apache.spark.scheduler.TaskSetManager:Lost task 275.0
in stage 659.0 (TID 150805) (host-6 executor 30):
FetchFailed(BlockManagerId(34, host-2, 43943, None), shuffleId=164,
mapIndex=85, mapId=150117, reduceId=275, ...)
12:36:32,117 WARN org.apache.spark.scheduler.TaskSetManager:Lost task 401.0
in stage 659.0 (TID 150890) (host-6 executor 30):
FetchFailed(BlockManagerId(34, host-2, 43943, None), shuffleId=164,
mapIndex=86, mapId=150118, reduceId=401, ...)
12:36:32,285 WARN org.apache.spark.scheduler.TaskSetManager:Lost task 340.0
in stage 659.0 (TID 150848) (host-3 executor 33):
FetchFailed(BlockManagerId(25, host-5, 41631, None), shuffleId=164,
mapIndex=47, mapId=150079, reduceId=340, ...)
12:36:32,425 WARN org.apache.spark.scheduler.TaskSetManager:Lost task 418.0
in stage 659.0 (TID 150905) (host-7 executor 8): FetchFailed(BlockManagerId(34,
host-2, 43943, None), shuffleId=164, mapIndex=85, mapId=150117, reduceId=418,
...)
12:36:34,125 WARN org.apache.spark.scheduler.TaskSetManager:Lost task 424.0
in stage 659.0 (TID 150910) (host-8 executor 26):
FetchFailed(BlockManagerId(25, host-5, 41631, None), shuffleId=164,
mapIndex=47, mapId=150079, reduceId=424, ...)
12:36:34,270 WARN org.apache.spark.scheduler.TaskSetManager:Lost task 365.0
in stage 659.0 (TID 150868) (host-7 executor 8): FetchFailed(BlockManagerId(25,
host-5, 41631, None), shuffleId=164, mapIndex=47, mapId=150079, reduceId=365,
...)
12:36:34,828 WARN org.apache.spark.scheduler.TaskSetManager:Lost task 422.0
in stage 659.0 (TID 150909) (host-9 executor 24):
FetchFailed(BlockManagerId(25, host-5, 41631, None), shuffleId=164,
mapIndex=47, mapId=150079, reduceId=422, ...)
12:36:35,096 WARN org.apache.spark.scheduler.TaskSetManager:Lost task 409.0
in stage 659.0 (TID 150897) (host-9 executor 24):
FetchFailed(BlockManagerId(25, host-5, 41631, None), shuffleId=164,
mapIndex=47, mapId=150079, reduceId=409, ...)
12:36:35,489 WARN org.apache.spark.scheduler.TaskSetManager:Lost task 417.0
in stage 659.0 (TID 150904) (host-10 executor 35):
FetchFailed(BlockManagerId(34, host-2, 43943, None), shuffleId=164,
mapIndex=85, mapId=150117, reduceId=417, ...)
12:36:47,529 INFO org.apache.hudi.client.BaseHoodieWriteClient:Committing
20261112123519782 action commit
12:36:47,529 INFO
org.apache.hudi.utilities.streamer.StreamSync:instantTime=20261112123519782,
totalRecords=21390404, totalErrorRecords=0, totalSuccessfulRecords=21390404
12:36:47,529 INFO <app SparkListener>:SparkListenerJobEnd - Stage: 659
SparkRDDWriteClient.Committing_stats.collect_SparkRDDWriteClient_442.2:
duration 32259
12:36:50,228 INFO org.apache.hudi.client.BaseHoodieWriteClient:Committing
20261112123519782 action commit
12:36:50,270 INFO org.apache.hudi.table.HoodieTable:Removing duplicate files
created due to task retries before committing. Paths=[... 136 paths ...,
<partition>/e02198ad-58b4-4d6b-85f8-2607c2efecb5-1_406-659-150895_20261112123519782.parquet,
...,
<partition>/e02198ad-58b4-4d6b-85f8-2607c2efecb5-0_406-659-150895_20261112123519782.parquet,
...]
12:36:50,578 INFO
org.apache.hudi.common.table.timeline.versioning.v1.ActiveTimelineV1:Marking
instant complete [==>20261112123519782__commit__INFLIGHT]
12:36:50,578 INFO <app SparkListener>:SparkListenerJobEnd - Stage: 660
HoodieSparkCopyOnWriteTable.Delete_invalid_files_generated_during_th.collect_HoodieSparkEngineContext_136.1:
duration 284
12:36:50,578 INFO org.apache.hudi.client.BaseHoodieClient:Finalize write
elapsed time (milliseconds): 349
12:36:50,655 INFO
org.apache.hudi.common.table.timeline.versioning.v1.ActiveTimelineV1:Completed
[==>20261112123519782__commit__INFLIGHT]
12:36:52,966 INFO org.apache.hudi.common.fs.FSUtils:Removed directory at
<base-path>/.hoodie/.temp/20261112123519782
12:38:32,948 WARN org.apache.spark.scheduler.TaskSetManager:Lost task 406.0
in stage 659.0 (TID 150895) (host-4 executor 32): TaskKilled (Stage finished)
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]