stream2000 commented on PR #9887:
URL: https://github.com/apache/hudi/pull/9887#issuecomment-1777029605
> so in such case, files are always created already?
I added some logs to track one of the write tasks(task 62.0) and get the
following result:
```txt
31398 [dispatcher-event-loop-0] INFO org.apache.spark.executor.Executor []
- Executor is trying to kill task 62.0 in stage 38.0 (TID 845), reason: Stage
cancelled
33673 [ScalaTest-run-running-TestInsertTable] ERROR
org.apache.hudi.internal.DataSourceInternalWriterHelper [] - Commit
20231024112325400 aborted
33720 [Executor task launch worker for task 62.0 in stage 38.0 (TID 845)]
INFO org.apache.hudi.table.marker.TimelineServerBasedWriteMarkers [] -
[timeline-server-based] Created marker file
dt=2021-01-05/bb4c8ba7-475a-3e8f-b2ad-5ed55bee188f-0_62-845-0_20231024112325400.parquet.marker.CREATE
in 2390 ms
33725 [Executor task launch worker for task 62.0 in stage 38.0 (TID 845)]
INFO org.apache.hudi.io.storage.row.HoodieRowParquetWriteSupport [] -
Initialized Parquet WriteSupport with Catalyst schema:
33749 [Executor task launch worker for task 62.0 in stage 38.0 (TID 845)]
ERROR org.apache.spark.util.Utils [] - Aborting task
33753 [Executor task launch worker for task 62.0 in stage 38.0 (TID 845)]
ERROR
33753 [Executor task launch worker for task 62.0 in stage 38.0 (TID 845)]
ERROR org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask [] -
Aborting commit for partition 62 (task 845, attempt 0, stage 38.0)
33753 [Executor task launch worker for task 62.0 in stage 38.0 (TID 845)]
WARN org.apache.hudi.spark3.internal.HoodieBulkInsertDataInternalWriter [] -
Task Aborted
org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask [] -
Aborted commit for partition 62 (task 845, attempt 0, stage 38.0)
33753 [Executor task launch worker for task 62.0 in stage 38.0 (TID 845)]
WARN org.apache.hudi.spark3.internal.HoodieBulkInsertDataInternalWriter [] -
Task Closed
```
The event sequence is:
driver try to cancel task 62 -> driver aborted -> task 62 create marker ->
task 62 write parquet -> task 62 aborted -> task 62 closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
