boneanxs commented on issue #9615: URL: https://github.com/apache/hudi/issues/9615#issuecomment-1772233812
@KnightChess Have you seen any logs like this in executor side when this issue happens? ```java 23/03/23 02:28:45 INFO HoodieMergeHandle: MaxMemoryPerPartitionMerge => 1073741824 23/03/23 02:28:46 INFO Executor: Executor is trying to kill task 2.1 in stage 11.0 (TID 1471), reason: another attempt succeeded 23/03/23 02:28:46 INFO Executor: Executor is trying to kill task 2.1 in stage 11.0 (TID 1471), reason: Stage finished 23/03/23 02:28:47 INFO HoodieMergeHandle: Number of entries in MemoryBasedMap => 0, Total size in bytes of MemoryBasedMap => 0, Number of entries in BitCaskDiskMap => 0, Size of file spilled to disk => 0 23/03/23 02:28:47 INFO HoodieMergeHandle: partitionPath:grass_region=test, fileId to be merged:d3ee8406-4011-44a4-8913-8be0349a6686-0 ``` From our side, we actually see hudi ignore this kill signal and continue writing(`Executor is trying to kill task`). So here is actually 2 issues 1. Should fail task immediately if task obtains kill signal 2. How to handle duplicate files if reconcile commits is finished while task still writing. I think if the 1) is handled correctly, we can add an extra clean step(to clean files if already written) in the task side? The extra clean step should solve most cases, but may not solve all cases. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
