JingsongLi commented on code in PR #11182:
URL: https://github.com/apache/paimon/pull/11182#discussion_r4583089037
##########
paimon-core/src/main/java/org/apache/paimon/operation/FileStoreCommitImpl.java:
##########
@@ -757,16 +763,76 @@ public void abort(List<CommitMessage> commitMessages) {
for (DataFileMeta file :
commitMessage.compactIncrement().compactAfter()) {
file.collectFiles(dataPathFactory).stream()
.filter(path -> !compactInputPaths.contains(path))
- .forEach(fileIO::deleteQuietly);
+ .forEach(filesToDelete::add);
}
List<IndexFileMeta> indexFilesToDelete = new ArrayList<>();
indexFilesToDelete.addAll(commitMessage.newFilesIncrement().newIndexFiles());
indexFilesToDelete.addAll(commitMessage.compactIncrement().newIndexFiles());
for (IndexFileMeta file : indexFilesToDelete) {
- fileIO.deleteQuietly(indexPathFactory.toPath(file));
+ filesToDelete.add(indexPathFactory.toPath(file));
}
}
+ deleteConcurrently(filesToDelete);
+ }
+
+ /**
+ * Attempts every delete and waits for all of them even when the caller is
interrupted. The
+ * first unchecked failure in input order is rethrown as is, later ones
are suppressed into it.
+ * Do not call this from a {@link FileOperationThreadPool} thread, it may
wait for itself.
+ */
+ private void deleteConcurrently(List<Path> files) {
+ if (files.isEmpty()) {
+ return;
+ }
+
+ ExecutorService executor =
+
FileOperationThreadPool.getExecutorService(options.fileOperationThreadNum());
+ boolean interrupted = false;
+ // Run each delete with the caller's context class loader, as the
sequential abort did.
+ ClassLoader classLoader =
Thread.currentThread().getContextClassLoader();
+ List<FutureTask<?>> deletes = new ArrayList<>(files.size());
+ for (Path file : files) {
+ FutureTask<?> delete =
+ new FutureTask<>(
+ () -> {
+ Thread thread = Thread.currentThread();
+ ClassLoader previous =
thread.getContextClassLoader();
+ thread.setContextClassLoader(classLoader);
+ try {
+ fileIO.deleteQuietly(file);
+ } finally {
+ thread.setContextClassLoader(previous);
+ }
+ },
+ null);
+ // Clear the flag first, otherwise the bounded executor skips its
permit and the bound.
+ interrupted |= Thread.interrupted();
+ executor.execute(delete);
Review Comment:
[P2] Preserve the delete bound when interrupted while acquiring a permit
Clearing the interrupt flag before `execute` does not cover an interrupt
arriving while `SemaphoredDelegatingExecutor.execute` is blocked in
`Semaphore.acquire()`. That executor catches the interruption and dispatches
the command without a permit. A table limit below the shared pool size is a
normal configuration, so cancellation can bypass the advertised maximum
concurrent file operations.
I reproduced this through public `abort` using four real owned file paths,
`file-operation.thread-num=1`, and a shared pool with at least four threads.
Holding the first delete, waiting until the caller is blocked in semaphore
acquisition, then interrupting it once produces two simultaneous deletes; the
identical no-interrupt control produces one. Both finish all deletes and
preserve the interrupt flag.
Please make bounded submission continue waiting for a permit under
interruption (while remembering and restoring the flag), or use a fixed number
of delete workers, and cover interruption during submission. The existing tests
interrupt before submission or after every delete has already been submitted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]