dimas-b commented on code in PR #4850:
URL: https://github.com/apache/polaris/pull/4850#discussion_r3456307636
##########
runtime/service/src/main/java/org/apache/polaris/service/task/BatchFileCleanupTaskHandler.java:
##########
@@ -77,17 +77,12 @@ public boolean handleTask(TaskEntity task, CallContext
callContext) {
missingFiles.size());
}
- // Schedule the deletion for each file asynchronously
- List<CompletableFuture<Void>> deleteFutures =
- validFiles.stream()
- .map(file -> super.tryDelete(tableId, authorizedFileIO, null,
file, null, 1))
- .toList();
+ CompletableFuture<Void> deleteFutures =
+ tryDelete(
+ tableId, authorizedFileIO, validFiles,
cleanupTask.type().getValue(), true, null, 1);
Review Comment:
The `true` means "concurrent" here, I guess, and it goes into Iceberg's
`CatalogUtil.deleteFiles()`, which will use its own thread pool for "async"
work in batch deletes are not supported... I'm not sure delegating to the
Iceberg's SDK thread pool is a good idea here... 🤔
If batching is not supported, I'd rather Polaris handled individual delete
tasks on its own thread pool.
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
