amogh-jahagirdar commented on code in PR #5379:
URL: https://github.com/apache/iceberg/pull/5379#discussion_r933670895
##########
api/src/main/java/org/apache/iceberg/io/SupportsBulkOperations.java:
##########
@@ -18,12 +18,23 @@
*/
package org.apache.iceberg.io;
+import java.util.concurrent.ExecutorService;
+
public interface SupportsBulkOperations {
/**
* Delete the files at the given paths.
*
* @param pathsToDelete The paths to delete
- * @throws BulkDeletionFailureException in
+ * @throws BulkDeletionFailureException in case of failure to delete at
least 1 file
*/
void deleteFiles(Iterable<String> pathsToDelete) throws
BulkDeletionFailureException;
+
+ /**
+ * Delete the files at the given paths with the given executor service
+ *
+ * @param pathsToDelete The paths to delete
+ * @throws BulkDeletionFailureException in case of failure to delete at
least 1 file
+ */
+ void deleteFiles(Iterable<String> pathsToDelete, ExecutorService svc)
Review Comment:
Good point, I missed there's already a threadpool in the fileIO
implementation. For now, since it's unclear if passing it in the mixin
interface makes sense. I thought it did because I was thinking a compute layer
at a higher layer can determine what is the appropriate executor service to use
and then pass that to the file io layer for better control, but maybe it
doesn't matter. We can modify the interface when we know we want it.
I've updated to us the existing executor service in S3FileIO.
##########
api/src/main/java/org/apache/iceberg/io/SupportsBulkOperations.java:
##########
@@ -18,12 +18,23 @@
*/
package org.apache.iceberg.io;
+import java.util.concurrent.ExecutorService;
+
public interface SupportsBulkOperations {
/**
* Delete the files at the given paths.
*
* @param pathsToDelete The paths to delete
- * @throws BulkDeletionFailureException in
+ * @throws BulkDeletionFailureException in case of failure to delete at
least 1 file
*/
void deleteFiles(Iterable<String> pathsToDelete) throws
BulkDeletionFailureException;
+
+ /**
+ * Delete the files at the given paths with the given executor service
+ *
+ * @param pathsToDelete The paths to delete
+ * @throws BulkDeletionFailureException in case of failure to delete at
least 1 file
+ */
+ void deleteFiles(Iterable<String> pathsToDelete, ExecutorService svc)
Review Comment:
Good point, I missed there's already a threadpool in the S3fileIO
implementation. For now, since it's unclear if passing it in the mixin
interface makes sense. I thought it did because I was thinking a compute layer
at a higher layer can determine what is the appropriate executor service to use
and then pass that to the file io layer for better control, but maybe it
doesn't matter. We can modify the interface when we know we want it.
I've updated to us the existing executor service in S3FileIO.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
