aokolnychyi commented on a change in pull request #4469:
URL: https://github.com/apache/iceberg/pull/4469#discussion_r840971425
##########
File path:
spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/actions/BaseDeleteReachableFilesSparkAction.java
##########
@@ -57,37 +51,27 @@
@SuppressWarnings("UnnecessaryAnonymousClass")
public class BaseDeleteReachableFilesSparkAction
extends BaseSparkAction<DeleteReachableFiles, DeleteReachableFiles.Result>
implements DeleteReachableFiles {
- private static final Logger LOG =
LoggerFactory.getLogger(BaseDeleteReachableFilesSparkAction.class);
-
- private static final String CONTENT_FILE = "Content File";
- private static final String MANIFEST = "Manifest";
- private static final String MANIFEST_LIST = "Manifest List";
- private static final String OTHERS = "Others";
- private static final String STREAM_RESULTS = "stream-results";
+ public static final String STREAM_RESULTS = "stream-results";
+ public static final boolean STREAM_RESULTS_DEFAULT = false;
- // Creates an executor service that runs each task in the thread that
invokes execute/submit.
- private static final ExecutorService DEFAULT_DELETE_EXECUTOR_SERVICE = null;
-
- private final TableMetadata tableMetadata;
+ private static final Logger LOG =
LoggerFactory.getLogger(BaseDeleteReachableFilesSparkAction.class);
+ private final String metadataFileLocation;
private final Consumer<String> defaultDelete = new Consumer<String>() {
@Override
public void accept(String file) {
io.deleteFile(file);
}
};
- private Consumer<String> removeFunc = defaultDelete;
- private ExecutorService removeExecutorService =
DEFAULT_DELETE_EXECUTOR_SERVICE;
+ private Consumer<String> deleteFunc = defaultDelete;
+ private ExecutorService deleteExecutorService = null;
private FileIO io = new HadoopFileIO(spark().sessionState().newHadoopConf());
- public BaseDeleteReachableFilesSparkAction(SparkSession spark, String
metadataLocation) {
+ public BaseDeleteReachableFilesSparkAction(SparkSession spark, String
metadataFileLocation) {
super(spark);
- this.tableMetadata = TableMetadataParser.read(io, metadataLocation);
Review comment:
We should not try to read the metadata using the default `FileIO` in the
constructor. I moved this check to execution once we have the actual `FileIO`
the user has configured.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
