sivabalan narayanan created HUDI-6039:
-----------------------------------------
Summary: Fix FS based listing in clean planner
Key: HUDI-6039
URL: https://issues.apache.org/jira/browse/HUDI-6039
Project: Apache Hudi
Issue Type: Improvement
Components: cleaning, writer-core
Reporter: sivabalan narayanan
we are using FS based listing when full cleaning is invoked w/ clean planner.
{code:java}
/** * Scan and list all partitions for cleaning. * @return all partitions
paths for the dataset. */ private List<String>
getPartitionPathsForFullCleaning() { // Go to brute force mode of scanning
all partitions try { // Because the partition of BaseTableMetadata has
been deleted, // all partition information can only be obtained from
FileSystemBackedTableMetadata. FileSystemBackedTableMetadata
fsBackedTableMetadata = new FileSystemBackedTableMetadata(context,
context.getHadoopConf(), config.getBasePath(),
config.shouldAssumeDatePartitioning()); return
fsBackedTableMetadata.getAllPartitionPaths(); } catch (IOException e) {
return Collections.emptyList(); } } {code}
we should be using Metadata based listing if metadata is enabled.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
