[
https://issues.apache.org/jira/browse/HUDI-6039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ethan Guo updated HUDI-6039:
----------------------------
Fix Version/s: 0.14.0
(was: 0.13.1)
> Fix FS based listing in clean planner
> -------------------------------------
>
> Key: HUDI-6039
> URL: https://issues.apache.org/jira/browse/HUDI-6039
> Project: Apache Hudi
> Issue Type: Improvement
> Components: cleaning, writer-core
> Reporter: sivabalan narayanan
> Priority: Major
> Labels: pull-request-available
> Fix For: 0.14.0
>
>
> we are using FS based listing when full cleaning is invoked w/ clean planner.
> {code:java}
> /** * Scan and list all partitions for cleaning. * @return all partitions
> paths for the dataset. */ private List<String>
> getPartitionPathsForFullCleaning() { // Go to brute force mode of scanning
> all partitions try { // Because the partition of BaseTableMetadata
> has been deleted, // all partition information can only be obtained from
> FileSystemBackedTableMetadata. FileSystemBackedTableMetadata
> fsBackedTableMetadata = new FileSystemBackedTableMetadata(context,
> context.getHadoopConf(), config.getBasePath(),
> config.shouldAssumeDatePartitioning()); return
> fsBackedTableMetadata.getAllPartitionPaths(); } catch (IOException e) {
> return Collections.emptyList(); } } {code}
>
> we should be using Metadata based listing if metadata is enabled.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
