yihua commented on code in PR #11067:
URL: https://github.com/apache/hudi/pull/11067#discussion_r1577326679
##########
hudi-common/src/main/java/org/apache/hudi/metadata/MetadataPartitionType.java:
##########
@@ -18,30 +18,52 @@
package org.apache.hudi.metadata;
+import org.apache.hudi.common.config.HoodieMetadataConfig;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+
+import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;
import java.util.Set;
+import java.util.function.BiPredicate;
+import java.util.function.Predicate;
import java.util.stream.Collectors;
/**
* Partition types for metadata table.
*/
public enum MetadataPartitionType {
- FILES(HoodieTableMetadataUtil.PARTITION_NAME_FILES, "files-"),
- COLUMN_STATS(HoodieTableMetadataUtil.PARTITION_NAME_COLUMN_STATS,
"col-stats-"),
- BLOOM_FILTERS(HoodieTableMetadataUtil.PARTITION_NAME_BLOOM_FILTERS,
"bloom-filters-"),
- RECORD_INDEX(HoodieTableMetadataUtil.PARTITION_NAME_RECORD_INDEX,
"record-index-"),
-
FUNCTIONAL_INDEX(HoodieTableMetadataUtil.PARTITION_NAME_FUNCTIONAL_INDEX_PREFIX,
"func-index-");
+ FILES(HoodieTableMetadataUtil.PARTITION_NAME_FILES, "files-",
+ HoodieMetadataConfig::enabled,
Review Comment:
For checking the write config, should we have the `ConfigProperty` reference
here, instead of using the method, so it's more clear?
##########
hudi-common/src/main/java/org/apache/hudi/metadata/MetadataPartitionType.java:
##########
@@ -70,6 +92,19 @@ public static Set<String> getAllPartitionPaths() {
.collect(Collectors.toSet());
}
+ /**
+ * Returns the list of metadata partition types enabled based on the
metadata config and table config.
+ */
+ public static List<MetadataPartitionType>
getEnabledPartitions(HoodieMetadataConfig metadataConfig, HoodieTableMetaClient
metaClient) {
+ List<MetadataPartitionType> enabledTypes = new ArrayList<>(4);
Review Comment:
We can use sth like to avoid initializing the list:
`return Arrays.stream(values()).filter(...).collect(Collectors.toList());`
##########
hudi-common/src/main/java/org/apache/hudi/metadata/MetadataPartitionType.java:
##########
@@ -18,30 +18,52 @@
package org.apache.hudi.metadata;
+import org.apache.hudi.common.config.HoodieMetadataConfig;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+
+import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;
import java.util.Set;
+import java.util.function.BiPredicate;
+import java.util.function.Predicate;
import java.util.stream.Collectors;
/**
* Partition types for metadata table.
*/
public enum MetadataPartitionType {
- FILES(HoodieTableMetadataUtil.PARTITION_NAME_FILES, "files-"),
- COLUMN_STATS(HoodieTableMetadataUtil.PARTITION_NAME_COLUMN_STATS,
"col-stats-"),
- BLOOM_FILTERS(HoodieTableMetadataUtil.PARTITION_NAME_BLOOM_FILTERS,
"bloom-filters-"),
- RECORD_INDEX(HoodieTableMetadataUtil.PARTITION_NAME_RECORD_INDEX,
"record-index-"),
-
FUNCTIONAL_INDEX(HoodieTableMetadataUtil.PARTITION_NAME_FUNCTIONAL_INDEX_PREFIX,
"func-index-");
+ FILES(HoodieTableMetadataUtil.PARTITION_NAME_FILES, "files-",
+ HoodieMetadataConfig::enabled,
+ (metaClient, partitionType) ->
metaClient.getTableConfig().isMetadataPartitionAvailable(partitionType)),
+ COLUMN_STATS(HoodieTableMetadataUtil.PARTITION_NAME_COLUMN_STATS,
"col-stats-",
+ HoodieMetadataConfig::isColumnStatsIndexEnabled,
+ (metaClient, partitionType) ->
metaClient.getTableConfig().isMetadataPartitionAvailable(partitionType)),
Review Comment:
This can be the default implementation of the method.
##########
hudi-common/src/main/java/org/apache/hudi/metadata/MetadataPartitionType.java:
##########
@@ -18,30 +18,52 @@
package org.apache.hudi.metadata;
+import org.apache.hudi.common.config.HoodieMetadataConfig;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+
+import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;
import java.util.Set;
+import java.util.function.BiPredicate;
+import java.util.function.Predicate;
import java.util.stream.Collectors;
/**
* Partition types for metadata table.
*/
public enum MetadataPartitionType {
- FILES(HoodieTableMetadataUtil.PARTITION_NAME_FILES, "files-"),
- COLUMN_STATS(HoodieTableMetadataUtil.PARTITION_NAME_COLUMN_STATS,
"col-stats-"),
- BLOOM_FILTERS(HoodieTableMetadataUtil.PARTITION_NAME_BLOOM_FILTERS,
"bloom-filters-"),
- RECORD_INDEX(HoodieTableMetadataUtil.PARTITION_NAME_RECORD_INDEX,
"record-index-"),
-
FUNCTIONAL_INDEX(HoodieTableMetadataUtil.PARTITION_NAME_FUNCTIONAL_INDEX_PREFIX,
"func-index-");
+ FILES(HoodieTableMetadataUtil.PARTITION_NAME_FILES, "files-",
+ HoodieMetadataConfig::enabled,
+ (metaClient, partitionType) ->
metaClient.getTableConfig().isMetadataPartitionAvailable(partitionType)),
Review Comment:
I assume `partitionType` is the same as `this`, so we can directly check
against `this` instead of passing in the argument
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
