yihua commented on code in PR #17724:
URL: https://github.com/apache/hudi/pull/17724#discussion_r2802738315
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieMetadataConfig.java:
##########
@@ -604,6 +604,27 @@ public final class HoodieMetadataConfig extends
HoodieConfig {
+ "honor the set value for number of tasks. If not, number of write
status's from data "
+ "table writes will be used for metadata table record preparation");
+ public static final ConfigProperty<Integer> CLEANER_PARALLELISM =
ConfigProperty
+ .key(METADATA_PREFIX + ".cleaner.parallelism")
+ .defaultValue(512)
+ .markAdvanced()
+ .sinceVersion("0.14.2")
+ .withDocumentation("Cleaner parallelism to use for metadata table.");
+
+ public static final ConfigProperty<Integer> ROLLBACK_PARALLELISM =
ConfigProperty
+ .key(METADATA_PREFIX + ".rollback.parallelism")
+ .defaultValue(512)
+ .markAdvanced()
+ .sinceVersion("0.14.2")
+ .withDocumentation("Rollback parallelism to use for metadata table.");
+
+ public static final ConfigProperty<Integer> FINALIZE_WRITES_PARALLELISM =
ConfigProperty
Review Comment:
```suggestion
public static final ConfigProperty<Integer> FINALIZE_WRITE_PARALLELISM =
ConfigProperty
```
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataWriteUtils.java:
##########
@@ -233,8 +226,8 @@ public static HoodieWriteConfig createMetadataWriteConfig(
.withBloomFilterFpp(writeConfig.getMetadataConfig().getBloomFilterFpp())
.withBloomFilterDynamicMaxEntries(writeConfig.getMetadataConfig().getDynamicBloomFilterMaxNumEntries())
.build())
- .withRollbackParallelism(MDT_DEFAULT_PARALLELISM)
- .withFinalizeWriteParallelism(MDT_DEFAULT_PARALLELISM)
+
.withRollbackParallelism(writeConfig.getMetadataConfig().getRollbackParallelism())
+
.withFinalizeWriteParallelism(writeConfig.getMetadataConfig().getFinalizeWritesParallelism())
Review Comment:
```suggestion
.withFinalizeWriteParallelism(writeConfig.getMetadataConfig().getFinalizeWriteParallelism())
```
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieMetadataConfig.java:
##########
@@ -909,6 +930,18 @@ public boolean isDropMetadataIndex(String indexName) {
return subIndexNameToDrop.contains(indexName);
}
+ public int getCleanerParallelism() {
+ return getInt(CLEANER_PARALLELISM);
+ }
+
+ public int getRollbackParallelism() {
+ return getInt(ROLLBACK_PARALLELISM);
+ }
+
+ public int getFinalizeWritesParallelism() {
Review Comment:
```suggestion
public int getFinalizeWriteParallelism() {
```
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieMetadataConfig.java:
##########
@@ -604,6 +604,27 @@ public final class HoodieMetadataConfig extends
HoodieConfig {
+ "honor the set value for number of tasks. If not, number of write
status's from data "
+ "table writes will be used for metadata table record preparation");
+ public static final ConfigProperty<Integer> CLEANER_PARALLELISM =
ConfigProperty
+ .key(METADATA_PREFIX + ".cleaner.parallelism")
+ .defaultValue(512)
+ .markAdvanced()
+ .sinceVersion("0.14.2")
Review Comment:
Same for other configs
```suggestion
.sinceVersion("1.2.0")
```
##########
hudi-common/src/test/java/org/apache/hudi/common/config/TestHoodieMetadataConfig.java:
##########
@@ -31,6 +31,7 @@
* Tests {@link HoodieMetadataConfig}.
*/
class TestHoodieMetadataConfig {
+
Review Comment:
nit: remove empty line
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieMetadataConfig.java:
##########
@@ -604,6 +604,27 @@ public final class HoodieMetadataConfig extends
HoodieConfig {
+ "honor the set value for number of tasks. If not, number of write
status's from data "
+ "table writes will be used for metadata table record preparation");
+ public static final ConfigProperty<Integer> CLEANER_PARALLELISM =
ConfigProperty
+ .key(METADATA_PREFIX + ".cleaner.parallelism")
+ .defaultValue(512)
+ .markAdvanced()
+ .sinceVersion("0.14.2")
+ .withDocumentation("Cleaner parallelism to use for metadata table.");
+
+ public static final ConfigProperty<Integer> ROLLBACK_PARALLELISM =
ConfigProperty
+ .key(METADATA_PREFIX + ".rollback.parallelism")
+ .defaultValue(512)
+ .markAdvanced()
+ .sinceVersion("0.14.2")
+ .withDocumentation("Rollback parallelism to use for metadata table.");
+
+ public static final ConfigProperty<Integer> FINALIZE_WRITES_PARALLELISM =
ConfigProperty
+ .key(METADATA_PREFIX + ".finalize.writes.parallelism")
+ .defaultValue(512)
+ .markAdvanced()
+ .sinceVersion("0.14.2")
+ .withDocumentation("Finalize writes parallelism to use for metadata
table.");
Review Comment:
```suggestion
.withDocumentation("Finalize write parallelism to use for metadata
table.");
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
