xushiyan commented on code in PR #6347:
URL: https://github.com/apache/hudi/pull/6347#discussion_r955245813
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java:
##########
@@ -192,18 +194,31 @@ public void addPartitionsToTable(String tableName,
List<String> partitionsToAdd)
LOG.info("Adding partitions " + partitionsToAdd.size() + " to table " +
tableName);
try {
StorageDescriptor sd = client.getTable(databaseName, tableName).getSd();
- List<Partition> partitionList = partitionsToAdd.stream().map(partition
-> {
+ if (syncConfig.getIntOrDefault(HIVE_BATCH_SYNC_PARTITION_NUM) <= 0) {
+ throw new HoodieHiveSyncException("batch-sync-num for sync hive table
must be greater than 0, pls check your parameter");
+ }
Review Comment:
this validation can be done at higher level like in the sync config, instead
of being buried deep down in DDLExecutor level.
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java:
##########
@@ -192,18 +194,31 @@ public void addPartitionsToTable(String tableName,
List<String> partitionsToAdd)
LOG.info("Adding partitions " + partitionsToAdd.size() + " to table " +
tableName);
try {
StorageDescriptor sd = client.getTable(databaseName, tableName).getSd();
- List<Partition> partitionList = partitionsToAdd.stream().map(partition
-> {
+ if (syncConfig.getIntOrDefault(HIVE_BATCH_SYNC_PARTITION_NUM) <= 0) {
+ throw new HoodieHiveSyncException("batch-sync-num for sync hive table
must be greater than 0, pls check your parameter");
+ }
+ List<Partition> partitionList = new ArrayList<>();
+ int batchSyncPartitionNum =
syncConfig.getIntOrDefault(HIVE_BATCH_SYNC_PARTITION_NUM);
+ for (int idx = 0; idx < partitionsToAdd.size(); idx++) {
Review Comment:
see if you can make use of
`org.apache.hudi.common.util.CollectionUtils#batches` or
`org.apache.hudi.common.util.CollectionUtils#batchesAsStream` for this scenario
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java:
##########
@@ -192,18 +194,31 @@ public void addPartitionsToTable(String tableName,
List<String> partitionsToAdd)
LOG.info("Adding partitions " + partitionsToAdd.size() + " to table " +
tableName);
try {
StorageDescriptor sd = client.getTable(databaseName, tableName).getSd();
- List<Partition> partitionList = partitionsToAdd.stream().map(partition
-> {
+ if (syncConfig.getIntOrDefault(HIVE_BATCH_SYNC_PARTITION_NUM) <= 0) {
+ throw new HoodieHiveSyncException("batch-sync-num for sync hive table
must be greater than 0, pls check your parameter");
+ }
Review Comment:
Also make use of `ValidationUtils` wherever applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
