xushiyan commented on code in PR #6347:
URL: https://github.com/apache/hudi/pull/6347#discussion_r957968992
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HiveSyncConfig.java:
##########
@@ -69,6 +70,7 @@ public static String getBucketSpec(String bucketCols, int
bucketNum) {
public HiveSyncConfig(Properties props) {
super(props);
+ validateParameters();
Review Comment:
since validation is done in constructor. we don't need to check in
JDBCExecutor either right?
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java:
##########
@@ -191,19 +195,27 @@ public void addPartitionsToTable(String tableName,
List<String> partitionsToAdd)
}
LOG.info("Adding partitions " + partitionsToAdd.size() + " to table " +
tableName);
try {
+
ValidationUtils.checkArgument(syncConfig.getIntOrDefault(HIVE_BATCH_SYNC_PARTITION_NUM)
> 0,
+ "batch-sync-num for sync hive table must be greater than 0, pls
check your parameter");
Review Comment:
then this check can be removed?
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java:
##########
@@ -191,19 +195,27 @@ public void addPartitionsToTable(String tableName,
List<String> partitionsToAdd)
}
LOG.info("Adding partitions " + partitionsToAdd.size() + " to table " +
tableName);
try {
+
ValidationUtils.checkArgument(syncConfig.getIntOrDefault(HIVE_BATCH_SYNC_PARTITION_NUM)
> 0,
+ "batch-sync-num for sync hive table must be greater than 0, pls
check your parameter");
StorageDescriptor sd = client.getTable(databaseName, tableName).getSd();
- List<Partition> partitionList = partitionsToAdd.stream().map(partition
-> {
- StorageDescriptor partitionSd = new StorageDescriptor();
- partitionSd.setCols(sd.getCols());
- partitionSd.setInputFormat(sd.getInputFormat());
- partitionSd.setOutputFormat(sd.getOutputFormat());
- partitionSd.setSerdeInfo(sd.getSerdeInfo());
- String fullPartitionPath =
FSUtils.getPartitionPath(syncConfig.getString(META_SYNC_BASE_PATH),
partition).toString();
- List<String> partitionValues =
partitionValueExtractor.extractPartitionValuesInPath(partition);
- partitionSd.setLocation(fullPartitionPath);
- return new Partition(partitionValues, databaseName, tableName, 0, 0,
partitionSd, null);
- }).collect(Collectors.toList());
- client.add_partitions(partitionList, true, false);
+ List<Partition> partitionList = new ArrayList<>();
Review Comment:
let's not re-use the same variable. create new var for each batch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
