xushiyan commented on code in PR #5664:
URL: https://github.com/apache/hudi/pull/5664#discussion_r884231341
##########
hudi-spark-datasource/hudi-spark-common/src/main/java/org/apache/hudi/internal/BulkInsertDataInternalWriterHelper.java:
##########
@@ -128,7 +133,11 @@ public void write(InternalRow record) throws IOException {
if (!keyGeneratorOpt.isPresent()) { // NoPartitionerKeyGen
partitionPath = "";
} else if (simpleKeyGen) { // SimpleKeyGen
- partitionPath = (record.get(simplePartitionFieldIndex,
simplePartitionFieldDataType)).toString();
+ Object parititionPathValue = record.get(simplePartitionFieldIndex,
simplePartitionFieldDataType);
+ partitionPath = parititionPathValue != null ?
parititionPathValue.toString() :
PartitionPathEncodeUtils.DEFAULT_PARTITION_PATH;
+ if (isHiveStylePartitioning) {
+ partitionPath =
(keyGeneratorOpt.get()).getPartitionPathFields().get(0) + "=" + partitionPath;
Review Comment:
`(keyGeneratorOpt.get())` the bracket is redundant
##########
hudi-spark-datasource/hudi-spark-common/src/main/java/org/apache/hudi/internal/BulkInsertDataInternalWriterHelper.java:
##########
@@ -87,6 +89,9 @@ public BulkInsertDataInternalWriterHelper(HoodieTable
hoodieTable, HoodieWriteCo
this.populateMetaFields = populateMetaFields;
this.arePartitionRecordsSorted = arePartitionRecordsSorted;
this.fileIdPrefix = UUID.randomUUID().toString();
+ this.isHiveStylePartitioning =
writeConfig.getProps().containsKey(DataSourceWriteOptions.HIVE_STYLE_PARTITIONING().key())
+ ? Boolean.parseBoolean((String)
writeConfig.getProps().get(DataSourceWriteOptions.HIVE_STYLE_PARTITIONING().key()))
+ :
Boolean.parseBoolean(DataSourceWriteOptions.HIVE_STYLE_PARTITIONING().defaultValue());
Review Comment:
this is very verbose.. why not add an API in write config say
`isHiveStylePartitioning()` ?
##########
hudi-spark-datasource/hudi-spark-common/src/main/java/org/apache/hudi/internal/BulkInsertDataInternalWriterHelper.java:
##########
@@ -128,7 +133,11 @@ public void write(InternalRow record) throws IOException {
if (!keyGeneratorOpt.isPresent()) { // NoPartitionerKeyGen
partitionPath = "";
} else if (simpleKeyGen) { // SimpleKeyGen
- partitionPath = (record.get(simplePartitionFieldIndex,
simplePartitionFieldDataType)).toString();
+ Object parititionPathValue = record.get(simplePartitionFieldIndex,
simplePartitionFieldDataType);
+ partitionPath = parititionPathValue != null ?
parititionPathValue.toString() :
PartitionPathEncodeUtils.DEFAULT_PARTITION_PATH;
+ if (isHiveStylePartitioning) {
+ partitionPath =
(keyGeneratorOpt.get()).getPartitionPathFields().get(0) + "=" + partitionPath;
Review Comment:
why only get the first partition path field and add `=` sign? what about
multi-level partition like `year=2020/month=01/day=01` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
