yihua commented on code in PR #11163:
URL: https://github.com/apache/hudi/pull/11163#discussion_r1594770498
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/HoodieSparkFileWriterFactory.java:
##########
@@ -58,16 +58,16 @@ protected HoodieFileWriter newParquetFileWriter(
config.getIntOrDefault(HoodieStorageConfig.PARQUET_BLOCK_SIZE),
config.getIntOrDefault(HoodieStorageConfig.PARQUET_PAGE_SIZE),
config.getLongOrDefault(HoodieStorageConfig.PARQUET_MAX_FILE_SIZE),
- conf.unwrapAs(Configuration.class),
+ writeSupport.getHadoopConf(),
config.getDoubleOrDefault(HoodieStorageConfig.PARQUET_COMPRESSION_RATIO_FRACTION),
config.getBooleanOrDefault(HoodieStorageConfig.PARQUET_DICTIONARY_ENABLED));
- parquetConfig.getHadoopConf().addResource(writeSupport.getHadoopConf());
+ writeSupport.getHadoopConf().addResource(writeSupport.getHadoopConf());
Review Comment:
It looks like it's adding itself.
##########
hudi-common/src/main/java/org/apache/hudi/common/model/BootstrapIndexType.java:
##########
@@ -32,7 +31,7 @@
@EnumDescription("Bootstrap index type to use for mapping between skeleton and
actual data files.")
public enum BootstrapIndexType {
@EnumFieldDescription("Maintains mapping in HFile format.")
- HFILE(HFileBootstrapIndex.class.getName()),
+ HFILE("org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex"),
Review Comment:
I think we can keep `HFileBootstrapIndex` class in `hudi-common`, with the
changes to make `HFileBootstrapIndex` independent of Hadoop, separating the
logic out that reads HFile using HBase HFile reader, which you're working on in
#11171.
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/HoodieSparkFileWriterFactory.java:
##########
@@ -82,7 +82,7 @@ protected HoodieFileWriter newParquetFileWriter(
config.getLong(HoodieStorageConfig.PARQUET_MAX_FILE_SIZE),
writeSupport.getHadoopConf(),
config.getDouble(HoodieStorageConfig.PARQUET_COMPRESSION_RATIO_FRACTION),
config.getBooleanOrDefault(HoodieStorageConfig.PARQUET_DICTIONARY_ENABLED));
- parquetConfig.getHadoopConf().addResource(writeSupport.getHadoopConf());
+ writeSupport.getHadoopConf().addResource(writeSupport.getHadoopConf());
Review Comment:
Similar here.
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/HoodieSparkFileWriterFactory.java:
##########
@@ -58,16 +58,16 @@ protected HoodieFileWriter newParquetFileWriter(
config.getIntOrDefault(HoodieStorageConfig.PARQUET_BLOCK_SIZE),
config.getIntOrDefault(HoodieStorageConfig.PARQUET_PAGE_SIZE),
config.getLongOrDefault(HoodieStorageConfig.PARQUET_MAX_FILE_SIZE),
- conf.unwrapAs(Configuration.class),
+ writeSupport.getHadoopConf(),
Review Comment:
Can this be different from directly using `conf`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
