codope commented on code in PR #11302:
URL: https://github.com/apache/hudi/pull/11302#discussion_r1614632735
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/HoodieSparkParquetReader.java:
##########
@@ -69,7 +69,7 @@ public HoodieSparkParquetReader(HoodieStorage storage,
StoragePath path) {
this.storage = storage.newInstance(path, storage.getConf().newInstance());
// Avoid adding record in list element when convert parquet schema to avro
schema
this.storage.getConf().set(ADD_LIST_ELEMENT_RECORDS, "false");
- this.parquetUtils = FileFormatUtils.getInstance(HoodieFileFormat.PARQUET);
+ this.parquetUtils = new
HoodieSparkIOFactory(storage).getFileFormatUtils(HoodieFileFormat.PARQUET);
Review Comment:
can't we use
`HoodieIOFactory.getIOFactory(storage).getFileFormatUtils(HoodieFileFormat.PARQUET)`?
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/SparkHelpers.scala:
##########
@@ -140,7 +141,8 @@ class SparkHelper(sqlContext: SQLContext, fs: FileSystem) {
* @return <pre>true</pre> if all keys are added to the bloom filter;
<pre>false</pre> otherwise.
*/
def fileKeysAgainstBF(storage: HoodieStorage, sqlContext: SQLContext, file:
String): Boolean = {
- val bf =
FileFormatUtils.getInstance(HoodieFileFormat.PARQUET).readBloomFilterFromMetadata(storage,
new StoragePath(file))
+ val bf = new
HoodieSparkIOFactory(storage).getFileFormatUtils(HoodieFileFormat.PARQUET)
Review Comment:
same here
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/bootstrap/BootstrapOperator.java:
##########
@@ -202,7 +203,8 @@ protected void loadRecords(String partitionPath) throws
Exception {
Option<HoodieInstant> latestCommitTime =
commitsTimeline.filterCompletedAndCompactionInstants().lastInstant();
if (latestCommitTime.isPresent()) {
- FileFormatUtils fileUtils =
FileFormatUtils.getInstance(this.hoodieTable.getBaseFileFormat());
+ FileFormatUtils fileUtils =
HoodieIOFactory.getIOFactory(hoodieTable.getStorage())
+ .getFileFormatUtils(this.hoodieTable.getBaseFileFormat());
Review Comment:
```suggestion
.getFileFormatUtils(hoodieTable.getBaseFileFormat());
```
##########
hudi-hadoop-common/src/main/java/org/apache/hudi/io/hadoop/HoodieAvroParquetReader.java:
##########
@@ -68,7 +68,7 @@ public HoodieAvroParquetReader(HoodieStorage storage,
StoragePath path) {
// by the Reader (for proper config propagation to Parquet components)
this.storage = storage.newInstance(path,
tryOverrideDefaultConfigs(storage.getConf().newInstance()));
this.path = path;
- this.parquetUtils = FileFormatUtils.getInstance(HoodieFileFormat.PARQUET);
+ this.parquetUtils = new
HoodieHadoopIOFactory(storage).getFileFormatUtils(HoodieFileFormat.PARQUET);
Review Comment:
same here, can we use `HoodieIOFactory` instead of specific impl of
HoodieIOFactory?
##########
hudi-hadoop-common/src/main/java/org/apache/hudi/io/hadoop/HoodieAvroOrcReader.java:
##########
@@ -58,7 +58,7 @@ public class HoodieAvroOrcReader extends HoodieAvroFileReader
{
public HoodieAvroOrcReader(HoodieStorage storage, StoragePath path) {
this.storage = storage;
this.path = path;
- this.orcUtils = FileFormatUtils.getInstance(HoodieFileFormat.ORC);
+ this.orcUtils = new
HoodieHadoopIOFactory(storage).getFileFormatUtils(HoodieFileFormat.ORC);
Review Comment:
same here, can we use
`HoodieIOFactory.getIOFactory(storage).getFileFormatUtils(HoodieFileFormat.ORC)`?
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieMetadataTableValidator.java:
##########
@@ -1440,10 +1440,11 @@ public List<HoodieColumnRangeMetadata<Comparable>>
getSortedColumnStatsList(
.collect(Collectors.toList());
} else {
return baseFileNameList.stream().flatMap(filename ->
-
FileFormatUtils.getInstance(HoodieFileFormat.PARQUET).readColumnStatsFromMetadata(
- metaClient.getStorage(),
- new
StoragePath(FSUtils.constructAbsolutePath(metaClient.getBasePathV2(),
partitionPath), filename),
- allColumnNameList).stream())
+ new
HoodieSparkIOFactory(metaClient.getStorage()).getFileFormatUtils(HoodieFileFormat.PARQUET)
Review Comment:
here also, let;s see if we can use `HoodieIOFactory`. And would be good to
create the util before stream.
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/SparkHelpers.scala:
##########
@@ -48,7 +48,8 @@ object SparkHelpers {
sourceFile: StoragePath,
destinationFile: StoragePath,
keysToSkip: Set[String]) {
- val sourceRecords =
FileFormatUtils.getInstance(HoodieFileFormat.PARQUET).readAvroRecords(storage,
sourceFile).asScala
+ val sourceRecords = new
HoodieSparkIOFactory(storage).getFileFormatUtils(HoodieFileFormat.PARQUET)
Review Comment:
same comment, use `HoodieIOFactory` if posible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
