yihua commented on code in PR #10304:
URL: https://github.com/apache/hudi/pull/10304#discussion_r1425903940
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DefaultSource.scala:
##########
@@ -235,15 +236,11 @@ object DefaultSource {
Option(schema)
}
- val useNewParquetFileFormat =
- parameters.getOrElse(
- USE_NEW_HUDI_PARQUET_FILE_FORMAT.key,
- USE_NEW_HUDI_PARQUET_FILE_FORMAT.defaultValue).toBoolean &&
- !metaClient.isMetadataTable &&
- (globPaths == null || globPaths.isEmpty) &&
- parameters.getOrElse(REALTIME_MERGE.key(), REALTIME_MERGE.defaultValue())
- .equalsIgnoreCase(REALTIME_PAYLOAD_COMBINE_OPT_VAL)
-
+ val useNewParquetFileFormat =
parameters.getOrDefault(HoodieReaderConfig.FILE_GROUP_READER_ENABLED.key(),
Review Comment:
Discussed offline. We should keep one config and use
`hoodie.file.group.reader.enabled` to control whether new parquet file format
is used, along with new file group reader, in Spark.
`hoodie.file.group.reader.enabled` should be used for other engines too, to
avoid introducing config per engine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
