TeRS-K commented on a change in pull request #2740:
URL: https://github.com/apache/hudi/pull/2740#discussion_r610789102
##########
File path: hudi-cli/src/main/scala/org/apache/hudi/cli/SparkHelpers.scala
##########
@@ -40,7 +40,7 @@ import scala.collection.mutable._
object SparkHelpers {
@throws[Exception]
def skipKeysAndWriteNewFile(instantTime: String, fs: FileSystem, sourceFile:
Path, destinationFile: Path, keysToSkip: Set[String]) {
- val sourceRecords = ParquetUtils.readAvroRecords(fs.getConf, sourceFile)
+ val sourceRecords = new ParquetUtils().readAvroRecords(fs.getConf,
sourceFile)
Review comment:
I removed all instances of `new ParquetUtils()` except for the one in
`HoodieSparkBootstrapSchemaProvider::getBootstrapSourceSchema` as
`readSchema()` returns a schema per file type. For parquet, `readSchema`
returns a `org.apache.parquet.schema.MessageType`, for ORC (in future work),
`readSchema` would return a `org.apache.orc.TypeDescription`, so this method
cannot be extended from the base class. Does that make sense?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
