alexeykudinkin commented on code in PR #5723:
URL: https://github.com/apache/hudi/pull/5723#discussion_r901976253
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/MergeOnReadSnapshotRelation.scala:
##########
@@ -104,14 +104,22 @@ class MergeOnReadSnapshotRelation(sqlContext: SQLContext,
val fileSlices = fileIndex.listFileSlices(convertedPartitionFilters)
buildSplits(fileSlices.values.flatten.toSeq)
} else {
- // TODO refactor to avoid iterating over listed files multiple times
- val partitions = listLatestBaseFiles(globPaths,
convertedPartitionFilters, dataFilters)
- val partitionPaths = partitions.keys.toSeq
+ val inMemoryFileIndex = HoodieInMemoryFileIndex.create(sparkSession,
globPaths)
Review Comment:
@boneanxs appreciate your effort of addressing this TODO.
Although, the better way to refactor this would be to instead change
`listLatestBaseFiles` to be `listLatestFileSlices` and return file-slices
instead of just the base-files. Right now we've essentially duplicated this
method implementation in here. I'd suggest we avoid the duplication by changing
the method as noted above and keeping this code mostly intact.
Keep in mind: one of the goals of the refactoring of COW/MOR relations was
to bring implementations closer together making them bifurcate only at a points
where it's necessary (for ex, in a way they actually read file-slices) and
everywhere else keep them mostly identical.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
