nsivabalan commented on code in PR #12125:
URL: https://github.com/apache/hudi/pull/12125#discussion_r1805963250
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieFileIndex.scala:
##########
@@ -274,8 +274,13 @@ case class HoodieFileIndex(spark: SparkSession,
.collect(Collectors.toSet[String])
val baseFileStatusOpt =
getBaseFileInfo(Option.apply(fs.getBaseFile.orElse(null)))
baseFileStatusOpt.exists(f =>
fileSliceFiles.add(f.getPath.getName))
- // NOTE: This predicate is true when {@code Option} is empty
- candidateFilesNamesOpt.forall(files => files.exists(elem =>
fileSliceFiles.contains(elem)))
Review Comment:
there is no correctness issue w/ this code. just some perf optimization.
here, we are taking each entry from candidateFilesNames list and then
checking if its part of fileSliceFiles. and on any match, we bail out.
I am just flipping the comparison in this patch.
For every file in fileSliceFiles, if we find any in candidateFilesNames, we
include the fileSlice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
