yihua commented on code in PR #12642:
URL: https://github.com/apache/hudi/pull/12642#discussion_r1926253358
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieBaseRelation.scala:
##########
@@ -350,6 +350,7 @@ abstract class HoodieBaseRelation(val sqlContext:
SQLContext,
* NOTE: DO NOT OVERRIDE THIS METHOD
*/
override final def buildScan(requiredColumns: Array[String], filters:
Array[Filter]): RDD[Row] = {
+ fileIndex.refresh()
Review Comment:
Does master branch using the new reader logic, e.g.,
`HoodieMergeOnReadSnapshotHadoopFsRelationFactory`, have this problem? In the
new factory, the relation factory and file index are created at the time of
instantiation. Wondering if that need to be refreshed too.
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieBaseRelation.scala:
##########
@@ -288,7 +288,7 @@ abstract class HoodieBaseRelation(val sqlContext:
SQLContext,
protected def timeline: HoodieTimeline =
// NOTE: We're including compaction here since it's not considering a
"commit" operation
- metaClient.getCommitsAndCompactionTimeline.filterCompletedInstants
+
metaClient.reloadActiveTimeline().getCommitsAndCompactionTimeline.filterCompletedInstants
Review Comment:
To clarify, if we're refreshing the file index, do we still need to reload
the active timeline?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
