JingsongLi commented on code in PR #11265:
URL: https://github.com/apache/paimon/pull/11265#discussion_r4624891027


##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/read/PaimonSupportsRuntimeFiltering.scala:
##########
@@ -38,6 +47,93 @@ trait PaimonSupportsRuntimeFiltering extends BaseScan with 
SupportsRuntimeV2Filt
   private var _inputSplits: Array[Split] = _
   private var _inputPartitions: Seq[PaimonInputPartition] = _
 
+  // Pin metadata discovery and every subsequent scan to the same snapshot. In 
particular,
+  // replanning after the build side completes must not pick up a concurrent 
append.
+  private lazy val runtimeReadSnapshot: Option[(FileStoreTable, Snapshot)] = {
+    val eligibleScan = this match {
+      case scan: PaimonScan => !scan.preserveDataGrouping
+      case _ => false
+    }
+    if (
+      !eligibleScan || pushedLimit.isDefined || pushedTopN.isDefined ||
+      pushedVectorSearch.isDefined || pushedHybridSearch.isDefined ||
+      pushedFullTextSearch.isDefined || !coreOptions.dataEvolutionEnabled() ||
+      !coreOptions.globalIndexEnabled() || coreOptions.queryAuthEnabled() ||
+      coreOptions.ignoreIndexColumnUpdate() ||
+      coreOptions.startupMode() != StartupMode.LATEST_FULL
+    ) {
+      None
+    } else {
+      table match {
+        case fileTable: FileStoreTable if fileTable.primaryKeys().isEmpty =>

Review Comment:
   **[P1] Preserve fallback-branch snapshots on ordinary reads**
   
   `FallbackReadFileStoreTable` also implements `FileStoreTable`. The new 
`readTable` override pins it to main's latest snapshot even for a plain 
`SELECT`. Pinning the wrapper also rewrites the fallback branch's snapshot 
selection using main's commit time.
   
   I reproduced this with a partitioned Data Evolution table (`bucket=-1`) and 
an empty fallback branch. I inserted `(1,'main')` on main, then 
`(2,'fallback')` on the fallback branch, and set `scan.fallback-branch` on 
main. A plain `SELECT` on the baseline returned both rows. The merged PR throws 
because the fallback's first snapshot is later than main's. With an earlier 
fallback snapshot present, the same query succeeds and silently omits a newer 
fallback-only partition. Setting `global-index.enabled=false` on this build 
restores both rows.
   
   Keep fallback-branch reads on their existing path and add a SQL regression 
covering a fallback commit newer than main without any runtime filter.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to