JingsongLi commented on code in PR #11265:
URL: https://github.com/apache/paimon/pull/11265#discussion_r4624891027
##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/read/PaimonSupportsRuntimeFiltering.scala:
##########
@@ -38,6 +47,93 @@ trait PaimonSupportsRuntimeFiltering extends BaseScan with
SupportsRuntimeV2Filt
private var _inputSplits: Array[Split] = _
private var _inputPartitions: Seq[PaimonInputPartition] = _
+ // Pin metadata discovery and every subsequent scan to the same snapshot. In
particular,
+ // replanning after the build side completes must not pick up a concurrent
append.
+ private lazy val runtimeReadSnapshot: Option[(FileStoreTable, Snapshot)] = {
+ val eligibleScan = this match {
+ case scan: PaimonScan => !scan.preserveDataGrouping
+ case _ => false
+ }
+ if (
+ !eligibleScan || pushedLimit.isDefined || pushedTopN.isDefined ||
+ pushedVectorSearch.isDefined || pushedHybridSearch.isDefined ||
+ pushedFullTextSearch.isDefined || !coreOptions.dataEvolutionEnabled() ||
+ !coreOptions.globalIndexEnabled() || coreOptions.queryAuthEnabled() ||
+ coreOptions.ignoreIndexColumnUpdate() ||
+ coreOptions.startupMode() != StartupMode.LATEST_FULL
+ ) {
+ None
+ } else {
+ table match {
+ case fileTable: FileStoreTable if fileTable.primaryKeys().isEmpty =>
Review Comment:
**[P1] Preserve fallback-branch snapshots on ordinary reads**
`FallbackReadFileStoreTable` also implements `FileStoreTable`. The new
`readTable` override pins it to main's latest snapshot even for a plain
`SELECT`. Pinning the wrapper also rewrites the fallback branch's snapshot
selection using main's commit time.
I reproduced this with a partitioned Data Evolution table (`bucket=-1`) and
an empty fallback branch. I inserted `(1,'main')` on main, then
`(2,'fallback')` on the fallback branch, and set `scan.fallback-branch` on
main. A plain `SELECT` on the baseline returned both rows. The merged PR throws
because the fallback's first snapshot is later than main's. With an earlier
fallback snapshot present, the same query succeeds and silently omits a newer
fallback-only partition. Setting `global-index.enabled=false` on this build
restores both rows.
Keep fallback-branch reads on their existing path and add a SQL regression
covering a fallback commit newer than main without any runtime filter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]