hubgeter opened a new pull request, #70618:
URL: https://github.com/apache/doris/pull/70618

   ### Background
   
   Partition-following asynchronous materialized views save base-partition 
versions and compare them before refresh or transparent rewrite. Iceberg's 
partitions metadata derives its last-updated snapshot from live manifest 
entries.
   
   ### What problem does this PR solve?
   
   Issue Number: None
   Related PR: None
   PR Type: Bugfix
   
   Problem Summary:
   
   Removing some data files from a partition can leave its live-entry snapshot 
marker unchanged, allowing a stale MV partition to remain eligible for rewrite 
or be skipped by AUTO refresh. Bind the saved partition version to the table 
snapshot already retained with its partition view. Previously saved partition 
versions are treated as stale until refreshed.
   
   ### Details
   
   - **Frequency:** After a data commit changes a partition while the latest 
snapshot marker among its surviving entries stays unchanged.
   - **Root cause:** PCT base tables are excluded from the whole-table 
comparison, while the previous partition snapshot equality checked only the 
live-entry marker, which can miss removed files.
   - **Why existing tests missed it:** The range-view tests cover appended 
partitions and their individual snapshot ids, but do not change the table 
snapshot while retaining the same partition marker or test comparison with 
legacy saved versions.
   - **Fix:** Add an optional table snapshot id to the existing persisted 
snapshot representation. Both single and batched partition snapshot APIs 
populate it from the same retained pin; equality and hashing include it. The 
existing one-argument constructor keeps whole-table and legacy snapshot 
behavior. No connector SPI, FE/BE protocol, configuration or metadata RPC is 
added.
   - **Other paths and branches:** Timestamp-based versions keep their existing 
path. Branch-4.2 has the same root cause in the legacy 
IcebergUtils.partitionSnapshotOf helper used by both APIs; it needs a separate 
semantic backport.
   - **Behavior change:** Any change to the pinned data snapshot can invalidate 
followed MV partitions, including partitions untouched by that commit. Legacy 
partition versions with no table snapshot scope cannot match newly scoped 
versions.
   
   ### Testing
   
   - Local supported FE UT entry point, scoped to fe-core and its upstream 
reactor through CUSTOM_MVN: 155 tests in PluginDrivenMvccExternalTableTest, 
MTMVSnapshotIdSnapshotTest, MTMVRefreshSnapshotTest, MTMVPartitionUtilTest and 
MTMVRefreshContextBatchTest; zero failures, errors or skips. Maven build cache 
was disabled.
   - With the original bridge restored while retaining the optional snapshot 
field, the new pin/consumer regression fails at the unchanged-version 
assertion. With both bridge paths fixed, it passes, including historical-pin 
reuse, a numerically smaller table snapshot id and single/batch equivalence.
   - Equality and persistence tests cover legacy-token collisions after 
rollback, polymorphic snapshot serialization and nested saved MV refresh state.
   - Checkstyle on all four changed Java files: zero violations. Staged 
whitespace and public-text checks passed.
   
   SQL/BE integration and branch-4.2 runtime validation remain pending.
   
   ### Notes for Reviewers
   
   The optional field also distinguishes legacy saved versions: after a 
deletion and rollback, simply substituting a table snapshot id can equal a 
legacy live-entry id even though the MV contains different data. Existing 
snapshot types remain readable, and upgraded FEs reject that legacy-to-scoped 
comparison.
   
   ### Release note
   
   Prevent snapshot-based partition-following materialized views from reusing 
stale partition results after partial data-file removal. Legacy saved partition 
versions require refresh before reuse, and commits affecting other partitions 
can cause additional refreshes.
   
   ### Check List (For Author)
   
   - Test: Unit tests and changed-file Checkstyle (see Testing).
   - Behavior changed: Yes (see Details).
   - Does this need documentation: No new setting or syntax.
   - Pre-submission status: Partially verified: local focused tests and 
changed-file style checks completed; live SQL/BE integration and release-branch 
validation remain pending.
   
   ### AI Disclosure
   
   - Tool: OpenAI Codex.
   - Generated: Implementation, unit tests and this description.
   - Human review: Not reported; agent self-review performed.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to