hubgeter opened a new pull request, #70618: URL: https://github.com/apache/doris/pull/70618
### Background Partition-following asynchronous materialized views save base-partition versions and compare them before refresh or transparent rewrite. Iceberg's partitions metadata derives its last-updated snapshot from live manifest entries. ### What problem does this PR solve? Issue Number: None Related PR: None PR Type: Bugfix Problem Summary: Removing some data files from a partition can leave its live-entry snapshot marker unchanged, allowing a stale MV partition to remain eligible for rewrite or be skipped by AUTO refresh. Bind the saved partition version to the table snapshot already retained with its partition view. Previously saved partition versions are treated as stale until refreshed. ### Details - **Frequency:** After a data commit changes a partition while the latest snapshot marker among its surviving entries stays unchanged. - **Root cause:** PCT base tables are excluded from the whole-table comparison, while the previous partition snapshot equality checked only the live-entry marker, which can miss removed files. - **Why existing tests missed it:** The range-view tests cover appended partitions and their individual snapshot ids, but do not change the table snapshot while retaining the same partition marker or test comparison with legacy saved versions. - **Fix:** Add an optional table snapshot id to the existing persisted snapshot representation. Both single and batched partition snapshot APIs populate it from the same retained pin; equality and hashing include it. The existing one-argument constructor keeps whole-table and legacy snapshot behavior. No connector SPI, FE/BE protocol, configuration or metadata RPC is added. - **Other paths and branches:** Timestamp-based versions keep their existing path. Branch-4.2 has the same root cause in the legacy IcebergUtils.partitionSnapshotOf helper used by both APIs; it needs a separate semantic backport. - **Behavior change:** Any change to the pinned data snapshot can invalidate followed MV partitions, including partitions untouched by that commit. Legacy partition versions with no table snapshot scope cannot match newly scoped versions. ### Testing - Local supported FE UT entry point, scoped to fe-core and its upstream reactor through CUSTOM_MVN: 155 tests in PluginDrivenMvccExternalTableTest, MTMVSnapshotIdSnapshotTest, MTMVRefreshSnapshotTest, MTMVPartitionUtilTest and MTMVRefreshContextBatchTest; zero failures, errors or skips. Maven build cache was disabled. - With the original bridge restored while retaining the optional snapshot field, the new pin/consumer regression fails at the unchanged-version assertion. With both bridge paths fixed, it passes, including historical-pin reuse, a numerically smaller table snapshot id and single/batch equivalence. - Equality and persistence tests cover legacy-token collisions after rollback, polymorphic snapshot serialization and nested saved MV refresh state. - Checkstyle on all four changed Java files: zero violations. Staged whitespace and public-text checks passed. SQL/BE integration and branch-4.2 runtime validation remain pending. ### Notes for Reviewers The optional field also distinguishes legacy saved versions: after a deletion and rollback, simply substituting a table snapshot id can equal a legacy live-entry id even though the MV contains different data. Existing snapshot types remain readable, and upgraded FEs reject that legacy-to-scoped comparison. ### Release note Prevent snapshot-based partition-following materialized views from reusing stale partition results after partial data-file removal. Legacy saved partition versions require refresh before reuse, and commits affecting other partitions can cause additional refreshes. ### Check List (For Author) - Test: Unit tests and changed-file Checkstyle (see Testing). - Behavior changed: Yes (see Details). - Does this need documentation: No new setting or syntax. - Pre-submission status: Partially verified: local focused tests and changed-file style checks completed; live SQL/BE integration and release-branch validation remain pending. ### AI Disclosure - Tool: OpenAI Codex. - Generated: Implementation, unit tests and this description. - Human review: Not reported; agent self-review performed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
