suxiaogang223 opened a new pull request, #70594: URL: https://github.com/apache/doris/pull/70594
### Background Paimon branches maintain independent snapshot and schema histories. A fast-forward replaces part of the main branch history with the selected branch history and can reuse snapshot and schema IDs. [Paimon branch management](https://paimon.apache.org/docs/2.0/maintenance/manage-branches/) ### What problem does this PR solve? Issue Number: None Related PR: None PR Type: Bugfix Problem Summary: After an external fast-forward, a historical query with a warm Doris cache could bind columns from the previous branch history while scanning the replacement snapshot. This change carries the selected snapshot and schema identities from column binding through scan planning. Once fast-forward completes, historical queries resolve the replacement schema without refreshing the Doris table. ### Details - **Change map:** 233 changed production lines capture, propagate and validate metadata identities. 121 test lines cover replacement at the same IDs in the Paimon connector and in a shared Spark/Doris warehouse. The 17-line regression output was generated by the test runner. - **Frequency:** Reproducible when a historical query warms the Doris schema cache and an external fast-forward replaces its snapshot and schema files under the same IDs. - **Root cause:** The historical schema memo used table, branch and schema ID as its identity even when fast-forward replaced the file at that ID. - **Why existing tests missed it:** The existing branch regression refreshes the Doris table before checking the fast-forward result. - **Fix:** Key historical memo entries by the captured metadata identities, refresh stale SDK state, and validate the selected metadata around scan planning. Both column binding and the scan consume the selected schema, including the native reader's schema dictionary. - **Other paths and branches:** On master, this covers resolved snapshot selectors, including version IDs, timestamps and snapshot-valued table options. - **Behavior change:** A historical query after fast-forward reads the replacement metadata. If that metadata changes during planning, Doris reports that the statement should be retried. ### Testing - Built the patched FE on Fedora with `./build.sh --fe -j6`. The integration cluster used the existing BE at `679c38eef69`. - Ran 78 Paimon FE unit tests across `PaimonHistoricalSchemaOptionsTest`, `PaimonConnectorMetadataMvccTest`, `PaimonConnectorMetadataOptionsTagPinTest` and `PaimonSchemaAtMemoTest`. All passed. - Ran `test_paimon_fast_forward_historical_schema`, `test_paimon_renamed_field_time_travel` and `test_paimon_schema_only_snapshot_precision` on Fedora. All passed. The new regression reproduced `RequiredField main_metric not found in schema` on the baseline. With the fix, forced JNI and automatic scanning both returned `id INT`, `dev_metric BIGINT` and rows `(1, 10)`, `(3, 6000000000)`, matching Spark 4.0.3 with Paimon 2.1.0. `EXPLAIN` confirmed 2/2 native splits in automatic mode. ### Release note Paimon historical queries pick up the replacement schema after an external branch fast-forward reuses a snapshot ID. The Doris table does not need to be refreshed. ### Check List (For Author) - Behavior changed: Yes (see Details). - Does this need documentation: No. - Pre-submission status: Ready. ### AI Disclosure - Tool: OpenAI Codex. - Generated: Implementation, tests and PR description. - Human review: Pending. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
