suxiaogang223 opened a new pull request, #70594:
URL: https://github.com/apache/doris/pull/70594

   ### Background
   
   Paimon branches maintain independent snapshot and schema histories. A 
fast-forward replaces part of the main branch history with the selected branch 
history and can reuse snapshot and schema IDs. [Paimon branch 
management](https://paimon.apache.org/docs/2.0/maintenance/manage-branches/)
   
   ### What problem does this PR solve?
   
   Issue Number: None
   Related PR: None
   PR Type: Bugfix
   
   Problem Summary:
   
   After an external fast-forward, a historical query with a warm Doris cache 
could bind columns from the previous branch history while scanning the 
replacement snapshot. This change carries the selected snapshot and schema 
identities from column binding through scan planning. Once fast-forward 
completes, historical queries resolve the replacement schema without refreshing 
the Doris table.
   
   ### Details
   
   - **Change map:** 233 changed production lines capture, propagate and 
validate metadata identities. 121 test lines cover replacement at the same IDs 
in the Paimon connector and in a shared Spark/Doris warehouse. The 17-line 
regression output was generated by the test runner.
   - **Frequency:** Reproducible when a historical query warms the Doris schema 
cache and an external fast-forward replaces its snapshot and schema files under 
the same IDs.
   - **Root cause:** The historical schema memo used table, branch and schema 
ID as its identity even when fast-forward replaced the file at that ID.
   - **Why existing tests missed it:** The existing branch regression refreshes 
the Doris table before checking the fast-forward result.
   - **Fix:** Key historical memo entries by the captured metadata identities, 
refresh stale SDK state, and validate the selected metadata around scan 
planning. Both column binding and the scan consume the selected schema, 
including the native reader's schema dictionary.
   - **Other paths and branches:** On master, this covers resolved snapshot 
selectors, including version IDs, timestamps and snapshot-valued table options.
   - **Behavior change:** A historical query after fast-forward reads the 
replacement metadata. If that metadata changes during planning, Doris reports 
that the statement should be retried.
   
   ### Testing
   
   - Built the patched FE on Fedora with `./build.sh --fe -j6`. The integration 
cluster used the existing BE at `679c38eef69`.
   - Ran 78 Paimon FE unit tests across `PaimonHistoricalSchemaOptionsTest`, 
`PaimonConnectorMetadataMvccTest`, `PaimonConnectorMetadataOptionsTagPinTest` 
and `PaimonSchemaAtMemoTest`. All passed.
   - Ran `test_paimon_fast_forward_historical_schema`, 
`test_paimon_renamed_field_time_travel` and 
`test_paimon_schema_only_snapshot_precision` on Fedora. All passed. The new 
regression reproduced `RequiredField main_metric not found in schema` on the 
baseline. With the fix, forced JNI and automatic scanning both returned `id 
INT`, `dev_metric BIGINT` and rows `(1, 10)`, `(3, 6000000000)`, matching Spark 
4.0.3 with Paimon 2.1.0. `EXPLAIN` confirmed 2/2 native splits in automatic 
mode.
   
   ### Release note
   
   Paimon historical queries pick up the replacement schema after an external 
branch fast-forward reuses a snapshot ID. The Doris table does not need to be 
refreshed.
   
   ### Check List (For Author)
   
   - Behavior changed: Yes (see Details).
   - Does this need documentation: No.
   - Pre-submission status: Ready.
   
   ### AI Disclosure
   
   - Tool: OpenAI Codex.
   - Generated: Implementation, tests and PR description.
   - Human review: Pending.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to