morningman opened a new pull request, #69569:
URL: https://github.com/apache/doris/pull/69569
### What problem does this PR solve?
Issue Number: None
Related PR: #68897, #69343
Problem Summary:
**In short.** This PR only adds tests. Some Paimon system tables are planned
on the BE. For them, the JNI scanner rebuilds the system table from the source
table that the FE sends in `doris.serialized-system-source`. In a
version-managed catalog, that payload must be pinned to the snapshot the
catalog publishes. #69343 added this pin. This PR adds the two cases its tests
do not cover: a `scan.fallback-branch` pair, and an explicit relation snapshot
that must still take precedence over the pin.
**Background.** Six Paimon system tables are planned on the BE: `$files`,
`$file_indexes`, `$partitions`, `$manifests`, `$statistics` and
`$table_indexes` (`PaimonScanParams#resolvesSnapshotOnBackend`). For these
tables the FE sends two payloads:
- `serialized_table` is built by `tableForBackend`.
- `doris.serialized-system-source` was added in #66247 so that a smaller BE
can cap the manifest parallelism. The FE always sends it when the system table
has a source. When it is present,
`PaimonJniScanner#applyBackendManifestParallelism` rebuilds the system table
from this payload instead of from `serialized_table`.
A version-managed catalog (REST) owns the committed snapshot pointer. Once
the catalog loader is removed, Paimon falls back to listing the snapshot
directory, and that directory can already contain a snapshot the catalog has
not published. #65867 therefore pins `scan.snapshot-id` to the catalog's
pointer (`PaimonScanPlanProvider#pinCatalogSnapshot`).
**The problem, and what it cost.** Before #69343, only `serialized_table`
carried that pin. The side channel shipped the source unpinned, and the BE
rebuilt from the side channel. So in a REST catalog these six system tables
were planned on the newest snapshot in storage rather than the published one,
on every branch. Inside the publication window, `$partitions` could list
partitions of a snapshot the catalog had not published yet. The review of
#68897 pointed this out for chain tables.
#69343 (the Paimon 2.1.0 upgrade) now prepares the side-channel source in
the same order as `tableForBackend` (`prepareSystemSource`):
1. Pin the catalog snapshot, branch by branch.
2. Apply the relation's `@options` over the pin.
3. Drop the catalog loader.
Its tests cover `$file_indexes` on a single-branch table, on the FE and
through the JNI rebuild.
**How this PR fixes it.** Two FE unit tests in `PaimonBackendBoundTableTest`
read the side channel out of `paimon.options_json`:
- `systemSourcePinsEachFallbackBranchToItsOwnCatalogSnapshot` builds
`$partitions` over a fallback pair whose two catalogs point at snapshots 2 and
1. Each branch of the transported source carries its own pointer and no catalog
loader. A chain table nests the same pairs.
- `relationSnapshotOutranksTheCatalogPinOnTheSystemSource` uses a catalog
that points at snapshot 2 and `@options('scan.snapshot-id'='1')`. The
transported source says 1.
Paimon 2.1.0 still has the two limitations documented on
`pinCatalogSnapshot`:
- `$files` emits its marker splits from `SnapshotReader#partitions`.
- `$snapshots` and `$buckets` read without `scan.snapshot-id`.
So that javadoc stays as written.
**Results.**
| `prepareSystemSource` | `PaimonBackendBoundTableTest` |
|---|---|
| as on master (pin, then `@options`) | 38 / 38 pass |
| without the pin | the new fallback case and the existing `$file_indexes`
case fail |
| pin applied after the relation options | the new relation-snapshot case
fails (`expected: <1> but was: <2>`) |
All 847 tests of fe-connector-paimon pass, with 1 skipped as before, and
checkstyle passes. No regression suite is added: it would need a
version-managed (REST) Paimon catalog whose pointer lags the snapshot
directory, and the docker fixtures do not provide one.
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [x] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
