This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 76fb880330c [test](paimon) Cover the catalog snapshot pin on the
Paimon system-table source (#69569)
76fb880330c is described below
commit 76fb880330cbee03b19e070daf79e7c9f67e88af
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Sun Oct 25 08:16:48 2026 +0000
[test](paimon) Cover the catalog snapshot pin on the Paimon system-table
source (#69569)
### What problem does this PR solve?
Issue Number: None
Related PR: #68897, #69343
Problem Summary:
**In short.** This PR only adds tests. Some Paimon system tables are
planned on the BE. For them, the JNI scanner rebuilds the system table
from the source table that the FE sends in
`doris.serialized-system-source`. In a version-managed catalog, that
payload must be pinned to the snapshot the catalog publishes. #69343
added this pin. This PR adds the two cases its tests do not cover: a
`scan.fallback-branch` pair, and an explicit relation snapshot that must
still take precedence over the pin.
**Background.** Six Paimon system tables are planned on the BE:
`$files`, `$file_indexes`, `$partitions`, `$manifests`, `$statistics`
and `$table_indexes` (`PaimonScanParams#resolvesSnapshotOnBackend`). For
these tables the FE sends two payloads:
- `serialized_table` is built by `tableForBackend`.
- `doris.serialized-system-source` was added in #66247 so that a smaller
BE can cap the manifest parallelism. The FE always sends it when the
system table has a source. When it is present,
`PaimonJniScanner#applyBackendManifestParallelism` rebuilds the system
table from this payload instead of from `serialized_table`.
A version-managed catalog (REST) owns the committed snapshot pointer.
Once the catalog loader is removed, Paimon falls back to listing the
snapshot directory, and that directory can already contain a snapshot
the catalog has not published. #65867 therefore pins `scan.snapshot-id`
to the catalog's pointer (`PaimonScanPlanProvider#pinCatalogSnapshot`).
**The problem, and what it cost.** Before #69343, only
`serialized_table` carried that pin. The side channel shipped the source
unpinned, and the BE rebuilt from the side channel. So in a REST catalog
these six system tables were planned on the newest snapshot in storage
rather than the published one, on every branch. Inside the publication
window, `$partitions` could list partitions of a snapshot the catalog
had not published yet. The review of #68897 pointed this out for chain
tables.
#69343 (the Paimon 2.1.0 upgrade) now prepares the side-channel source
in the same order as `tableForBackend` (`prepareSystemSource`):
1. Pin the catalog snapshot, branch by branch.
2. Apply the relation's `@options` over the pin.
3. Drop the catalog loader.
Its tests cover `$file_indexes` on a single-branch table, on the FE and
through the JNI rebuild.
**How this PR fixes it.** Two FE unit tests in
`PaimonBackendBoundTableTest` read the side channel out of
`paimon.options_json`:
- `systemSourcePinsEachFallbackBranchToItsOwnCatalogSnapshot` builds
`$partitions` over a fallback pair whose two catalogs point at snapshots
2 and 1. Each branch of the transported source carries its own pointer
and no catalog loader. A chain table nests the same pairs.
- `relationSnapshotOutranksTheCatalogPinOnTheSystemSource` uses a
catalog that points at snapshot 2 and
`@options('scan.snapshot-id'='1')`. The transported source says 1.
Paimon 2.1.0 still has the two limitations documented on
`pinCatalogSnapshot`:
- `$files` emits its marker splits from `SnapshotReader#partitions`.
- `$snapshots` and `$buckets` read without `scan.snapshot-id`.
So that javadoc stays as written.
**Results.**
| `prepareSystemSource` | `PaimonBackendBoundTableTest` |
|---|---|
| as on master (pin, then `@options`) | 38 / 38 pass |
| without the pin | the new fallback case and the existing
`$file_indexes` case fail |
| pin applied after the relation options | the new relation-snapshot
case fails (`expected: <1> but was: <2>`) |
All 847 tests of fe-connector-paimon pass, with 1 skipped as before, and
checkstyle passes. No regression suite is added: it would need a
version-managed (REST) Paimon catalog whose pointer lags the snapshot
directory, and the docker fixtures do not provide one.
---
.../paimon/PaimonBackendBoundTableTest.java | 65 ++++++++++++++++++++++
1 file changed, 65 insertions(+)
diff --git
a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonBackendBoundTableTest.java
b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonBackendBoundTableTest.java
index 882b7ac5851..69d7958ac04 100644
---
a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonBackendBoundTableTest.java
+++
b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonBackendBoundTableTest.java
@@ -562,6 +562,63 @@ public class PaimonBackendBoundTableTest {
Assertions.assertEquals(2L,
onDisk.snapshotManager().latestSnapshotIdFromFileSystem());
}
+ @Test
+ public void
systemSourcePinsEachFallbackBranchToItsOwnCatalogSnapshot(@TempDir Path
warehouse)
+ throws Exception {
+ // JNI rebuilds a system table that plans on the BE from
doris.serialized-system-source rather
+ // than from serialized_table, so that payload must carry each
branch's own catalog pointer too.
+ FileStoreTable onDisk = commit(newRealTable(warehouse,
"fallback_system_source"), 2);
+ VersionManagedCatalog mainCatalog = new
VersionManagedCatalog(warehouse, true);
+ VersionManagedCatalog fallbackCatalog = new
VersionManagedCatalog(warehouse, true);
+ mainCatalog.pointAt(onDisk, 2L);
+ fallbackCatalog.pointAt(onDisk, 1L);
+ Map<String, String> mainOptions = new HashMap<>();
+ mainOptions.put("scan.fallback-branch", "fb");
+ Map<String, String> fallbackOptions = new HashMap<>();
+ fallbackOptions.put("branch", "fb");
+ FileStoreTable pair = new FallbackReadFileStoreTable(
+
withCatalogEnvironment(onDisk.copyWithoutTimeTravel(mainOptions),
+ catalogEnvironment(mainCatalog)),
+
withCatalogEnvironment(onDisk.copyWithoutTimeTravel(fallbackOptions),
+ catalogEnvironment(fallbackCatalog)),
+ true);
+ PaimonTableHandle handle = sysHandle("partitions", pair);
+ handle.setPaimonTable(SystemTableLoader.load("partitions", pair));
+ handle.setSystemTableSource(pair);
+
+ FileStoreTable transported = transportedSystemSource(handle);
+
+ Assertions.assertTrue(transported instanceof
FallbackReadFileStoreTable);
+ FallbackReadFileStoreTable transportedPair =
(FallbackReadFileStoreTable) transported;
+ Assertions.assertEquals("2",
transportedPair.wrapped().options().get("scan.snapshot-id"));
+ Assertions.assertEquals("1",
transportedPair.other().options().get("scan.snapshot-id"),
+ "the fallback branch must use its own catalog pointer, not its
newest snapshot file");
+
Assertions.assertNull(transportedPair.wrapped().catalogEnvironment().catalogLoader());
+
Assertions.assertNull(transportedPair.other().catalogEnvironment().catalogLoader());
+ }
+
+ @Test
+ public void
relationSnapshotOutranksTheCatalogPinOnTheSystemSource(@TempDir Path warehouse)
+ throws Exception {
+ // The catalog pin is taken first and the relation's @options are
applied over it, so an explicit
+ // snapshot still selects what the BE rebuilds the system table on.
+ VersionManagedCatalog catalog = new VersionManagedCatalog(warehouse,
true);
+ FileStoreTable onDisk = commit(newRealTable(warehouse,
"options_system_source"), 2);
+ catalog.pointAt(onDisk, 2L);
+ FileStoreTable versionManaged = withCatalogEnvironment(onDisk,
catalogEnvironment(catalog));
+ Map<String, String> relationOptions = PaimonScanParams.markAsOptions(
+ Collections.singletonMap("scan.snapshot-id", "1"));
+ PaimonTableHandle handle = sysHandle("partitions",
versionManaged).withScanOptions(relationOptions);
+ handle.setPaimonTable(PaimonScanParams.applyOptions(
+ SystemTableLoader.load("partitions", versionManaged),
relationOptions));
+ handle.setSystemTableSource(versionManaged);
+
+ FileStoreTable transported = transportedSystemSource(handle);
+
+ Assertions.assertEquals("1",
transported.options().get("scan.snapshot-id"));
+ Assertions.assertTrue(catalog.loadSnapshotCalls > 0, "the catalog pin
is still resolved on the FE");
+ }
+
@Test
public void
keepsThePinOffSystemTablesWhoseRowTypeFollowsTheBaseTable(@TempDir Path
warehouse)
throws Exception {
@@ -1023,6 +1080,14 @@ public class PaimonBackendBoundTableTest {
}
/** Decodes {@code paimon.serialized_table} the way {@code
PaimonJniScanner#initTable} does. */
+ /** The catalog-less source that JNI rebuilds a backend-planned system
table from. */
+ private static FileStoreTable transportedSystemSource(PaimonTableHandle
handle) throws Exception {
+ Map<String, String> props = provider().getScanNodeProperties(null,
handle,
+ Collections.emptyList(), Optional.empty());
+ Map<?, ?> options =
JsonSerdeUtil.fromJson(props.get("paimon.options_json"), Map.class);
+ return deserializeTable((String)
options.get("doris.serialized-system-source"));
+ }
+
private static FileStoreTable deserializeTable(String encoded) throws
Exception {
byte[] raw = encoded.getBytes(StandardCharsets.UTF_8);
byte[] bytes;
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
