morningman commented on code in PR #65867:
URL: https://github.com/apache/doris/pull/65867#discussion_r3670864303
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonScanNode.java:
##########
@@ -203,7 +213,225 @@ protected void doInitialize() throws UserException {
private void serializeProcessedTable() throws UserException {
// System-table splits are materialized by the BE JNI reader, so it
must receive the same
// option-bearing table copy that FE uses to plan the split.
- serializedTable = PaimonUtil.encodeObjectToString(getProcessedTable());
+ serializedTable =
PaimonUtil.encodeObjectToString(getPaimonTableForBackend());
+ }
+
+ /**
+ * Build the Paimon table object that is serialized to the BE.
+ *
+ * <p>Every table loaded from a metastore-backed Paimon catalog (HMS /
DLF) carries a Paimon
+ * {@code HiveCatalogLoader} in its {@link
org.apache.paimon.table.CatalogEnvironment}. The BE
+ * only reads — via FE-resolved splits and the object store — and never
needs the catalog, yet
+ * deserializing that loader forces the whole Hive metastore stack onto
the BE classpath:
+ * {@code HiveConf}, the metastore API, and, when a system table resolves
its latest snapshot,
+ * even the metastore client (DLF's {@code ProxyMetaStoreClient} and its
REST stack). So we
+ * serialize a catalog-less table to the BE:
+ * <ul>
+ * <li>data table: drop the catalog loader. A {@link FileStoreTable} is
fully defined by
+ * fileIO / location / schema / catalogEnvironment, and its dynamic
options (time travel,
+ * incremental) are merged into the schema by {@code copy(...)}, so
rebuilding from
+ * fileIO / location / schema preserves everything except the
catalog loader.</li>
+ * <li>system table (e.g. {@code $snapshots}): rebuild it over a
catalog-less data table so
+ * {@code SnapshotManager#latestSnapshotId} lists the snapshot
directory on the filesystem
+ * instead of calling the metastore. The base table is the one the
FE-side wrapper was
+ * built over, and for the system tables that pick their snapshot on
the BE ({@link
+ * #resolvesSnapshotOnBackend}) what the catalog would have done
there is done here
+ * instead: see {@link #authorizeDeferredScan} and {@link
#pinCatalogSnapshot}. Every
+ * other system table reads what the FE already planned, so it is
handed over untouched.
+ * The relation-scoped scan params the FE applied to the original
wrapper are re-applied
+ * to the rebuilt one by {@link #reapplyScanParams(Table)}.</li>
+ * </ul>
+ */
+ private Table getPaimonTableForBackend() throws UserException {
+ Table paimonTable = getProcessedTable();
+ if (paimonTable instanceof FileStoreTable) {
+ // copy(...) merges the relation's dynamic options into the
schema, and the rebuild
+ // below goes through that schema, so this branch needs no
re-application.
+ return dropCatalogLoader((FileStoreTable) paimonTable);
+ }
+ if (!(source.getExternalTable() instanceof PaimonSysExternalTable)) {
+ return paimonTable;
+ }
+ PaimonSysExternalTable sysTable = (PaimonSysExternalTable)
source.getExternalTable();
+ // The very same base table the FE-side wrapper was built over, so
that the BE never sees a
+ // different schema generation than the one this query was planned
with.
+ FileStoreTable dataTable = sysTable.getSysBaseTable();
+ if (dataTable == null) {
+ return paimonTable;
+ }
+ String sysTableType = sysTable.getSysTableType();
+ if (PAIMON_FILES_SYSTEM_TABLE_TYPE.equalsIgnoreCase(sysTableType)) {
+ authorizeDeferredScan(dataTable);
+ }
+ FileStoreTable baseForBackend = dropCatalogLoader(dataTable);
+ if (resolvesSnapshotOnBackend(sysTableType)) {
+ baseForBackend = pinCatalogSnapshot(baseForBackend, dataTable);
+ }
+ Table catalogLessSysTable = SystemTableLoader.load(sysTableType,
baseForBackend);
+ return catalogLessSysTable == null ? paimonTable :
reapplyScanParams(catalogLessSysTable);
+ }
+
+ /**
+ * Re-apply the relation-scoped scan params to a rebuilt system-table
wrapper.
+ *
+ * <p>{@link #getProcessedTable()} applies {@code @incr} / {@code
@options} to the wrapper the
+ * meta cache holds, and every Paimon system table delegates {@code
copy(...)} to the data table
+ * it wraps. The wrapper rebuilt above is a different object, so the same
copy has to be redone
+ * on it, otherwise the BE would materialize its splits against the
unpinned latest state.
+ *
+ * <p>This runs last on purpose: an explicit relation option outranks
anything this class pins
+ * on the rebuilt table, and {@code copy(...)} lets the option win.
+ */
+ private Table reapplyScanParams(Table rebuiltSysTable) throws
UserException {
+ TableScanParams theScanParams = getScanParams();
+ if (theScanParams == null) {
+ return rebuiltSysTable;
+ }
+ if (theScanParams.incrementalRead()) {
+ return rebuiltSysTable.copy(getIncrReadParams());
+ }
+ if (theScanParams.isOptions()) {
+ return PaimonScanParams.applyOptions(rebuiltSysTable,
+
theScanParams.getResolvedMapParams().orElse(Collections.emptyMap()));
+ }
+ return rebuiltSysTable;
+ }
+
+ /**
+ * The system tables whose rows are not fixed by what the FE planned: the
FE only sends marker
+ * splits and the snapshot is picked inside the BE reader. Two shapes,
both honoring
+ * {@code scan.snapshot-id}:
+ * <ul>
+ * <li>{@code $files} / {@code $partitions} re-plan the base table on
the BE
+ * ({@code FilesTable.FilesRead#createReader} -> {@code
DataTableScan#plan()},
+ * {@code PartitionsTable.PartitionsRead#createReader} ->
+ * {@code newScan().listPartitionEntries()});</li>
+ * <li>{@code $manifests} / {@code $table_indexes} / {@code $statistics}
resolve it directly
+ * ({@code TimeTravelUtil#tryTravelOrLatest}, reached from {@code
ManifestsTable},
+ * {@code TableIndexesTable} and {@code
AbstractFileStoreTable#statistics}).</li>
+ * </ul>
+ * All five have a fixed row type, so {@link #pinCatalogSnapshot} cannot
rewind the BE schema.
+ */
+ private static boolean resolvesSnapshotOnBackend(String sysTableType) {
+ return PAIMON_FILES_SYSTEM_TABLE_TYPE.equalsIgnoreCase(sysTableType)
+ ||
PAIMON_PARTITIONS_SYSTEM_TABLE_TYPE.equalsIgnoreCase(sysTableType)
+ ||
PAIMON_MANIFESTS_SYSTEM_TABLE_TYPE.equalsIgnoreCase(sysTableType)
+ ||
PAIMON_STATISTICS_SYSTEM_TABLE_TYPE.equalsIgnoreCase(sysTableType)
+ ||
PAIMON_TABLE_INDEXES_SYSTEM_TABLE_TYPE.equalsIgnoreCase(sysTableType);
+ }
+
+ /**
+ * {@code $files} plans only partition-level splits on the FE and re-plans
the base table on the
+ * BE through {@code DataTableScan#plan()} ({@code
FilesTable.FilesRead#createReader}). That
+ * deferred plan normally authorizes itself through the catalog loader
+ * ({@code CatalogEnvironment#tableQueryAuth} -> {@code
Catalog#authTableQuery}); once the
+ * loader is dropped it silently allows everything. So authorize here,
while the loader is still
+ * around. Paimon discards the predicates the call returns (row level
access control is a TODO in
+ * {@code AbstractDataTableScan#authQuery}), so running it on the FE loses
nothing.
+ *
+ * <p>Only {@code $files} may do this: {@code auth(null)} means "every
column" to
+ * {@code Catalog#authTableQuery}, and the system tables that keep
planning on the FE
+ * ({@code $ro}, {@code $row_tracking}, {@code $audit_log}, {@code
$binlog}) already authorize
+ * themselves through {@code DataTableBatchScan} with the slot projection
the query really reads.
+ * Authorizing those again for every column would reject a user allowed to
read only some of the
+ * base columns. {@code $partitions} never reaches {@code plan()} on
either side, so it has no
+ * authorization to transfer.
+ */
+ private static void authorizeDeferredScan(FileStoreTable dataTable) {
+ CoreOptions options = dataTable.coreOptions();
+ if (options.queryAuthEnabled()) {
+ dataTable.catalogEnvironment().tableQueryAuth(options).auth(null);
+ }
+ }
+
+ /**
+ * For catalogs that manage versions themselves (Paimon REST / DLF REST)
the committed snapshot
+ * is the one the catalog points at, not the newest file in the snapshot
directory: Paimon
+ * publishes the snapshot file before the pointer moves, and a rollback
leaves newer files
+ * behind. Without the catalog loader {@code SnapshotManager} falls back
to listing that
+ * directory, so the BE could plan on a snapshot the catalog has not
published while the FE
+ * planned on the previous one. Pin the catalog-visible snapshot instead.
+ *
+ * <p>Only for {@link #resolvesSnapshotOnBackend} system tables. {@code
PaimonJniScanner#initTable}
+ * unconditionally calls {@code table.copy(table.options())}, and every
Paimon system-table
+ * wrapper delegates {@code copy} to {@code FileStoreTable#copy}, which
time-travels the schema
+ * to {@code scan.snapshot-id}. For a wrapper whose row type follows the
base table
+ * ({@code $ro}, {@code $row_tracking}, {@code $audit_log}, {@code
$binlog}) that would rewind
+ * the BE schema: a column added after the latest snapshot is planned by
the FE and then
+ * rejected by {@code PaimonJniScanner#getProjected} with "RequiredField
... not found in
+ * schema". The pinned ones all have a fixed row type, so the pin is safe
there.
+ *
+ * <p>Known gap: {@code scan.snapshot-id} only bounds plans that go through
+ * {@code DataTableScan}. {@code $snapshots} ({@code
SnapshotManager#snapshotsWithinRange}) and
+ * {@code $buckets} ({@code SnapshotReader#bucketEntries}) ignore it, so
on the BE they observe
+ * the snapshot directory rather than the catalog's pointer. Both are
read-only metadata tables,
+ * so this shows up only as a transient extra row inside the publication
window of a
+ * version-managed catalog.
+ *
+ * <p>Known gap: {@code $files} is planned in two phases and only the
second one can be pinned.
+ * The FE emits one marker split per partition of {@code
SnapshotManager#latestSnapshot()} at
+ * split generation time ({@code FilesTable.FilesScan#innerPlan} ->
+ * {@code SnapshotReader#partitions}), and that path reads {@code
ManifestsReader#read(null, ..)},
+ * which ignores {@code scan.snapshot-id} - so the marker set cannot be
bound to the pin, and
+ * {@code FilesSplit} is private to {@code FilesTable}, so Doris cannot
emit a snapshot
+ * independent marker either. A commit landing between initialization and
split generation that drops a partition
+ * therefore hides that partition's rows even though the BE stays on the
older snapshot. Paimon's
+ * two-phase plan has this gap regardless: before the pin the BE resolved
the latest snapshot at
+ * read time, i.e. across a strictly wider window, so this only moves
which side of the window the
+ * mismatch falls on.
+ *
+ * <p>Known gap: on a table with {@code scan.fallback-branch} this bounds
the main branch only.
+ * {@code FallbackReadFileStoreTable#copyWithoutTimeTravel} derives the
fallback branch's own
+ * bound from the pin ({@code rewriteFallbackOptions} ->
+ * {@code SnapshotManager#earlierOrEqualTimeMills}), and that branch has
no catalog loader left
+ * either, so its bound comes from the snapshot directory too - same
transient window on the same
+ * read-only metadata tables.
+ */
+ private static FileStoreTable pinCatalogSnapshot(FileStoreTable
catalogLessTable, FileStoreTable dataTable) {
+ if (!dataTable.catalogEnvironment().supportsVersionManagement()) {
+ return catalogLessTable;
+ }
+ Long snapshotId = dataTable.snapshotManager().latestSnapshotId();
+ if (snapshotId == null) {
+ return catalogLessTable;
+ }
+ // Without time travel: pin which snapshot the BE plans on, leave
schema resolution alone.
+ return catalogLessTable.copyWithoutTimeTravel(
+ Collections.singletonMap(CoreOptions.SCAN_SNAPSHOT_ID.key(),
String.valueOf(snapshotId)));
+ }
+
+ /**
+ * Return an equivalent {@link FileStoreTable} without the catalog loader,
so the BE never
+ * deserializes a {@code HiveCatalogLoader} (and snapshot loading uses the
filesystem instead of
+ * the catalog's metastore). fileIO / location / schema (and the schema's
options) are preserved.
+ *
+ * <p>A {@code scan.fallback-branch} table must be rebuilt branch by
branch.
+ * {@link FallbackReadFileStoreTable} exposes only its main branch through
{@code schema()}, and
+ * the plain {@code FileStoreTableFactory#create} re-expands the fallback
branch from
+ * {@code SchemaManager(fallbackBranch).latest()} instead of the object
the FE captured. That
+ * would ship a main/fallback pair from two different generations: after
external DDL publishes
+ * a new schema on both branches, the FE keeps planning on the cached
M1/F1 while the BE gets
+ * M1/F2 and fails in {@code FallbackReadFileStoreTable#validateSchema}.
So rebuild each branch
+ * from the schema the FE really planned with, and re-wrap.
+ */
+ private static FileStoreTable dropCatalogLoader(FileStoreTable dataTable) {
+ if (dataTable.catalogEnvironment().catalogLoader() == null) {
+ return dataTable;
+ }
+ FileStoreTable undecorated =
PaimonUtil.unwrapToFallbackOrBase(dataTable);
+ if (undecorated instanceof FallbackReadFileStoreTable) {
+ FallbackReadFileStoreTable fallbackReadTable =
(FallbackReadFileStoreTable) undecorated;
+ return new FallbackReadFileStoreTable(
+ rebuildWithoutCatalogLoader(fallbackReadTable.wrapped()),
+ rebuildWithoutCatalogLoader(fallbackReadTable.fallback()));
Review Comment:
Confirmed and fixed in e692e978351.
Verified against paimon `release-1.3.1`:
- The fallback branch does get its own environment:
`FileStoreTableFactory#create` (`:107-123`)
builds it with `catalogEnvironment.copy(new Identifier(db, table,
fallbackBranch))`, i.e. a
branch-qualified identifier.
- Paimon does authorize the two branches independently:
`FallbackReadFileStoreTable#newScan`
(`:184-187`) is `new FallbackReadScan(wrapped.newScan(),
fallback.newScan())`, each
`AbstractFileStoreTable#newScan` (`:268-275`) passes its own
`catalogEnvironment.tableQueryAuth(coreOptions())` into
`DataTableBatchScan`, and `plan()` calls
`authQuery()` (`DataTableBatchScan:93`).
- And the check really was lost:
`DelegatedFileStoreTable#catalogEnvironment` (`:168-169`) returns
`wrapped.catalogEnvironment()`, so the single call only ever reached the
main branch, while
`CatalogEnvironment.empty()#tableQueryAuth` returns `select -> emptyList()`
(`CatalogEnvironment:145-148`) - a permanent allow on the BE.
Fix: `authorizeDeferredScan` peels decorators and authorizes each captured
branch with its own
environment and its own `coreOptions()`, since `query-auth.enabled` is
per-branch too. Test
`testFallbackBranchIsAuthorizedBeforeDroppingTheCatalogLoader` asserts
`Catalog#authTableQuery` is
called once per identifier - `db.tbl` and `db.tbl$branch_fb`. The fallback
expectation fails on the
previous code.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonScanNode.java:
##########
@@ -203,7 +213,225 @@ protected void doInitialize() throws UserException {
private void serializeProcessedTable() throws UserException {
// System-table splits are materialized by the BE JNI reader, so it
must receive the same
// option-bearing table copy that FE uses to plan the split.
- serializedTable = PaimonUtil.encodeObjectToString(getProcessedTable());
+ serializedTable =
PaimonUtil.encodeObjectToString(getPaimonTableForBackend());
+ }
+
+ /**
+ * Build the Paimon table object that is serialized to the BE.
+ *
+ * <p>Every table loaded from a metastore-backed Paimon catalog (HMS /
DLF) carries a Paimon
+ * {@code HiveCatalogLoader} in its {@link
org.apache.paimon.table.CatalogEnvironment}. The BE
+ * only reads — via FE-resolved splits and the object store — and never
needs the catalog, yet
+ * deserializing that loader forces the whole Hive metastore stack onto
the BE classpath:
+ * {@code HiveConf}, the metastore API, and, when a system table resolves
its latest snapshot,
+ * even the metastore client (DLF's {@code ProxyMetaStoreClient} and its
REST stack). So we
+ * serialize a catalog-less table to the BE:
+ * <ul>
+ * <li>data table: drop the catalog loader. A {@link FileStoreTable} is
fully defined by
+ * fileIO / location / schema / catalogEnvironment, and its dynamic
options (time travel,
+ * incremental) are merged into the schema by {@code copy(...)}, so
rebuilding from
+ * fileIO / location / schema preserves everything except the
catalog loader.</li>
+ * <li>system table (e.g. {@code $snapshots}): rebuild it over a
catalog-less data table so
+ * {@code SnapshotManager#latestSnapshotId} lists the snapshot
directory on the filesystem
+ * instead of calling the metastore. The base table is the one the
FE-side wrapper was
+ * built over, and for the system tables that pick their snapshot on
the BE ({@link
+ * #resolvesSnapshotOnBackend}) what the catalog would have done
there is done here
+ * instead: see {@link #authorizeDeferredScan} and {@link
#pinCatalogSnapshot}. Every
+ * other system table reads what the FE already planned, so it is
handed over untouched.
+ * The relation-scoped scan params the FE applied to the original
wrapper are re-applied
+ * to the rebuilt one by {@link #reapplyScanParams(Table)}.</li>
+ * </ul>
+ */
+ private Table getPaimonTableForBackend() throws UserException {
+ Table paimonTable = getProcessedTable();
+ if (paimonTable instanceof FileStoreTable) {
+ // copy(...) merges the relation's dynamic options into the
schema, and the rebuild
+ // below goes through that schema, so this branch needs no
re-application.
+ return dropCatalogLoader((FileStoreTable) paimonTable);
+ }
+ if (!(source.getExternalTable() instanceof PaimonSysExternalTable)) {
+ return paimonTable;
+ }
+ PaimonSysExternalTable sysTable = (PaimonSysExternalTable)
source.getExternalTable();
+ // The very same base table the FE-side wrapper was built over, so
that the BE never sees a
+ // different schema generation than the one this query was planned
with.
+ FileStoreTable dataTable = sysTable.getSysBaseTable();
+ if (dataTable == null) {
+ return paimonTable;
+ }
+ String sysTableType = sysTable.getSysTableType();
+ if (PAIMON_FILES_SYSTEM_TABLE_TYPE.equalsIgnoreCase(sysTableType)) {
+ authorizeDeferredScan(dataTable);
+ }
+ FileStoreTable baseForBackend = dropCatalogLoader(dataTable);
+ if (resolvesSnapshotOnBackend(sysTableType)) {
+ baseForBackend = pinCatalogSnapshot(baseForBackend, dataTable);
+ }
+ Table catalogLessSysTable = SystemTableLoader.load(sysTableType,
baseForBackend);
+ return catalogLessSysTable == null ? paimonTable :
reapplyScanParams(catalogLessSysTable);
+ }
+
+ /**
+ * Re-apply the relation-scoped scan params to a rebuilt system-table
wrapper.
+ *
+ * <p>{@link #getProcessedTable()} applies {@code @incr} / {@code
@options} to the wrapper the
+ * meta cache holds, and every Paimon system table delegates {@code
copy(...)} to the data table
+ * it wraps. The wrapper rebuilt above is a different object, so the same
copy has to be redone
+ * on it, otherwise the BE would materialize its splits against the
unpinned latest state.
+ *
+ * <p>This runs last on purpose: an explicit relation option outranks
anything this class pins
+ * on the rebuilt table, and {@code copy(...)} lets the option win.
+ */
+ private Table reapplyScanParams(Table rebuiltSysTable) throws
UserException {
+ TableScanParams theScanParams = getScanParams();
+ if (theScanParams == null) {
+ return rebuiltSysTable;
+ }
+ if (theScanParams.incrementalRead()) {
+ return rebuiltSysTable.copy(getIncrReadParams());
Review Comment:
Confirmed and fixed in e692e978351.
Verified the chain against paimon `release-1.3.1`, not taken on trust:
- The pin really is dropped: `PaimonScanParams#isolateIncrementalRead` nulls
every inherited
read-state key, `scan.snapshot-id` included, and `copy(...)` removes a key
that maps to null
(`AbstractFileStoreTable#copyInternal:328-335`). But it could not have
bounded this scan either
way: `AbstractDataTableScan#createStartingScanner` (`:204-286`) switches
on `options.startupMode()`,
and `INCREMENTAL` routes to `createIncrementalStartingScanner`, which
never reads
`options.scanSnapshotId()`.
- The timestamp form is the exposed one.
`IncrementalDeltaStartingScanner#betweenTimestamps`
(`:168-185`) turns both endpoints into snapshot ids through
`SnapshotManager#earlierOrEqualTimeMills`, whose binary search runs up to
`latestSnapshotId()`
(`SnapshotManager:288`), and falls back to `latestSnapshot().id()` when
the end resolves to
nothing; `AbstractDataTableScan:407-425` reads `latestSnapshot()` again
for its empty-range guard.
All three go through `snapshotLoader` when it exists
(`SnapshotManager:168-192`) - the catalog
pointer - and list the snapshot directory once it does not.
- Scope is exactly `$partitions`: `INCREMENTAL_SYSTEM_TABLES` intersected
with
`resolvesSnapshotOnBackend` is `{partitions}`. `$audit_log` / `$binlog` /
`$ro` / `$row_tracking`
are planned on the FE and the BE only materializes their `DataSplit`s, and
the snapshot-id form
(`startSnapshotId` / `endSnapshotId`) names its endpoints outright.
Fix: `bindIncrementalRangeToCatalog` resolves the range here, while the
loader is still around, and
hands the BE the explicit `incremental-between=startId,endId` that
`betweenTimestamps` would have
computed from the catalog's own view. Two things keep that faithful rather
than approximate:
- Only the delta scanner's start rule is reachable -
`incremental-between-scan-mode` is cleared with
the rest of the read state and its default `AUTO` never selects the diff
scanner
(`AbstractDataTableScan:427-436`, `toSnapshotScanMode`).
- The rewrite only fires when the requested end reaches the catalog's
snapshot. An older end already
resolves to the same id on both sides, because every snapshot the catalog
has not published yet is
younger than the one it points at - which also keeps the `endTimestamp <
earliest.timeMillis()`
empty-range corner on Paimon's own path.
Tests: `testIncrementalRangeIsResolvedOnTheCatalogVisibleSnapshot` - catalog
snapshot 42 at t=5000
with 43 already on the filesystem resolves to `20,42`, plus the past-end and
non-version-managed
no-ops - and `testIncrementalPartitionsScanBindsItsRangeBeforeTheBackend`,
the catalog-N /
filesystem-N+1 phase barrier through `getPaimonTableForBackend`. The second
one fails on the
previous code: `earlierOrEqualTimeMills` is never called, because nothing
resolved the endpoint
before the BE.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonSysExternalTable.java:
##########
@@ -115,17 +118,48 @@ private static long generateSysTableId(long
sourceTableId, String sysTableType)
/**
* Returns the Paimon system table instance (e.g., snapshots, binlog).
* Note: system tables currently ignore snapshot semantics.
+ *
+ * <p>The wrapper is built over the cached base table, exactly the way the
Paimon catalog builds
+ * it ({@code CatalogUtils#createSystemTable}). Loading it from the
catalog instead would give
+ * this wrapper its own schema generation, independent of the cached base
table: the FE would
+ * then plan on one generation while {@code PaimonScanNode} serializes a
system table rebuilt
+ * over the other one, and a system table whose row type follows the base
schema
+ * ({@code $audit_log}, {@code $binlog}, {@code $ro}) could reach the BE
without the columns the
+ * FE asked for.
+ *
+ * <p>"Exactly the way the catalog builds it" includes building it over
the <i>undecorated</i>
+ * base: {@code CatalogUtils#loadTable} hands {@code createSystemTable}
the raw
+ * {@code FileStoreTableFactory#create} result, and the decorator a
catalog may add afterwards
+ * never reaches a system table, because it only wraps a {@code
FileStoreTable} and no system
+ * table is one. The meta cache, in contrast, holds the decorated table.
That matters for
+ * {@code $ro}: {@code ReadOptimizedTable#newScan} builds its two-branch
{@code FallbackReadScan}
+ * only when its immediate wrapped object is a {@code
FallbackReadFileStoreTable}, so with a
+ * {@code PrivilegedFileStoreTable} in between it would plan the main
branch alone through the
+ * pair's inherited {@code newSnapshotReader()} and silently drop every
fallback-only partition.
*/
public Table getSysPaimonTable() {
if (paimonSysTable == null) {
synchronized (this) {
if (paimonSysTable == null) {
- PaimonExternalCatalog catalog = (PaimonExternalCatalog)
getCatalog();
- paimonSysTable = catalog.getPaimonTable(
- sourceTable.getOrBuildNameMapping(),
- "main", // branch
- sysTableType // queryType: snapshots, binlog, etc.
- );
+ sourceTable.makeSureInitialized();
+ // System tables ignore snapshot semantics, so the empty
snapshot resolves the base table.
+ Table baseTable =
sourceTable.getPaimonTable(Optional.empty());
Review Comment:
Fixed in e692e978351, with one correction to the framing.
The split lookup is not something this PR introduced.
`getSysPaimonTable(TableScanParams)` and its
`sourceTable.getBasePaimonTable()` resolution are untouched upstream code,
and before this PR the
wrapper came from an independent `catalog.getPaimonTable(nameMapping,
"main", sysTableType)` load
rather than from the meta cache - so the wrapper and the resolution base
were already two different
sources, diverging across a strictly wider window. What this PR changed is
that both now come from
the same cache, at two different times.
Worth closing anyway, and it is this PR's own thesis, so: the resolution now
runs against the base
`getSysPaimonTable()` captured (`getOptionsResolutionTable()`). Only the
catalog fallback - format /
object tables, which have no captured base - still looks the table up. The
captured base is the
undecorated one, but every call `resolveOptions` makes on it (`options()`,
`snapshotManager()`,
`changelogManager()`, `coreOptions()`, `latestSnapshot()`, `copy()`)
delegates through Paimon's
decorators unchanged, so the resolved selector is identical.
Test `testSystemTableOptionsResolveOnTheCapturedBaseGeneration`: the
captured generation is at
snapshot 1, the refreshed cache at snapshot 2, and
`@options('scan.mode'='latest')` has to freeze
`scan.snapshot-id=1`. It resolves to 2 on the previous code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
