github-actions[bot] commented on code in PR #66247:
URL: https://github.com/apache/doris/pull/66247#discussion_r3694600164
##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonConnectorMetadata.java:
##########
@@ -1301,30 +1324,41 @@ private ConnectorTableKey
partitionViewCacheKey(PaimonTableHandle paimonHandle)
* the partition columns and escapes path-special characters in the name
via the Paimon SDK.
*/
private List<ConnectorPartitionInfo> collectPartitions(PaimonTableHandle
paimonHandle) {
+ if (PaimonScanParams.isPinnedEmptyScan(paimonHandle.getScanOptions()))
{
+ // Do not reopen latest metadata after the statement fenced an
empty table.
+ return Collections.emptyList();
+ }
List<String> partitionKeys = paimonHandle.getPartitionKeys();
// Legacy never lists partitions for unpartitioned tables:
PaimonPartitionInfoLoader.load
// returns EMPTY when partitionColumns is empty, so guard before
touching the seam.
if (partitionKeys == null || partitionKeys.isEmpty()) {
return Collections.emptyList();
}
- // Partition enumeration is intentionally BASE-only: branch /
time-travel reads carry EMPTY
- // partition info (legacy PaimonPartitionInfo.EMPTY) and never reach
this path, so for the
- // (non-branch) handles that do, resolveTable returns the base table
and the base-Identifier
- // listing below is consistent. (A branch handle would otherwise mix
branch schema metadata
- // here with the base partition list — but that combination does not
occur by design.)
- Table table = resolveTable(paimonHandle);
+ Table resolvedTable = resolveTable(paimonHandle);
+ boolean optionsPin =
PaimonScanParams.isOptionsPin(paimonHandle.getScanOptions());
+ Table table;
+ if (optionsPin) {
+ table = PaimonScanParams.applyOptions(resolvedTable,
paimonHandle.getScanOptions());
+ } else {
Review Comment:
[P2] Apply the positive fence while hydrating partitions
For an options-first/plain-second alias pair,
`materializeLatest(existingFence)` now passes this method a handle carrying
ordinary `scan.snapshot-id=S`. Only an OPTIONS-marked map is applied here, so
this `else` discards that positive fence and lists current partitions at S+1,
while split planning later honors S. This leaves EXPLAIN's partition universe
inconsistent; under COUNT(*) pushdown the native scanned-partition correction
is suppressed, so an S+1 partition drop can also under-report the partitions
scanned at S and bypass a `sql_block_rule` limit. Please apply ordinary
positive snapshot pins during partition hydration (or publish no live partition
view where REST cannot enumerate that version), and cover an options-first race
where S+1 changes the partition set.
##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonConnectorMetadata.java:
##########
@@ -1474,8 +1511,13 @@ public Optional<ConnectorTableStatistics>
getTableStatistics(
Table table = resolveTable(pinned);
Map<String, String> scanOptions = pinned.getScanOptions();
if (scanOptions != null && !scanOptions.isEmpty()) {
- table = table.copy(scanOptions);
+ table = PaimonScanParams.isOptionsPin(scanOptions)
+ ? PaimonScanParams.applyOptions(table, scanOptions)
Review Comment:
[P2] Preserve the pinned-empty fence in snapshot statistics
When an `@options` relation pins an empty latest table, `applySnapshot` puts
the internal pinned-empty marker in `scanOptions`. Split planning recognizes
that marker and returns no splits, but this branch only applies the user
options to the current live table, discards the marker, and then calls
`rowCount`. If the table receives its first commit between binding and CBO
row-count planning, `getVersionedSnapshot` reaches this overload and it reports
the new rows even though the same relation remains fenced empty at execution,
skewing the statement's plan. Please handle `isPinnedEmptyScan` before planning
statistics (and cover the first-commit race for the snapshot-statistics path).
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/StatementContext.java:
##########
@@ -1047,8 +1050,35 @@ public void loadSnapshots(TableIf specificTable,
Optional<TableSnapshot> tableSn
MvccTableInfo mvccTableInfo = new MvccTableInfo(specificTable,
versionKeyOf(tableSnapshot, scanParams));
if (!snapshots.containsKey(mvccTableInfo)) {
- snapshots.put(mvccTableInfo,
- ((MvccTable)
specificTable).loadSnapshot(tableSnapshot, scanParams));
+ MvccTable mvccTable = (MvccTable) specificTable;
+ MvccSnapshot snapshot;
+ if (mvccTable.requiresLatestSnapshotFence(tableSnapshot,
scanParams)) {
+ MvccTableInfo latestKey = new MvccTableInfo(specificTable);
+ MvccSnapshot latestFence =
latestSnapshotFences.computeIfAbsent(latestKey,
+ key -> latestSnapshots.containsKey(key)
+ ? latestSnapshots.get(key) :
mvccTable.loadLatestSnapshotFence());
+ // Different planning projections remain separate, but
their version selector
+ // comes from one statement fence instead of repeated
mutable latest reads.
+ snapshot = mvccTable.loadSnapshot(tableSnapshot,
scanParams, Optional.of(latestFence));
+ } else if (!tableSnapshot.isPresent() &&
!scanParams.isPresent()) {
+ snapshot = latestSnapshots.computeIfAbsent(mvccTableInfo,
+ key -> latestSnapshotFences.containsKey(key)
Review Comment:
[P1] Reset the snapshot fence between prepared executions
`ExecuteCommand` reuses this `StatementContext` and resets each relation's
`resolvedMapParams` before every `EXECUTE`, but none of `snapshots`,
`latestSnapshots`, `latestSnapshotFences`, or `resolvedSnapshotScanParams` is
cleared. On the second execution this key is already in `snapshots`, so
connector resolution is skipped and the new `else if` restores the first
execution's resolved options. A prepared selector-free `@options` query
therefore remains pinned to snapshot S after a commit creates S+1 (with the
latest cache disabled or invalidated), returning stale rows on later
executions. Please reset the MVCC/fence maps at the prepared-execution boundary
and cover two EXECUTEs separated by a commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
