github-actions[bot] commented on code in PR #66247:
URL: https://github.com/apache/doris/pull/66247#discussion_r3688450289
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/StatementContext.java:
##########
@@ -1047,8 +1049,28 @@ public void loadSnapshots(TableIf specificTable,
Optional<TableSnapshot> tableSn
MvccTableInfo mvccTableInfo = new MvccTableInfo(specificTable,
versionKeyOf(tableSnapshot, scanParams));
if (!snapshots.containsKey(mvccTableInfo)) {
- snapshots.put(mvccTableInfo,
- ((MvccTable)
specificTable).loadSnapshot(tableSnapshot, scanParams));
+ MvccTable mvccTable = (MvccTable) specificTable;
+ MvccSnapshot snapshot;
+ if (mvccTable.requiresLatestSnapshotFence(tableSnapshot,
scanParams)) {
+ MvccTableInfo latestKey = new MvccTableInfo(specificTable);
+ MvccSnapshot latestFence =
latestSnapshots.computeIfAbsent(latestKey,
Review Comment:
[P1] Reuse an explicitly seeded statement snapshot for this fence. MTMV
refresh captures snapshot S before planning and injects it with
`setSnapshot(defaultKey, S)`, but that setter populates only `snapshots`, not
`latestSnapshots`. A reader/planning-only `@options` relation has a non-default
projection key, misses `snapshots`, and reaches this `computeIfAbsent`,
performing a live latest lookup S+1. If a commit lands after MTMV's capture,
the option alias scans S+1 while a plain alias in the same refresh still reads
seeded S. Please seed `latestSnapshots` when a default snapshot is supplied (or
prefer the existing default entry here), and add an MTMV-style seeded-fence
alias-order test.
##########
fe/be-java-extensions/paimon-scanner/src/main/java/org/apache/doris/paimon/PaimonJniScanner.java:
##########
@@ -575,12 +676,98 @@ static Optional<Long> parseDataSizeBytes(String value) {
private void initTable() {
Preconditions.checkState(params.containsKey("serialized_table"));
table = PaimonUtils.deserialize(params.get("serialized_table"));
+ table = applyBackendManifestParallelism(table,
+ params.get(PAIMON_OPTION_PREFIX +
DORIS_MANIFEST_PARALLELISM_CAP),
+ Runtime.getRuntime().availableProcessors());
+ validateSerializedReaderOptions(table);
paimonAllFieldNames = PaimonUtils.getFieldNames(this.table.rowType());
if (LOG.isDebugEnabled()) {
LOG.debug("paimonAllFieldNames:{}", paimonAllFieldNames);
}
}
+ static Table applyBackendManifestParallelism(
+ Table table, String feParallelismCap, int localCapacity) {
+ int safeParallelism;
+ if (feParallelismCap != null) {
+ safeParallelism =
parsePositiveManifestParallelism(feParallelismCap);
+ if (safeParallelism <= localCapacity) {
+ return table;
+ }
+ safeParallelism = localCapacity;
+ } else {
+ List<Integer> configuredValues = new ArrayList<>();
+ collectManifestParallelism(table, configuredValues);
+ if (configuredValues.isEmpty()
+ || configuredValues.stream().noneMatch(value -> value >
localCapacity)) {
+ return table;
+ }
+ safeParallelism = Math.min(
+
configuredValues.stream().mapToInt(Integer::intValue).min().getAsInt(),
+ localCapacity);
+ }
+ Map<String, String> cap = Collections.singletonMap(
+ CoreOptions.SCAN_MANIFEST_PARALLELISM.key(),
String.valueOf(safeParallelism));
+ // File-store copies must retain the FE-selected schema while only
lowering an execution
+ // bound; ordinary copy can re-resolve time travel and undo schema
pinning.
+ return table instanceof FileStoreTable
+ ? ((FileStoreTable) table).copyWithoutTimeTravel(cap)
+ : table.copy(cap);
Review Comment:
[P1] Preserve the schema fence when capping system wrappers. For a deferred
`$files`/`$partitions` object this arm receives a system wrapper, so Paimon
delegates `table.copy(cap)` to the hidden `storeTable.copy`. Because
`tableForBackend` retained `scan.snapshot-id` with `copyWithoutTimeTravel`,
this ordinary copy re-runs time travel and rewinds the hidden base schema,
undoing that deliberate fence. After a schema-only ALTER with no new snapshot,
a smaller BE can therefore return `$files` stats maps keyed by the old fields
(`FilesRead` seeds them from `storeTable.schema().id()`) while the FE has bound
the current schema. Please cap the hidden base with `copyWithoutTimeTravel` and
rebuild the same wrapper, and cover a real deferred system wrapper at lower BE
capacity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
