github-actions[bot] commented on code in PR #68648:
URL: https://github.com/apache/doris/pull/68648#discussion_r4370114859
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/RefreshMTMVCommand.java:
##########
@@ -194,12 +197,31 @@ protected LogicalPlan createRefreshCommand(MTMV mtmv,
StatementContext statement
statementContext.setExcludedTriggerTables(mtmv.getExcludedTriggerTables());
return createIvmIncrRefreshManager().buildInsertCommand(mtmv);
case COMPLETE:
+
statementContext.setExcludedTriggerTables(mtmv.getExcludedTriggerTables());
+ Set<String> completeRefreshPartitions =
getCompleteRefreshPartitions(mtmv);
+ Map<TableIf, String> tableWithPartKey =
getIncrementalTableMap(mtmv);
+ MTMVRefreshContext refreshContext =
MTMVRefreshContext.buildContext(mtmv, Maps.newHashMap());
Review Comment:
[P2] Pin external snapshots before deriving COMPLETE EXPLAIN scope
This runs before planner binding, while the new explain StatementContext has
no input snapshots; getSnapshot only reads its empty map. For an Iceberg
DAY(ts) source with N partitions and an MV PARTITION
BY(date_trunc(ts,'month')), rollUpRange calls generateRollUpPartitionKeyDesc
for each descriptor, which calls getPartitionColumns(Optional.empty()).
PluginDrivenMvccExternalTable then rematerializes all N RangePartitionItems
each time. That adds O(N²) FE partition conversions/allocations (roughly 100
million for 10,000 partitions), even when the connector caches its partition
view. The real refresh avoids this by loading and reusing an engine snapshot
before building its mapping.
The same missing pin can also fail COMPLETE EXPLAIN for Hive with
meta.cache.hive.partition_view.enable=false and
meta.cache.hive.partition_names.enable=false: if a remote DROP removes a
partition after this mapping enumerates it, from() obtains a newer external map
and throws the new "base partition ... is not in the snapshot" error. Pin
external inputs in the explain StatementContext before building this context,
and reuse those pins for predicate construction and scan binding. Add a test
that checks one engine snapshot materialization per input; the existing scope
assertions do not cover this pre-binding path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
