github-actions[bot] commented on code in PR #67725:
URL: https://github.com/apache/doris/pull/67725#discussion_r4256724195
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PruneFileScanPartition.java:
##########
@@ -79,6 +84,18 @@ public Rule build() {
}).toRule(RuleType.FILE_SCAN_PARTITION_PRUNE);
}
+ /**
+ * Builds binary-search ranges for a connector-declined fallback from the
partition map already recorded for
+ * this statement. The deferred partition view must not be enumerated a
second time here: a remote mutation
+ * between the two listings could make the range generation disagree with
the partition map and either abort
+ * planning or silently omit a matching partition.
+ */
+ static Optional<SortedPartitionRanges<String>>
fallbackSortedPartitionRanges(
+ SelectedPartitions selectedPartitions, Map<String, PartitionItem>
partitionItems) {
+ return selectedPartitions.sortedPartitionRanges
+ .or(() ->
Optional.ofNullable(SortedPartitionRanges.build(partitionItems)));
Review Comment:
[P2] Preserve cached ranges for materialized connector snapshots
For repeated `WHERE p = 1` queries on an unchanged partitioned Paimon table,
`ExternalTable.initSelectedPartitions` creates a materialized full map with
`sortedPartitionRanges` empty, so this helper rebuilds and sorts every
partition on every query. The removed `externalTable.getSortedPartitionRanges`
lookup was the only entry to the existing cross-query cache; Paimon's ranges
and version are read from the same immutable MVCC pin, so this path has no
deferred Hive re-enumeration mismatch to avoid. This adds O(N log N) work and
O(N) temporary range objects to every selective query on a large table.
Preserve cached ranges for materialized pinned views while building from the
recorded map for deferred/connector-filtered views, and cover two queries over
an unchanged pin.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/mvcc/PluginDrivenMvccExternalTable.java:
##########
@@ -728,11 +706,163 @@ public boolean
supportInternalPartitionPruned(Optional<MvccSnapshot> snapshot) {
@Override
public Map<String, PartitionItem>
getNameToPartitionItems(Optional<MvccSnapshot> snapshot) {
+ if (supportsConnectorPartitionPruning()) {
+ PluginDrivenMvccSnapshot pin = getOrMaterialize(snapshot);
+ if (pin.isPartitionViewMaterialized()) {
+ return pin.getNameToPartitionItem();
+ }
+ if (pin.isPartitionViewUnavailable()) {
+ return Collections.emptyMap();
+ }
+ Optional<ExternalTablePreloadInfo> sharedView =
sharedLatestPreloadInfo(snapshot);
+ if (sharedView.isPresent() &&
sharedView.get().hasScanPartitionView()
+ && sharedView.get().getScanPartitionView().isPresent()) {
+ return sharedView.get().getScanPartitionView().get();
+ }
+ // The latest Hive query pin intentionally carries no partition
map so selective scans can send a
+ // predicate to HMS first. Consumers that explicitly ask for a
partition map (MTMV alignment,
+ // no-filter scan finalization, and a connector-declined pruning
fallback) require the real full
+ // view instead of treating that query-only pin as an empty table.
An unavailable shared scan view
+ // is not an authoritative empty map either: checked MTMV
consumers reject it in
+ // getAndCopyPartitionItems, while other explicit map consumers
retain this fail-loud full-view path.
+ return super.getNameToPartitionItems(snapshot);
+ }
return getOrMaterialize(snapshot).getNameToPartitionItem();
}
@Override
- public Map<String, PartitionItem>
getAndCopyPartitionItems(Optional<MvccSnapshot> snapshot) {
+ public Optional<Map<String, PartitionItem>>
getNameToPartitionItemsForScan(Optional<MvccSnapshot> snapshot) {
+ if (snapshot.isPresent() && snapshot.get() instanceof
PluginDrivenMvccSnapshot
+ && ((PluginDrivenMvccSnapshot)
snapshot.get()).isPartitionViewUnavailable()) {
+ return Optional.empty();
+ }
+ if (supportsConnectorPartitionPruning()) {
+ PluginDrivenMvccSnapshot pin = getOrMaterialize(snapshot);
+ if (pin.isPartitionViewMaterialized()) {
+ // This statement's pin already carries a materialized view:
reuse it instead of paying another
+ // connector round-trip that could observe a different remote
generation.
+ return Optional.of(pin.getNameToPartitionItem());
+ }
+ }
+ return super.getNameToPartitionItemsForScan(snapshot);
+ }
+
+ @Override
+ public Optional<ConnectorFilteredPartitionView>
applyPartitionFilterForScan(Optional<MvccSnapshot> snapshot,
+ ConnectorExpression partitionFilter) {
+ if (snapshot.isPresent() && snapshot.get() instanceof
PluginDrivenMvccSnapshot
+ && ((PluginDrivenMvccSnapshot)
snapshot.get()).isPartitionViewUnavailable()) {
+ return Optional.empty();
+ }
+ return super.applyPartitionFilterForScan(snapshot, partitionFilter);
+ }
+
+ /**
+ * Threads this statement's MVCC pin onto the handle the partition view is
enumerated from, so a
+ * time-travel / {@code @options} query never prunes against the latest
generation: the data scan reads the
+ * pinned snapshot, and a latest view can be missing (or contain)
partitions it will never read.
+ */
+ @Override
+ protected ConnectorTableHandle pinPartitionViewHandle(ConnectorTableHandle
handle,
+ ConnectorMetadata metadata, ConnectorSession session,
Optional<MvccSnapshot> snapshot) {
+ if (snapshot.isPresent() && snapshot.get() instanceof
PluginDrivenMvccSnapshot) {
+ return metadata.applySnapshot(session, handle,
+ ((PluginDrivenMvccSnapshot)
snapshot.get()).getConnectorSnapshot());
+ }
+ return handle;
+ }
+
+ /**
+ * Materializes the complete partition view for an MTMV refresh before it
acquires base-table locks.
+ * The lightweight Hive query pin keeps the connector snapshot/freshness
kind but deliberately omits the
+ * partition map; MTMV alignment needs that map and must not load it while
holding internal table locks.
+ */
+ public MvccSnapshot materializePartitionViewForMtmv(MvccSnapshot snapshot)
{
+ if (!supportsConnectorPartitionPruning() || !(snapshot instanceof
PluginDrivenMvccSnapshot)) {
+ return snapshot;
+ }
+ PluginDrivenMvccSnapshot pin = (PluginDrivenMvccSnapshot) snapshot;
+ if (!pin.isPartitionViewDeferred()) {
+ return pin;
+ }
+ // The warmup runs BEFORE the task takes its base-table locks, and a
SELF_MANAGE MTMV must still be
+ // refreshed when ONE partition cannot be represented in its column
type (an HMS/API-created DATE
+ // partition `dt=not-a-date` is the canonical case). Enumerate with
the LIST path's per-partition
+ // contract (listPartitions: log the bad partition, skip it, leave the
item map short of the listed
+ // names) rather than the fail-loud eager MTMV-alignment build: the
short map makes
+ // isPartitionInvalid() true, which degrades this base table to
UNPARTITIONED so the whole-MV refresh
+ // continues as a scan-all instead of aborting the task.
+ PluginDrivenExternalCatalog pluginCatalog =
(PluginDrivenExternalCatalog) catalog;
+ Connector connector = pluginCatalog.getConnector();
+ ConnectorSession session = pluginCatalog.buildConnectorSession();
+ ConnectorMetadata metadata = PluginDrivenMetadata.get(session,
connector);
+ Optional<ConnectorTableHandle> handleOpt =
resolveConnectorTableHandle(session, metadata);
+ if (!handleOpt.isPresent()) {
+ // No handle (e.g. the base table was concurrently dropped): the
view cannot be materialized at
+ // all, so publish it UNAVAILABLE instead of an authoritative
empty map that the consumers would
+ // read as a real zero-partition table.
+ return
PluginDrivenMvccSnapshot.unavailable(pin.getConnectorSnapshot(), null);
+ }
+ // Enumerate from the SNAPSHOT-PINNED handle - the same generation the
data scan reads - never the
+ // freshly resolved latest one (see pinPartitionViewHandle).
+ ConnectorTableHandle pinnedHandle =
+ pinPartitionViewHandle(handleOpt.get(), metadata, session,
Optional.of(pin));
+ Map<String, PartitionItem> nameToPartitionItem = Maps.newHashMap();
+ Map<String, Long> nameToLastModifiedMillis = Maps.newHashMap();
+ listPartitions(metadata, session, pinnedHandle,
getPartitionColumns(Optional.of(pin)),
+ nameToPartitionItem, nameToLastModifiedMillis);
+ return new PluginDrivenMvccSnapshot(pin.getConnectorSnapshot(),
nameToPartitionItem,
+ nameToLastModifiedMillis, null);
+ }
+
+ /**
+ * Returns the latest-view preload record materialized by the statement
before internal table locks were
+ * acquired, or empty when this request does not refer to that latest view.
+ *
+ * <p>The record keeps the scan view and the async-MV mapping view
separate. A filtered scan intentionally
+ * leaves its scan view unwarmed so connector predicate pruning remains
effective, while query-time async-MV
+ * validation and union compensation read the complete mapping view
materialized before planner locks. Each
+ * view also distinguishes UNAVAILABLE ({@link Optional#empty()}) from an
authoritative empty map. A
+ * supplied pin is accepted only when it is the statement's own latest
pin; historical/time-travel pins and
+ * calls without a statement context keep their existing snapshot
semantics.</p>
+ */
+ private Optional<ExternalTablePreloadInfo>
sharedLatestPreloadInfo(Optional<MvccSnapshot> snapshot) {
+ ConnectContext connectContext = ConnectContext.get();
+ StatementContext statementContext = connectContext == null ? null :
connectContext.getStatementContext();
+ if (statementContext == null) {
+ return Optional.empty();
+ }
+ Optional<ExternalTablePreloadInfo> preloadInfo =
statementContext.getExternalTablePreloadInfo(getId());
+ if (!preloadInfo.isPresent() ||
!preloadInfo.get().hasLatestOnlyRelation()) {
+ return Optional.empty();
+ }
+ Optional<MvccSnapshot> latestSnapshot =
statementContext.getSnapshot(this);
+ boolean isLatestRequest = !snapshot.isPresent()
+ || (latestSnapshot.isPresent() && latestSnapshot.get() ==
snapshot.get());
+ return isLatestRequest ? preloadInfo : Optional.empty();
+ }
+
+ @Override
+ public Map<String, PartitionItem>
getAndCopyPartitionItems(Optional<MvccSnapshot> snapshot)
+ throws AnalysisException {
+ Optional<ExternalTablePreloadInfo> sharedView =
sharedLatestPreloadInfo(snapshot);
+ if (sharedView.isPresent()) {
Review Comment:
[P2] Restrict deferred-view reuse to connector-pruning tables
An ordinary latest Iceberg reference is registered for preload because
Iceberg declares `SUPPORTS_METADATA_PRELOAD`, but neither new partition-view
warmup populates its entry because it lacks
`SUPPORTS_CONNECTOR_PARTITION_PRUNING`. This branch therefore throws even
though its MVCC snapshot already contains the complete partition map. For the
reduced plan `Filter(dt = ...) -> IcebergScan` with a partitioned async MV,
`MTMV.calculatePartitionMappings -> MTMVRelatedPartitionDescInitGenerator ->
getAndCopyPartitionItems` reaches that exception and
`AbstractMaterializedViewRule` abandons the eligible rewrite. EXPR range MVs
also reach it through `MTMVPartitionExpander`. Gate this reuse branch on
connector partition pruning (or first reuse a materialized pin), and cover a
latest Iceberg reference with a preload record.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
