zhaorongsheng commented on code in PR #67725:
URL: https://github.com/apache/doris/pull/67725#discussion_r4255902907
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalTable.java:
##########
@@ -949,17 +952,57 @@ public boolean supportInternalPartitionPruned() {
return true;
}
+ @Override
+ public SelectedPartitions initSelectedPartitions(Optional<MvccSnapshot>
snapshot) {
+ if (supportsConnectorPartitionPruning()) {
+ return SelectedPartitions.DEFERRED_PARTITION_PRUNING;
+ }
+ return super.initSelectedPartitions(snapshot);
+ }
+
+ /** Whether this table defers partition materialization until Nereids
supplies a connector predicate. */
+ public boolean supportsConnectorPartitionPruning() {
+ return
hasCapability(ConnectorCapability.SUPPORTS_CONNECTOR_PARTITION_PRUNING);
+ }
+
@Override
public Map<String, PartitionItem>
getNameToPartitionItems(Optional<MvccSnapshot> snapshot) {
+ return getNameToPartitionItems(snapshot, Optional.empty());
+ }
+
+ /**
+ * Builds the generic partition map from a connector-filtered partition
view. Callers use this only after
+ * converting a Nereids predicate into the neutral connector expression
grammar.
+ */
+ public Optional<Map<String, PartitionItem>>
getNameToPartitionItemsByFilter(Optional<MvccSnapshot> snapshot,
+ ConnectorExpression partitionFilter) {
List<Column> partitionColumns = getPartitionColumns(snapshot);
if (partitionColumns.isEmpty()) {
- return Collections.emptyMap();
+ return Optional.empty();
}
- List<String> remoteNames = getSchemaCacheValue(snapshot)
- .map(value -> ((PluginDrivenSchemaCacheValue)
value).getPartitionColumnRemoteNames())
- .orElse(Collections.emptyList());
- List<Type> types =
partitionColumns.stream().map(Column::getType).collect(Collectors.toList());
+ PluginDrivenExternalCatalog pluginCatalog =
(PluginDrivenExternalCatalog) catalog;
+ Connector connector = pluginCatalog.getConnector();
+ ConnectorSession session = pluginCatalog.buildConnectorSession();
+ ConnectorMetadata metadata = PluginDrivenMetadata.get(session,
connector);
+ Optional<ConnectorTableHandle> handleOpt =
resolveConnectorTableHandle(session, metadata);
+ if (!handleOpt.isPresent()) {
+ return Optional.empty();
+ }
+ Optional<FilterApplicationResult<ConnectorTableHandle>> filterResult =
metadata.applyFilter(
Review Comment:
Addressed. Logical pruning now carries its connector-filter result forward
instead of discarding it: `SelectedPartitions` stores the applied
`FilterApplicationResult<ConnectorTableHandle>` plus the covered predicate, and
`PluginDrivenScanNode.convertPredicate` reuses it
(`reusedConnectorFilterResult` / `connectorFilterCoversCurrentPredicate` /
`applyConnectorFilterResult`) whenever the current predicate is still covered,
so the physical scan does not re-apply the filter to a fresh handle across an
HMS change and cannot combine names from one generation with
`prunedPartitionsByName` from another.
##########
fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/ThriftHmsClient.java:
##########
@@ -239,6 +240,21 @@ static short toThriftMaxParts(int maxParts) {
return maxParts <= 0 ? (short) -1 : (short) maxParts;
}
+ @Override
+ public List<HmsPartitionInfo> listPartitionsByFilter(String dbName, String
tableName, String filter) {
+ List<Partition> partitions = execute(client ->
client.listPartitionsByFilter(
+ dbName, tableName, filter, (short) (MAX_FILTERED_PARTITIONS +
1)));
+ if (isFilteredPartitionResponseSaturated(partitions.size())) {
Review Comment:
Addressed. The saturation decision now uses the RAW pre-hook page size: the
vendored client exposes
`HmsRawPartitionFilterPageSource.listPartitionsByFilterRawPage`, and
`ThriftHmsClient.listPartitionsByFilter` throws
`HmsPartitionFilterSaturatedException` based on `page.rawCount` (the
pre-`MetaStoreFilterHook` size) rather than the hook-filtered list size, so a
hook-hidden entry in a truncated page can no longer mask a saturated response.
##########
fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java:
##########
@@ -1288,6 +1277,87 @@ private List<ConnectorPartitionInfo>
listPartitionsUncached(HiveTableHandle hive
return result;
}
+ private PartitionPruningResult prunePartitions(ConnectorSession session,
HiveTableHandle hiveHandle,
+ ConnectorExpression expression) {
+ List<String> partKeyNames = hiveHandle.getPartitionKeyNames();
+ Map<String, List<String>> partitionPredicates =
extractPartitionPredicates(expression, partKeyNames);
+ if (partitionPredicates.isEmpty()) {
+ return null;
+ }
+
+ String hmsFilter = buildHmsPartitionFilter(partKeyNames,
hiveHandle.getPartitionKeyTypes(),
+ partitionPredicates);
+ if (hmsFilter != null) {
+ try {
+ List<HmsPartitionInfo> prunedPartitions =
hmsClient.listPartitionsByFilter(
+ hiveHandle.getDbName(), hiveHandle.getTableName(),
hmsFilter);
+ LOG.info("Partition pruning through HMS filter: {}.{}
filter={} pruned={}",
Review Comment:
Addressed. The INFO records now log only bounded counts
(`predicateValues=<count> pruned=<count>`, and `all=<count> pruned=<count>` on
the local path); the generated predicate is emitted only at DEBUG through
`summarizeHmsFilterForDebug`, and the deliberate saturation fallback logs at
INFO without a stack trace (the exception path is reserved for genuine
failures).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
