github-actions[bot] commented on code in PR #67725:
URL: https://github.com/apache/doris/pull/67725#discussion_r3977833479
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/mvcc/PluginDrivenMvccExternalTable.java:
##########
@@ -671,6 +687,13 @@ static boolean schemaCacheDisabled(Connector connector) {
@Override
public Map<String, PartitionItem>
getNameToPartitionItems(Optional<MvccSnapshot> snapshot) {
+ if (supportsConnectorPartitionPruning()) {
+ // The latest Hive query pin intentionally carries no partition
map so selective scans can send a
+ // predicate to HMS first. Consumers that explicitly ask for a
partition map (MTMV alignment,
+ // no-filter scan finalization, and a connector-declined pruning
fallback) require the real full
+ // view instead of treating that query-only pin as an empty table.
+ return super.getNameToPartitionItems(snapshot);
Review Comment:
[P1] Rehydrate this MTMV partition view before taking base-table locks.
`MTMVTask.beforeMTMVRefresh` now stores only the lightweight empty pin, but
`syncPartitionsIfNeeded` acquires every base table's read lock before
`alignMvPartition` reaches this live full-view call; `buildRefreshContext`
repeats the same path under the same locks. For an MV joining Hive with an
internal table, a slow or very large HMS listing therefore blocks metadata
writers/DDL on the internal table for the full RPC and O(all partitions) map
construction. This is distinct from the existing Nereids `planWithLock` thread:
it is the scheduled MTMV refresh path introduced while fixing the empty-pin
correctness issue. Capture and reuse the full Hive view before
`MetaLockUtils.readLockTables` (or otherwise keep connector I/O outside that
scope), and add a lock-timing regression test.
##########
fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanPlanProvider.java:
##########
@@ -200,7 +200,11 @@ public List<ConnectorScanRange> planScan(ConnectorSession
session, ConnectorScan
public boolean supportsBatchScan(ConnectorSession session,
ConnectorTableHandle handle) {
HiveTableHandle hiveHandle = (HiveTableHandle) handle;
List<String> partKeyNames = hiveHandle.getPartitionKeyNames();
- return partKeyNames != null && !partKeyNames.isEmpty() &&
!hiveHandle.isTransactional();
+ // A connector-filtered handle already carries the exact HMS Partition
objects. Keeping it on the normal
+ // scan path avoids reconstructing partition names from key/value
pairs and fetching them again per batch,
+ // which can lose the metastore's locale-sensitive canonical key
spelling.
+ return partKeyNames != null && !partKeyNames.isEmpty() &&
!hiveHandle.isTransactional()
+ && hiveHandle.getPrunedPartitions() == null;
Review Comment:
[P1] Keep large connector-pruned scans eligible for batched split
generation. Physical `applyFilter` installs this pruned handle before the
generic 1,024-partition gate runs, so returning false here makes `planScan`
synchronously list and split every surviving partition before producing any
work. A direct HMS result can contain 5,000 partitions, and the saturation
fallback can materialize an arbitrarily larger matched subset, so a valid
low-selectivity predicate can regress from bounded asynchronous batches to a
long planning stall and large FE allocation. Preserve the metastore-native
partition objects/identity in a batchable representation and add a
greater-than-threshold filtered-scan test that proves split discovery is
streamed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
