github-actions[bot] commented on code in PR #67725:
URL: https://github.com/apache/doris/pull/67725#discussion_r4237135533
##########
fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java:
##########
@@ -2495,53 +2591,269 @@ static String hiveDateTimeString(LocalDateTime ldt) {
* Prunes partition names based on extracted equality predicates.
* Partition names follow the Hive convention: key1=val1/key2=val2
*/
- private List<String> prunePartitionNames(List<String> allPartNames,
- List<String> partKeyNames, Map<String, List<String>> predicates) {
+ private List<String> prunePartitionNames(List<String> allPartNames,
List<String> partKeyNames,
+ Map<String, String> partKeyTypes, Map<String, List<String>>
predicates) {
List<String> matched = new ArrayList<>();
for (String partName : allPartNames) {
Map<String, String> partValues = parsePartitionName(partName,
partKeyNames);
- if (matchesPredicates(partValues, predicates)) {
+ // A name this prefilter cannot decode is KEPT, never dropped: its
result becomes the logical
+ // selected view, so dropping a name only because the raw text was
not interpretable loses rows the
+ // query must read. The typed PartitionPruner re-prunes the
survivors, so a superset only costs the
+ // lost optimization.
+ if (partValues == null || matchesPredicates(partValues,
partKeyTypes, predicates)) {
matched.add(partName);
}
}
return matched;
}
+ private static String buildHmsPartitionFilter(List<String> partKeyNames,
Map<String, String> partKeyTypes,
+ Map<String, String> partKeyHiveTypes, Map<String, List<String>>
partitionPredicates) {
+ List<String> filters = new ArrayList<>();
+ for (String partKeyName : partKeyNames) {
+ List<String> values = partitionPredicates.get(partKeyName);
+ if (values == null || values.isEmpty()) {
+ continue;
+ }
+ if (!isHmsFilterIdentifier(partKeyName)) {
+ return null;
+ }
+ if (isHmsStringType(partKeyTypes.get(partKeyName))
+ && !isHmsStringType(partKeyHiveTypes.get(partKeyName))) {
+ return null;
+ }
+ List<String> valueFilters = new ArrayList<>();
+ for (String value : values) {
+ String literal = toHmsFilterLiteral(value,
partKeyTypes.get(partKeyName));
+ if (literal == null) {
+ return null;
+ }
+ valueFilters.add(partKeyName + " = " + literal);
Review Comment:
[P1] Decline direct HMS filtering for uppercase partition keys. An
HMS/API-created Hive table with STRING key `P` stores partition `p=x`, but this
line emits `P = 'x'` for `WHERE P = 'x'`. When HMS uses JDO (direct SQL
disabled or failed),
[ExpressionTree](https://github.com/apache/hive/blob/rel/release-3.1.3/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/parser/ExpressionTree.java#L373-L465)
accepts the key case-insensitively yet compares its `P=x` fragment with the
[stored lowercase
name](https://github.com/apache/hive/blob/rel/release-3.1.3/common/src/java/org/apache/hadoop/hive/common/FileUtils.java#L150-L160),
returning a successful empty result. Doris accepts it as the logical and
physical selected view, silently missing rows. Use the positional local
fallback for uppercase keys and test this JDO path. This differs from the
existing special-key local-fallback thread.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/mvcc/PluginDrivenMvccExternalTable.java:
##########
@@ -714,11 +692,142 @@ static boolean schemaCacheDisabled(Connector connector) {
@Override
public Map<String, PartitionItem>
getNameToPartitionItems(Optional<MvccSnapshot> snapshot) {
+ if (supportsConnectorPartitionPruning()) {
+ PluginDrivenMvccSnapshot pin = getOrMaterialize(snapshot);
+ if (pin.isPartitionViewMaterialized()) {
+ return pin.getNameToPartitionItem();
+ }
+ if (pin.isPartitionViewUnavailable()) {
+ return Collections.emptyMap();
+ }
+ Optional<ExternalTablePreloadInfo> sharedView =
sharedLatestPreloadInfo(snapshot);
+ if (sharedView.isPresent() &&
sharedView.get().hasScanPartitionView()
+ && sharedView.get().getScanPartitionView().isPresent()) {
+ return sharedView.get().getScanPartitionView().get();
+ }
+ // The latest Hive query pin intentionally carries no partition
map so selective scans can send a
+ // predicate to HMS first. Consumers that explicitly ask for a
partition map (MTMV alignment,
+ // no-filter scan finalization, and a connector-declined pruning
fallback) require the real full
+ // view instead of treating that query-only pin as an empty table.
An unavailable shared scan view
+ // is not an authoritative empty map either: checked MTMV
consumers reject it in
+ // getAndCopyPartitionItems, while other explicit map consumers
retain this fail-loud full-view path.
+ return super.getNameToPartitionItems(snapshot);
+ }
return getOrMaterialize(snapshot).getNameToPartitionItem();
}
@Override
- public Map<String, PartitionItem>
getAndCopyPartitionItems(Optional<MvccSnapshot> snapshot) {
+ public Optional<Map<String, PartitionItem>>
getNameToPartitionItemsForScan(Optional<MvccSnapshot> snapshot) {
+ if (snapshot.isPresent() && snapshot.get() instanceof
PluginDrivenMvccSnapshot
+ && ((PluginDrivenMvccSnapshot)
snapshot.get()).isPartitionViewUnavailable()) {
+ return Optional.empty();
+ }
+ if (supportsConnectorPartitionPruning()) {
+ PluginDrivenMvccSnapshot pin = getOrMaterialize(snapshot);
+ if (pin.isPartitionViewMaterialized()) {
+ // This statement's pin already carries a materialized view:
reuse it instead of paying another
+ // connector round-trip that could observe a different remote
generation.
+ return Optional.of(pin.getNameToPartitionItem());
+ }
+ }
+ return super.getNameToPartitionItemsForScan(snapshot);
+ }
+
+ @Override
+ public Optional<ConnectorFilteredPartitionView>
applyPartitionFilterForScan(Optional<MvccSnapshot> snapshot,
+ ConnectorExpression partitionFilter) {
+ if (snapshot.isPresent() && snapshot.get() instanceof
PluginDrivenMvccSnapshot
+ && ((PluginDrivenMvccSnapshot)
snapshot.get()).isPartitionViewUnavailable()) {
+ return Optional.empty();
+ }
+ return super.applyPartitionFilterForScan(snapshot, partitionFilter);
+ }
+
+ /**
+ * Threads this statement's MVCC pin onto the handle the partition view is
enumerated from, so a
+ * time-travel / {@code @options} query never prunes against the latest
generation: the data scan reads the
+ * pinned snapshot, and a latest view can be missing (or contain)
partitions it will never read.
+ */
+ @Override
+ protected ConnectorTableHandle pinPartitionViewHandle(ConnectorTableHandle
handle,
+ ConnectorMetadata metadata, ConnectorSession session,
Optional<MvccSnapshot> snapshot) {
+ if (snapshot.isPresent() && snapshot.get() instanceof
PluginDrivenMvccSnapshot) {
+ return metadata.applySnapshot(session, handle,
+ ((PluginDrivenMvccSnapshot)
snapshot.get()).getConnectorSnapshot());
+ }
+ return handle;
+ }
+
+ /**
+ * Materializes the complete partition view for an MTMV refresh before it
acquires base-table locks.
+ * The lightweight Hive query pin keeps the connector snapshot/freshness
kind but deliberately omits the
+ * partition map; MTMV alignment needs that map and must not load it while
holding internal table locks.
+ */
+ public MvccSnapshot materializePartitionViewForMtmv(MvccSnapshot snapshot)
{
+ if (!supportsConnectorPartitionPruning() || !(snapshot instanceof
PluginDrivenMvccSnapshot)) {
+ return snapshot;
+ }
+ PluginDrivenMvccSnapshot pin = (PluginDrivenMvccSnapshot) snapshot;
+ if (!pin.isPartitionViewDeferred()) {
+ return pin;
+ }
+ Map<String, PartitionItem> partitionItems =
super.getNameToPartitionItems(Optional.of(pin));
Review Comment:
[P1] Preserve per-partition degradation in the scheduled MTMV warmup.
`MTMVTask.beforeMTMVRefresh` invokes this new full-view call for a partitioned
Hive base even when the MV is `SELF_MANAGE`. An HMS/API-created DATE partition
`dt=not-a-date` makes `TablePartitionValues.addPartitions` throw
`CacheException`, aborting the task before its whole-MV refresh. Before this
change, `loadSnapshot` built items with a per-partition catch in
`listPartitions`, marked the pin invalid, and let a `SELF_MANAGE` MTMV continue
with a scan-all refresh. Use that failure contract here and test a refresh with
one unrepresentable typed partition. The existing malformed-partition thread
concerns scan planning; this is the new scheduled-refresh caller.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
