yujun777 commented on code in PR #68648:
URL: https://github.com/apache/doris/pull/68648#discussion_r4278417177
##########
fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRelatedPartitionDescTransferGenerator.java:
##########
@@ -46,17 +46,27 @@ public class MTMVRelatedPartitionDescTransferGenerator
implements MTMVRelatedPar
public void apply(MTMVPartitionInfo mvPartitionInfo, Map<String, String>
mvProperties,
RelatedPartitionDescResult lastResult, List<Column>
partitionColumns,
Map<List<String>, Set<String>> queryUsedPartitionMap)
throws AnalysisException {
- Map<MTMVRelatedTableIf, Map<PartitionKeyDesc, Set<String>>> descs =
lastResult.getDescs();
+ Map<PartitionKeyDesc, Map<MTMVRelatedTableIf, Set<String>>> merged =
+ mergeOverlappingListDescs(lastResult.getDescs());
+ // Now that the keys are grouped -- across the tables of the MV, and
before any query filter -- the
+ // partitions the query does not read are dropped per table, and an MV
partition no table of which
+ // feeds the query is dropped whole: what the query reads is what is
left.
Map<PartitionKeyDesc, Map<MTMVRelatedTableIf, Set<String>>> res =
Maps.newHashMap();
- for (Entry<MTMVRelatedTableIf, Map<PartitionKeyDesc, Set<String>>>
entry : descs.entrySet()) {
- MTMVRelatedTableIf pctTable = entry.getKey();
- Map<PartitionKeyDesc, Set<String>> onePctDescs = entry.getValue();
- for (Entry<PartitionKeyDesc, Set<String>> onePctEntry :
onePctDescs.entrySet()) {
- PartitionKeyDesc partitionKeyDesc = onePctEntry.getKey();
- Set<String> partitionNames = onePctEntry.getValue();
- Map<MTMVRelatedTableIf, Set<String>> partitionKeyDescMap =
res.computeIfAbsent(partitionKeyDesc,
- k -> new HashMap<>());
- partitionKeyDescMap.put(pctTable, partitionNames);
+ for (Entry<PartitionKeyDesc, Map<MTMVRelatedTableIf, Set<String>>>
entry : merged.entrySet()) {
+ Map<MTMVRelatedTableIf, Set<String>> onePartition =
Maps.newHashMap();
+ for (Entry<MTMVRelatedTableIf, Set<String>> tableEntry :
entry.getValue().entrySet()) {
+ Set<String> names = Sets.newHashSet(tableEntry.getValue());
+ Set<String> queryUsed =
queryUsedPartitionMap.get(tableEntry.getKey().getFullQualifiers());
+ if (queryUsed != null) {
+ names.retainAll(queryUsed);
Review Comment:
Fixed in 1c3645aacdd. A component the query reads keeps every partition it
was recorded with, and the query's partitions only select which component is
read (one it reads none of is still left out), so the name set the freshness
check compares to the snapshot is the recorded one.
`MTMVRelatedPartitionDescGeneratorTest`'s pruned-query cases now expect the
component's members (`{p_single, p_double}`, and `{t15_p1, t15_p23}` for the
cross-table one).
End to end, `mtmv_p0/test_mtmv_base_partition_read_scope` gains
`merged_component`: `p_single = ((1,'EU'))` and `p_double =
((1,'US'),(2,'US'))` over a LIST(d, region) base, a fresh MV, and the ordinary
`WHERE d = 2` -- the rewrite is asserted through the same
`is_partition_statistics_ready` guard the suite uses, and measured the other
way it fails at that assertion with the narrowing in place.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
