yujun777 commented on code in PR #68648:
URL: https://github.com/apache/doris/pull/68648#discussion_r4280800429
##########
fe/fe-core/src/main/java/org/apache/doris/job/extensions/mtmv/MTMVTask.java:
##########
@@ -1290,6 +1290,13 @@ private void
executePartitionBasedRefresh(MTMVRefreshContext context, RefreshMod
int end = start + refreshPartitionNum;
Set<String> execPartitionNames = Sets.newHashSet(partitions
.subList(start, Math.min(end, partitions.size())));
+ // What this batch reads and what it records are the same set,
both decided here from the one
+ // mapping the snapshots below are generated from; see
MTMVPartitionUtil#mappedBasePartitions.
+ Map<BaseTableInfo, Set<String>> readableBasePartitions =
mtmv.isIvm()
+ ? null :
MTMVPartitionUtil.mappedBasePartitions(tableWithPartKey, context,
execPartitionNames);
Review Comment:
Confirmed by reading the three links, and it goes on the follow-up list with
the external-PCT thread above rather than into this change, because closing it
is a decision about the mapping that the two share:
- `MTMV#calculatePartitionMappings` gives an MV partition that no base
partition describes an entry in `undescribedMvPartitions` and an empty inner
map, so the mapping names no table for it at all;
- `MTMVTask#collectPctResetPartitionIds` walks only the tables the mapping
names, so a PCT table all of whose partitions the window left out has no entry
there;
- `IvmFullRefreshMTMV#rewriteScan` then reads
`getFullRefreshResetPartitionIds` as empty for that table, and since it is a
PCT table it returns the ordinary scan -- the batch reads it and records
nothing.
Both ways to close it change what the mapping means, which is why I would
rather settle it with a test than here: scoping that read to the batch's
mapping makes the MV partition lose rows the join does produce for the keys it
does cover, while recording what the read takes makes the mapping -- and the
incremental delta that reads it -- admit a partition the window dropped. That
is the same fork as the external-PCT finding above, whose read is unscoped for
the same reason (`mappedBasePartitions` scopes `OlapTable` only), so both want
a COMPLETE -> DELETE -> INCREMENTAL case on a merged component before either
side is picked.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
