github-actions[bot] commented on code in PR #68995:
URL: https://github.com/apache/doris/pull/68995#discussion_r4266624323
##########
fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudTabletRebalancer.java:
##########
@@ -1775,48 +1805,48 @@ private void handleWarmupCompletion(InfightTask task,
String clusterId, boolean
}
}
- private void updateBeToTablets(Long tabletId, Long srcBe, Long destBe,
- ConcurrentHashMap<Long, Set<Long>>
globalBeToTablets,
- ConcurrentHashMap<Long,
ConcurrentHashMap<Long, Set<Long>>> beToTabletsInTable,
- ConcurrentHashMap<Long,
ConcurrentHashMap<Long, ConcurrentHashMap<Long,
- Set<Long>>>> partToTablets) {
+ private void updateBeToTablets(Long tabletId, Long srcBe, Long destBe,
String clusterId,
+ RouteVariant variant,
ConcurrentHashMap<Long, Set<Long>> globalBeToTablets) {
TabletMeta tabletMeta =
Env.getCurrentEnv().getTabletInvertedIndex().getTabletMeta(tabletId);
if (tabletMeta == null) {
LOG.warn("tablet {} meta not found in inverted index, skip
updateBeToTablets", tabletId);
return;
}
Long tableId = tabletMeta.getTableId();
- Long partId = tabletMeta.getPartitionId();
- Long indexId = tabletMeta.getIndexId();
-
Set<Long> globalSrcTablets = globalBeToTablets.get(srcBe);
if (globalSrcTablets == null || !globalSrcTablets.remove(tabletId)) {
- LOG.debug("skip updateBeToTablets for tablet {}: srcBe {} not in
globalBeToTablets", tabletId, srcBe);
- return;
+ // A live view can be newer than the round-global route snapshot.
+ for (Long be :
Preconditions.checkNotNull(clusterToBes.get(clusterId))) {
+ Set<Long> tablets = globalBeToTablets.get(be);
+ if (tablets != null) {
+ tablets.remove(tabletId);
+ }
+ }
}
- ConcurrentHashMap<Long, Set<Long>> tableBeMap =
beToTabletsInTable.get(tableId);
- if (tableBeMap == null) {
- return;
+ if (destBe != null) {
+ fillGlobalBeToTablets(destBe, tabletId,
DEFAULT_GLOBAL_TABLET_SET_FACTORY, globalBeToTablets);
}
- Set<Long> tableSrcTablets = tableBeMap.get(srcBe);
- if (tableSrcTablets != null) {
- tableSrcTablets.remove(tabletId);
+ if (clusterId.equals(tableCountClusterId)) {
+ Map<Long, Integer> counts = (variant == RouteVariant.CURRENT ?
currentTableCounts : futureTableCounts)
+ .get(tableId);
+ if (counts != null) {
+ counts.merge(srcBe, -1, Integer::sum);
+ if (destBe != null) {
+ counts.merge(destBe, 1, Integer::sum);
+ }
+ }
}
-
- ConcurrentHashMap<Long, ConcurrentHashMap<Long, Set<Long>>> indexMap =
partToTablets.get(partId);
- if (indexMap != null) {
- ConcurrentHashMap<Long, Set<Long>> beMap = indexMap.get(indexId);
- if (beMap != null) {
- Set<Long> partSrcTablets = beMap.get(srcBe);
- if (partSrcTablets != null) {
- partSrcTablets.remove(tabletId);
+ if (clusterId.equals(indexCountClusterId)) {
+ Map<Long, Integer> counts = (variant == RouteVariant.CURRENT ?
currentIndexCounts : futureIndexCounts)
+ .get(Pair.of(tabletMeta.getPartitionId(),
tabletMeta.getIndexId()));
+ if (counts != null) {
+ counts.merge(srcBe, -1, Integer::sum);
Review Comment:
[P2] Derive CURRENT cache deltas from the actual primary
After a switch from `async_warmup` to `peer_read_async_warmup`, the checker
retains old tasks, so a tablet primary on A but warming to B can be picked from
FUTURE B for a direct B-to-C transfer. This CURRENT update decrements B
although the actual current source is A. Reconciling global membership does not
make that delta correct.
With partition balancing disabled and `cloud_rebalance_number_threshold=1`,
use four BEs A/B/C/D and one table whose partitions have CURRENT counts
`(4,3,2,0)`, `(1,2,0,0)`, `(1,0,0,4)`. Let one A tablet in the first partition
be warming to B; TABLE FUTURE totals are `(5,6,2,4)` and the default budget
permits four attempts. Choose B-to-C and A-to-C in the target, B-to-A in the
second partition, then A-to-B in the target. Before the fourth move, new
CURRENT target counts are `(3,2,4,0)` but actual counts are `(2,3,4,0)`; the
old current graph is `(3,3,3,0)` and rejects that move. The new check permits
it, worsening actual placement to `(1,4,4,0)`.
Derive CURRENT deltas from the actual primary, or rebuild/invalidate that
cache when the selected FUTURE source differs; apply this to table counts too.
Add a residual-task mode-change test. This instance uses an uncanceled task and
CURRENT counts, separate from the earlier FUTURE cancellation issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
