Copilot commented on code in PR #61930:
URL: https://github.com/apache/doris/pull/61930#discussion_r3013825477
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/CloudTabletStatMgr.java:
##########
@@ -360,7 +367,31 @@ private GetTabletStatsResponse
getTabletStats(GetTabletStatsRequest request)
return response;
}
- public Map<Pair<Long, Long>, OlapTable.Statistics> getCloudTableStatsMap()
{
- return this.cloudTableStatsMap;
+ public List<OlapTable.Statistics> getCloudTableStats() {
+ return this.cloudTableStatsList;
+ }
Review Comment:
getCloudTableStats() returns the internal mutable List instance. To prevent
accidental external mutation (and potential ConcurrentModificationException
during iteration), consider returning an unmodifiable view or a defensive copy,
and/or storing an immutable list in the field.
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/CloudTabletStatMgr.java:
##########
@@ -149,20 +128,49 @@ protected void runAfterCatalogReady() {
LOG.info("finished to get tablet stat of all backends. cost: {} ms",
(System.currentTimeMillis() - start));
+ return dbIds;
+ }
+ private Future<Void> submitGetTabletStatsTask(GetTabletStatsRequest req) {
+ return GET_TABLET_STATS_THREAD_POOL.submit(() -> {
+ GetTabletStatsResponse resp;
+ try {
+ resp = getTabletStats(req);
+ } catch (RpcException e) {
+ LOG.warn("get tablet stats exception:", e);
+ return null;
+ }
+ if (resp.getStatus().getCode() != MetaServiceCode.OK) {
+ LOG.warn("get tablet stats return failed.");
+ return null;
+ }
+ if (LOG.isDebugEnabled()) {
+ int i = 0;
+ for (TabletIndexPB idx : req.getTabletIdxList()) {
+ LOG.debug("db_id: {} table_id: {} index_id: {} tablet_id:
{} size: {}",
+ idx.getDbId(), idx.getTableId(), idx.getIndexId(),
+ idx.getTabletId(),
resp.getTabletStats(i++).getDataSize());
+ }
+ }
+ updateTabletStat(resp);
+ return null;
+ });
+ }
+
+ private void updateStatInfo(List<Long> dbIds) {
// after update replica in all backends, update index row num
- start = System.currentTimeMillis();
+ long start = System.currentTimeMillis();
Pair<String, Long> maxTabletSize = Pair.of(/* tablet id= */null, /*
byte size= */0L);
Pair<String, Long> maxPartitionSize = Pair.of(/* partition id= */null,
/* byte size= */0L);
Pair<String, Long> maxTableSize = Pair.of(/* table id= */null, /* byte
size= */0L);
Pair<String, Long> minTabletSize = Pair.of(/* tablet id= */null, /*
byte size= */Long.MAX_VALUE);
Pair<String, Long> minPartitionSize = Pair.of(/* partition id= */null,
/* byte size= */Long.MAX_VALUE);
Pair<String, Long> minTableSize = Pair.of(/* tablet id= */null, /*
byte size= */Long.MAX_VALUE);
- Long totalTableSize = 0L;
- Long tabletCount = 0L;
- Long partitionCount = 0L;
- Long tableCount = 0L;
- Map<Pair<Long, Long>, OlapTable.Statistics> newCloudTableStatsMap =
new HashMap<>();
+ long totalTableSize = 0L;
+ long tabletCount = 0L;
+ long partitionCount = 0L;
+ long tableCount = 0L;
+ List<OlapTable.Statistics> newCloudTableStatsList = new ArrayList<>();
for (Long dbId : dbIds) {
Review Comment:
updateStatInfo() collects statistics for all tables into
newCloudTableStatsList and only filters to the top-N at the end. If the number
of tables is very large, this can still create a large temporary list/array
each stat cycle. Consider maintaining the top-N PriorityQueue incrementally
during the scan so you never retain references for every table at once.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
