yujun777 opened a new pull request, #66488:
URL: https://github.com/apache/doris/pull/66488
### What problem does this PR solve?
Problem Summary:
Every column task deep-copies the whole `partitionUpdateRows` map from the
analyze job through the `AnalysisInfo` copy constructor. With a large number of
partitions this yields an O(Task x Partition) memory amplification: the copied
maps are retained by all history records in `analysisJobInfoMap` /
`analysisTaskInfoMap` (up to `analyze_record_limit` each), so a 4200-partition
table analyzed with many columns keeps tens of millions of `ConcurrentHashMap`
nodes alive in the FE heap.
This change makes tasks share the job's map reference instead of
deep-copying it, and clears the shared map once at the job terminal state, so
the memory held by every job/task record is released at once.
### Release note
None
### Check List (For Author)
- Test:
- [ ] BE UT
- [ ] FE UT (`AnalysisManagerTest` + `StatisticsAutoCollectorTest`, 20
cases passed)
- [ ] Regression Test
- [x] Manual Test
- Behavior changed: No
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
