SWJTU-ZhangLei commented on code in PR #22889:
URL: https://github.com/apache/doris/pull/22889#discussion_r1439181237
##########
fe/fe-core/src/main/java/org/apache/doris/load/loadv2/LoadManager.java:
##########
@@ -90,7 +92,7 @@
public class LoadManager implements Writable {
private static final Logger LOG = LogManager.getLogger(LoadManager.class);
- private Map<Long, LoadJob> idToLoadJob = Maps.newConcurrentMap();
+ private Map<Long, LoadJob> idToLoadJob =
Collections.synchronizedMap(Maps.newLinkedHashMap());
private Map<Long, Map<String, List<LoadJob>>> dbIdToLabelToLoadJobs =
Maps.newConcurrentMap();
Review Comment:
Why replace concurrentMap with synchronizedMap ? what is the advantages and
disadvantages?
##########
fe/fe-core/src/main/java/org/apache/doris/load/loadv2/LoadManager.java:
##########
@@ -399,13 +401,26 @@ public long getLoadJobNum(JobState jobState, EtlJobType
jobType) {
**/
public void removeOldLoadJob() {
long currentTimeMs = System.currentTimeMillis();
+ removeLoadJobIf(job -> job.isExpired(currentTimeMs), -1);
+ }
+
+ /**
+ * Remove completed jobs if total job num exceed Config.label_num_threshold
+ */
+ public void removeOverLimitLoadJob() {
+ if (idToLoadJob.size() <= Config.label_num_threshold) {
+ return;
+ }
+ removeLoadJobIf(LoadJob::isCompleted, Config.label_num_threshold);
+ }
+ private void removeLoadJobIf(Predicate<LoadJob> pred, int numLimit) {
writeLock();
try {
Iterator<Map.Entry<Long, LoadJob>> iter =
idToLoadJob.entrySet().iterator();
- while (iter.hasNext()) {
+ while (iter.hasNext() && idToLoadJob.size() > numLimit) {
Review Comment:
do we need think about sorting by finishTime ? I think we keep the last
recently label_num_threshold final job maybe better
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
