imay commented on a change in pull request #846: Add report queue size limit to
avoid too many report requests
URL: https://github.com/apache/incubator-doris/pull/846#discussion_r270761298
##########
File path: fe/src/main/java/org/apache/doris/master/ReportHandler.java
##########
@@ -84,7 +86,30 @@
private BlockingQueue<ReportTask> reportQueue =
Queues.newLinkedBlockingQueue();
+ // This threshold is to avoid piling up too many report task in FE, which
may cause OOM exception.
+ // In some large Doris cluster, eg: 100 Backends with ten million
replicas, a tablet report may cost
+ // several seconds after some modification of metadata(drop partition,
etc..).
+ // And one Backend will report tablets info every 1 min, so unlimited
receiving reports is unacceptable.
+ // TODO(cmy): we will optimize the processing speed of tablet report in
future, but now, just discard
+ // the report if queue size exceeding limit.
+ // Some online time cost:
+ // 1. disk report: 0-1 ms
+ // 2. task report: 0-1 ms
+ // 3. tablet report
+ // 10000 replicas: 200ms
+ private final static int MAX_QUEUE_SIZE = 100;
Review comment:
I think a config is better
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
