devmadhuu commented on code in PR #9584:
URL: https://github.com/apache/ozone/pull/9584#discussion_r2719740559
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/DataNodeMetricsService.java:
##########
@@ -294,16 +300,23 @@ private void resetState() {
totalNodesFailed = 0;
}
- public DataNodeMetricsServiceResponse getCollectedMetrics() {
+ public DataNodeMetricsServiceResponse getCollectedMetrics(Integer limit) {
Review Comment:
@priyeshkaratha yes it may have performance issues if number of DNs are high
and if some DN may respond slow and 10min is quite high for timeout
[here](https://github.com/apache/ozone/blob/8eaca4c3bf2972a7e9f84062ff4c1d49fa2ea6db/hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/DataNodeMetricsService.java#L191),
because we wait for all futures to be completed, and in worst case 10 mins can
be the time taken for this all futures to be completed and return the pending
blocks data. Now if we are displaying top 10 or top 20 datanodes sorted by size
(descending), then we could have implemented the above collectMetrics data in
such a way that instead of waiting for all datanodes to return the data after
their futures is completed, we could have started pushing their results in
queue and another thread (any client program) who needs this data like API end
point here could just start picking results from that queue and can implement
priority queue to start sorting for to
p 10 or 20, so this way I think memory footprint will be less and since on UI,
we display only top 10 or top 20, then backend logic based on priority can
keep only that much and keep discarding remaining. And UI should have a scroll
dynamic logic that should send request to backend if user goes beyond those top
10 or top 20. But we can have this logic later sometime in future and assume
that in case of large clusters where 1000+ datanodes can be there, jmx response
from all datanodes can come quickly or else worst 10 mins.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
