[
https://issues.apache.org/jira/browse/YARN-10276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17117694#comment-17117694
]
Gergely Pollak commented on YARN-10276:
---------------------------------------
I've conducted the following test:
Replaced the actual queue requirement, with a dummy queue, which stores only
name, parent object, and full path. And made a test storing 10 000 of these
queues, the memory overhead for storing the queues was less than the memory
used by the 10 000 dummy queues, which had only 2 strings and an object
reference, the actual leaf queue object would use much more memory, since it
has a lot more references and internal data structures, so the storage overhead
will diminish compared to the actual object's memory consumption.
The total consumption for the 10 000 queues and store was 20MB, which is a bit
large, but this use case is a bit extreme, and for an RM with over 10 000
queues, we will need a lot of RAM anyway, where the 20MB won't matter much.
> Check and improve memory footprint of CapacityScheduler CSQueueStore
> --------------------------------------------------------------------
>
> Key: YARN-10276
> URL: https://issues.apache.org/jira/browse/YARN-10276
> Project: Hadoop YARN
> Issue Type: Task
> Reporter: Gergely Pollak
> Assignee: Gergely Pollak
> Priority: Major
>
> The class creates a lot of Set instances, which might have a bit bigger
> memory overhead than necessary, this might be not a critical issue, but let's
> examine if we can or should create a more memory efficient solution while
> keeping the performance.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
