[
https://issues.apache.org/jira/browse/HDDS-1785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16886777#comment-16886777
]
Xudong Cao commented on HDDS-1785:
----------------------------------
sizeof VolumeProcessor: 2621008
> OOM error in Freon due to the concurrency handling
> --------------------------------------------------
>
> Key: HDDS-1785
> URL: https://issues.apache.org/jira/browse/HDDS-1785
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Reporter: Elek, Marton
> Assignee: Doroszlai, Attila
> Priority: Blocker
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> HDDS-1532 modified the concurrent framework usage of Freon
> (RandomKeyGenerator).
> The new approach uses separated tasks (Runnable) to create the
> volumes/buckets/keys.
> Unfortunately it doesn't work very well in some cases.
> # When Freon starts it creates an executor with fixed number of threads (10)
> # The first loop submits numOfVolumes (10) VolumeProcessor tasks to the
> executor
> # The 10 threads starts to execute the 10 VolumeProcessor tasks
> # Each VolumeProcessor tasks creates numOfBuckets (1000) BucketProcessor
> tasks. All together 10000 tasks are submitted to the executor.
> # The 10 threads starts to execute the first 10 BucketProcessor tasks, they
> starts to create the KeyProcessor tasks: 500 000 * 10 tasks are submitted.
> # At this point of the time no keys are generated, but the next 10
> BucketProcessor tasks are started to execute..
> # To execute the first key creation we should process all the
> BucketProcessor tasks which means that all the Key creation tasks (10 * 1000
> * 500 000) are created and added to the executor
> # Which requires a huge amount of time and memory
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
