[
https://issues.apache.org/jira/browse/HDDS-1785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16887152#comment-16887152
]
Elek, Marton commented on HDDS-1785:
------------------------------------
Thanks the detailed analyses [~xudongcao]. As (If I understood well) you are
happy with the proposed approach, I committed it to the trunk (and 0.4.1) and
we can improve it further in other patches.
I tested the patch to generate 25M keys in k8s and worked well...
> OOM error in Freon due to the concurrency handling
> --------------------------------------------------
>
> Key: HDDS-1785
> URL: https://issues.apache.org/jira/browse/HDDS-1785
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Reporter: Elek, Marton
> Assignee: Doroszlai, Attila
> Priority: Blocker
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> HDDS-1532 modified the concurrent framework usage of Freon
> (RandomKeyGenerator).
> The new approach uses separated tasks (Runnable) to create the
> volumes/buckets/keys.
> Unfortunately it doesn't work very well in some cases.
> # When Freon starts it creates an executor with fixed number of threads (10)
> # The first loop submits numOfVolumes (10) VolumeProcessor tasks to the
> executor
> # The 10 threads starts to execute the 10 VolumeProcessor tasks
> # Each VolumeProcessor tasks creates numOfBuckets (1000) BucketProcessor
> tasks. All together 10000 tasks are submitted to the executor.
> # The 10 threads starts to execute the first 10 BucketProcessor tasks, they
> starts to create the KeyProcessor tasks: 500 000 * 10 tasks are submitted.
> # At this point of the time no keys are generated, but the next 10
> BucketProcessor tasks are started to execute..
> # To execute the first key creation we should process all the
> BucketProcessor tasks which means that all the Key creation tasks (10 * 1000
> * 500 000) are created and added to the executor
> # Which requires a huge amount of time and memory
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
