[
https://issues.apache.org/jira/browse/YARN-3139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15491678#comment-15491678
]
Wangda Tan commented on YARN-3139:
----------------------------------
I just tried to use SLS to benchmark the performance after applied patches:
YARN-3139/YARN-3140/YARN-3141, I created a mini framework to run the comparison
tests: https://github.com/leftnoteasy/yarn_application_synthesizer.
Because of some unknown issue, I cannot run the benchmark with FairScheduler,
filed YARN-5654. Capacity scheduler runs fine.
Overview of test data:
- 3000+ applications, ~1000 applications running in parallel
- 3000 NMs, each node configured 128G capacity
- Size of containers ranges from 8G to 32G
- Lifespan of containers ranges from 5 sec to 2 mins
- Total 245,301 containers launched
- Total time spent on the test is < 10 mins (can increase #apps / #containers
to increase the total time), thus, there're around 500 containers allocated for
each second.
For original CS, time:
{code}
real 8m51.774s
user 8m41.998s
sys 1m28.158s
{code}
For CS with R/W locking changes:
{code}
real 8m48.426s
user 8m31.398s
sys 1m29.861s
{code}
Summary:
- No regression in performance, didn't see deadlock happens.
- No significant performance improvement either, because existing scheduler
allocation is still in single thread.
Hope this can with code review.
cc: [~jianhe], [~kasha], [~templedf], [~vinodkv], [~jlowe].
> Improve locks in AbstractYarnScheduler/CapacityScheduler/FairScheduler
> ----------------------------------------------------------------------
>
> Key: YARN-3139
> URL: https://issues.apache.org/jira/browse/YARN-3139
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager, scheduler
> Reporter: Wangda Tan
> Assignee: Wangda Tan
> Attachments: YARN-3139.0.patch
>
>
> Enhance locks in AbstractYarnScheduler/CapacityScheduler/FairScheduler, as
> mentioned in YARN-3091, a possible solution is using read/write lock. Other
> fine-graind locks for specific purposes / bugs should be addressed in
> separated tickets.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
