[
https://issues.apache.org/jira/browse/YARN-8995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16862866#comment-16862866
]
Tao Yang edited comment on YARN-8995 at 6/13/19 9:29 AM:
---------------------------------------------------------
I did a simple test (details in TestStreamPerf.java) on performance comparison
between sequential stream and parallel stream in a similar scenario: count a
blocking queue with 100 distinct keys and 1w/10w/100w/200w total length, it
seems that parallel stream indeed lead to more overhead than sequential stream,
results of this test are as follows (suffix "_S" refers to sequential stream
and suffix "_PS" refers to parallel stream):
{noformat}
TestStreamPerf.test_100_100w_PS: [measured 10 out of 15 rounds, threads: 1
(sequential)]
round: 0.03 [+- 0.00], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00],
GC.calls: 1, GC.time: 0.01, time.total: 0.64, time.warmup: 0.31, time.bench:
0.32
TestStreamPerf.test_100_100w_S: [measured 10 out of 15 rounds, threads: 1
(sequential)]
round: 0.02 [+- 0.00], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00],
GC.calls: 0, GC.time: 0.00, time.total: 0.37, time.warmup: 0.15, time.bench:
0.22
TestStreamPerf.test_100_10w_PS: [measured 10 out of 15 rounds, threads: 1
(sequential)]
round: 0.00 [+- 0.00], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00],
GC.calls: 0, GC.time: 0.00, time.total: 0.08, time.warmup: 0.05, time.bench:
0.04
TestStreamPerf.test_100_10w_S: [measured 10 out of 15 rounds, threads: 1
(sequential)]
round: 0.00 [+- 0.00], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00],
GC.calls: 0, GC.time: 0.00, time.total: 0.04, time.warmup: 0.01, time.bench:
0.03
TestStreamPerf.test_100_1w_PS: [measured 10 out of 15 rounds, threads: 1
(sequential)]
round: 0.00 [+- 0.00], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00],
GC.calls: 0, GC.time: 0.00, time.total: 0.01, time.warmup: 0.00, time.bench:
0.01
TestStreamPerf.test_100_1w_S: [measured 10 out of 15 rounds, threads: 1
(sequential)]
round: 0.00 [+- 0.00], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00],
GC.calls: 0, GC.time: 0.00, time.total: 0.01, time.warmup: 0.00, time.bench:
0.00
TestStreamPerf.test_100_200w_PS: [measured 10 out of 15 rounds, threads: 1
(sequential)]
round: 0.07 [+- 0.00], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00],
GC.calls: 0, GC.time: 0.00, time.total: 1.03, time.warmup: 0.37, time.bench:
0.66
TestStreamPerf.test_100_200w_S: [measured 10 out of 15 rounds, threads: 1
(sequential)]
round: 0.04 [+- 0.00], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00],
GC.calls: 0, GC.time: 0.00, time.total: 0.70, time.warmup: 0.25, time.bench:
0.45
{noformat}
was (Author: tao yang):
I did a simple test on performance comparison between sequential stream and
parallel stream in a similar scenario: count a blocking queue with 100 distinct
keys and 1w/10w/100w/200w total length, it seems that parallel stream indeed
lead to more overhead than sequential stream, results of this test are as
follows (suffix "_S" refers to sequential stream and suffix "_PS" refers to
parallel stream):
{noformat}
TestStreamPerf.test_100_1w_S: [measured 10 out of 15 rounds, threads: 1
(sequential)]
round: 0.00 [+- 0.00], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00],
GC.calls: 0, GC.time: 0.00, time.total: 0.00, time.warmup: 0.00, time.bench:
0.00
TestStreamPerf.test_100_1w_PS: [measured 10 out of 15 rounds, threads: 1
(sequential)]
round: 0.00 [+- 0.00], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00],
GC.calls: 0, GC.time: 0.00, time.total: 0.01, time.warmup: 0.00, time.bench:
0.01
TestStreamPerf.test_100_10w_S: [measured 10 out of 15 rounds, threads: 1
(sequential)]
round: 0.00 [+- 0.00], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00],
GC.calls: 0, GC.time: 0.00, time.total: 0.04, time.warmup: 0.01, time.bench:
0.03
TestStreamPerf.test_100_10w_PS: [measured 10 out of 15 rounds, threads: 1
(sequential)]
round: 0.00 [+- 0.00], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00],
GC.calls: 0, GC.time: 0.00, time.total: 0.14, time.warmup: 0.09, time.bench:
0.05
TestStreamPerf.test_100_100w_S: [measured 10 out of 15 rounds, threads: 1
(sequential)]
round: 0.03 [+- 0.00], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00],
GC.calls: 0, GC.time: 0.00, time.total: 0.43, time.warmup: 0.17, time.bench:
0.26
TestStreamPerf.test_100_100w_PS: [measured 10 out of 15 rounds, threads: 1
(sequential)]
round: 0.04 [+- 0.01], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00],
GC.calls: 0, GC.time: 0.00, time.total: 0.56, time.warmup: 0.20, time.bench:
0.36
TestStreamPerf.test_100_200w_S: [measured 10 out of 15 rounds, threads: 1
(sequential)]
round: 0.05 [+- 0.00], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00],
GC.calls: 0, GC.time: 0.00, time.total: 0.75, time.warmup: 0.25, time.bench:
0.50
TestStreamPerf.test_100_200w_PS: [measured 10 out of 15 rounds, threads: 1
(sequential)]
round: 0.07 [+- 0.01], round.block: 0.01 [+- 0.00], round.gc: 0.00 [+- 0.00],
GC.calls: 0, GC.time: 0.00, time.total: 1.06, time.warmup: 0.35, time.bench:
0.71
{noformat}
> Log the event type of the too big AsyncDispatcher event queue size, and add
> the information to the metrics.
> ------------------------------------------------------------------------------------------------------------
>
> Key: YARN-8995
> URL: https://issues.apache.org/jira/browse/YARN-8995
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: metrics, nodemanager, resourcemanager
> Affects Versions: 3.2.0
> Reporter: zhuqi
> Assignee: zhuqi
> Priority: Major
> Attachments: YARN-8995.001.patch, YARN-8995.002.patch,
> YARN-8995.003.patch
>
>
> In our growing cluster，there are unexpected situations that cause some event
> queues to block the performance of the cluster, such as the bug of
> https://issues.apache.org/jira/browse/YARN-5262 . I think it's necessary to
> log the event type of the too big event queue size, and add the information
> to the metrics, and the threshold of queue size is a parametor which can be
> changed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
