[
https://issues.apache.org/jira/browse/YARN-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13932623#comment-13932623
]
Zhijie Shen commented on YARN-1530:
-----------------------------------
I've done some stress test about the timeline service. I've taken two steps:
1. I setup a yarn cluster of 5 nodes (1 master and 4 slaves). I ran 7200
mapreduce example jobs with tez framework (which will post tez entities to the
timeline service) in 10 hours. The max concurrent job was 7, because it was not
a big cluster with 32G memory only, but the real concurrency should be higher
because of multiple mappers/reducers. The workload was kept almost full in 10
hours. All the job were succeeded, and the tez entities were stored in the
timeline store without exceptions. The leveldb based timeline store has grown
to about 220MB (not very big because of small example jobs).
2. Afterwards, I tested the the concurrent reads/writes together. On the write
part, I did the same thing as step 1. On the read part, I set up 4 timeline
query clients, one on each slave node. Each client starts 10 parallel threads
to send requests to the timeline service for 10 hours as well. Each client sent
more than 6 million queries during the 10 hours with the combination of three
RESTful APIs (24+ million total for 4 clients). In general, the timeline
service was still working well. I just saw one query was responded with not
found exception, and some other JVM warnings. The query of get entities takes
0.X on average while the query of get entity/events take 0.0x.
Therefore, the timeline service with leveldb store works so far so good. I'll
do more stress testing with big entity, and update to you once I've some
metrics.
> [Umbrella] Store, manage and serve per-framework application-timeline data
> --------------------------------------------------------------------------
>
> Key: YARN-1530
> URL: https://issues.apache.org/jira/browse/YARN-1530
> Project: Hadoop YARN
> Issue Type: Bug
> Reporter: Vinod Kumar Vavilapalli
> Attachments: application timeline design-20140108.pdf, application
> timeline design-20140116.pdf, application timeline design-20140130.pdf,
> application timeline design-20140210.pdf
>
>
> This is a sibling JIRA for YARN-321.
> Today, each application/framework has to do store, and serve per-framework
> data all by itself as YARN doesn't have a common solution. This JIRA attempts
> to solve the storage, management and serving of per-framework data from
> various applications, both running and finished. The aim is to change YARN to
> collect and store data in a generic manner with plugin points for frameworks
> to do their own thing w.r.t interpretation and serving.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
