Amit Tiwari updated YARN-2556:
    Attachment: YARN-2556.patch

Hi guys,
I've done the following enhancements to the previous patches that were posted:
1) Earlier, the payload was getting set as the entityId. Since the entityId is 
used as a key, by LevelDB it was crashing under moderate loads, because each 
key size was ~2MB. Hence I've changed it to send the payload as a part of 
OtherInfo. This is handled well.
2) Instead of posting a string of repeated 'a's as a payload, I choose from a 
set of characters. This ensures that the LevelDB does not get away easily with 
compression ('cos algos can easily compress a string if it comprises a single 
repeated character)

Here are some of the performance numbers that I've got:
I run 20 concurrent jobs, with the argument -m 300 -s 10 -t 20 
On a 36 node cluster, this results in ~830 concurrent containers (e.g maps), 
each firing 10KB of payload, 20 times.

Level DB seems to hold up fine.

Would you have other ways that I could stress/load the system even more?

> Tool to measure the performance of the timeline server
> ------------------------------------------------------
>                 Key: YARN-2556
>                 URL: https://issues.apache.org/jira/browse/YARN-2556
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Jonathan Eagles
>            Assignee: Chang Li
>         Attachments: YARN-2556-WIP.patch, YARN-2556-WIP.patch, 
> YARN-2556.patch, yarn2556.patch, yarn2556.patch, yarn2556_wip.patch
> We need to be able to understand the capacity model for the timeline server 
> to give users the tools they need to deploy a timeline server with the 
> correct capacity.
> I propose we create a mapreduce job that can measure timeline server write 
> and read performance. Transactions per second, I/O for both read and write 
> would be a good start.
> This could be done as an example or test job that could be tied into gridmix.

This message was sent by Atlassian JIRA

Reply via email to