[ 
https://issues.apache.org/jira/browse/YARN-3378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14387762#comment-14387762
 ] 

Sangjin Lee commented on YARN-3378:
-----------------------------------

Looking into this bit more, I realize that converting this into something 
against the v.2 API is a little more involved than I originally thought. 
Basically it has to do with the distributed timeline collectors and the fact we 
need to use non-existent apps to be able to pump in data into the timeline 
service.

Some challenges:
- we need to trigger creating app-level timeline collectors for these fake apps 
without real AMs: we may need to add RPC for the per-node timeline collector to 
trigger adding and removing app-level timeline collectors
- when the app-level timeline collectors are added to the timeline collector 
manager, the collector manager needs to talk to NM to get the context but NM 
won't know the context as these are for non-existent apps
- we also need to remove these fake apps properly from the per-node timeline 
collectors, NMs, and RMs
- we need a way to spray the traffic to various NMs

Another idea may be to create per-node timeline collectors in a special manner 
for these load tests so that you can simply direct traffic to them. We'll need 
to make sure this mode is removed or cleaned up properly before we merge this.

Thoughts? Comments?

> a load test client that can replay a volume of history files
> ------------------------------------------------------------
>
>                 Key: YARN-3378
>                 URL: https://issues.apache.org/jira/browse/YARN-3378
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Sangjin Lee
>            Assignee: Sangjin Lee
>
> It might be good to create a load test client that can replay a large volume 
> of history files into the timeline service. One can envision running such a 
> load test client as a mapreduce job and generate a fair amount of load. It 
> would be useful to spot check correctness, and more importantly observe 
> performance characteristic.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to