[jira] [Commented] (YARN-3378) a load test client that can replay a volume of history files
[ https://issues.apache.org/jira/browse/YARN-3378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14387762#comment-14387762 ] Sangjin Lee commented on YARN-3378: --- Looking into this bit more, I realize that converting this into something against the v.2 API is a little more involved than I originally thought. Basically it has to do with the distributed timeline collectors and the fact we need to use non-existent apps to be able to pump in data into the timeline service. Some challenges: - we need to trigger creating app-level timeline collectors for these fake apps without real AMs: we may need to add RPC for the per-node timeline collector to trigger adding and removing app-level timeline collectors - when the app-level timeline collectors are added to the timeline collector manager, the collector manager needs to talk to NM to get the context but NM won't know the context as these are for non-existent apps - we also need to remove these fake apps properly from the per-node timeline collectors, NMs, and RMs - we need a way to spray the traffic to various NMs Another idea may be to create per-node timeline collectors in a special manner for these load tests so that you can simply direct traffic to them. We'll need to make sure this mode is removed or cleaned up properly before we merge this. Thoughts? Comments? > a load test client that can replay a volume of history files > > > Key: YARN-3378 > URL: https://issues.apache.org/jira/browse/YARN-3378 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Sangjin Lee > > It might be good to create a load test client that can replay a large volume > of history files into the timeline service. One can envision running such a > load test client as a mapreduce job and generate a fair amount of load. It > would be useful to spot check correctness, and more importantly observe > performance characteristic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3378) a load test client that can replay a volume of history files
[ https://issues.apache.org/jira/browse/YARN-3378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14382435#comment-14382435 ] Sangjin Lee commented on YARN-3378: --- cc [~jeagles], [~lichangleo] I'm working on this based on what you have on YARN-2556, with major differences being - write it against the v.2 API (obviously) - add an ability to replay things like a bunch of history files to generate more realistic and non-trivial entities and data We'll also look into benchmarks more appropriate for the v.2 work as Li mentioned. We need a little bit of discussion on how this will proceed in parallel with YARN-2556. I'm taking the latest patch on YARN-2556 as the basis. Should we go ahead and commit the work done in YARN-2556 first? Thoughts? > a load test client that can replay a volume of history files > > > Key: YARN-3378 > URL: https://issues.apache.org/jira/browse/YARN-3378 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Sangjin Lee > > It might be good to create a load test client that can replay a large volume > of history files into the timeline service. One can envision running such a > load test client as a mapreduce job and generate a fair amount of load. It > would be useful to spot check correctness, and more importantly observe > performance characteristic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3378) a load test client that can replay a volume of history files
[ https://issues.apache.org/jira/browse/YARN-3378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14371626#comment-14371626 ] Li Lu commented on YARN-3378: - Thanks [~Naganarasimha] for reminding me YARN-2556. I looked at it. It seems like there are some storage level implementations missing in our v2 branch to adopt that patch. Since we're benchmarking on a WIP project, maybe we'd like to organize the benchmarks in a different way? I'll definitely keep an eye on it when working on this one for v2. > a load test client that can replay a volume of history files > > > Key: YARN-3378 > URL: https://issues.apache.org/jira/browse/YARN-3378 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Li Lu > > It might be good to create a load test client that can replay a large volume > of history files into the timeline service. One can envision running such a > load test client as a mapreduce job and generate a fair amount of load. It > would be useful to spot check correctness, and more importantly observe > performance characteristic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3378) a load test client that can replay a volume of history files
[ https://issues.apache.org/jira/browse/YARN-3378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14371586#comment-14371586 ] Naganarasimha G R commented on YARN-3378: - Thanks [~sjlee0], thats fine with me :) . Anyway Jonathan has shared a patch for YARN-2556 two days back, may be you and [~gtCarrera9] can have a look at it . > a load test client that can replay a volume of history files > > > Key: YARN-3378 > URL: https://issues.apache.org/jira/browse/YARN-3378 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Li Lu > > It might be good to create a load test client that can replay a large volume > of history files into the timeline service. One can envision running such a > load test client as a mapreduce job and generate a fair amount of load. It > would be useful to spot check correctness, and more importantly observe > performance characteristic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3378) a load test client that can replay a volume of history files
[ https://issues.apache.org/jira/browse/YARN-3378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14371560#comment-14371560 ] Sangjin Lee commented on YARN-3378: --- Thanks [~Naganarasimha] for reminding me of YARN-2556. I forgot about that one. I do agree that the purpose of these JIRAs is quite similar. It would be ideal if we can use what comes out of YARN-2556 with no or little modifications. In the meantime, we can leave this open until we're getting close to do this. If YARN-2556 can be used as is, we could close this one then. Does that sound reasonable? > a load test client that can replay a volume of history files > > > Key: YARN-3378 > URL: https://issues.apache.org/jira/browse/YARN-3378 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Li Lu > > It might be good to create a load test client that can replay a large volume > of history files into the timeline service. One can envision running such a > load test client as a mapreduce job and generate a fair amount of load. It > would be useful to spot check correctness, and more importantly observe > performance characteristic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3378) a load test client that can replay a volume of history files
[ https://issues.apache.org/jira/browse/YARN-3378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14370888#comment-14370888 ] Naganarasimha G R commented on YARN-3378: - Hi [~gtCarrera9], I think you took me wrongly its not that i am against having this jira and infact i feel this kind of test is very much important, as i faced similar issue with ATS V1 and i was checking with Zhijie. My intent was about reuse, IIUC YARN-2256, they too have MR task which takes arguments like number of mappers, number of bytes to send and the interval. and they are using Timeline client(v1) to post the entities based on these input. So my view was to take one more configuration here to take the ATS version to which to Entites needs to be posted (and other conf information related ATSV2) and make use of the same code to publish Data to ATS V2. But if we are planning to have much more functionality then we might not be able reuse... > a load test client that can replay a volume of history files > > > Key: YARN-3378 > URL: https://issues.apache.org/jira/browse/YARN-3378 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Li Lu > > It might be good to create a load test client that can replay a large volume > of history files into the timeline service. One can envision running such a > load test client as a mapreduce job and generate a fair amount of load. It > would be useful to spot check correctness, and more importantly observe > performance characteristic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3378) a load test client that can replay a volume of history files
[ https://issues.apache.org/jira/browse/YARN-3378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14370784#comment-14370784 ] Li Lu commented on YARN-3378: - Hi [~Naganarasimha], YARN-2556 was mainly opened for measuring the existing ATS v1's performance. In this JIRA our main focus we'd like to build a client to generate reasonable load to guide v2 timeline service's storage design. From our discussion about Phoenix/hbase I believe this is a quite necessary step for us to understand our v2 design. These two JIRAs also work on two different branches. Just like timeline service v1 and v2 may co-exist (so do all related jiras) in YARN-2928 branch, I don't see any reason to prevent both JIRAs exist. If you have any special concerns about this feel free to let us know. Thanks! > a load test client that can replay a volume of history files > > > Key: YARN-3378 > URL: https://issues.apache.org/jira/browse/YARN-3378 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Li Lu > > It might be good to create a load test client that can replay a large volume > of history files into the timeline service. One can envision running such a > load test client as a mapreduce job and generate a fair amount of load. It > would be useful to spot check correctness, and more importantly observe > performance characteristic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3378) a load test client that can replay a volume of history files
[ https://issues.apache.org/jira/browse/YARN-3378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14370772#comment-14370772 ] Naganarasimha G R commented on YARN-3378: - Hi [~sjlee0] & [~gtCarrera], Is this jira planning to do something differently from YARN-2556, Already patch is available for it there and if the current scope of the jira is not targeting to do anything differently than the older one, then we can close this jira and continue to do with YARN-2556, else may be we can close the older jira and leverage the patch here or continue with the new changes... > a load test client that can replay a volume of history files > > > Key: YARN-3378 > URL: https://issues.apache.org/jira/browse/YARN-3378 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Li Lu > > It might be good to create a load test client that can replay a large volume > of history files into the timeline service. One can envision running such a > load test client as a mapreduce job and generate a fair amount of load. It > would be useful to spot check correctness, and more importantly observe > performance characteristic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3378) a load test client that can replay a volume of history files
[ https://issues.apache.org/jira/browse/YARN-3378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14369837#comment-14369837 ] Sangjin Lee commented on YARN-3378: --- Once the write part of the storage is ready, we could use a tool like this to see scalability. > a load test client that can replay a volume of history files > > > Key: YARN-3378 > URL: https://issues.apache.org/jira/browse/YARN-3378 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee > > It might be good to create a load test client that can replay a large volume > of history files into the timeline service. One can envision running such a > load test client as a mapreduce job and generate a fair amount of load. It > would be useful to spot check correctness, and more importantly observe > performance characteristic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
