[jira] [Updated] (YARN-422) Add AM-NM client library
[ https://issues.apache.org/jira/browse/YARN-422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-422: - Attachment: YARN-422.1.patch Here's the first patch, which is ready for review. Based on the previous definition file, the patch has the following updates: 1. Refactor some code (add some more logs, paraphrase the javadoc, and etc) 2. Rename AMNMClient to NMClient since not only AM will use this client. 3. In NMClientAsync, join the threadpool's thread, which is set to non-daeomon. 4. Enhance the test cases. As the patch is already big, I suggest to defer the code changes of using the client in AM and RM in the follow-up patches. In addition, as I found, maven seems not to distinguish scope when checking cyclic dependency. Therefore, making resourcemanager project depend on client (to use NMClient in AMLauncher) will fail the build. Add AM-NM client library Key: YARN-422 URL: https://issues.apache.org/jira/browse/YARN-422 Project: Hadoop YARN Issue Type: Sub-task Reporter: Bikas Saha Assignee: Zhijie Shen Attachments: AMNMClient_Defination.txt, AMNMClient_Definition_Updated_With_Tests.txt, proposal_v1.pdf, YARN-422.1.patch Create a simple wrapper over the AM-NM container protocol to provide hide the details of the protocol implementation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-422) Add AM-NM client library
[ https://issues.apache.org/jira/browse/YARN-422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-422: - Attachment: YARN-422.2.patch Try to fix the test failure. The problem seems that the mockNMClient is changed before all the test cases are completed. So, split the success and the failure tests. Add AM-NM client library Key: YARN-422 URL: https://issues.apache.org/jira/browse/YARN-422 Project: Hadoop YARN Issue Type: Sub-task Reporter: Bikas Saha Assignee: Zhijie Shen Attachments: AMNMClient_Defination.txt, AMNMClient_Definition_Updated_With_Tests.txt, proposal_v1.pdf, YARN-422.1.patch, YARN-422.2.patch Create a simple wrapper over the AM-NM container protocol to provide hide the details of the protocol implementation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-422) Add AM-NM client library
[ https://issues.apache.org/jira/browse/YARN-422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-422: - Attachment: AMNMClient_Definition_Updated_With_Tests.txt I've updated the definition of AMNMClient, fixing the minor bugs, synchronizing NMCommunicator setter/getter and drafting the javadoc. In addition, I've added the test to demonstrate how AMNMClientImpl work with AMRMClientImpl together on YarnMiniCluster. And I've added the test for AMNMClientAsync. The remaining stuff in this ticket could be using AMNMClient in AMLauncher. There're two concerns: 1. Semantically, it is a bit strange RM use AMNMClient. 2. Technically, hadoop-yarn-client has dependency on hadoop-yarn-server-resourcemanager in test scope. If we want to use AMNMClient in AMLauncher, hadoop-yarn-server-resourcemanager needs to add the dependency on hadoop-yarn-client, forming a circular dependency. Add AM-NM client library Key: YARN-422 URL: https://issues.apache.org/jira/browse/YARN-422 Project: Hadoop YARN Issue Type: Sub-task Reporter: Bikas Saha Assignee: Zhijie Shen Attachments: AMNMClient_Defination.txt, AMNMClient_Definition_Updated_With_Tests.txt, proposal_v1.pdf Create a simple wrapper over the AM-NM container protocol to provide hide the details of the protocol implementation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-422) Add AM-NM client library
[ https://issues.apache.org/jira/browse/YARN-422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-422: - Attachment: AMNMClient_Defination.txt I've drafted a footprint of AMNMClient library, without documentation, tests and etc. While I'm going on towards a complete patch, please have a look at the client's definition, and share you ideas. I define AMNMClient by referring AMRMClient. In general, there're the following parts in the library. 1. AMNMClient defines three basic APIs 2. NMCommunicator is a wrapper of the communications of only one container, defined in ContainerManager. It is also the inner class of AMNMClientImpl. 2. AMNMClientImpl implements the APIs. It maintain one-to-many relationship with all the containers that are to be started. It contains a collection of NMCommunicator. 3. AMNMClienAsync is the ultimate class that AM wants to use. It implements the three APIs in the non-blocking way. Internally, there's an event dispatcher, which starts when AMNMClienAsync starts. Calling the three APIs are just scheduling an event on the dispatcher. The dispatcher will deliver the event to an idle thread in the thread pool, where AMNMClientImpl is called do the real work. This part refers the design of ContainerLaucherImpl. In addition, as the execution is asynchronous, an Callback interface is exposed to AM. Add AM-NM client library Key: YARN-422 URL: https://issues.apache.org/jira/browse/YARN-422 Project: Hadoop YARN Issue Type: Sub-task Reporter: Bikas Saha Assignee: Zhijie Shen Attachments: AMNMClient_Defination.txt, proposal_v1.pdf Create a simple wrapper over the AM-NM container protocol to provide hide the details of the protocol implementation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-422) Add AM-NM client library
[ https://issues.apache.org/jira/browse/YARN-422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-422: - Attachment: proposal_v1.pdf Some initial thoughts about creating AMNMClient. Your comments, please. Add AM-NM client library Key: YARN-422 URL: https://issues.apache.org/jira/browse/YARN-422 Project: Hadoop YARN Issue Type: Sub-task Reporter: Bikas Saha Assignee: Zhijie Shen Attachments: proposal_v1.pdf Create a simple wrapper over the AM-NM container protocol to provide hide the details of the protocol implementation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
