[jira] [Commented] (YARN-2517) Implement TimelineClientAsync
[ https://issues.apache.org/jira/browse/YARN-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14238158#comment-14238158 ] Zhijie Shen commented on YARN-2517: --- [~ozawa], we may hay to hang on a bit about the async call implementation. Recently folks have some offline discussion around the timeline server next gen. Some architecture may be changed in the future, Would you please keep an eye on YARN-2928? Implement TimelineClientAsync - Key: YARN-2517 URL: https://issues.apache.org/jira/browse/YARN-2517 Project: Hadoop YARN Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Tsuyoshi OZAWA Attachments: YARN-2517.1.patch, YARN-2517.2.patch In some scenarios, we'd like to put timeline entities in another thread no to block the current one. It's good to have a TimelineClientAsync like AMRMClientAsync and NMClientAsync. It can buffer entities, put them in a separate thread, and have callback to handle the responses. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2517) Implement TimelineClientAsync
[ https://issues.apache.org/jira/browse/YARN-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14236763#comment-14236763 ] Tsuyoshi OZAWA commented on YARN-2517: -- Thanks for your comments, [~zjshen], [~sjlee0], [~mitdesai], [~hitesh]. If we don't care communication problem, fire-and-forget approach in putEntitiesAsync sounds reasonable to me. In this case, it's reasonable to me to use Future without callbacks. v2 design is based on Future, so I appreciate if you have feedbacks. One possible API to be added is flush() to assure to send all pending requests. It can be added on separate JIRA. Implement TimelineClientAsync - Key: YARN-2517 URL: https://issues.apache.org/jira/browse/YARN-2517 Project: Hadoop YARN Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Tsuyoshi OZAWA Attachments: YARN-2517.1.patch, YARN-2517.2.patch In some scenarios, we'd like to put timeline entities in another thread no to block the current one. It's good to have a TimelineClientAsync like AMRMClientAsync and NMClientAsync. It can buffer entities, put them in a separate thread, and have callback to handle the responses. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2517) Implement TimelineClientAsync
[ https://issues.apache.org/jira/browse/YARN-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14225420#comment-14225420 ] Sangjin Lee commented on YARN-2517: --- Thanks for the clarification [~zjshen]. If there is a data integrity issue and the client receives an error due to such a data integrity problem, I wonder what it can do upon receiving it. Can you talk about possible non-trivial action here? To me, it seems like what it can do is really limited. Implement TimelineClientAsync - Key: YARN-2517 URL: https://issues.apache.org/jira/browse/YARN-2517 Project: Hadoop YARN Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Tsuyoshi OZAWA Attachments: YARN-2517.1.patch, YARN-2517.2.patch In some scenarios, we'd like to put timeline entities in another thread no to block the current one. It's good to have a TimelineClientAsync like AMRMClientAsync and NMClientAsync. It can buffer entities, put them in a separate thread, and have callback to handle the responses. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2517) Implement TimelineClientAsync
[ https://issues.apache.org/jira/browse/YARN-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223091#comment-14223091 ] Mit Desai commented on YARN-2517: - I had similar concerns. Do we really need this at this point? And as Hitesh pointed out, if this may hinder the design in future. bq. Also, is the timeline layer meant to eventually be reliable and always up? As far as what I am aware of, this is not going to be in near future. Implement TimelineClientAsync - Key: YARN-2517 URL: https://issues.apache.org/jira/browse/YARN-2517 Project: Hadoop YARN Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Tsuyoshi OZAWA Attachments: YARN-2517.1.patch, YARN-2517.2.patch In some scenarios, we'd like to put timeline entities in another thread no to block the current one. It's good to have a TimelineClientAsync like AMRMClientAsync and NMClientAsync. It can buffer entities, put them in a separate thread, and have callback to handle the responses. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2517) Implement TimelineClientAsync
[ https://issues.apache.org/jira/browse/YARN-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223299#comment-14223299 ] Sangjin Lee commented on YARN-2517: --- It might be conceptually cleaner (and simpler to the clients) to have the clients post the events but not deal with what would happen if the result was not successfully sent. Even the very concept of whether the result was sent is problematic. An ATS writer implementation could make a decision of buffering a certain amount of data before it physically writes it to the backing storage (as optimization). If the client needs to know whether the event was truly sent to the backing storage and also deal with its failure, it may lead to ATS writer implementations leaking to clients and may limit the way an ATS write implementation can be optimized, etc. How about a sync write (as it stands now) for critical data and an async write which is basically fire-and-forget? The understanding there would be the async write is basically a best effort and it would fall on the implementation of ATS writes to try to deliver the events to the backing storage as reliably and optimally as it can (but in theory no guarantees still). Then the async write can even be enabled with a boolean flag. Implement TimelineClientAsync - Key: YARN-2517 URL: https://issues.apache.org/jira/browse/YARN-2517 Project: Hadoop YARN Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Tsuyoshi OZAWA Attachments: YARN-2517.1.patch, YARN-2517.2.patch In some scenarios, we'd like to put timeline entities in another thread no to block the current one. It's good to have a TimelineClientAsync like AMRMClientAsync and NMClientAsync. It can buffer entities, put them in a separate thread, and have callback to handle the responses. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2517) Implement TimelineClientAsync
[ https://issues.apache.org/jira/browse/YARN-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223923#comment-14223923 ] Zhijie Shen commented on YARN-2517: --- Thanks for sharing your idea, Hitesh, Mit and Sangjin! I'd like to make some clarification. The error that the handler wants to take care of is not the communication problem, but the problem that happens when the server is processing the posted timeline entity (See TimelinePutResponse). It could be the data integrity issue that is messed up by the app. For example, the posted Entity A in Domain 1 is trying to relate to Entity B in Domain 2. It's fine if in some use cases, the app doesn't want to care about it, and consequently doesn't require an Ack. The app can just go ahead without providing the handler. However, it's still better to be generic enough to cover the other use cases where the app wants to make sure it has the timeline data persisted, or at least know it is succeeded or not. Queueing/messaging layer may help to mitigate the communication problem, but aforementioned data problem still could happen, such that the app may still want to hear about the put response. However, it sounds right that the implementation of this layer will affect that of async call. It makes sense to wait until we make it clear how to make client - TS communication reliable. If we eventually find handler in async call is difficult and will further prevent optimization, a sync write (as it stands now) for critical data and an async write which is basically fire-and-forge sounds a reasonable alternative plan. Implement TimelineClientAsync - Key: YARN-2517 URL: https://issues.apache.org/jira/browse/YARN-2517 Project: Hadoop YARN Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Tsuyoshi OZAWA Attachments: YARN-2517.1.patch, YARN-2517.2.patch In some scenarios, we'd like to put timeline entities in another thread no to block the current one. It's good to have a TimelineClientAsync like AMRMClientAsync and NMClientAsync. It can buffer entities, put them in a separate thread, and have callback to handle the responses. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2517) Implement TimelineClientAsync
[ https://issues.apache.org/jira/browse/YARN-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14222582#comment-14222582 ] Hitesh Shah commented on YARN-2517: --- Sorry for coming in late on this. Why is this needed? [~zjshen] [~ozawa] Is a callback layer back to the user really needed? How would the user make use of this callback information except maybe to log an error or fail the application? Also, is the timeline layer meant to eventually be reliable and always up? Let us say that sometime later, the timeline client started using some form of queueing/messaging layer to send the data to the server. When would a callback be invoked? When the data is written to the messaging layer or when the server actually processes the data? Implementing this now will inhibit the design down the line if it needs to retain the same semantics for the callback invocation ( which would be done today when the server finishes processing the data ) Implement TimelineClientAsync - Key: YARN-2517 URL: https://issues.apache.org/jira/browse/YARN-2517 Project: Hadoop YARN Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Tsuyoshi OZAWA Attachments: YARN-2517.1.patch, YARN-2517.2.patch In some scenarios, we'd like to put timeline entities in another thread no to block the current one. It's good to have a TimelineClientAsync like AMRMClientAsync and NMClientAsync. It can buffer entities, put them in a separate thread, and have callback to handle the responses. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2517) Implement TimelineClientAsync
[ https://issues.apache.org/jira/browse/YARN-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14222338#comment-14222338 ] Tsuyoshi OZAWA commented on YARN-2517: -- [~zjshen], thanks for your comment. {code} // Async call - Type (1) void asyncCall(Input, CallBackHandler); {code} If we choose Type(1), TimelineClient need to manage all objects of callback handlers it's passed. It can get more complex. I think it's better to add {{registerAsyncCallbackHandler(CallBackHandler)}} to add TimelineClient separately or pass the callback via constructor for the simplicity. Or, do we have use cases to switch CallBackHandler for each method calls? {quote} Maybe compromise now is to add putEntitiesAsync to TimelineClient. In the future, let's see if we want to have a separate TimelineClientAsync that contains a bunch of async APIs. {quote} If we will have a plan to migrate from putEntitiesAsync to TimelineClientAsync, I think we should add TimelineClientAsync from the beginning. I think it's enough to add *Async methods to current TimelineClient for now since users don't need to create specific clients for async calls. [~mitdesai] thanks for your help. Looking forward to your opinion. We're discussing the design based on [Vinod's suggestion|https://issues.apache.org/jira/browse/YARN-2517?focusedCommentId=14128819page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14128819]. Implement TimelineClientAsync - Key: YARN-2517 URL: https://issues.apache.org/jira/browse/YARN-2517 Project: Hadoop YARN Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Tsuyoshi OZAWA Attachments: YARN-2517.1.patch, YARN-2517.2.patch In some scenarios, we'd like to put timeline entities in another thread no to block the current one. It's good to have a TimelineClientAsync like AMRMClientAsync and NMClientAsync. It can buffer entities, put them in a separate thread, and have callback to handle the responses. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2517) Implement TimelineClientAsync
[ https://issues.apache.org/jira/browse/YARN-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14221027#comment-14221027 ] Mit Desai commented on YARN-2517: - Yes I have been working on timeline service recently. I will take a look. Implement TimelineClientAsync - Key: YARN-2517 URL: https://issues.apache.org/jira/browse/YARN-2517 Project: Hadoop YARN Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Tsuyoshi OZAWA Attachments: YARN-2517.1.patch, YARN-2517.2.patch In some scenarios, we'd like to put timeline entities in another thread no to block the current one. It's good to have a TimelineClientAsync like AMRMClientAsync and NMClientAsync. It can buffer entities, put them in a separate thread, and have callback to handle the responses. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2517) Implement TimelineClientAsync
[ https://issues.apache.org/jira/browse/YARN-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14221249#comment-14221249 ] Zhijie Shen commented on YARN-2517: --- I'm not sure Future is going to help the use case. Say I want to use TimelineClientAsync to do async put entity operations. With Future, putEntitiesAsync returns immediately. However, to know if my put entity operation is successful not, I still have to been blocked at Future#get(), or create a separate thread to wait for the response. But IMHO, one goal of TimelineClientAsync is to relieve users from multithreading details, such that Type (1) sounds better to me. Rethink whether we create a separate TimelineClientAsync or add async method in TimelineClient. We have putEntities and putDomain, and in the future we will have more get APIs. For now, the most concerned API is putEntities, as we don't want it to block the normal execution logic of an app. Maybe compromise now is to add putEntitiesAsync to TimelineClient. In the future, let's see if we want to have a separate TimelineClientAsync that contains a bunch of async APIs. Thoughts? Implement TimelineClientAsync - Key: YARN-2517 URL: https://issues.apache.org/jira/browse/YARN-2517 Project: Hadoop YARN Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Tsuyoshi OZAWA Attachments: YARN-2517.1.patch, YARN-2517.2.patch In some scenarios, we'd like to put timeline entities in another thread no to block the current one. It's good to have a TimelineClientAsync like AMRMClientAsync and NMClientAsync. It can buffer entities, put them in a separate thread, and have callback to handle the responses. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2517) Implement TimelineClientAsync
[ https://issues.apache.org/jira/browse/YARN-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14219554#comment-14219554 ] Tsuyoshi OZAWA commented on YARN-2517: -- [~mitdesai] I think you're one of the users of TimelineClient. If you have any feedbacks about the interface, please let me know. Implement TimelineClientAsync - Key: YARN-2517 URL: https://issues.apache.org/jira/browse/YARN-2517 Project: Hadoop YARN Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Tsuyoshi OZAWA Attachments: YARN-2517.1.patch, YARN-2517.2.patch In some scenarios, we'd like to put timeline entities in another thread no to block the current one. It's good to have a TimelineClientAsync like AMRMClientAsync and NMClientAsync. It can buffer entities, put them in a separate thread, and have callback to handle the responses. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2517) Implement TimelineClientAsync
[ https://issues.apache.org/jira/browse/YARN-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14219587#comment-14219587 ] Hadoop QA commented on YARN-2517: - {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12682679/YARN-2517.2.patch against trunk revision a9a0cc3. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/5893//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5893//console This message is automatically generated. Implement TimelineClientAsync - Key: YARN-2517 URL: https://issues.apache.org/jira/browse/YARN-2517 Project: Hadoop YARN Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Tsuyoshi OZAWA Attachments: YARN-2517.1.patch, YARN-2517.2.patch In some scenarios, we'd like to put timeline entities in another thread no to block the current one. It's good to have a TimelineClientAsync like AMRMClientAsync and NMClientAsync. It can buffer entities, put them in a separate thread, and have callback to handle the responses. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2517) Implement TimelineClientAsync
[ https://issues.apache.org/jira/browse/YARN-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14162820#comment-14162820 ] Zhijie Shen commented on YARN-2517: --- bq. Clearly, async calls need call-back handlers just for errors. As of today, there are no APIs that really need to send back results (not error) asynchronously. I realize that another requirement of TimelineClient is to wrap the read APIs to facilitate the java app developers. These APIs are going to return results. Implement TimelineClientAsync - Key: YARN-2517 URL: https://issues.apache.org/jira/browse/YARN-2517 Project: Hadoop YARN Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Tsuyoshi OZAWA Attachments: YARN-2517.1.patch In some scenarios, we'd like to put timeline entities in another thread no to block the current one. It's good to have a TimelineClientAsync like AMRMClientAsync and NMClientAsync. It can buffer entities, put them in a separate thread, and have callback to handle the responses. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2517) Implement TimelineClientAsync
[ https://issues.apache.org/jira/browse/YARN-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14131087#comment-14131087 ] Tsuyoshi OZAWA commented on YARN-2517: -- I see. I'll try to prototype the new async method to post entities. {quote} If you guys think there is a lot more functionality coming in an async class in the future, can we hear about some of them here? {quote} Currently, TimelineClient just have APIs to post entities. I think there is a possibility to have helper methods like {{TimelineCilent#getEntities()}} for applications of Timeline Server. Implement TimelineClientAsync - Key: YARN-2517 URL: https://issues.apache.org/jira/browse/YARN-2517 Project: Hadoop YARN Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Tsuyoshi OZAWA Attachments: YARN-2517.1.patch In some scenarios, we'd like to put timeline entities in another thread no to block the current one. It's good to have a TimelineClientAsync like AMRMClientAsync and NMClientAsync. It can buffer entities, put them in a separate thread, and have callback to handle the responses. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2517) Implement TimelineClientAsync
[ https://issues.apache.org/jira/browse/YARN-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14128156#comment-14128156 ] Zhijie Shen commented on YARN-2517: --- Scan through the patch, the approach is quite close to AMRMClientAsync/NMClientAsync. It looks fine to me in general. Later on, we can improve the client step by step. For example, according to the discussion on the umbrella, we can persist the queued entities to be reliable. And we may want to allow multiple threads to put entities. Implement TimelineClientAsync - Key: YARN-2517 URL: https://issues.apache.org/jira/browse/YARN-2517 Project: Hadoop YARN Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Tsuyoshi OZAWA Attachments: YARN-2517.1.patch In some scenarios, we'd like to put timeline entities in another thread no to block the current one. It's good to have a TimelineClientAsync like AMRMClientAsync and NMClientAsync. It can buffer entities, put them in a separate thread, and have callback to handle the responses. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2517) Implement TimelineClientAsync
[ https://issues.apache.org/jira/browse/YARN-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14128567#comment-14128567 ] Tsuyoshi OZAWA commented on YARN-2517: -- Thanks for your review, Zhijie. I think batch optimization and persisting entities can be done in sync client since the async client uses sync client. Submitting a patch again for merging. Please let me know if you have additional review comments. Implement TimelineClientAsync - Key: YARN-2517 URL: https://issues.apache.org/jira/browse/YARN-2517 Project: Hadoop YARN Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Tsuyoshi OZAWA Attachments: YARN-2517.1.patch In some scenarios, we'd like to put timeline entities in another thread no to block the current one. It's good to have a TimelineClientAsync like AMRMClientAsync and NMClientAsync. It can buffer entities, put them in a separate thread, and have callback to handle the responses. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2517) Implement TimelineClientAsync
[ https://issues.apache.org/jira/browse/YARN-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14128819#comment-14128819 ] Vinod Kumar Vavilapalli commented on YARN-2517: --- I am +1 about a client that makes async calls. The question is whether we need a a new client class (and thus a public interface) or not. Clearly, async calls need call-back handlers _just_ for errors. As of today, there are no APIs that really need to send back *results* (not error) asynchronously. The way you usually handle it is through one of the following {code} // Sync call Result call(Input); // Async call - Type (1) void asyncCall(Input, CallBackHandler); // Async call - Type (2) Future asyncCall(Input); {code} You can do type (1). Having an entire separate client side interface isn't mandatory. If you guys think there is a lot more functionality coming in an async class in the future, can we hear about some of them here? Implement TimelineClientAsync - Key: YARN-2517 URL: https://issues.apache.org/jira/browse/YARN-2517 Project: Hadoop YARN Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Tsuyoshi OZAWA Attachments: YARN-2517.1.patch In some scenarios, we'd like to put timeline entities in another thread no to block the current one. It's good to have a TimelineClientAsync like AMRMClientAsync and NMClientAsync. It can buffer entities, put them in a separate thread, and have callback to handle the responses. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2517) Implement TimelineClientAsync
[ https://issues.apache.org/jira/browse/YARN-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14128097#comment-14128097 ] Tsuyoshi OZAWA commented on YARN-2517: -- [~zjshen], [~vinodkv], can we go with current design(v1 patch)? Implement TimelineClientAsync - Key: YARN-2517 URL: https://issues.apache.org/jira/browse/YARN-2517 Project: Hadoop YARN Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Tsuyoshi OZAWA Attachments: YARN-2517.1.patch In some scenarios, we'd like to put timeline entities in another thread no to block the current one. It's good to have a TimelineClientAsync like AMRMClientAsync and NMClientAsync. It can buffer entities, put them in a separate thread, and have callback to handle the responses. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2517) Implement TimelineClientAsync
[ https://issues.apache.org/jira/browse/YARN-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125664#comment-14125664 ] Hadoop QA commented on YARN-2517: - {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12667168/YARN-2517.1.patch against trunk revision 0974f43. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/4842//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4842//console This message is automatically generated. Implement TimelineClientAsync - Key: YARN-2517 URL: https://issues.apache.org/jira/browse/YARN-2517 Project: Hadoop YARN Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Tsuyoshi OZAWA Attachments: YARN-2517.1.patch In some scenarios, we'd like to put timeline entities in another thread no to block the current one. It's good to have a TimelineClientAsync like AMRMClientAsync and NMClientAsync. It can buffer entities, put them in a separate thread, and have callback to handle the responses. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2517) Implement TimelineClientAsync
[ https://issues.apache.org/jira/browse/YARN-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125714#comment-14125714 ] Vinod Kumar Vavilapalli commented on YARN-2517: --- I am not entirely sure we need a parallel client for this. The other clients needed async clients because - they had loads of functionality that made sense in the blocking and non-blocking modes - the client code really needed call-back hooks to act on the results. Timeline Client's only responsibility is to post events. There are only two use-cases: Clients need a sync write through, or an asynchronous write, the end of which they don't care about. I think we should simply have a mode in the existing client to post events asynchronously without any further need for call-back handlers. What do others think? Implement TimelineClientAsync - Key: YARN-2517 URL: https://issues.apache.org/jira/browse/YARN-2517 Project: Hadoop YARN Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Tsuyoshi OZAWA Attachments: YARN-2517.1.patch In some scenarios, we'd like to put timeline entities in another thread no to block the current one. It's good to have a TimelineClientAsync like AMRMClientAsync and NMClientAsync. It can buffer entities, put them in a separate thread, and have callback to handle the responses. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2517) Implement TimelineClientAsync
[ https://issues.apache.org/jira/browse/YARN-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125760#comment-14125760 ] Tsuyoshi OZAWA commented on YARN-2517: -- Thanks for your comment, Vinod. {quote} an asynchronous write, the end of which they don't care about. I think we should simply have a mode in the existing client to post events asynchronously without any further need for call-back handlers. {quote} Make sense. We can assure at-most-once semantics without any callbacks. How about adding a {{flush()}} API to TimelineClient for asynchronous mode? It helps users to know whether contents of current buffer are written to Timeline Server or not. Implement TimelineClientAsync - Key: YARN-2517 URL: https://issues.apache.org/jira/browse/YARN-2517 Project: Hadoop YARN Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Tsuyoshi OZAWA Attachments: YARN-2517.1.patch In some scenarios, we'd like to put timeline entities in another thread no to block the current one. It's good to have a TimelineClientAsync like AMRMClientAsync and NMClientAsync. It can buffer entities, put them in a separate thread, and have callback to handle the responses. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2517) Implement TimelineClientAsync
[ https://issues.apache.org/jira/browse/YARN-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125777#comment-14125777 ] Zhijie Shen commented on YARN-2517: --- [~vinodkv], thanks for your feedback. The reason why an async client (or async HTTP REST call) is going to be good is to unblock the current thread if it is doing the important management logic. For example, in YARN-2033, we have a bunch of logic to dispatch the entity putting action on a separate thread, to make the application life cycle management move on. Given an async client, it could be far more simplified. I think from the user point of view, it may be a useful feature as well. I'm fine whether we can two classes, sync for one and async for the other, or one class for both modes, while the former option complies with the previous client design. I think the callback is necessary, at least onError. TimelinePutResponse will give the user a summary of why his uploaded entity is not accepted by the timeline server. Based on the response, the user can determine whether the app should neglect the problem and move on, or stop immediately. Implement TimelineClientAsync - Key: YARN-2517 URL: https://issues.apache.org/jira/browse/YARN-2517 Project: Hadoop YARN Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Tsuyoshi OZAWA Attachments: YARN-2517.1.patch In some scenarios, we'd like to put timeline entities in another thread no to block the current one. It's good to have a TimelineClientAsync like AMRMClientAsync and NMClientAsync. It can buffer entities, put them in a separate thread, and have callback to handle the responses. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2517) Implement TimelineClientAsync
[ https://issues.apache.org/jira/browse/YARN-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14126557#comment-14126557 ] Tsuyoshi OZAWA commented on YARN-2517: -- As Zhijie mentioned, we should have the callback if we need to check errors. IMHO, if we have a thread for the callback onError, we should also have onEntitiesPut since the complexity doesn't increase so much and it's useful to distinguish connection-level exceptions from entity-level errors. Implement TimelineClientAsync - Key: YARN-2517 URL: https://issues.apache.org/jira/browse/YARN-2517 Project: Hadoop YARN Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Tsuyoshi OZAWA Attachments: YARN-2517.1.patch In some scenarios, we'd like to put timeline entities in another thread no to block the current one. It's good to have a TimelineClientAsync like AMRMClientAsync and NMClientAsync. It can buffer entities, put them in a separate thread, and have callback to handle the responses. -- This message was sent by Atlassian JIRA (v6.3.4#6332)