[jira] [Commented] (YARN-2517) Implement TimelineClientAsync

2014-12-08 Thread Zhijie Shen (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14238158#comment-14238158
 ] 

Zhijie Shen commented on YARN-2517:
---

[~ozawa], we may hay to hang on a bit about the async call implementation. 
Recently folks have some offline discussion around the timeline server next 
gen. Some architecture may be changed in the future, Would you please keep an 
eye on YARN-2928? 

 Implement TimelineClientAsync
 -

 Key: YARN-2517
 URL: https://issues.apache.org/jira/browse/YARN-2517
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Tsuyoshi OZAWA
 Attachments: YARN-2517.1.patch, YARN-2517.2.patch


 In some scenarios, we'd like to put timeline entities in another thread no to 
 block the current one.
 It's good to have a TimelineClientAsync like AMRMClientAsync and 
 NMClientAsync. It can buffer entities, put them in a separate thread, and 
 have callback to handle the responses.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-2517) Implement TimelineClientAsync

2014-12-06 Thread Tsuyoshi OZAWA (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14236763#comment-14236763
 ] 

Tsuyoshi OZAWA commented on YARN-2517:
--

Thanks for your comments, [~zjshen], [~sjlee0], [~mitdesai], [~hitesh]. 

If we don't care communication problem, fire-and-forget approach in 
putEntitiesAsync sounds reasonable to me. In this case, it's reasonable to me 
to use Future without callbacks. v2 design is based on Future, so I appreciate 
if you have feedbacks. One possible API to be added is flush() to assure to 
send all pending requests. It can be added on separate JIRA.

 Implement TimelineClientAsync
 -

 Key: YARN-2517
 URL: https://issues.apache.org/jira/browse/YARN-2517
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Tsuyoshi OZAWA
 Attachments: YARN-2517.1.patch, YARN-2517.2.patch


 In some scenarios, we'd like to put timeline entities in another thread no to 
 block the current one.
 It's good to have a TimelineClientAsync like AMRMClientAsync and 
 NMClientAsync. It can buffer entities, put them in a separate thread, and 
 have callback to handle the responses.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-2517) Implement TimelineClientAsync

2014-11-25 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14225420#comment-14225420
 ] 

Sangjin Lee commented on YARN-2517:
---

Thanks for the clarification [~zjshen].

If there is a data integrity issue and the client receives an error due to such 
a data integrity problem, I wonder what it can do upon receiving it. Can you 
talk about possible non-trivial action here? To me, it seems like what it can 
do is really limited.

 Implement TimelineClientAsync
 -

 Key: YARN-2517
 URL: https://issues.apache.org/jira/browse/YARN-2517
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Tsuyoshi OZAWA
 Attachments: YARN-2517.1.patch, YARN-2517.2.patch


 In some scenarios, we'd like to put timeline entities in another thread no to 
 block the current one.
 It's good to have a TimelineClientAsync like AMRMClientAsync and 
 NMClientAsync. It can buffer entities, put them in a separate thread, and 
 have callback to handle the responses.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-2517) Implement TimelineClientAsync

2014-11-24 Thread Mit Desai (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223091#comment-14223091
 ] 

Mit Desai commented on YARN-2517:
-

I had similar concerns. Do we really need this at this point? And as Hitesh 
pointed out, if this may hinder the design in future.

bq.  Also, is the timeline layer meant to eventually be reliable and always up? 
As far as what I am aware of, this is not going to be in near future.

 Implement TimelineClientAsync
 -

 Key: YARN-2517
 URL: https://issues.apache.org/jira/browse/YARN-2517
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Tsuyoshi OZAWA
 Attachments: YARN-2517.1.patch, YARN-2517.2.patch


 In some scenarios, we'd like to put timeline entities in another thread no to 
 block the current one.
 It's good to have a TimelineClientAsync like AMRMClientAsync and 
 NMClientAsync. It can buffer entities, put them in a separate thread, and 
 have callback to handle the responses.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-2517) Implement TimelineClientAsync

2014-11-24 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223299#comment-14223299
 ] 

Sangjin Lee commented on YARN-2517:
---

It might be conceptually cleaner (and simpler to the clients) to have the 
clients post the events but not deal with what would happen if the result was 
not successfully sent. Even the very concept of whether the result was sent 
is problematic. An ATS writer implementation could make a decision of buffering 
a certain amount of data before it physically writes it to the backing storage 
(as optimization).

If the client needs to know whether the event was truly sent to the backing 
storage and also deal with its failure, it may lead to ATS writer 
implementations leaking to clients and may limit the way an ATS write 
implementation can be optimized, etc.

How about a sync write (as it stands now) for critical data and an async write 
which is basically fire-and-forget? The understanding there would be the async 
write is basically a best effort and it would fall on the implementation of ATS 
writes to try to deliver the events to the backing storage as reliably and 
optimally as it can (but in theory no guarantees still). Then the async write 
can even be enabled with a boolean flag.

 Implement TimelineClientAsync
 -

 Key: YARN-2517
 URL: https://issues.apache.org/jira/browse/YARN-2517
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Tsuyoshi OZAWA
 Attachments: YARN-2517.1.patch, YARN-2517.2.patch


 In some scenarios, we'd like to put timeline entities in another thread no to 
 block the current one.
 It's good to have a TimelineClientAsync like AMRMClientAsync and 
 NMClientAsync. It can buffer entities, put them in a separate thread, and 
 have callback to handle the responses.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-2517) Implement TimelineClientAsync

2014-11-24 Thread Zhijie Shen (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223923#comment-14223923
 ] 

Zhijie Shen commented on YARN-2517:
---

Thanks for sharing your idea, Hitesh, Mit and Sangjin! I'd like to make some 
clarification. The error that the handler wants to take care of is not the 
communication problem, but the problem that happens when the server is 
processing the posted timeline entity (See TimelinePutResponse). It could be 
the data integrity issue that is messed up by the app. For example, the posted 
Entity A in Domain 1 is trying to relate to Entity B in Domain 2. It's fine if 
in some use cases, the app doesn't want to care about it, and consequently 
doesn't require an Ack. The app can just go ahead without providing the 
handler. However, it's still better to be generic enough to cover the other use 
cases where the app wants to make sure it has the timeline data persisted, or 
at least know it is succeeded or not.

Queueing/messaging layer may help to mitigate the communication problem, but 
aforementioned data problem still could happen, such that the app may still 
want to hear about the put response. However, it sounds right that the 
implementation of this layer will affect that of async call. It makes sense to 
wait until we make it clear how to make client - TS communication reliable. 
If we eventually find handler in async call is difficult and will further 
prevent optimization, a sync write (as it stands now) for critical data and an 
async write which is basically fire-and-forge sounds a reasonable alternative 
plan.

 Implement TimelineClientAsync
 -

 Key: YARN-2517
 URL: https://issues.apache.org/jira/browse/YARN-2517
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Tsuyoshi OZAWA
 Attachments: YARN-2517.1.patch, YARN-2517.2.patch


 In some scenarios, we'd like to put timeline entities in another thread no to 
 block the current one.
 It's good to have a TimelineClientAsync like AMRMClientAsync and 
 NMClientAsync. It can buffer entities, put them in a separate thread, and 
 have callback to handle the responses.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-2517) Implement TimelineClientAsync

2014-11-23 Thread Hitesh Shah (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14222582#comment-14222582
 ] 

Hitesh Shah commented on YARN-2517:
---

Sorry for coming in late on this. Why is this needed? 

[~zjshen] [~ozawa] Is a callback layer back to the user really needed? How 
would the user make use of this callback information except maybe to log an 
error or fail the application? Also, is the timeline layer meant to eventually 
be reliable and always up? Let us say that sometime later, the timeline client 
started using some form of queueing/messaging layer to send the data to the 
server. When would a callback be invoked? When the data is written to the 
messaging layer or when the server actually processes the data? Implementing 
this now will inhibit the design down the line if it needs to retain the same 
semantics for the callback invocation ( which would be done today when the 
server finishes processing the data )





 Implement TimelineClientAsync
 -

 Key: YARN-2517
 URL: https://issues.apache.org/jira/browse/YARN-2517
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Tsuyoshi OZAWA
 Attachments: YARN-2517.1.patch, YARN-2517.2.patch


 In some scenarios, we'd like to put timeline entities in another thread no to 
 block the current one.
 It's good to have a TimelineClientAsync like AMRMClientAsync and 
 NMClientAsync. It can buffer entities, put them in a separate thread, and 
 have callback to handle the responses.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-2517) Implement TimelineClientAsync

2014-11-22 Thread Tsuyoshi OZAWA (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14222338#comment-14222338
 ] 

Tsuyoshi OZAWA commented on YARN-2517:
--

[~zjshen], thanks for your comment.

{code}
// Async call - Type (1)
void asyncCall(Input, CallBackHandler);
{code}

If we choose Type(1), TimelineClient need to manage all objects of callback 
handlers it's passed. It can get more complex. I think it's better to add 
{{registerAsyncCallbackHandler(CallBackHandler)}} to add TimelineClient 
separately or pass the callback via constructor for the simplicity. Or, do we 
have use cases to switch CallBackHandler for each method calls?

{quote}
Maybe compromise now is to add putEntitiesAsync to TimelineClient. In the 
future, let's see if we want to have a separate TimelineClientAsync that 
contains a bunch of async APIs.
{quote}

If we will have a plan to migrate from putEntitiesAsync to TimelineClientAsync, 
I think we should add TimelineClientAsync from the beginning. I think it's 
enough to add *Async methods to current TimelineClient for now since users 
don't need to create specific clients for async calls.

[~mitdesai] thanks for your help. Looking forward to your opinion. We're 
discussing the design based on [Vinod's 
suggestion|https://issues.apache.org/jira/browse/YARN-2517?focusedCommentId=14128819page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14128819].

 Implement TimelineClientAsync
 -

 Key: YARN-2517
 URL: https://issues.apache.org/jira/browse/YARN-2517
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Tsuyoshi OZAWA
 Attachments: YARN-2517.1.patch, YARN-2517.2.patch


 In some scenarios, we'd like to put timeline entities in another thread no to 
 block the current one.
 It's good to have a TimelineClientAsync like AMRMClientAsync and 
 NMClientAsync. It can buffer entities, put them in a separate thread, and 
 have callback to handle the responses.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-2517) Implement TimelineClientAsync

2014-11-21 Thread Mit Desai (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14221027#comment-14221027
 ] 

Mit Desai commented on YARN-2517:
-

Yes I have been working on timeline service recently. I will take a look.

 Implement TimelineClientAsync
 -

 Key: YARN-2517
 URL: https://issues.apache.org/jira/browse/YARN-2517
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Tsuyoshi OZAWA
 Attachments: YARN-2517.1.patch, YARN-2517.2.patch


 In some scenarios, we'd like to put timeline entities in another thread no to 
 block the current one.
 It's good to have a TimelineClientAsync like AMRMClientAsync and 
 NMClientAsync. It can buffer entities, put them in a separate thread, and 
 have callback to handle the responses.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-2517) Implement TimelineClientAsync

2014-11-21 Thread Zhijie Shen (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14221249#comment-14221249
 ] 

Zhijie Shen commented on YARN-2517:
---

I'm not sure Future is going to help the use case. Say I want to use 
TimelineClientAsync to do async put entity operations. With Future, 
putEntitiesAsync returns immediately. However, to know if my put entity 
operation is successful not, I still have to been blocked at Future#get(), or 
create a separate thread to wait for the response. But IMHO, one goal of 
TimelineClientAsync is to relieve users from multithreading details, such that 
Type (1) sounds better to me.

Rethink whether we create a separate TimelineClientAsync or add async method in 
TimelineClient. We have putEntities and putDomain, and in the future we will 
have more get APIs. For now, the most concerned API is putEntities, as we don't 
want it to block the normal execution logic of an app. Maybe compromise now is 
to add putEntitiesAsync to TimelineClient. In the future, let's see if we want 
to have a separate TimelineClientAsync that contains a bunch of async APIs.

Thoughts?

 Implement TimelineClientAsync
 -

 Key: YARN-2517
 URL: https://issues.apache.org/jira/browse/YARN-2517
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Tsuyoshi OZAWA
 Attachments: YARN-2517.1.patch, YARN-2517.2.patch


 In some scenarios, we'd like to put timeline entities in another thread no to 
 block the current one.
 It's good to have a TimelineClientAsync like AMRMClientAsync and 
 NMClientAsync. It can buffer entities, put them in a separate thread, and 
 have callback to handle the responses.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-2517) Implement TimelineClientAsync

2014-11-20 Thread Tsuyoshi OZAWA (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14219554#comment-14219554
 ] 

Tsuyoshi OZAWA commented on YARN-2517:
--

[~mitdesai] I think you're one of the users of TimelineClient. If you have any 
feedbacks about the interface, please let me know.

 Implement TimelineClientAsync
 -

 Key: YARN-2517
 URL: https://issues.apache.org/jira/browse/YARN-2517
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Tsuyoshi OZAWA
 Attachments: YARN-2517.1.patch, YARN-2517.2.patch


 In some scenarios, we'd like to put timeline entities in another thread no to 
 block the current one.
 It's good to have a TimelineClientAsync like AMRMClientAsync and 
 NMClientAsync. It can buffer entities, put them in a separate thread, and 
 have callback to handle the responses.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-2517) Implement TimelineClientAsync

2014-11-20 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14219587#comment-14219587
 ] 

Hadoop QA commented on YARN-2517:
-

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12682679/YARN-2517.2.patch
  against trunk revision a9a0cc3.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/5893//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5893//console

This message is automatically generated.

 Implement TimelineClientAsync
 -

 Key: YARN-2517
 URL: https://issues.apache.org/jira/browse/YARN-2517
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Tsuyoshi OZAWA
 Attachments: YARN-2517.1.patch, YARN-2517.2.patch


 In some scenarios, we'd like to put timeline entities in another thread no to 
 block the current one.
 It's good to have a TimelineClientAsync like AMRMClientAsync and 
 NMClientAsync. It can buffer entities, put them in a separate thread, and 
 have callback to handle the responses.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-2517) Implement TimelineClientAsync

2014-10-07 Thread Zhijie Shen (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14162820#comment-14162820
 ] 

Zhijie Shen commented on YARN-2517:
---

bq. Clearly, async calls need call-back handlers just for errors. As of today, 
there are no APIs that really need to send back results (not error) 
asynchronously. 

I realize that another requirement of TimelineClient is to wrap the read APIs 
to facilitate the java app developers. These APIs are going to return results.

 Implement TimelineClientAsync
 -

 Key: YARN-2517
 URL: https://issues.apache.org/jira/browse/YARN-2517
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Tsuyoshi OZAWA
 Attachments: YARN-2517.1.patch


 In some scenarios, we'd like to put timeline entities in another thread no to 
 block the current one.
 It's good to have a TimelineClientAsync like AMRMClientAsync and 
 NMClientAsync. It can buffer entities, put them in a separate thread, and 
 have callback to handle the responses.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-2517) Implement TimelineClientAsync

2014-09-11 Thread Tsuyoshi OZAWA (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14131087#comment-14131087
 ] 

Tsuyoshi OZAWA commented on YARN-2517:
--

I see. I'll try to prototype the new async method to post entities.

{quote}
If you guys think there is a lot more functionality coming in an async class in 
the future, can we hear about some of them here?
{quote}

Currently, TimelineClient just have APIs to post entities. I think there is a 
possibility to have helper methods like {{TimelineCilent#getEntities()}} for 
applications of Timeline Server.

 Implement TimelineClientAsync
 -

 Key: YARN-2517
 URL: https://issues.apache.org/jira/browse/YARN-2517
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Tsuyoshi OZAWA
 Attachments: YARN-2517.1.patch


 In some scenarios, we'd like to put timeline entities in another thread no to 
 block the current one.
 It's good to have a TimelineClientAsync like AMRMClientAsync and 
 NMClientAsync. It can buffer entities, put them in a separate thread, and 
 have callback to handle the responses.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-2517) Implement TimelineClientAsync

2014-09-10 Thread Zhijie Shen (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14128156#comment-14128156
 ] 

Zhijie Shen commented on YARN-2517:
---

Scan through the patch, the approach is quite close to 
AMRMClientAsync/NMClientAsync. It looks fine to me in general. Later on, we can 
improve the client step by step. For example, according to the discussion on 
the umbrella, we can persist the queued entities to be reliable. And we may 
want to allow multiple threads to put entities.

 Implement TimelineClientAsync
 -

 Key: YARN-2517
 URL: https://issues.apache.org/jira/browse/YARN-2517
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Tsuyoshi OZAWA
 Attachments: YARN-2517.1.patch


 In some scenarios, we'd like to put timeline entities in another thread no to 
 block the current one.
 It's good to have a TimelineClientAsync like AMRMClientAsync and 
 NMClientAsync. It can buffer entities, put them in a separate thread, and 
 have callback to handle the responses.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-2517) Implement TimelineClientAsync

2014-09-10 Thread Tsuyoshi OZAWA (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14128567#comment-14128567
 ] 

Tsuyoshi OZAWA commented on YARN-2517:
--

Thanks for your review, Zhijie. I think batch optimization and persisting 
entities can be done in sync client since the async client uses sync client. 

Submitting a patch again for merging. Please let me know if you have additional 
review comments.

 Implement TimelineClientAsync
 -

 Key: YARN-2517
 URL: https://issues.apache.org/jira/browse/YARN-2517
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Tsuyoshi OZAWA
 Attachments: YARN-2517.1.patch


 In some scenarios, we'd like to put timeline entities in another thread no to 
 block the current one.
 It's good to have a TimelineClientAsync like AMRMClientAsync and 
 NMClientAsync. It can buffer entities, put them in a separate thread, and 
 have callback to handle the responses.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-2517) Implement TimelineClientAsync

2014-09-10 Thread Vinod Kumar Vavilapalli (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14128819#comment-14128819
 ] 

Vinod Kumar Vavilapalli commented on YARN-2517:
---

I am +1 about a client that makes async calls. The question is whether we need 
a a new client class (and thus a public interface) or not.

Clearly, async calls need call-back handlers _just_ for errors. As of today, 
there are no APIs that really need to send back *results* (not error) 
asynchronously. The way you usually handle it is through one of the following
{code}
// Sync call
Result call(Input);
// Async call - Type (1)
void asyncCall(Input, CallBackHandler);
// Async call - Type (2)
Future asyncCall(Input);
{code}
You can do type (1). Having an entire separate client side interface isn't 
mandatory.

If you guys think there is a lot more functionality coming in an async class in 
the future, can we hear about some of them here?

 Implement TimelineClientAsync
 -

 Key: YARN-2517
 URL: https://issues.apache.org/jira/browse/YARN-2517
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Tsuyoshi OZAWA
 Attachments: YARN-2517.1.patch


 In some scenarios, we'd like to put timeline entities in another thread no to 
 block the current one.
 It's good to have a TimelineClientAsync like AMRMClientAsync and 
 NMClientAsync. It can buffer entities, put them in a separate thread, and 
 have callback to handle the responses.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-2517) Implement TimelineClientAsync

2014-09-09 Thread Tsuyoshi OZAWA (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14128097#comment-14128097
 ] 

Tsuyoshi OZAWA commented on YARN-2517:
--

[~zjshen], [~vinodkv], can we go with current design(v1 patch)?

 Implement TimelineClientAsync
 -

 Key: YARN-2517
 URL: https://issues.apache.org/jira/browse/YARN-2517
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Tsuyoshi OZAWA
 Attachments: YARN-2517.1.patch


 In some scenarios, we'd like to put timeline entities in another thread no to 
 block the current one.
 It's good to have a TimelineClientAsync like AMRMClientAsync and 
 NMClientAsync. It can buffer entities, put them in a separate thread, and 
 have callback to handle the responses.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-2517) Implement TimelineClientAsync

2014-09-08 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125664#comment-14125664
 ] 

Hadoop QA commented on YARN-2517:
-

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12667168/YARN-2517.1.patch
  against trunk revision 0974f43.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/4842//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4842//console

This message is automatically generated.

 Implement TimelineClientAsync
 -

 Key: YARN-2517
 URL: https://issues.apache.org/jira/browse/YARN-2517
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Tsuyoshi OZAWA
 Attachments: YARN-2517.1.patch


 In some scenarios, we'd like to put timeline entities in another thread no to 
 block the current one.
 It's good to have a TimelineClientAsync like AMRMClientAsync and 
 NMClientAsync. It can buffer entities, put them in a separate thread, and 
 have callback to handle the responses.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-2517) Implement TimelineClientAsync

2014-09-08 Thread Vinod Kumar Vavilapalli (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125714#comment-14125714
 ] 

Vinod Kumar Vavilapalli commented on YARN-2517:
---

I am not entirely sure we need a parallel client for this. The other clients 
needed async clients because
 - they had loads of functionality that made sense in the blocking and 
non-blocking modes
 - the client code really needed call-back hooks to act on the results.

Timeline Client's only responsibility is to post events. There are only two 
use-cases: Clients need a sync write through, or an asynchronous write, the end 
of which they don't care about. I think we should simply have a mode in the 
existing client to post events asynchronously without any further need for 
call-back handlers.

What do others think?

 Implement TimelineClientAsync
 -

 Key: YARN-2517
 URL: https://issues.apache.org/jira/browse/YARN-2517
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Tsuyoshi OZAWA
 Attachments: YARN-2517.1.patch


 In some scenarios, we'd like to put timeline entities in another thread no to 
 block the current one.
 It's good to have a TimelineClientAsync like AMRMClientAsync and 
 NMClientAsync. It can buffer entities, put them in a separate thread, and 
 have callback to handle the responses.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-2517) Implement TimelineClientAsync

2014-09-08 Thread Tsuyoshi OZAWA (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125760#comment-14125760
 ] 

Tsuyoshi OZAWA commented on YARN-2517:
--

Thanks for your comment, Vinod.

{quote}
an asynchronous write, the end of which they don't care about. I think we 
should simply have a mode in the existing client to post events asynchronously 
without any further need for call-back handlers.
{quote}

Make sense. We can assure at-most-once semantics without any callbacks. How 
about adding a {{flush()}} API to TimelineClient for asynchronous mode? It 
helps users to know whether contents of current buffer are written to Timeline 
Server or not.

 Implement TimelineClientAsync
 -

 Key: YARN-2517
 URL: https://issues.apache.org/jira/browse/YARN-2517
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Tsuyoshi OZAWA
 Attachments: YARN-2517.1.patch


 In some scenarios, we'd like to put timeline entities in another thread no to 
 block the current one.
 It's good to have a TimelineClientAsync like AMRMClientAsync and 
 NMClientAsync. It can buffer entities, put them in a separate thread, and 
 have callback to handle the responses.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-2517) Implement TimelineClientAsync

2014-09-08 Thread Zhijie Shen (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125777#comment-14125777
 ] 

Zhijie Shen commented on YARN-2517:
---

[~vinodkv], thanks for your feedback. The reason why an async client (or async 
HTTP REST call) is going to be good is to unblock the current thread if it is 
doing the important management logic. For example, in YARN-2033, we have a 
bunch of logic to dispatch the entity putting action on a separate thread, to 
make the application life cycle management move on. Given an async client, it 
could be far more simplified. I think from the user point of view, it may be a 
useful feature as well.

I'm fine whether we can two classes, sync for one and async for the other,  or 
one class for both modes, while the former option complies with the previous 
client design. I think the callback is necessary, at least onError. 
TimelinePutResponse will give the user a summary of why his uploaded entity is 
not accepted by the timeline server. Based on the response, the user can 
determine whether the app should neglect the problem and move on, or stop 
immediately.

 Implement TimelineClientAsync
 -

 Key: YARN-2517
 URL: https://issues.apache.org/jira/browse/YARN-2517
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Tsuyoshi OZAWA
 Attachments: YARN-2517.1.patch


 In some scenarios, we'd like to put timeline entities in another thread no to 
 block the current one.
 It's good to have a TimelineClientAsync like AMRMClientAsync and 
 NMClientAsync. It can buffer entities, put them in a separate thread, and 
 have callback to handle the responses.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-2517) Implement TimelineClientAsync

2014-09-08 Thread Tsuyoshi OZAWA (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14126557#comment-14126557
 ] 

Tsuyoshi OZAWA commented on YARN-2517:
--

As Zhijie mentioned, we should have the callback if we need to check errors. 
IMHO, if we have a thread for the callback onError, we should also have 
onEntitiesPut since the complexity doesn't increase so much and it's useful 
to distinguish connection-level exceptions from entity-level errors.

 Implement TimelineClientAsync
 -

 Key: YARN-2517
 URL: https://issues.apache.org/jira/browse/YARN-2517
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Tsuyoshi OZAWA
 Attachments: YARN-2517.1.patch


 In some scenarios, we'd like to put timeline entities in another thread no to 
 block the current one.
 It's good to have a TimelineClientAsync like AMRMClientAsync and 
 NMClientAsync. It can buffer entities, put them in a separate thread, and 
 have callback to handle the responses.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)