subject:"\[jira\] \[Commented\] \(YARN\-3047\) \[Data Serving\] Set up ATS reader with basic request serving structure and lifecycle"


[ 
https://issues.apache.org/jira/browse/YARN-3047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14383402#comment-14383402
 ] 

Varun Saxena commented on YARN-3047:


We can discuss offline or in YARN-3051 regarding support of metrics. Store 
interface patch I will upload once this goes in as it will be on top of this 
one.
Coming to metrics, do we plan to match on the basis of metrics value ? Will 
include it in the interface. But that will be part of YARN-3051.

 [Data Serving] Set up ATS reader with basic request serving structure and 
 lifecycle
 ---

 Key: YARN-3047
 URL: https://issues.apache.org/jira/browse/YARN-3047
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Sangjin Lee
Assignee: Varun Saxena
 Attachments: Timeline_Reader(draft).pdf, YARN-3047.001.patch, 
 YARN-3047.003.patch, YARN-3047.005.patch, YARN-3047.02.patch, 
 YARN-3047.04.patch


 Per design in YARN-2938, set up the ATS reader as a service and implement the 
 basic structure as a service. It includes lifecycle management, request 
 serving, and so on.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-3047) [Data Serving] Set up ATS reader with basic request serving structure and lifecycle


[ 
https://issues.apache.org/jira/browse/YARN-3047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14383444#comment-14383444
 ] 

Varun Saxena commented on YARN-3047:


You are applying on a WINDOWS machine or LINUX ? Might have to do with CRLF 
characters. I converted the yarn.cmd part of the patch using unix2dos command 
because in normal Jenkins server, it doesn't apply without this.

 [Data Serving] Set up ATS reader with basic request serving structure and 
 lifecycle
 ---

 Key: YARN-3047
 URL: https://issues.apache.org/jira/browse/YARN-3047
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Sangjin Lee
Assignee: Varun Saxena
 Attachments: Timeline_Reader(draft).pdf, YARN-3047.001.patch, 
 YARN-3047.003.patch, YARN-3047.005.patch, YARN-3047.02.patch, 
 YARN-3047.04.patch


 Per design in YARN-2938, set up the ATS reader as a service and implement the 
 basic structure as a service. It includes lifecycle management, request 
 serving, and so on.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-3047) [Data Serving] Set up ATS reader with basic request serving structure and lifecycle


[ 
https://issues.apache.org/jira/browse/YARN-3047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14383440#comment-14383440
 ] 

Varun Saxena commented on YARN-3047:


Ok, we can see it later and implement it if required.

 [Data Serving] Set up ATS reader with basic request serving structure and 
 lifecycle
 ---

 Key: YARN-3047
 URL: https://issues.apache.org/jira/browse/YARN-3047
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Sangjin Lee
Assignee: Varun Saxena
 Attachments: Timeline_Reader(draft).pdf, YARN-3047.001.patch, 
 YARN-3047.003.patch, YARN-3047.005.patch, YARN-3047.02.patch, 
 YARN-3047.04.patch


 Per design in YARN-2938, set up the ATS reader as a service and implement the 
 basic structure as a service. It includes lifecycle management, request 
 serving, and so on.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-3047) [Data Serving] Set up ATS reader with basic request serving structure and lifecycle

2015-03-27 Thread Zhijie Shen (JIRA)

[
https://issues.apache.org/jira/browse/YARN-3047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14383438#comment-14383438
]

Zhijie Shen commented on YARN-3047:
---

Yeah, I did {{patch -p0}}, but applying failed at {{yarn.cmd}}. [~gtCarrera9],
does this patch work on you box?

bq. My assumption here is that we only care about per-application partial
order, but not a total order on the timeline server.

IMHO, we may care about total order. To be more accurate, it may not be
total, but among the applications of the same flow. I think eventually we
should support reading from collector to get up-to-date/realtime timeline data.

bq. If later we deem that the view is too stale and we need to get the more
recent data from the timeline collector, we could add that functionality in a
separate effort.

Exactly, I agree with this prioritization. Let's start from reading the
backend, and we will get a better sense how stale the timeline data will be.

[Data Serving] Set up ATS reader with basic request serving structure and
lifecycle
---

Key: YARN-3047
URL: https://issues.apache.org/jira/browse/YARN-3047
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Reporter: Sangjin Lee
Assignee: Varun Saxena
Attachments: Timeline_Reader(draft).pdf, YARN-3047.001.patch,
YARN-3047.003.patch, YARN-3047.005.patch, YARN-3047.02.patch,
YARN-3047.04.patch

Per design in YARN-2938, set up the ATS reader as a service and implement the
basic structure as a service. It includes lifecycle management, request
serving, and so on.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-3047) [Data Serving] Set up ATS reader with basic request serving structure and lifecycle


[ 
https://issues.apache.org/jira/browse/YARN-3047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14383504#comment-14383504
 ] 

Varun Saxena commented on YARN-3047:


And I think max number of HttpServer threads should not be merely 10. Filed 
YARN-3407 for it

 [Data Serving] Set up ATS reader with basic request serving structure and 
 lifecycle
 ---

 Key: YARN-3047
 URL: https://issues.apache.org/jira/browse/YARN-3047
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Sangjin Lee
Assignee: Varun Saxena
 Attachments: Timeline_Reader(draft).pdf, YARN-3047.001.patch, 
 YARN-3047.003.patch, YARN-3047.005.patch, YARN-3047.02.patch, 
 YARN-3047.04.patch


 Per design in YARN-2938, set up the ATS reader as a service and implement the 
 basic structure as a service. It includes lifecycle management, request 
 serving, and so on.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-3047) [Data Serving] Set up ATS reader with basic request serving structure and lifecycle


[ 
https://issues.apache.org/jira/browse/YARN-3047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14383494#comment-14383494
 ] 

Varun Saxena commented on YARN-3047:


[~gtCarrera], the threads I am talking about in the design doc are the 
HttpServer2 threads. HttpServer2 has a thread pool and it will launch multiple 
threads to process requests. And hence our REST calls will also be in parallel. 
The thread pool in HttpServer2 works like this.

If an idle thread is available a job, it is directly dispatched, otherwise the 
job is queued.  After queuing a job, if the total number of threads is less 
than the maximum pool size, a new thread is spawned.

This is what I was talking about in the doc. Am I missing something ?

 [Data Serving] Set up ATS reader with basic request serving structure and 
 lifecycle
 ---

 Key: YARN-3047
 URL: https://issues.apache.org/jira/browse/YARN-3047
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Sangjin Lee
Assignee: Varun Saxena
 Attachments: Timeline_Reader(draft).pdf, YARN-3047.001.patch, 
 YARN-3047.003.patch, YARN-3047.005.patch, YARN-3047.02.patch, 
 YARN-3047.04.patch


 Per design in YARN-2938, set up the ATS reader as a service and implement the 
 basic structure as a service. It includes lifecycle management, request 
 serving, and so on.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-3047) [Data Serving] Set up ATS reader with basic request serving structure and lifecycle


[ 
https://issues.apache.org/jira/browse/YARN-3047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14384244#comment-14384244
 ] 

Sangjin Lee commented on YARN-3047:
---

bq. My assumption here is that we only care about per-application partial 
order, but not a total order on the timeline server. So I assume there are 
multiple timelines on the server, one for each application, but not a global 
timeline (which cost much more). Based on this assumption I think we can focus 
on reading from storage level firstly.

I'm not quite sure what you meant by this. Could you kindly elaborate on the 
partial order and the total order? Maybe I missed some context.


 [Data Serving] Set up ATS reader with basic request serving structure and 
 lifecycle
 ---

 Key: YARN-3047
 URL: https://issues.apache.org/jira/browse/YARN-3047
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Sangjin Lee
Assignee: Varun Saxena
 Attachments: Timeline_Reader(draft).pdf, YARN-3047.001.patch, 
 YARN-3047.003.patch, YARN-3047.005.patch, YARN-3047.006.patch, 
 YARN-3047.02.patch, YARN-3047.04.patch


 Per design in YARN-2938, set up the ATS reader as a service and implement the 
 basic structure as a service. It includes lifecycle management, request 
 serving, and so on.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-3047) [Data Serving] Set up ATS reader with basic request serving structure and lifecycle

2015-03-27 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-3047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14384035#comment-14384035
 ] 

Hadoop QA commented on YARN-3047:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12707786/YARN-3047.006.patch
  against trunk revision 05499b1.

{color:red}-1 patch{color}.  The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7125//console

This message is automatically generated.

 [Data Serving] Set up ATS reader with basic request serving structure and 
 lifecycle
 ---

 Key: YARN-3047
 URL: https://issues.apache.org/jira/browse/YARN-3047
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Sangjin Lee
Assignee: Varun Saxena
 Attachments: Timeline_Reader(draft).pdf, YARN-3047.001.patch, 
 YARN-3047.003.patch, YARN-3047.005.patch, YARN-3047.006.patch, 
 YARN-3047.02.patch, YARN-3047.04.patch


 Per design in YARN-2938, set up the ATS reader as a service and implement the 
 basic structure as a service. It includes lifecycle management, request 
 serving, and so on.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-3047) [Data Serving] Set up ATS reader with basic request serving structure and lifecycle


[ 
https://issues.apache.org/jira/browse/YARN-3047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14384024#comment-14384024
 ] 

Varun Saxena commented on YARN-3047:


[~zjshen], have uploaded another patch {{YARN-3047.006.patch}}. Check if this 
applies. In this patch I have not changed the line endings. Windows script 
normally has issues in applying due to line endings. You may have cloned the 
repo with line endings which are not same as whats in my directory. 
Previous patch was according to what applies in Jenkins. I had faced this issue 
with another patch earlier.

 [Data Serving] Set up ATS reader with basic request serving structure and 
 lifecycle
 ---

 Key: YARN-3047
 URL: https://issues.apache.org/jira/browse/YARN-3047
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Sangjin Lee
Assignee: Varun Saxena
 Attachments: Timeline_Reader(draft).pdf, YARN-3047.001.patch, 
 YARN-3047.003.patch, YARN-3047.005.patch, YARN-3047.006.patch, 
 YARN-3047.02.patch, YARN-3047.04.patch


 Per design in YARN-2938, set up the ATS reader as a service and implement the 
 basic structure as a service. It includes lifecycle management, request 
 serving, and so on.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-3047) [Data Serving] Set up ATS reader with basic request serving structure and lifecycle


[ 
https://issues.apache.org/jira/browse/YARN-3047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14384434#comment-14384434
 ] 

Varun Saxena commented on YARN-3047:


Updated a new patch

 [Data Serving] Set up ATS reader with basic request serving structure and 
 lifecycle
 ---

 Key: YARN-3047
 URL: https://issues.apache.org/jira/browse/YARN-3047
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Sangjin Lee
Assignee: Varun Saxena
 Attachments: Timeline_Reader(draft).pdf, YARN-3047.001.patch, 
 YARN-3047.003.patch, YARN-3047.005.patch, YARN-3047.006.patch, 
 YARN-3047.007.patch, YARN-3047.02.patch, YARN-3047.04.patch


 Per design in YARN-2938, set up the ATS reader as a service and implement the 
 basic structure as a service. It includes lifecycle management, request 
 serving, and so on.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-3047) [Data Serving] Set up ATS reader with basic request serving structure and lifecycle


[ 
https://issues.apache.org/jira/browse/YARN-3047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14384427#comment-14384427
 ] 

Varun Saxena commented on YARN-3047:


I think 005 patch would work for you then. This issue has to do with line 
endings.

TimelineEvents has been copied over from ATS v1 to support getEvents API. I 
guess we have to support this v1 API.

 [Data Serving] Set up ATS reader with basic request serving structure and 
 lifecycle
 ---

 Key: YARN-3047
 URL: https://issues.apache.org/jira/browse/YARN-3047
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Sangjin Lee
Assignee: Varun Saxena
 Attachments: Timeline_Reader(draft).pdf, YARN-3047.001.patch, 
 YARN-3047.003.patch, YARN-3047.005.patch, YARN-3047.006.patch, 
 YARN-3047.007.patch, YARN-3047.02.patch, YARN-3047.04.patch


 Per design in YARN-2938, set up the ATS reader as a service and implement the 
 basic structure as a service. It includes lifecycle management, request 
 serving, and so on.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-3047) [Data Serving] Set up ATS reader with basic request serving structure and lifecycle


[ 
https://issues.apache.org/jira/browse/YARN-3047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14384285#comment-14384285
 ] 

Sangjin Lee commented on YARN-3047:
---

Took a quick look at the latest patch (v06). Looks good for the most part, but 
Field.java and TestTimelineReaderServer.java are missing the license. Could you 
please fix that quickly? The patch applies cleanly for me BTW.

bq. And I think max number of HttpServer threads should not be merely 10. Filed 
YARN-3407 for it

If I'm not mistaken, it looks like the number of threads can be controlled by 
HTTP_MAX_THREADS? And also by default jetty maxes out at 250? Wouldn't that be 
enough?

 [Data Serving] Set up ATS reader with basic request serving structure and 
 lifecycle
 ---

 Key: YARN-3047
 URL: https://issues.apache.org/jira/browse/YARN-3047
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Sangjin Lee
Assignee: Varun Saxena
 Attachments: Timeline_Reader(draft).pdf, YARN-3047.001.patch, 
 YARN-3047.003.patch, YARN-3047.005.patch, YARN-3047.006.patch, 
 YARN-3047.02.patch, YARN-3047.04.patch


 Per design in YARN-2938, set up the ATS reader as a service and implement the 
 basic structure as a service. It includes lifecycle management, request 
 serving, and so on.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-3047) [Data Serving] Set up ATS reader with basic request serving structure and lifecycle

2015-03-27 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-3047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14384415#comment-14384415
 ] 

Hadoop QA commented on YARN-3047:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12707853/YARN-3047.007.patch
  against trunk revision 05499b1.

{color:red}-1 patch{color}.  The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7127//console

This message is automatically generated.

 [Data Serving] Set up ATS reader with basic request serving structure and 
 lifecycle
 ---

 Key: YARN-3047
 URL: https://issues.apache.org/jira/browse/YARN-3047
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Sangjin Lee
Assignee: Varun Saxena
 Attachments: Timeline_Reader(draft).pdf, YARN-3047.001.patch, 
 YARN-3047.003.patch, YARN-3047.005.patch, YARN-3047.006.patch, 
 YARN-3047.007.patch, YARN-3047.02.patch, YARN-3047.04.patch


 Per design in YARN-2938, set up the ATS reader as a service and implement the 
 basic structure as a service. It includes lifecycle management, request 
 serving, and so on.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-3047) [Data Serving] Set up ATS reader with basic request serving structure and lifecycle


[ 
https://issues.apache.org/jira/browse/YARN-3047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14384361#comment-14384361
 ] 

Sangjin Lee commented on YARN-3047:
---

One more thing: I notice that TimelineReaderManager uses object types (Long) 
for primitive types in its methods. Is there a strong reason to do this? If 
not, I would prefer staying with primitive types to avoid unnecessary boxing 
and unboxing.

I'm also going to go over the document.

 [Data Serving] Set up ATS reader with basic request serving structure and 
 lifecycle
 ---

 Key: YARN-3047
 URL: https://issues.apache.org/jira/browse/YARN-3047
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Sangjin Lee
Assignee: Varun Saxena
 Attachments: Timeline_Reader(draft).pdf, YARN-3047.001.patch, 
 YARN-3047.003.patch, YARN-3047.005.patch, YARN-3047.006.patch, 
 YARN-3047.02.patch, YARN-3047.04.patch


 Per design in YARN-2938, set up the ATS reader as a service and implement the 
 basic structure as a service. It includes lifecycle management, request 
 serving, and so on.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-3047) [Data Serving] Set up ATS reader with basic request serving structure and lifecycle

2015-03-27 Thread Li Lu (JIRA)

[
https://issues.apache.org/jira/browse/YARN-3047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14384381#comment-14384381
]

Li Lu commented on YARN-3047:
-

I tried the latest (006) patch on my machine. There were some hunk fails with
.cmd file:
{code}
patching file hadoop-yarn-project/hadoop-yarn/bin/yarn
patching file hadoop-yarn-project/hadoop-yarn/bin/yarn.cmd
Hunk #1 FAILED at 151.
Hunk #2 FAILED at 242.
Hunk #3 FAILED at 317.
3 out of 3 hunks FAILED -- saving rejects to file
hadoop-yarn-project/hadoop-yarn/bin/yarn.cmd.rej
{code}

BTW, why TimelineEvents.java is still here?

For the documentation, I know what a thread pool is but my point is maybe we
don't need to reach into the HttpServer2 level for now. We may also want to
focus on building the whole thing up rather than deciding the best parameters
for performance tuning.

[Data Serving] Set up ATS reader with basic request serving structure and
lifecycle
---

Per design in YARN-2938, set up the ATS reader as a service and implement the
basic structure as a service. It includes lifecycle management, request
serving, and so on.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-3047) [Data Serving] Set up ATS reader with basic request serving structure and lifecycle


[ 
https://issues.apache.org/jira/browse/YARN-3047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14384645#comment-14384645
 ] 

Varun Saxena commented on YARN-3047:


This JIRA was just meant to start reader as a daemon and start some basic 
services. This can act as base code which we add to as we reach consensus on 
reader API. Thoughts ?

 [Data Serving] Set up ATS reader with basic request serving structure and 
 lifecycle
 ---

 Key: YARN-3047
 URL: https://issues.apache.org/jira/browse/YARN-3047
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Sangjin Lee
Assignee: Varun Saxena
 Attachments: Timeline_Reader(draft).pdf, YARN-3047.001.patch, 
 YARN-3047.003.patch, YARN-3047.005.patch, YARN-3047.006.patch, 
 YARN-3047.007.patch, YARN-3047.02.patch, YARN-3047.04.patch


 Per design in YARN-2938, set up the ATS reader as a service and implement the 
 basic structure as a service. It includes lifecycle management, request 
 serving, and so on.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-3047) [Data Serving] Set up ATS reader with basic request serving structure and lifecycle

2015-03-27 Thread Li Lu (JIRA)

[
https://issues.apache.org/jira/browse/YARN-3047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14384609#comment-14384609
]

Li Lu commented on YARN-3047:
-

Given the current discussion in YARN-3051, seems for now we have not reached a
rough conclusion on the V2 timeline reader APIs. In the patch of this JIRA,
we're simply assuming the very least subset of reader APIs, but in the
foreseeable future we may probably add more to support more v2 queries. We may
want to minimize such kind of iterations. IMHO, the reader layer should depend
on the reader APIs, but not the other way around. That said, we may want to
pause this JIRA until we have a relatively workable v2 API. After we feel we're
fine with the APIs, we can then rebase the patch to connect the relatively
simple wire up works here.

[Data Serving] Set up ATS reader with basic request serving structure and
lifecycle
---

Per design in YARN-2938, set up the ATS reader as a service and implement the
basic structure as a service. It includes lifecycle management, request
serving, and so on.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-3047) [Data Serving] Set up ATS reader with basic request serving structure and lifecycle

[
https://issues.apache.org/jira/browse/YARN-3047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14384998#comment-14384998
]

Sangjin Lee commented on YARN-3047:
---

I can understand both sides here. On the one hand, the scope of this JIRA
initially was to work out the basic structure of the timeline reader as a
daemon. On the other hand, the frontend APIs are intimately tied to the storage
reader API as well as the types of queries we need to support. Changes in one
are very likely to lead to changes in the other.

Either we commit this with the understanding that the frontend APIs are
basically a placeholder, or we combine this with the storage reader API work
(YARN-3051) and tackle it as a whole. Personally I'm fine with either approach,
whichever help us make progress...

[Data Serving] Set up ATS reader with basic request serving structure and
lifecycle
---

Per design in YARN-2938, set up the ATS reader as a service and implement the
basic structure as a service. It includes lifecycle management, request
serving, and so on.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-3047) [Data Serving] Set up ATS reader with basic request serving structure and lifecycle

2015-03-26 Thread Varun Saxena (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-3047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14382352#comment-14382352
 ] 

Varun Saxena commented on YARN-3047:


[~zjshen], the patch YARN-3047.04.patch applies for me using {{patch -p0}}. I 
had updated latest code as well. May I know where is it failing for you ?

 [Data Serving] Set up ATS reader with basic request serving structure and 
 lifecycle
 ---

 Key: YARN-3047
 URL: https://issues.apache.org/jira/browse/YARN-3047
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Sangjin Lee
Assignee: Varun Saxena
 Attachments: Timeline_Reader(draft).pdf, YARN-3047.001.patch, 
 YARN-3047.003.patch, YARN-3047.02.patch, YARN-3047.04.patch


 Per design in YARN-2938, set up the ATS reader as a service and implement the 
 basic structure as a service. It includes lifecycle management, request 
 serving, and so on.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-3047) [Data Serving] Set up ATS reader with basic request serving structure and lifecycle

2015-03-26 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-3047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14382690#comment-14382690
 ] 

Hadoop QA commented on YARN-3047:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12707592/YARN-3047.005.patch
  against trunk revision 61df1b2.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:red}-1 javac{color:red}.  The patch appears to cause the build to 
fail.

Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7117//console

This message is automatically generated.

 [Data Serving] Set up ATS reader with basic request serving structure and 
 lifecycle
 ---

 Key: YARN-3047
 URL: https://issues.apache.org/jira/browse/YARN-3047
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Sangjin Lee
Assignee: Varun Saxena
 Attachments: Timeline_Reader(draft).pdf, YARN-3047.001.patch, 
 YARN-3047.003.patch, YARN-3047.005.patch, YARN-3047.02.patch, 
 YARN-3047.04.patch


 Per design in YARN-2938, set up the ATS reader as a service and implement the 
 basic structure as a service. It includes lifecycle management, request 
 serving, and so on.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-3047) [Data Serving] Set up ATS reader with basic request serving structure and lifecycle

2015-03-26 Thread Li Lu (JIRA)

[
https://issues.apache.org/jira/browse/YARN-3047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14382866#comment-14382866
]

Li Lu commented on YARN-3047:
-

Hi [~varun_saxena], thanks for the doc! I have two general questions about your
proposed plan:
# I'm a little bit confused on Timeline Reader will be a single daemon(in the
initial phase). In reader overview section there are multiple threads in the
reader, are those threads managed in YARN-3047? Specifically, what is the
concrete plan for Phase 1 on reader's architecture, single daemon multiple
thread, or single daemon single thread? If it's the former, you may want to
update YARN-3047's patch, while if it's the latter, you may want to confirm
this and update the figure afterwards (not the top priority for now).
# On storage layer we're prioritizing timeline entities and metrics, it would
be great if there are some API support from reader level for metrics. For the
current progress on the storage layer, I'm not sure if we can finish V1 storage
support by the time you finish reader phase 1. We may probably need some
coordination on this.

[Data Serving] Set up ATS reader with basic request serving structure and
lifecycle
---

Per design in YARN-2938, set up the ATS reader as a service and implement the
basic structure as a service. It includes lifecycle management, request
serving, and so on.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-3047) [Data Serving] Set up ATS reader with basic request serving structure and lifecycle

2015-03-25 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-3047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380152#comment-14380152
 ] 

Hadoop QA commented on YARN-3047:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12707233/Timeline_Reader%28draft%29.pdf
  against trunk revision c906a1d.

{color:red}-1 patch{color}.  The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7105//console

This message is automatically generated.

 [Data Serving] Set up ATS reader with basic request serving structure and 
 lifecycle
 ---

 Key: YARN-3047
 URL: https://issues.apache.org/jira/browse/YARN-3047
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Sangjin Lee
Assignee: Varun Saxena
 Attachments: Timeline_Reader(draft).pdf, YARN-3047.001.patch, 
 YARN-3047.003.patch, YARN-3047.02.patch, YARN-3047.04.patch


 Per design in YARN-2938, set up the ATS reader as a service and implement the 
 basic structure as a service. It includes lifecycle management, request 
 serving, and so on.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-3047) [Data Serving] Set up ATS reader with basic request serving structure and lifecycle

2015-03-25 Thread Varun Saxena (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-3047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380148#comment-14380148
 ] 

Varun Saxena commented on YARN-3047:


Draft document for timeline reader attached.

 [Data Serving] Set up ATS reader with basic request serving structure and 
 lifecycle
 ---

 Key: YARN-3047
 URL: https://issues.apache.org/jira/browse/YARN-3047
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Sangjin Lee
Assignee: Varun Saxena
 Attachments: Timeline_Reader(draft).pdf, YARN-3047.001.patch, 
 YARN-3047.003.patch, YARN-3047.02.patch, YARN-3047.04.patch


 Per design in YARN-2938, set up the ATS reader as a service and implement the 
 basic structure as a service. It includes lifecycle management, request 
 serving, and so on.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-3047) [Data Serving] Set up ATS reader with basic request serving structure and lifecycle

2015-03-25 Thread Vinod Kumar Vavilapalli (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-3047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380495#comment-14380495
 ] 

Vinod Kumar Vavilapalli commented on YARN-3047:
---

Tx for the doc.

Re Multiple options to determine whether an app is active: NMs informing the 
readers is backwards to me, though it is proactive. We can just have readers 
talk to RM. We can further optimize the calls by letting the readers 
asynchronously talk to RM to maintain an in-memory cache of a limited number of 
applications that finished say in the last one hour - this should work given 
the locality of queries w.r.t time.

 [Data Serving] Set up ATS reader with basic request serving structure and 
 lifecycle
 ---

 Key: YARN-3047
 URL: https://issues.apache.org/jira/browse/YARN-3047
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Sangjin Lee
Assignee: Varun Saxena
 Attachments: Timeline_Reader(draft).pdf, YARN-3047.001.patch, 
 YARN-3047.003.patch, YARN-3047.02.patch, YARN-3047.04.patch


 Per design in YARN-2938, set up the ATS reader as a service and implement the 
 basic structure as a service. It includes lifecycle management, request 
 serving, and so on.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-3047) [Data Serving] Set up ATS reader with basic request serving structure and lifecycle

2015-03-25 Thread Zhijie Shen (JIRA)

[
https://issues.apache.org/jira/browse/YARN-3047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380988#comment-14380988
]

Zhijie Shen commented on YARN-3047:
---

Varun, thanks of uploading the new patch and the design doc. Some more comments
for the patch first:

1. The patch cannot apply cleanly.

2. Okay, let's keep it for now, and resolve it later, but can we change the
config name to yarn.timeline-service.reader.blah-blah? And we should add an
entry in yarn-default.xml.

bq. Anyways I am continuing with separate config for reader as of now. Let me
know if you have a difference in opinion owing to ease in migration.

3. In TestTimelineReaderWebServer, can we check if we can access the info
resource when the server is up?

4. AboutInfo is used in TimelineCollectorWebService too. Can we prevent
duplicating this class?

5. TimelineReaderServer only contains one web server, such that it will
terminates immediately as web server threads are set as daemon. It needs to be
fixed in some way. I also file a ticket: HADOOP-11749. See if it can be
resolved form HttpServer2.

I'll review the design doc a bit later.

[Data Serving] Set up ATS reader with basic request serving structure and
lifecycle
---

Per design in YARN-2938, set up the ATS reader as a service and implement the
basic structure as a service. It includes lifecycle management, request
serving, and so on.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-3047) [Data Serving] Set up ATS reader with basic request serving structure and lifecycle

2015-03-25 Thread Zhijie Shen (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-3047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14381103#comment-14381103
 ] 

Zhijie Shen commented on YARN-3047:
---

I've a general question about reading from collector to get the timeline data 
of a running app. I'm not sure why we cannot read the timeline data from the 
backend for a running app. While the app is running, it will send the timeline 
data to its corresponding collector, and collector won't hold this data until 
the app is finished to start persisting it in the backend, right? The problem I 
can think of is that the collector process the timeline data in async manner, 
and the app is sending the timeline data intensively, such that there's still 
some timeline data which is already sent to the collector, but not persisted in 
the backend. In this case, to completely answer a user's query, the reader 
needs to combine the persisted data in the backend and the pending data in 
collector. That said, for running app, we can and should still read from the 
backend. Thoughts?

Moreover, as the timeline data keeps flowing into the backend when the app is 
running, we can get the final consistent results from the backend only, if we 
don't need to support real-time use case immediately. IMHO, in terms of the 
steps to move the reader work forward, we can first focus on querying the data 
from the backend, and then from collector as well.

 [Data Serving] Set up ATS reader with basic request serving structure and 
 lifecycle
 ---

 Key: YARN-3047
 URL: https://issues.apache.org/jira/browse/YARN-3047
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Sangjin Lee
Assignee: Varun Saxena
 Attachments: Timeline_Reader(draft).pdf, YARN-3047.001.patch, 
 YARN-3047.003.patch, YARN-3047.02.patch, YARN-3047.04.patch


 Per design in YARN-2938, set up the ATS reader as a service and implement the 
 basic structure as a service. It includes lifecycle management, request 
 serving, and so on.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-3047) [Data Serving] Set up ATS reader with basic request serving structure and lifecycle

2015-03-25 Thread Varun Saxena (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-3047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14381394#comment-14381394
 ] 

Varun Saxena commented on YARN-3047:


Ok...Will make the changes as per your suggestion and update a patch by EOD 
today.

 [Data Serving] Set up ATS reader with basic request serving structure and 
 lifecycle
 ---

 Key: YARN-3047
 URL: https://issues.apache.org/jira/browse/YARN-3047
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Sangjin Lee
Assignee: Varun Saxena
 Attachments: Timeline_Reader(draft).pdf, YARN-3047.001.patch, 
 YARN-3047.003.patch, YARN-3047.02.patch, YARN-3047.04.patch


 Per design in YARN-2938, set up the ATS reader as a service and implement the 
 basic structure as a service. It includes lifecycle management, request 
 serving, and so on.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-3047) [Data Serving] Set up ATS reader with basic request serving structure and lifecycle

2015-03-25 Thread Varun Saxena (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-3047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14381398#comment-14381398
 ] 

Varun Saxena commented on YARN-3047:


As per initial design, container related data will be first buffered and then 
written to store because if we write it immediately, there might be too many 
writes for a big cluster. [~sjlee0], thoughts ?

 [Data Serving] Set up ATS reader with basic request serving structure and 
 lifecycle
 ---

 Key: YARN-3047
 URL: https://issues.apache.org/jira/browse/YARN-3047
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Sangjin Lee
Assignee: Varun Saxena
 Attachments: Timeline_Reader(draft).pdf, YARN-3047.001.patch, 
 YARN-3047.003.patch, YARN-3047.02.patch, YARN-3047.04.patch


 Per design in YARN-2938, set up the ATS reader as a service and implement the 
 basic structure as a service. It includes lifecycle management, request 
 serving, and so on.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-3047) [Data Serving] Set up ATS reader with basic request serving structure and lifecycle

2015-03-25 Thread Sangjin Lee (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-3047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14381416#comment-14381416
 ] 

Sangjin Lee commented on YARN-3047:
---

The main reason for going directly to the per-app timeline collector in the 
original design was to get the up-to-date view of the data. Having said that, I 
believe in subsequent discussions we felt that it was acceptable to have a 
somewhat stale view if it is due to data being flushed with some interval.

I am fine with moving ahead without talking to the timeline collector directly. 
If later we deem that the view is too stale and we need to get the more recent 
data from the timeline collector, we could add that functionality in a separate 
effort.

 [Data Serving] Set up ATS reader with basic request serving structure and 
 lifecycle
 ---

 Key: YARN-3047
 URL: https://issues.apache.org/jira/browse/YARN-3047
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Sangjin Lee
Assignee: Varun Saxena
 Attachments: Timeline_Reader(draft).pdf, YARN-3047.001.patch, 
 YARN-3047.003.patch, YARN-3047.02.patch, YARN-3047.04.patch


 Per design in YARN-2938, set up the ATS reader as a service and implement the 
 basic structure as a service. It includes lifecycle management, request 
 serving, and so on.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-3047) [Data Serving] Set up ATS reader with basic request serving structure and lifecycle

2015-03-24 Thread Zhijie Shen (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-3047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14379031#comment-14379031
 ] 

Zhijie Shen commented on YARN-3047:
---

[~varun_saxena], any luck to take a look at the latest comments? Thanks! - 
Zhijie

 [Data Serving] Set up ATS reader with basic request serving structure and 
 lifecycle
 ---

 Key: YARN-3047
 URL: https://issues.apache.org/jira/browse/YARN-3047
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Sangjin Lee
Assignee: Varun Saxena
 Attachments: YARN-3047.001.patch, YARN-3047.003.patch, 
 YARN-3047.02.patch


 Per design in YARN-2938, set up the ATS reader as a service and implement the 
 basic structure as a service. It includes lifecycle management, request 
 serving, and so on.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-3047) [Data Serving] Set up ATS reader with basic request serving structure and lifecycle

2015-03-24 Thread Varun Saxena (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-3047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14379218#comment-14379218
 ] 

Varun Saxena commented on YARN-3047:


None actually. Was mistaken. TimelineEvents is required because we will 
continue with three of the v1 APIs', one of which requires TimelineEvents.

 [Data Serving] Set up ATS reader with basic request serving structure and 
 lifecycle
 ---

 Key: YARN-3047
 URL: https://issues.apache.org/jira/browse/YARN-3047
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Sangjin Lee
Assignee: Varun Saxena
 Attachments: YARN-3047.001.patch, YARN-3047.003.patch, 
 YARN-3047.02.patch


 Per design in YARN-2938, set up the ATS reader as a service and implement the 
 basic structure as a service. It includes lifecycle management, request 
 serving, and so on.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-3047) [Data Serving] Set up ATS reader with basic request serving structure and lifecycle

2015-03-24 Thread Varun Saxena (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-3047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14379225#comment-14379225
 ] 

Varun Saxena commented on YARN-3047:


Ok.. Will upload a patch.

I meant for reader, you think we can use the same config as v1.
Anyways I am continuing with separate config for reader as of now. Let me know 
if you have a difference in opinion owing to ease in migration.

 [Data Serving] Set up ATS reader with basic request serving structure and 
 lifecycle
 ---

 Key: YARN-3047
 URL: https://issues.apache.org/jira/browse/YARN-3047
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Sangjin Lee
Assignee: Varun Saxena
 Attachments: YARN-3047.001.patch, YARN-3047.003.patch, 
 YARN-3047.02.patch


 Per design in YARN-2938, set up the ATS reader as a service and implement the 
 basic structure as a service. It includes lifecycle management, request 
 serving, and so on.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-3047) [Data Serving] Set up ATS reader with basic request serving structure and lifecycle

2015-03-24 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-3047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14379264#comment-14379264
 ] 

Hadoop QA commented on YARN-3047:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12707130/YARN-3047.04.patch
  against trunk revision 53a28af.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:red}-1 javac{color:red}.  The patch appears to cause the build to 
fail.

Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7098//console

This message is automatically generated.

 [Data Serving] Set up ATS reader with basic request serving structure and 
 lifecycle
 ---

 Key: YARN-3047
 URL: https://issues.apache.org/jira/browse/YARN-3047
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Sangjin Lee
Assignee: Varun Saxena
 Attachments: YARN-3047.001.patch, YARN-3047.003.patch, 
 YARN-3047.02.patch, YARN-3047.04.patch


 Per design in YARN-2938, set up the ATS reader as a service and implement the 
 basic structure as a service. It includes lifecycle management, request 
 serving, and so on.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-3047) [Data Serving] Set up ATS reader with basic request serving structure and lifecycle

2015-03-23 Thread Zhijie Shen (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-3047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14376109#comment-14376109
 ] 

Zhijie Shen commented on YARN-3047:
---

bq. We can probably move it to yarn-api.

I prefer to keeping it in the server module, unless it's supposed to be public 
to users.

bq. This has to be discussed though as Zhijie Shen thinks we can use the same 
v1 config.

My opinion is that collector should bind on a random port, which will be 
reported to timeline client. Reader on single daemon should start on a 
configured port, and users know it form the config.

bq. TimelineReaderWebServer

If you'd like to keep reader, I'm fine with it, but let's still say 
TimelineReaderServer. Meanwhile, TimelineReaderWebService - 
TimelineReaderWebService*s*.

 [Data Serving] Set up ATS reader with basic request serving structure and 
 lifecycle
 ---

 Key: YARN-3047
 URL: https://issues.apache.org/jira/browse/YARN-3047
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Sangjin Lee
Assignee: Varun Saxena
 Attachments: YARN-3047.001.patch, YARN-3047.003.patch, 
 YARN-3047.02.patch


 Per design in YARN-2938, set up the ATS reader as a service and implement the 
 basic structure as a service. It includes lifecycle management, request 
 serving, and so on.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-3047) [Data Serving] Set up ATS reader with basic request serving structure and lifecycle

2015-03-23 Thread Li Lu (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-3047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14376719#comment-14376719
 ] 

Li Lu commented on YARN-3047:
-

Hi [~varun_saxena], thanks for the new patch. Could you please elaborate more 
about which exact comment will be addressed in YARN-3051? Thanks! BTW, in 003 
patch I can still see TimelineEvents.java. Do we still need that? 

 [Data Serving] Set up ATS reader with basic request serving structure and 
 lifecycle
 ---

 Key: YARN-3047
 URL: https://issues.apache.org/jira/browse/YARN-3047
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Sangjin Lee
Assignee: Varun Saxena
 Attachments: YARN-3047.001.patch, YARN-3047.003.patch, 
 YARN-3047.02.patch


 Per design in YARN-2938, set up the ATS reader as a service and implement the 
 basic structure as a service. It includes lifecycle management, request 
 serving, and so on.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-3047) [Data Serving] Set up ATS reader with basic request serving structure and lifecycle

2015-03-21 Thread Varun Saxena (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-3047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14372710#comment-14372710
 ] 

Varun Saxena commented on YARN-3047:


Thanks for your comments [~zjshen].

We can probably name it {{TimelineReaderWebServer}} then. Included Reader in 
the name so that it becomes clear that this web server acts as reader
I will close YARN-3048 then.

 [Data Serving] Set up ATS reader with basic request serving structure and 
 lifecycle
 ---

 Key: YARN-3047
 URL: https://issues.apache.org/jira/browse/YARN-3047
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Sangjin Lee
Assignee: Varun Saxena
 Attachments: YARN-3047.001.patch, YARN-3047.02.patch


 Per design in YARN-2938, set up the ATS reader as a service and implement the 
 basic structure as a service. It includes lifecycle management, request 
 serving, and so on.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-3047) [Data Serving] Set up ATS reader with basic request serving structure and lifecycle

2015-03-21 Thread Varun Saxena (JIRA)

[
https://issues.apache.org/jira/browse/YARN-3047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14372708#comment-14372708
]

Varun Saxena commented on YARN-3047:

Thanks for the comments [~gtCarrera]

bq. NameValuePair duplicates the same file in ahs, timeline/NameValuePair.
We can probably move it to yarn-api. Thoughts, [~zjshen].

bq. We're using Java 7 now so we can use switch statements for strings:
ok

bq. TimelineReaderServer, notice that the following lines:
ok, will fix.

bq. This about info seems to be confusing.
Will change it to Timeline Reader API.

bq. In the original design, there was a reader pool that holds a set of readers.
This will be handled in next phase(check YARN-3118). Right now, we will have a
single daemon.

bq. In TimelineReaderStore, DEFAULT_LIMIT is never used.
It will be used by subclasses. This is like v1 where if the client didn't
specify a limit/number of records to be returned, this DEFAULT_LIMIT was used.

bq. Why are we catching Exceptions, rather than the precise exception readValue
may throw?
Yes, we can catch them.

bq. Just want to double check that we do want to have the same end point for
both timeline readers and collectors. (Seems fine with me, since the reader
process runs on different machines as our collectors. )
Anyways I was using a different host and port configuration for reader. This
has to be discussed though as [~zjshen] thinks we can use the same v1 config.

bq. I would be very appreciate if you could upload a simple writeup for the
reader design.
Yes, working on it. Sorry for the delay. Got busy with other work last week.

[Data Serving] Set up ATS reader with basic request serving structure and
lifecycle
---

Per design in YARN-2938, set up the ATS reader as a service and implement the
basic structure as a service. It includes lifecycle management, request
serving, and so on.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-3047) [Data Serving] Set up ATS reader with basic request serving structure and lifecycle


[ 
https://issues.apache.org/jira/browse/YARN-3047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14372973#comment-14372973
 ] 

Hadoop QA commented on YARN-3047:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12706183/YARN-3047.003.patch
  against trunk revision e1feb4e.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:red}-1 javac{color:red}.  The patch appears to cause the build to 
fail.

Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7067//console

This message is automatically generated.

 [Data Serving] Set up ATS reader with basic request serving structure and 
 lifecycle
 ---

 Key: YARN-3047
 URL: https://issues.apache.org/jira/browse/YARN-3047
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Sangjin Lee
Assignee: Varun Saxena
 Attachments: YARN-3047.001.patch, YARN-3047.003.patch, 
 YARN-3047.02.patch


 Per design in YARN-2938, set up the ATS reader as a service and implement the 
 basic structure as a service. It includes lifecycle management, request 
 serving, and so on.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-3047) [Data Serving] Set up ATS reader with basic request serving structure and lifecycle


[ 
https://issues.apache.org/jira/browse/YARN-3047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14372980#comment-14372980
 ] 

Hadoop QA commented on YARN-3047:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12706184/YARN-3047.003.patch
  against trunk revision e1feb4e.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:red}-1 javac{color:red}.  The patch appears to cause the build to 
fail.

Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7068//console

This message is automatically generated.

 [Data Serving] Set up ATS reader with basic request serving structure and 
 lifecycle
 ---

 Key: YARN-3047
 URL: https://issues.apache.org/jira/browse/YARN-3047
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Sangjin Lee
Assignee: Varun Saxena
 Attachments: YARN-3047.001.patch, YARN-3047.003.patch, 
 YARN-3047.02.patch


 Per design in YARN-2938, set up the ATS reader as a service and implement the 
 basic structure as a service. It includes lifecycle management, request 
 serving, and so on.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-3047) [Data Serving] Set up ATS reader with basic request serving structure and lifecycle


[ 
https://issues.apache.org/jira/browse/YARN-3047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14372959#comment-14372959
 ] 

Hadoop QA commented on YARN-3047:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12706180/YARN-3047.03.patch
  against trunk revision e1feb4e.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:red}-1 javac{color:red}.  The patch appears to cause the build to 
fail.

Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7066//console

This message is automatically generated.

 [Data Serving] Set up ATS reader with basic request serving structure and 
 lifecycle
 ---

 Key: YARN-3047
 URL: https://issues.apache.org/jira/browse/YARN-3047
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Sangjin Lee
Assignee: Varun Saxena
 Attachments: YARN-3047.001.patch, YARN-3047.02.patch, 
 YARN-3047.03.patch


 Per design in YARN-2938, set up the ATS reader as a service and implement the 
 basic structure as a service. It includes lifecycle management, request 
 serving, and so on.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-3047) [Data Serving] Set up ATS reader with basic request serving structure and lifecycle