[jira] [Comment Edited] (YARN-4761) NMs reconnecting with changed capabilities can lead to wrong cluster resource calculations on fair scheduler

2018-01-30 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346254#comment-16346254
 ] 

Sangjin Lee edited comment on YARN-4761 at 1/31/18 5:12 AM:


+ [~templedf] [~yufeigu]

I'm not familiar enough with that piece of code. Daniel, Yufei, you guys know 
more about that code than I do. Thoughts?


was (Author: sjlee0):
+ [~templedf] [~yufeigu]

> NMs reconnecting with changed capabilities can lead to wrong cluster resource 
> calculations on fair scheduler
> 
>
> Key: YARN-4761
> URL: https://issues.apache.org/jira/browse/YARN-4761
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: fairscheduler
>Affects Versions: 2.6.4
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Major
> Fix For: 2.8.0, 2.7.3, 2.6.5, 3.0.0-alpha1
>
> Attachments: YARN-4761.01.patch, YARN-4761.02.patch
>
>
> YARN-3802 uncovered an issue with the scheduler where the resource 
> calculation can be incorrect due to async event handling. It was subsequently 
> fixed by YARN-4344, but it was never fixed for the fair scheduler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-4761) NMs reconnecting with changed capabilities can lead to wrong cluster resource calculations on fair scheduler

2018-01-30 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346254#comment-16346254
 ] 

Sangjin Lee commented on YARN-4761:
---

+ [~templedf] [~yufeigu]

> NMs reconnecting with changed capabilities can lead to wrong cluster resource 
> calculations on fair scheduler
> 
>
> Key: YARN-4761
> URL: https://issues.apache.org/jira/browse/YARN-4761
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: fairscheduler
>Affects Versions: 2.6.4
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Major
> Fix For: 2.8.0, 2.7.3, 2.6.5, 3.0.0-alpha1
>
> Attachments: YARN-4761.01.patch, YARN-4761.02.patch
>
>
> YARN-3802 uncovered an issue with the scheduler where the resource 
> calculation can be incorrect due to async event handling. It was subsequently 
> fixed by YARN-4344, but it was never fixed for the fair scheduler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Assigned] (YARN-6771) Use classloader inside configuration class to make new classes

2017-09-20 Thread Sangjin Lee (JIRA)

 [ 
https://issues.apache.org/jira/browse/YARN-6771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee reassigned YARN-6771:
-

Assignee: Jongyoul Lee

> Use classloader inside configuration class to make new classes 
> ---
>
> Key: YARN-6771
> URL: https://issues.apache.org/jira/browse/YARN-6771
> Project: Hadoop YARN
>  Issue Type: Bug
>Affects Versions: 2.8.1, 3.0.0-alpha4
>Reporter: Jongyoul Lee
>Assignee: Jongyoul Lee
> Attachments: YARN-6771-1.patch, YARN-6771-2.patch, YARN-6771-3.patch, 
> YARN-6771-4.patch, YARN-6771-5.patch, YARN-6771-6.patch, YARN-6771.patch
>
>
> While running {{RpcClientFactoryPBImpl.getClient}}, 
> {{RpcClientFactoryPBImpl}} uses {{localConf.getClassByName}}. But in case of 
> using custom classloader, we have to use {{conf.getClassByName}} because 
> custom classloader is already stored in {{Configuration}} class.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-6771) Use classloader inside configuration class to make new classes

2017-09-20 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-6771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16174281#comment-16174281
 ] 

Sangjin Lee commented on YARN-6771:
---

LGTM. Committing it shortly.

> Use classloader inside configuration class to make new classes 
> ---
>
> Key: YARN-6771
> URL: https://issues.apache.org/jira/browse/YARN-6771
> Project: Hadoop YARN
>  Issue Type: Bug
>Affects Versions: 2.8.1, 3.0.0-alpha4
>Reporter: Jongyoul Lee
> Attachments: YARN-6771-1.patch, YARN-6771-2.patch, YARN-6771-3.patch, 
> YARN-6771-4.patch, YARN-6771-5.patch, YARN-6771-6.patch, YARN-6771.patch
>
>
> While running {{RpcClientFactoryPBImpl.getClient}}, 
> {{RpcClientFactoryPBImpl}} uses {{localConf.getClassByName}}. But in case of 
> using custom classloader, we have to use {{conf.getClassByName}} because 
> custom classloader is already stored in {{Configuration}} class.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-6466) Provide shaded framework jar for containers

2017-08-11 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124097#comment-16124097
 ] 

Sangjin Lee commented on YARN-6466:
---

[~haibochen], I see you reassigned this to yourself. Let's resurrect the 
discussion on this one as it's been sitting idle for a while. Please have a 
look at the ongoing effort at HADOOP-13398 too. Thanks!

> Provide shaded framework jar for containers
> ---
>
> Key: YARN-6466
> URL: https://issues.apache.org/jira/browse/YARN-6466
> Project: Hadoop YARN
>  Issue Type: New Feature
>  Components: build, yarn
>Affects Versions: 3.0.0-alpha1
>Reporter: Sean Busbey
>Assignee: Haibo Chen
>
> We should build on the existing shading work to provide a jar with all of the 
> bits needed within a YARN application's container to talk to the resource 
> manager and node manager.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-6771) Use classloader inside configuration class to make new classes

2017-08-02 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-6771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16111829#comment-16111829
 ] 

Sangjin Lee commented on YARN-6771:
---

Thanks for the update. Could you please look at the checkstyle violations? They 
are related to your changes.

> Use classloader inside configuration class to make new classes 
> ---
>
> Key: YARN-6771
> URL: https://issues.apache.org/jira/browse/YARN-6771
> Project: Hadoop YARN
>  Issue Type: Bug
>Affects Versions: 2.8.1, 3.0.0-alpha4
>Reporter: Jongyoul Lee
> Fix For: 2.8.2
>
> Attachments: YARN-6771-1.patch, YARN-6771-2.patch, YARN-6771-3.patch, 
> YARN-6771.patch
>
>
> While running {{RpcClientFactoryPBImpl.getClient}}, 
> {{RpcClientFactoryPBImpl}} uses {{localConf.getClassByName}}. But in case of 
> using custom classloader, we have to use {{conf.getClassByName}} because 
> custom classloader is already stored in {{Configuration}} class.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-6771) Use classloader inside configuration class to make new classes

2017-07-21 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-6771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16096454#comment-16096454
 ] 

Sangjin Lee commented on YARN-6771:
---

Thanks for linking the ticket.

As for wider tests, I think it'd be great if you can run full YARN and 
mapreduce tests. If you have a working development environment, one could do
{code}
mvn clean install -DskipTests
cd hadoop-yarn-project/hadoop-yarn
mvn test -fn
cd ../../hadoop-mapreduce-project
mvn test -fn
{code}

Now, this is easier said than done. The reality is that this will take quite a 
while to run (around a couple of hours). And unfortunately I think there are 
some tests that are flaky or may fail depending on the environment. So this is 
not going to be exact science. The best effort is to run this on the current 
trunk, run it again with your patch, and see if there are "obvious" new 
failures that may be related to your patch. Let me know if that is something 
you can try. For setting up the dev environment, you can look at 
https://wiki.apache.org/hadoop/HowToSetupYourDevelopmentEnvironment.


> Use classloader inside configuration class to make new classes 
> ---
>
> Key: YARN-6771
> URL: https://issues.apache.org/jira/browse/YARN-6771
> Project: Hadoop YARN
>  Issue Type: Bug
>Affects Versions: 2.8.1, 3.0.0-alpha4
>Reporter: Jongyoul Lee
> Fix For: 2.8.2
>
> Attachments: YARN-6771-1.patch, YARN-6771-2.patch, YARN-6771.patch
>
>
> While running {{RpcClientFactoryPBImpl.getClient}}, 
> {{RpcClientFactoryPBImpl}} uses {{localConf.getClassByName}}. But in case of 
> using custom classloader, we have to use {{conf.getClassByName}} because 
> custom classloader is already stored in {{Configuration}} class.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-6771) Use classloader inside configuration class to make new classes

2017-07-19 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-6771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16093116#comment-16093116
 ] 

Sangjin Lee commented on YARN-6771:
---

Do you have a JIRA on the Zeppelin side that describes the actual issue? I 
suppose this is an issue when you need to load a class (that is only on your 
classpath) dynamically via {{Configuration}}?

Could you please update your patch to include {{RpcServerFactoryPBImpl}} too? 
Although the client alone would address your issue, from the YARN perspective, 
it would not be ideal to have an asymmetric behavior.

I think we also need to run a wider test suite than what jenkins runs out of 
the change analysis to ensure nothing changes as a result...

> Use classloader inside configuration class to make new classes 
> ---
>
> Key: YARN-6771
> URL: https://issues.apache.org/jira/browse/YARN-6771
> Project: Hadoop YARN
>  Issue Type: Bug
>Affects Versions: 2.8.1, 3.0.0-alpha4
>Reporter: Jongyoul Lee
> Fix For: 2.8.2
>
> Attachments: YARN-6771-1.patch, YARN-6771-2.patch, YARN-6771.patch
>
>
> While running {{RpcClientFactoryPBImpl.getClient}}, 
> {{RpcClientFactoryPBImpl}} uses {{localConf.getClassByName}}. But in case of 
> using custom classloader, we have to use {{conf.getClassByName}} because 
> custom classloader is already stored in {{Configuration}} class.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-6771) Use classloader inside configuration class to make new classes

2017-07-14 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-6771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16087770#comment-16087770
 ] 

Sangjin Lee commented on YARN-6771:
---

Thanks for the contribution [~jongyoul]. It appears that the same pattern 
exists in {{RpcServerFactoryPBImpl}} too. To be consistent, we should change 
both or neither.

That said, these are pretty basic classes that pin all YARN RPC communications. 
I don't see the full history, but it appears that using a clean configuration 
to load these classes was intentional. Could you please explain in bit more 
detail why you'd need to use provide a specific {{Configuration}} instance to 
create RPC clients/servers? Is there a JIRA or issue that describes an issue on 
your end?

> Use classloader inside configuration class to make new classes 
> ---
>
> Key: YARN-6771
> URL: https://issues.apache.org/jira/browse/YARN-6771
> Project: Hadoop YARN
>  Issue Type: Bug
>Affects Versions: 2.8.1, 3.0.0-alpha4
>Reporter: Jongyoul Lee
> Fix For: 2.8.2
>
> Attachments: YARN-6771-1.patch, YARN-6771-2.patch, YARN-6771.patch
>
>
> While running {{RpcClientFactoryPBImpl.getClient}}, 
> {{RpcClientFactoryPBImpl}} uses {{localConf.getClassByName}}. But in case of 
> using custom classloader, we have to use {{conf.getClassByName}} because 
> custom classloader is already stored in {{Configuration}} class.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-574) PrivateLocalizer does not support parallel resource download via ContainerLocalizer

2017-04-21 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15979326#comment-15979326
 ] 

Sangjin Lee commented on YARN-574:
--

Thanks for your contribution [~ajithshetty].

Regarding the use of {{AtomicInteger}}, why not use {{Semaphore}} for this? The 
semantics we're demanding is really that of a semaphore. We could also 
eliminate it altogether with Jason's suggestion but if we're going to use it, 
using a real counting semaphore is clearer.

Could you please update the patch for Jason's feedback and mine here too? 
Thanks!

> PrivateLocalizer does not support parallel resource download via 
> ContainerLocalizer
> ---
>
> Key: YARN-574
> URL: https://issues.apache.org/jira/browse/YARN-574
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Affects Versions: 2.6.0, 2.8.0, 2.7.1
>Reporter: Omkar Vinit Joshi
>Assignee: Ajith S
> Attachments: YARN-574.03.patch, YARN-574.04.patch, YARN-574.05.patch, 
> YARN-574.1.patch, YARN-574.2.patch
>
>
> At present private resources will be downloaded in parallel only if multiple 
> containers request the same resource. However otherwise it will be serial. 
> The protocol between PrivateLocalizer and ContainerLocalizer supports 
> multiple downloads however it is not used and only one resource is sent for 
> downloading at a time.
> I think we can increase / assure parallelism (even for single container 
> requesting resource) for private/application resources by making multiple 
> downloads per ContainerLocalizer.
> Total Parallelism before
> = number of threads allotted for PublicLocalizer [public resource] + number 
> of containers[private and application resource]
> Total Parallelism after
> = number of threads allotted for PublicLocalizer [public resource] + number 
> of containers * max downloads per container [private and application resource]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-6466) Provide shaded framework jar for containers

2017-04-18 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15973649#comment-15973649
 ] 

Sangjin Lee commented on YARN-6466:
---

I think the separate discussion that's happening on HADOOP-11656 is informative.

I would very much like to keep the isolating classloader feature at least for 
the container runtime. I have the main part of the work in the reviewable state 
on HADOOP-13398. Also, please note that the isolating classloader feature is an 
existing feature that works and many folks use. We're basically adding the 
stricter behavior for 3.0. I think it would be a loss if we abandon that. Let's 
discuss this.

bq. We probably don't need much of a footprint within the container in the 
first place.

I'm not quite sure if I understand this. Could you kindly elaborate?

> Provide shaded framework jar for containers
> ---
>
> Key: YARN-6466
> URL: https://issues.apache.org/jira/browse/YARN-6466
> Project: Hadoop YARN
>  Issue Type: New Feature
>  Components: build, yarn
>Affects Versions: 3.0.0-alpha1
>Reporter: Sean Busbey
>
> We should build on the existing shading work to provide a jar with all of the 
> bits needed within a YARN application's container to talk to the resource 
> manager and node manager.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-6466) Provide shaded framework jar for containers

2017-04-11 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15964646#comment-15964646
 ] 

Sangjin Lee commented on YARN-6466:
---

Hi [~busbey], thanks for raising this. How would this compare with the ongoing 
work at HADOOP-13070? Is your thinking that shading should be the/a mechanism 
for isolation for containers instead of the classloader isolation?

Note that the classloader isolation already exists and folks have been using it 
to accomplish isolation. I think we may need to sort out what our story on this 
is for 3.0 going forward.

> Provide shaded framework jar for containers
> ---
>
> Key: YARN-6466
> URL: https://issues.apache.org/jira/browse/YARN-6466
> Project: Hadoop YARN
>  Issue Type: New Feature
>  Components: build, yarn
>Reporter: Sean Busbey
>
> We should build on the existing shading work to provide a jar with all of the 
> bits needed within a YARN application's container to talk to the resource 
> manager and node manager.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Assigned] (YARN-4116) refactor ColumnHelper read* methods

2017-03-13 Thread Sangjin Lee (JIRA)

 [ 
https://issues.apache.org/jira/browse/YARN-4116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee reassigned YARN-4116:
-

Assignee: Vrushali C  (was: Sangjin Lee)

> refactor ColumnHelper read* methods
> ---
>
> Key: YARN-4116
> URL: https://issues.apache.org/jira/browse/YARN-4116
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Vrushali C
>  Labels: YARN-5355
>
> Currently we have several ColumnHelper.read* methods that are slightly 
> different in terms of the initial conditions and behave different 
> accordingly. We may want to refactor them so that the code reuse is strong 
> and also the API stays reasonable.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Assigned] (YARN-3741) consider nulling member maps/sets of TimelineEntity

2017-03-13 Thread Sangjin Lee (JIRA)

 [ 
https://issues.apache.org/jira/browse/YARN-3741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee reassigned YARN-3741:
-

Assignee: Vrushali C  (was: Sangjin Lee)

> consider nulling member maps/sets of TimelineEntity
> ---
>
> Key: YARN-3741
> URL: https://issues.apache.org/jira/browse/YARN-3741
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Vrushali C
>  Labels: YARN-5355
>
> Currently there are multiple collection members of TimelineEntity that are 
> always instantiated, regardless of whether they are used or not: info, 
> configs, metrics, events, isRelatedToEntities, and relatesToEntities.
> Since TimelineEntities will be created very often and in lots of cases many 
> of these members will be empty, creating these empty collections is wasteful 
> in terms of garbage collector pressure.
> It would be good to start out with null members, and instantiate these 
> collections only if they are actually used. Of course, we need to make that 
> contract very clear and refactor all client code to handle that scenario.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Assigned] (YARN-3907) create the flow-version table

2017-03-13 Thread Sangjin Lee (JIRA)

 [ 
https://issues.apache.org/jira/browse/YARN-3907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee reassigned YARN-3907:
-

Assignee: Vrushali C  (was: Sangjin Lee)

> create the flow-version table
> -
>
> Key: YARN-3907
> URL: https://issues.apache.org/jira/browse/YARN-3907
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Vrushali C
>  Labels: YARN-5355
>
> Per discussions on YARN-3815, create the flow-version table that maps flow 
> versions with various data about the versions.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Assigned] (YARN-3512) add more fine-grained metrics that measure write performance

2017-03-13 Thread Sangjin Lee (JIRA)

 [ 
https://issues.apache.org/jira/browse/YARN-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee reassigned YARN-3512:
-

Assignee: Vrushali C  (was: Sangjin Lee)

> add more fine-grained metrics that measure write performance
> 
>
> Key: YARN-3512
> URL: https://issues.apache.org/jira/browse/YARN-3512
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Sangjin Lee
>Assignee: Vrushali C
>
> We need more fine-grained metrics in the load testing tool that measure the 
> write performance of the timeline service. Currently it only captures the 
> number of writes and bytes per sec from the API point of view. But the actual 
> storage implementation may turn them into many more/fewer writes to the 
> storage itself. We need more fine-grained data about what's going on in terms 
> of actual writes to storage.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Assigned] (YARN-3378) a load test client that can replay a volume of history files

2017-03-13 Thread Sangjin Lee (JIRA)

 [ 
https://issues.apache.org/jira/browse/YARN-3378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee reassigned YARN-3378:
-

Assignee: Vrushali C  (was: Sangjin Lee)

> a load test client that can replay a volume of history files
> 
>
> Key: YARN-3378
> URL: https://issues.apache.org/jira/browse/YARN-3378
> Project: Hadoop YARN
>  Issue Type: New Feature
>  Components: timelineserver
>Reporter: Sangjin Lee
>Assignee: Vrushali C
>
> It might be good to create a load test client that can replay a large volume 
> of history files into the timeline service. One can envision running such a 
> load test client as a mapreduce job and generate a fair amount of load. It 
> would be useful to spot check correctness, and more importantly observe 
> performance characteristic.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Assigned] (YARN-3353) provide RPC metrics via JMX for timeline collectors and readers

2017-03-13 Thread Sangjin Lee (JIRA)

 [ 
https://issues.apache.org/jira/browse/YARN-3353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee reassigned YARN-3353:
-

Assignee: Vrushali C  (was: Sangjin Lee)

> provide RPC metrics via JMX for timeline collectors and readers
> ---
>
> Key: YARN-3353
> URL: https://issues.apache.org/jira/browse/YARN-3353
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Sangjin Lee
>Assignee: Vrushali C
>  Labels: YARN-5355
>
> We should provide RPC metrics via JMX for timeline collectors and readers. 
> One challenge we may have is it might be difficult to provide a stable view 
> for the metrics, given the distributed nature of the collectors.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Assigned] (YARN-451) Add more metrics to RM page

2017-03-13 Thread Sangjin Lee (JIRA)

 [ 
https://issues.apache.org/jira/browse/YARN-451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee reassigned YARN-451:


Assignee: Joep Rottinghuis  (was: Sangjin Lee)

> Add more metrics to RM page
> ---
>
> Key: YARN-451
> URL: https://issues.apache.org/jira/browse/YARN-451
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: resourcemanager
>Affects Versions: 2.0.3-alpha
>Reporter: Lohit Vijayarenu
>Assignee: Joep Rottinghuis
> Attachments: in_progress_2x.png, yarn-451-trunk-20130916.1.patch
>
>
> ResourceManager webUI shows list of RUNNING applications, but it does not 
> tell which applications are requesting more resource compared to others. With 
> cluster running hundreds of applications at once it would be useful to have 
> some kind of metric to show high-resource usage applications vs low-resource 
> usage ones. At the minimum showing number of containers is good option.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Assigned] (YARN-5355) YARN Timeline Service v.2: alpha 2

2017-03-13 Thread Sangjin Lee (JIRA)

 [ 
https://issues.apache.org/jira/browse/YARN-5355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee reassigned YARN-5355:
-

Assignee: Vrushali C  (was: Sangjin Lee)

> YARN Timeline Service v.2: alpha 2
> --
>
> Key: YARN-5355
> URL: https://issues.apache.org/jira/browse/YARN-5355
> Project: Hadoop YARN
>  Issue Type: New Feature
>  Components: timelineserver
>Reporter: Sangjin Lee
>Assignee: Vrushali C
>Priority: Critical
> Attachments: Timeline Service v2_ Ideas for Next Steps.pdf, 
> YARN-5355-branch-2.01.patch
>
>
> This is an umbrella JIRA for the alpha 2 milestone for YARN Timeline Service 
> v.2.
> This is developed on feature branches: {{YARN-5355}} for the trunk-based 
> development and {{YARN-5355-branch-2}} to maintain backports to branch-2. Any 
> subtask work on this JIRA will be committed to those 2 branches.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-5071) address HBase compatibility issues with trunk

2017-03-10 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-5071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15905968#comment-15905968
 ] 

Sangjin Lee commented on YARN-5071:
---

I believe this is largely addressed by your work (thanks!). I'm fine with 
closing this. [~vrushalic], let me know what you think.

> address HBase compatibility issues with trunk
> -
>
> Key: YARN-5071
> URL: https://issues.apache.org/jira/browse/YARN-5071
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Critical
>  Labels: YARN-5355
>
> The trunk is now adding or planning to add more and more 
> backward-incompatible changes. Some examples include
> - remove v.1 metrics classes (HADOOP-12504)
> - update jersey version (HADOOP-9613)
> - target java 8 by default (HADOOP-11858)
> This poses big challenges for the timeline service v.2 as we have a 
> dependency on hbase which depends on an older version of hadoop.
> We need to find a way to solve/contain/manage these risks before it is too 
> late.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-6318) timeline service schema creator fails if executed from a remote machine

2017-03-10 Thread Sangjin Lee (JIRA)

 [ 
https://issues.apache.org/jira/browse/YARN-6318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee updated YARN-6318:
--
Attachment: YARN-6318-YARN-5355.02.patch

Posted patch v.2 to address the javadoc issue.

> timeline service schema creator fails if executed from a remote machine
> ---
>
> Key: YARN-6318
> URL: https://issues.apache.org/jira/browse/YARN-6318
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: 3.0.0-alpha1
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Minor
>  Labels: yarn-5355-merge-blocker
> Attachments: YARN-6318-YARN-5355.01.patch, 
> YARN-6318-YARN-5355.02.patch
>
>
> The timeline service schema creator fails if executed from a remote machine 
> and the remote machine does not have the right {{hbase-site.xml}} file to 
> talk to that remote HBase cluster.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Assigned] (YARN-6318) timeline service schema creator fails if executed from a remote machine

2017-03-10 Thread Sangjin Lee (JIRA)

 [ 
https://issues.apache.org/jira/browse/YARN-6318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee reassigned YARN-6318:
-

Assignee: Sangjin Lee

> timeline service schema creator fails if executed from a remote machine
> ---
>
> Key: YARN-6318
> URL: https://issues.apache.org/jira/browse/YARN-6318
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: 3.0.0-alpha1
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Minor
>  Labels: yarn-5355-merge-blocker
> Attachments: YARN-6318-YARN-5355.01.patch
>
>
> The timeline service schema creator fails if executed from a remote machine 
> and the remote machine does not have the right {{hbase-site.xml}} file to 
> talk to that remote HBase cluster.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-6318) timeline service schema creator fails if executed from a remote machine

2017-03-10 Thread Sangjin Lee (JIRA)

 [ 
https://issues.apache.org/jira/browse/YARN-6318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee updated YARN-6318:
--
Attachment: YARN-6318-YARN-5355.01.patch

Posted patch v.1.

Passing in a real {{YarnConfiguration}} instance to properly exercise 
{{HBaseTimelineStorageUtils.getTimelineServiceHBaseConf()}}. Now calling that 
method with a null argument throws an NPE. Added a small test.

> timeline service schema creator fails if executed from a remote machine
> ---
>
> Key: YARN-6318
> URL: https://issues.apache.org/jira/browse/YARN-6318
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: 3.0.0-alpha1
>Reporter: Sangjin Lee
>Priority: Minor
>  Labels: yarn-5355-merge-blocker
> Attachments: YARN-6318-YARN-5355.01.patch
>
>
> The timeline service schema creator fails if executed from a remote machine 
> and the remote machine does not have the right {{hbase-site.xml}} file to 
> talk to that remote HBase cluster.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-6318) timeline service schema creator fails if executed from a remote machine

2017-03-10 Thread Sangjin Lee (JIRA)

 [ 
https://issues.apache.org/jira/browse/YARN-6318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee updated YARN-6318:
--
Priority: Minor  (was: Major)

Downgrading the issue to minor.

Yes, it is true that as long as {{hbase-site.xml}} is in the configuration 
(i.e. classpath) the schema creation works fine. But we should be able to 
create the schema using the separate config file location driven by 
{{yarn.timeline-service.hbase.configuration.file}}, and there is a bug that 
breaks that behavior. The fix should be very straightforward.

> timeline service schema creator fails if executed from a remote machine
> ---
>
> Key: YARN-6318
> URL: https://issues.apache.org/jira/browse/YARN-6318
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: 3.0.0-alpha1
>Reporter: Sangjin Lee
>Priority: Minor
>  Labels: yarn-5355-merge-blocker
>
> The timeline service schema creator fails if executed from a remote machine 
> and the remote machine does not have the right {{hbase-site.xml}} file to 
> talk to that remote HBase cluster.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Created] (YARN-6318) timeline service schema creator fails if executed from a remote machine

2017-03-09 Thread Sangjin Lee (JIRA)
Sangjin Lee created YARN-6318:
-

 Summary: timeline service schema creator fails if executed from a 
remote machine
 Key: YARN-6318
 URL: https://issues.apache.org/jira/browse/YARN-6318
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Affects Versions: 3.0.0-alpha1
Reporter: Sangjin Lee


The timeline service schema creator fails if executed from a remote machine and 
the remote machine does not have the right {{hbase-site.xml}} file to talk to 
that remote HBase cluster.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-6140) start time key in NM leveldb store should be removed when container is removed

2017-03-07 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-6140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15899804#comment-15899804
 ] 

Sangjin Lee commented on YARN-6140:
---

As mentioned above, This is not reproduced in the trunk version (YARN-5355) or 
the branch-2 version (YARN-5355-branch-2). I saw this when I backported 
timeline service to our internal branch based on 2.6. The unit test fails 
because remove calls did not remove the records fully.

I'm not too sure why the same code works against > 2.6 but not for 2.6. Since 
this is not happening in the latest versions, I don't think it's a serious 
issue. But I wanted to see if we can still delete the column to be on the safe 
side. It would be good to understand why it does not work against 2.6.

> start time key in NM leveldb store should be removed when container is removed
> --
>
> Key: YARN-6140
> URL: https://issues.apache.org/jira/browse/YARN-6140
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: nodemanager
>Affects Versions: YARN-5355
>Reporter: Sangjin Lee
>Assignee: Ajith S
>
> It appears that the start time key is not removed when the container is 
> removed. The key was introduced in YARN-5792.
> I found this while backporting the YARN-5355-branch-2 branch to our internal 
> branch loosely based on 2.6.0. The {{TestNMLeveldbStateStoreService}} test 
> was failing because of this.
> I'm not sure why we didn't see this earlier.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-6256) Add FROM_ID info key for timeline entities in reader response.

2017-03-07 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-6256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15899702#comment-15899702
 ] 

Sangjin Lee commented on YARN-6256:
---

+1. Sorry for the late reply. Thanks [~rohithsharma]!

> Add FROM_ID info key for timeline entities in reader response. 
> ---
>
> Key: YARN-6256
> URL: https://issues.apache.org/jira/browse/YARN-6256
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>  Labels: yarn-5355-merge-blocker
> Attachments: YARN-6256-YARN-5355.0001.patch, 
> YARN-6256-YARN-5355.0002.patch, YARN-6256-YARN-5355.0003.patch
>
>
> It is continuation with YARN-6027 to add FROM_ID key in all other timeline 
> entity responses which includes
> # Flow run entity response. 
> # Application entity response
> # Generic timeline entity response - Here we need to retrospect on idprefix 
> filter which is now separately provided. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-6256) Add FROM_ID info key for timeline entities in reader response.

2017-03-03 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-6256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15895239#comment-15895239
 ] 

Sangjin Lee commented on YARN-6256:
---

(TimelineReaderWebServices.java)
- l.270: need a space between "fromId." and "fromId" (the same for a few other 
places in this file)

(FlowRunEntityReader.java)
- l.257: I thought the check for single entity read was intentional (I guess 
this was introduced by YARN-4237). Do we no longer need it? cc [~varun_saxena]

One high level question. How would the UI use the fromId to render itself? For 
example, if you ask for 50 entities for the first page, UI would pick the last 
(50th) in that set for the fromId for the next page, right? But if you specify 
that as the fromId, that entity is returned again in the next call because the 
fromId is inclusive, right? Then how does UI deal with that? Does it ask for 51 
entities in the next page and drop the first (redundant) one? Am I 
misunderstanding this? Is making it precise (e.g. making fromId exclusive) 
feasible?


> Add FROM_ID info key for timeline entities in reader response. 
> ---
>
> Key: YARN-6256
> URL: https://issues.apache.org/jira/browse/YARN-6256
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>  Labels: yarn-5355-merge-blocker
> Attachments: YARN-6256-YARN-5355.0001.patch, 
> YARN-6256-YARN-5355.0002.patch
>
>
> It is continuation with YARN-6027 to add FROM_ID key in all other timeline 
> entity responses which includes
> # Flow run entity response. 
> # Application entity response
> # Generic timeline entity response - Here we need to retrospect on idprefix 
> filter which is now separately provided. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Comment Edited] (YARN-6256) Add FROM_ID info key for timeline entities in reader response.

2017-03-03 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-6256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15894621#comment-15894621
 ] 

Sangjin Lee edited comment on YARN-6256 at 3/3/17 4:01 PM:
---

Point taken. I know some entity ids and flow name etc. can get quite long, so 
it's pretty easy for such a from-id field to be in the hundreds of bytes. But 
again, if we return 100 entities, that would add only tens of kB.

If we are going to keep it for all entities, we might as well keep it for 
single-entity queries too for consistency.


was (Author: sjlee0):
Point taken. I know some entity ids and flow name etc. can get quite long, so 
it's pretty easy for such a from-id field to be in the hundreds of bytes. But 
again, if we return 100 entities, that would add only tens of kB.

> Add FROM_ID info key for timeline entities in reader response. 
> ---
>
> Key: YARN-6256
> URL: https://issues.apache.org/jira/browse/YARN-6256
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>  Labels: yarn-5355-merge-blocker
> Attachments: YARN-6256-YARN-5355.0001.patch
>
>
> It is continuation with YARN-6027 to add FROM_ID key in all other timeline 
> entity responses which includes
> # Flow run entity response. 
> # Application entity response
> # Generic timeline entity response - Here we need to retrospect on idprefix 
> filter which is now separately provided. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-6256) Add FROM_ID info key for timeline entities in reader response.

2017-03-03 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-6256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15894621#comment-15894621
 ] 

Sangjin Lee commented on YARN-6256:
---

Point taken. I know some entity ids and flow name etc. can get quite long, so 
it's pretty easy for such a from-id field to be in the hundreds of bytes. But 
again, if we return 100 entities, that would add only tens of kB.

> Add FROM_ID info key for timeline entities in reader response. 
> ---
>
> Key: YARN-6256
> URL: https://issues.apache.org/jira/browse/YARN-6256
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>  Labels: yarn-5355-merge-blocker
> Attachments: YARN-6256-YARN-5355.0001.patch
>
>
> It is continuation with YARN-6027 to add FROM_ID key in all other timeline 
> entity responses which includes
> # Flow run entity response. 
> # Application entity response
> # Generic timeline entity response - Here we need to retrospect on idprefix 
> filter which is now separately provided. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Comment Edited] (YARN-6256) Add FROM_ID info key for timeline entities in reader response.

2017-03-02 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-6256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15892935#comment-15892935
 ] 

Sangjin Lee edited comment on YARN-6256 at 3/2/17 8:35 PM:
---

Thanks for the patch [~rohithsharma]! I took a quick look at it, and I still 
need to do a more thorough review, but my observations so far are similar to 
Varun's above.

I do have another question (which also applies to YARN-6027). It seems that 
we're embedding the FROMID value for every single entity we return. Would it be 
possible to do this only for the "last" entity that gets returned if we're 
returning multiple entities? This might be a secondary optimization, but for 
all other entities but the last, the FROMID value would be ignored. In that 
sense, it would simply add to the payload size without providing benefits. 
Thoughts?


was (Author: sjlee0):
Thanks for the patch [~rohithsharma]! I took a quick look at it, and I still 
need to do a more thorough review, but my observations so far are similar to 
Varun's above.

I do have another questions (which also applies to YARN-6027). It seems that 
we're embedding the FROMID value for every single entity we return. Would it be 
possible to do this only for the "last" entity that gets returned if we're 
returning multiple entities? This might be a secondary optimization, but for 
all other entities but the last, the FROMID value would be ignored. In that 
sense, it would simply add to the payload size without providing benefits. 
Thoughts?

> Add FROM_ID info key for timeline entities in reader response. 
> ---
>
> Key: YARN-6256
> URL: https://issues.apache.org/jira/browse/YARN-6256
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>  Labels: yarn-5355-merge-blocker
> Attachments: YARN-6256-YARN-5355.0001.patch
>
>
> It is continuation with YARN-6027 to add FROM_ID key in all other timeline 
> entity responses which includes
> # Flow run entity response. 
> # Application entity response
> # Generic timeline entity response - Here we need to retrospect on idprefix 
> filter which is now separately provided. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-6256) Add FROM_ID info key for timeline entities in reader response.

2017-03-02 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-6256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15892935#comment-15892935
 ] 

Sangjin Lee commented on YARN-6256:
---

Thanks for the patch [~rohithsharma]! I took a quick look at it, and I still 
need to do a more thorough review, but my observations so far are similar to 
Varun's above.

I do have another questions (which also applies to YARN-6027). It seems that 
we're embedding the FROMID value for every single entity we return. Would it be 
possible to do this only for the "last" entity that gets returned if we're 
returning multiple entities? This might be a secondary optimization, but for 
all other entities but the last, the FROMID value would be ignored. In that 
sense, it would simply add to the payload size without providing benefits. 
Thoughts?

> Add FROM_ID info key for timeline entities in reader response. 
> ---
>
> Key: YARN-6256
> URL: https://issues.apache.org/jira/browse/YARN-6256
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>  Labels: yarn-5355-merge-blocker
> Attachments: YARN-6256-YARN-5355.0001.patch
>
>
> It is continuation with YARN-6027 to add FROM_ID key in all other timeline 
> entity responses which includes
> # Flow run entity response. 
> # Application entity response
> # Generic timeline entity response - Here we need to retrospect on idprefix 
> filter which is now separately provided. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-6027) Support fromid(offset) filter for /flows API

2017-03-01 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15891269#comment-15891269
 ] 

Sangjin Lee commented on YARN-6027:
---

Committing the addendum patch to YARN-5355-branch-2.

> Support fromid(offset) filter for /flows API
> 
>
> Key: YARN-6027
> URL: https://issues.apache.org/jira/browse/YARN-6027
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>  Labels: yarn-5355-merge-blocker
> Fix For: YARN-5355, YARN-5355-branch-2
>
> Attachments: YARN-6027-YARN-5355.0001.patch, 
> YARN-6027-YARN-5355.0002.patch, YARN-6027-YARN-5355.0003.patch, 
> YARN-6027-YARN-5355.0004.patch, YARN-6027-YARN-5355.0005.patch, 
> YARN-6027-YARN-5355.0006.patch, YARN-6027-YARN-5355.0007.patch, 
> YARN-6027-YARN-5355.0008.patch, YARN-6027-YARN-5355-branch-2.01.patch
>
>
> In YARN-5585 , fromId is supported for retrieving entities. We need similar 
> filter for flows/flowRun apps and flow run and flow as well. 
> Along with supporting fromId, this JIRA should also discuss following points
> * Should we throw an exception for entities/entity retrieval if duplicates 
> found?
> * TimelieEntity :
> ** Should equals method also check for idPrefix?
> ** Does idPrefix is part of identifiers?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-6027) Support fromid(offset) filter for /flows API

2017-03-01 Thread Sangjin Lee (JIRA)

 [ 
https://issues.apache.org/jira/browse/YARN-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee updated YARN-6027:
--
Attachment: YARN-6027-YARN-5355-branch-2.01.patch

The addendum patch for branch-2. This restores the original version of the code 
that's refactored. The tests pass.

> Support fromid(offset) filter for /flows API
> 
>
> Key: YARN-6027
> URL: https://issues.apache.org/jira/browse/YARN-6027
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>  Labels: yarn-5355-merge-blocker
> Fix For: YARN-5355, YARN-5355-branch-2
>
> Attachments: YARN-6027-YARN-5355.0001.patch, 
> YARN-6027-YARN-5355.0002.patch, YARN-6027-YARN-5355.0003.patch, 
> YARN-6027-YARN-5355.0004.patch, YARN-6027-YARN-5355.0005.patch, 
> YARN-6027-YARN-5355.0006.patch, YARN-6027-YARN-5355.0007.patch, 
> YARN-6027-YARN-5355.0008.patch, YARN-6027-YARN-5355-branch-2.01.patch
>
>
> In YARN-5585 , fromId is supported for retrieving entities. We need similar 
> filter for flows/flowRun apps and flow run and flow as well. 
> Along with supporting fromId, this JIRA should also discuss following points
> * Should we throw an exception for entities/entity retrieval if duplicates 
> found?
> * TimelieEntity :
> ** Should equals method also check for idPrefix?
> ** Does idPrefix is part of identifiers?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Comment Edited] (YARN-6027) Support fromid(offset) filter for /flows API

2017-03-01 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15891235#comment-15891235
 ] 

Sangjin Lee edited comment on YARN-6027 at 3/1/17 10:48 PM:


The YARN-5355-branch-2 branch is failing compilation at 
{{AbstractTimelineReaderHBaseTestBase.java}}:
{noformat}
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile 
(default-testCompile) on project 
hadoop-yarn-server-timelineservice-hbase-tests: Compilation failure: 
Compilation failure:
[ERROR] 
/Users/sjlee/git/hadoop-ats/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/reader/AbstractTimelineReaderHBaseTestBase.java:[90,9]
 method does not override or implement a method from a supertype
[ERROR] 
/Users/sjlee/git/hadoop-ats/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/reader/AbstractTimelineReaderHBaseTestBase.java:[122,29]
 cannot find symbol
[ERROR] symbol:   method getStatusInfo()
[ERROR] location: variable resp of type com.sun.jersey.api.client.ClientResponse
[ERROR] 
/Users/sjlee/git/hadoop-ats/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/reader/AbstractTimelineReaderHBaseTestBase.java:[126,34]
 cannot find symbol
[ERROR] symbol:   method getStatusInfo()
[ERROR] location: variable resp of type com.sun.jersey.api.client.ClientResponse
[ERROR] 
/Users/sjlee/git/hadoop-ats/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/reader/AbstractTimelineReaderHBaseTestBase.java:[140,13]
 cannot find symbol
[ERROR] symbol:   method getStatusInfo()
[ERROR] location: variable resp of type com.sun.jersey.api.client.ClientResponse
{noformat}

I'll post an addendum patch for the branch-2 commit.


was (Author: sjlee0):
The YARN-5355-branch-2 branch is failing compilation at 
{{AbstractTimelineReaderHBaseTestBase.java}}:
{noformat}
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile 
(default-testCompile) on project 
hadoop-yarn-server-timelineservice-hbase-tests: Compilation failure: 
Compilation failure:
[ERROR] 
/Users/sjlee/git/hadoop-ats/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/reader/AbstractTimelineReaderHBaseTestBase.java:[90,9]
 method does not override or implement a method from a supertype
[ERROR] 
/Users/sjlee/git/hadoop-ats/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/reader/AbstractTimelineReaderHBaseTestBase.java:[122,29]
 cannot find symbol
[ERROR] symbol:   method getStatusInfo()
[ERROR] location: variable resp of type com.sun.jersey.api.client.ClientResponse
[ERROR] 
/Users/sjlee/git/hadoop-ats/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/reader/AbstractTimelineReaderHBaseTestBase.java:[126,34]
 cannot find symbol
[ERROR] symbol:   method getStatusInfo()
[ERROR] location: variable resp of type com.sun.jersey.api.client.ClientResponse
[ERROR] 
/Users/sjlee/git/hadoop-ats/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/reader/AbstractTimelineReaderHBaseTestBase.java:[140,13]
 cannot find symbol
[ERROR] symbol:   method getStatusInfo()
[ERROR] location: variable resp of type com.sun.jersey.api.client.ClientResponse
{noformat}

I'll file a small JIRA to fix it.

> Support fromid(offset) filter for /flows API
> 
>
> Key: YARN-6027
> URL: https://issues.apache.org/jira/browse/YARN-6027
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>  Labels: yarn-5355-merge-blocker
> Fix For: YARN-5355, YARN-5355-branch-2
>
> Attachments: YARN-6027-YARN-5355.0001.patch, 
> YARN-6027-YARN-5355.0002.patch, YARN-6027-YARN-5355.0003.patch, 
> YARN-6027-YARN-5355.0004.patch, YARN-6027-YARN-5355.0005.patch, 
> YARN-6027-YARN-5355.0006.patch, YARN-6027-YARN-5355.0007.patch, 
> YARN-6027-YARN-5355.0008.patch
>
>
> In YARN-5585 , fromId is supported for retrieving entities. We need similar 
> filter for flows/flowRun apps and flow run and flow as 

[jira] [Commented] (YARN-6027) Support fromid(offset) filter for /flows API

2017-03-01 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15891235#comment-15891235
 ] 

Sangjin Lee commented on YARN-6027:
---

The YARN-5355-branch-2 branch is failing compilation at 
{{AbstractTimelineReaderHBaseTestBase.java}}:
{noformat}
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile 
(default-testCompile) on project 
hadoop-yarn-server-timelineservice-hbase-tests: Compilation failure: 
Compilation failure:
[ERROR] 
/Users/sjlee/git/hadoop-ats/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/reader/AbstractTimelineReaderHBaseTestBase.java:[90,9]
 method does not override or implement a method from a supertype
[ERROR] 
/Users/sjlee/git/hadoop-ats/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/reader/AbstractTimelineReaderHBaseTestBase.java:[122,29]
 cannot find symbol
[ERROR] symbol:   method getStatusInfo()
[ERROR] location: variable resp of type com.sun.jersey.api.client.ClientResponse
[ERROR] 
/Users/sjlee/git/hadoop-ats/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/reader/AbstractTimelineReaderHBaseTestBase.java:[126,34]
 cannot find symbol
[ERROR] symbol:   method getStatusInfo()
[ERROR] location: variable resp of type com.sun.jersey.api.client.ClientResponse
[ERROR] 
/Users/sjlee/git/hadoop-ats/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/reader/AbstractTimelineReaderHBaseTestBase.java:[140,13]
 cannot find symbol
[ERROR] symbol:   method getStatusInfo()
[ERROR] location: variable resp of type com.sun.jersey.api.client.ClientResponse
{noformat}

I'll file a small JIRA to fix it.

> Support fromid(offset) filter for /flows API
> 
>
> Key: YARN-6027
> URL: https://issues.apache.org/jira/browse/YARN-6027
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>  Labels: yarn-5355-merge-blocker
> Fix For: YARN-5355, YARN-5355-branch-2
>
> Attachments: YARN-6027-YARN-5355.0001.patch, 
> YARN-6027-YARN-5355.0002.patch, YARN-6027-YARN-5355.0003.patch, 
> YARN-6027-YARN-5355.0004.patch, YARN-6027-YARN-5355.0005.patch, 
> YARN-6027-YARN-5355.0006.patch, YARN-6027-YARN-5355.0007.patch, 
> YARN-6027-YARN-5355.0008.patch
>
>
> In YARN-5585 , fromId is supported for retrieving entities. We need similar 
> filter for flows/flowRun apps and flow run and flow as well. 
> Along with supporting fromId, this JIRA should also discuss following points
> * Should we throw an exception for entities/entity retrieval if duplicates 
> found?
> * TimelieEntity :
> ** Should equals method also check for idPrefix?
> ** Does idPrefix is part of identifiers?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-6027) Support fromid(offset) filter for /flows API

2017-03-01 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15890484#comment-15890484
 ] 

Sangjin Lee commented on YARN-6027:
---

+1. Thanks [~rohithsharma]!

> Support fromid(offset) filter for /flows API
> 
>
> Key: YARN-6027
> URL: https://issues.apache.org/jira/browse/YARN-6027
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>  Labels: yarn-5355-merge-blocker
> Attachments: YARN-6027-YARN-5355.0001.patch, 
> YARN-6027-YARN-5355.0002.patch, YARN-6027-YARN-5355.0003.patch, 
> YARN-6027-YARN-5355.0004.patch, YARN-6027-YARN-5355.0005.patch, 
> YARN-6027-YARN-5355.0006.patch, YARN-6027-YARN-5355.0007.patch, 
> YARN-6027-YARN-5355.0008.patch
>
>
> In YARN-5585 , fromId is supported for retrieving entities. We need similar 
> filter for flows/flowRun apps and flow run and flow as well. 
> Along with supporting fromId, this JIRA should also discuss following points
> * Should we throw an exception for entities/entity retrieval if duplicates 
> found?
> * TimelieEntity :
> ** Should equals method also check for idPrefix?
> ** Does idPrefix is part of identifiers?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-6253) FlowAcitivityColumnPrefix.store(byte[] rowKey, ...) drops timestamp

2017-02-28 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-6253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15889125#comment-15889125
 ] 

Sangjin Lee commented on YARN-6253:
---

LGTM. Will commit shortly.

I think it is fine to merge it to the YARN-5355 branch and have it go to trunk 
when we merge that feature branch. Are you fine with that?

> FlowAcitivityColumnPrefix.store(byte[] rowKey, ...) drops timestamp
> ---
>
> Key: YARN-6253
> URL: https://issues.apache.org/jira/browse/YARN-6253
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Affects Versions: 3.0.0-alpha2
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>  Labels: yarn-5355-merge-blocker
> Fix For: 3.0.0-alpha3
>
> Attachments: YARN-6253.01.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-6253) FlowAcitivityColumnPrefix.store(byte[] rowKey, ...) drops timestamp

2017-02-28 Thread Sangjin Lee (JIRA)

 [ 
https://issues.apache.org/jira/browse/YARN-6253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee updated YARN-6253:
--
Labels: yarn-5355-merge-blocker  (was: )

> FlowAcitivityColumnPrefix.store(byte[] rowKey, ...) drops timestamp
> ---
>
> Key: YARN-6253
> URL: https://issues.apache.org/jira/browse/YARN-6253
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Affects Versions: 3.0.0-alpha2
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>  Labels: yarn-5355-merge-blocker
> Fix For: 3.0.0-alpha3
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-4061) [Fault tolerance] Fault tolerant writer for timeline v2

2017-02-28 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15888979#comment-15888979
 ] 

Sangjin Lee commented on YARN-4061:
---

[~haibochen], is your comment a general comment on what that method is doing, 
independent of this JIRA? If so, I think that is correct. It must be a bug. 
Today there is no caller for that flavor of the {{store()}} method, but we 
should still fix it. Do you mind opening a new JIRA to fix that? Have you 
checked other {{store()}} methods to see if there is any other similar issue?

> [Fault tolerance] Fault tolerant writer for timeline v2
> ---
>
> Key: YARN-4061
> URL: https://issues.apache.org/jira/browse/YARN-4061
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Li Lu
>Assignee: Joep Rottinghuis
>  Labels: YARN-5355, yarn-5355-merge-blocker
> Attachments: FaulttolerantwriterforTimelinev2.pdf
>
>
> We need to build a timeline writer that can be resistant to backend storage 
> down time and timeline collector failures. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-6027) Support fromid(offset) filter for /flows API

2017-02-28 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15888630#comment-15888630
 ] 

Sangjin Lee commented on YARN-6027:
---

OK how about this? The {{TimelineUIDConverter}} class also uses these 
characters (the same ones). Can we at least standardize on one set of 
definitions? With this patch we would have 
{{TimelineUIDConverter#UID_DELIMETER_CHAR}} and 
{{TimelineUIDConverter#UID_ESCAPE_CHAR}}, and 
{{TimelineReaderUtils#DEFAULT_DELIMETER_CHAR}} and 
{{TimelineReaderUtils#DEFAULT_ESCAPE_CHAR}}. Specifically my proposal is to 
change the existing calls by {{TimelineUIDConverter}} to use the new constants 
in {{TimelineReaderUtils}} (or simply call {{split(String)}}).

> Support fromid(offset) filter for /flows API
> 
>
> Key: YARN-6027
> URL: https://issues.apache.org/jira/browse/YARN-6027
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>  Labels: yarn-5355-merge-blocker
> Attachments: YARN-6027-YARN-5355.0001.patch, 
> YARN-6027-YARN-5355.0002.patch, YARN-6027-YARN-5355.0003.patch, 
> YARN-6027-YARN-5355.0004.patch, YARN-6027-YARN-5355.0005.patch, 
> YARN-6027-YARN-5355.0006.patch
>
>
> In YARN-5585 , fromId is supported for retrieving entities. We need similar 
> filter for flows/flowRun apps and flow run and flow as well. 
> Along with supporting fromId, this JIRA should also discuss following points
> * Should we throw an exception for entities/entity retrieval if duplicates 
> found?
> * TimelieEntity :
> ** Should equals method also check for idPrefix?
> ** Does idPrefix is part of identifiers?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-6140) start time key in NM leveldb store should be removed when container is removed

2017-02-28 Thread Sangjin Lee (JIRA)

 [ 
https://issues.apache.org/jira/browse/YARN-6140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee updated YARN-6140:
--
Labels:   (was: yarn-5355-merge-blocker)

> start time key in NM leveldb store should be removed when container is removed
> --
>
> Key: YARN-6140
> URL: https://issues.apache.org/jira/browse/YARN-6140
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: nodemanager
>Affects Versions: YARN-5355
>Reporter: Sangjin Lee
>Assignee: Ajith S
>
> It appears that the start time key is not removed when the container is 
> removed. The key was introduced in YARN-5792.
> I found this while backporting the YARN-5355-branch-2 branch to our internal 
> branch loosely based on 2.6.0. The {{TestNMLeveldbStateStoreService}} test 
> was failing because of this.
> I'm not sure why we didn't see this earlier.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-6140) start time key in NM leveldb store should be removed when container is removed

2017-02-28 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-6140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15888607#comment-15888607
 ] 

Sangjin Lee commented on YARN-6140:
---

I'm going to remove the blocker label. In reality this issue doesn't manifest 
itself on trunk or on branch-2. I'd love to get this in soon (as this is fairly 
trivial), but I don't think this is a blocker per se.

> start time key in NM leveldb store should be removed when container is removed
> --
>
> Key: YARN-6140
> URL: https://issues.apache.org/jira/browse/YARN-6140
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: nodemanager
>Affects Versions: YARN-5355
>Reporter: Sangjin Lee
>Assignee: Ajith S
>
> It appears that the start time key is not removed when the container is 
> removed. The key was introduced in YARN-5792.
> I found this while backporting the YARN-5355-branch-2 branch to our internal 
> branch loosely based on 2.6.0. The {{TestNMLeveldbStateStoreService}} test 
> was failing because of this.
> I'm not sure why we didn't see this earlier.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-6199) Support for listing flows with filter userid

2017-02-28 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-6199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15888604#comment-15888604
 ] 

Sangjin Lee commented on YARN-6199:
---

Quick question: do you think this should be a merge blocker?

> Support for listing flows with filter userid
> 
>
> Key: YARN-6199
> URL: https://issues.apache.org/jira/browse/YARN-6199
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelinereader
>Reporter: Rohith Sharma K S
>  Labels: yarn-5355-merge-blocker
>
> Currently */flows* API retrieves flow entities for all the users by default. 
> It is required to provide filter user i.e */flows?user=rohith* . This is 
> critical filter in secured environment. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-6027) Support fromid(offset) filter for /flows API

2017-02-28 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15888594#comment-15888594
 ] 

Sangjin Lee commented on YARN-6027:
---

Thanks for the updated patch [~rohithsharma].

I'm not sure what the deal is with the findbugs issue. {{EntityColumnPrefix}} 
is not declared to be serializable, nor is it touched in this patch...

The checkstyle violations seem to be trivial to fix, and so are javadoc errors. 
Could you please look into them?


> Support fromid(offset) filter for /flows API
> 
>
> Key: YARN-6027
> URL: https://issues.apache.org/jira/browse/YARN-6027
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>  Labels: yarn-5355-merge-blocker
> Attachments: YARN-6027-YARN-5355.0001.patch, 
> YARN-6027-YARN-5355.0002.patch, YARN-6027-YARN-5355.0003.patch, 
> YARN-6027-YARN-5355.0004.patch, YARN-6027-YARN-5355.0005.patch, 
> YARN-6027-YARN-5355.0006.patch
>
>
> In YARN-5585 , fromId is supported for retrieving entities. We need similar 
> filter for flows/flowRun apps and flow run and flow as well. 
> Along with supporting fromId, this JIRA should also discuss following points
> * Should we throw an exception for entities/entity retrieval if duplicates 
> found?
> * TimelieEntity :
> ** Should equals method also check for idPrefix?
> ** Does idPrefix is part of identifiers?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-6027) Support fromid(offset) filter for /flows API

2017-02-27 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886893#comment-15886893
 ] 

Sangjin Lee commented on YARN-6027:
---

To me {{RowKeyConverter}} is really an orthogonal conversion from the existing 
key conversion and operates on the string representation. In that sense, I 
would slightly prefer if it is not an extension of {{KeyConverter}}. Any 
converter class that does both can implement both separate interfaces.

Also, perhaps we could find a better name for {{RowKeyConverter}}. The key 
thing is not so much that it is dealing with a row key as opposed to other 
{{KeyConverter}} types aren't. Most of {{KeyConverter}} implementations do deal 
with row keys after all. I think what separates this interface is the fact that 
it deals with the string representation (to be suitable for embedding it in 
URLs). Perhaps something like {{KeyConverterToString}} or some other name 
that's explicit to its purpose would be good.

I am also not 100% sold on {{RowKey}}. Again, it is not really about them being 
a row key type. Should we not introduce this interface (and even the converter 
interface for that matter) for now until it becomes clear we need polymorphism 
on this?

In terms of method names, how about making them very explicit about dealing 
with strings? I would be more comfortable with names such as 
{{encodeAsString()}}, {{decodeFromString()}}, {{getRowKeyAsString()}}, 
{{parseRowKeyFromString()}}, and so on.

In terms of separator characters, I am of a little different opinion from 
Varun's. IMO it would be better if we could reuse the same separator character 
(Separator.QUALIFIERS) as a constant (not a new constant with the same value). 
It is not so easy to keep track of many encoding schemes, and it would be hard 
to keep track of all encoding characters. If we can reuse the same whenever 
possible, it would help understand and maintain this code. I can be persuaded 
otherwise, but I wanted to state my current take for the record.

bq. utils classes are not expected to be subclassed. Right? They just 
encapsulate a bunch of static methods and constants.

Agree. It can be both public and final.

> Support fromid(offset) filter for /flows API
> 
>
> Key: YARN-6027
> URL: https://issues.apache.org/jira/browse/YARN-6027
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>  Labels: yarn-5355-merge-blocker
> Attachments: YARN-6027-YARN-5355.0001.patch, 
> YARN-6027-YARN-5355.0002.patch, YARN-6027-YARN-5355.0003.patch, 
> YARN-6027-YARN-5355.0004.patch, YARN-6027-YARN-5355.0005.patch
>
>
> In YARN-5585 , fromId is supported for retrieving entities. We need similar 
> filter for flows/flowRun apps and flow run and flow as well. 
> Along with supporting fromId, this JIRA should also discuss following points
> * Should we throw an exception for entities/entity retrieval if duplicates 
> found?
> * TimelieEntity :
> ** Should equals method also check for idPrefix?
> ** Does idPrefix is part of identifiers?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-6140) start time key in NM leveldb store should be removed when container is removed

2017-02-24 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-6140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15883609#comment-15883609
 ] 

Sangjin Lee commented on YARN-6140:
---

Any update on this [~ajithshetty]?

> start time key in NM leveldb store should be removed when container is removed
> --
>
> Key: YARN-6140
> URL: https://issues.apache.org/jira/browse/YARN-6140
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: nodemanager
>Affects Versions: YARN-5355
>Reporter: Sangjin Lee
>Assignee: Ajith S
>  Labels: yarn-5355-merge-blocker
>
> It appears that the start time key is not removed when the container is 
> removed. The key was introduced in YARN-5792.
> I found this while backporting the YARN-5355-branch-2 branch to our internal 
> branch loosely based on 2.6.0. The {{TestNMLeveldbStateStoreService}} test 
> was failing because of this.
> I'm not sure why we didn't see this earlier.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-6027) Support fromid(offset) filter for /flows API

2017-02-24 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15883397#comment-15883397
 ] 

Sangjin Lee commented on YARN-6027:
---

Thanks for the clarification. I forgot that this is something that would be 
part of URLs and thus needs a string representation. Yes, at least a separate 
interface might be desirable. Mixing them in with the RowKey classes might be a 
little confusing.

> Support fromid(offset) filter for /flows API
> 
>
> Key: YARN-6027
> URL: https://issues.apache.org/jira/browse/YARN-6027
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>  Labels: yarn-5355-merge-blocker
> Attachments: YARN-6027-YARN-5355.0001.patch, 
> YARN-6027-YARN-5355.0002.patch, YARN-6027-YARN-5355.0003.patch, 
> YARN-6027-YARN-5355.0004.patch
>
>
> In YARN-5585 , fromId is supported for retrieving entities. We need similar 
> filter for flows/flowRun apps and flow run and flow as well. 
> Along with supporting fromId, this JIRA should also discuss following points
> * Should we throw an exception for entities/entity retrieval if duplicates 
> found?
> * TimelieEntity :
> ** Should equals method also check for idPrefix?
> ** Does idPrefix is part of identifiers?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-6027) Support fromid(offset) filter for /flows API

2017-02-23 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15881523#comment-15881523
 ] 

Sangjin Lee commented on YARN-6027:
---

In addition to [~varun_saxena]'s comments, I have a couple of comments.

(FlowActivityRowKey.java)
- In addition to Varun's comment above, in fact the new {{decode()}} method 
seems largely redundant with {{FlowActivityRowKeyConverter.decode()}}. Is there 
a reason not to simply use it? The same comment goes for 
{{getRowKeyAsString()}}. Converters are how we apply encoding and decoding 
consistently, and we should try to stick to that as much as possible.

(AbstractHBaseTestBase.java)
- I am a little curious about this refactoring. Is the idea to split the 
current reader unit tests based on this abstract class? At least for now, only 
{{TestTimelineReaderWebServicesHBaseStorage}} is using it.
- Is the name the best? Also, if this is specifically for reader unit tests, it 
might be better to place it in the reader package and indicate that in the 
class name also; e.g. {{AbstractTimelineReaderHBaseTestBase}}



> Support fromid(offset) filter for /flows API
> 
>
> Key: YARN-6027
> URL: https://issues.apache.org/jira/browse/YARN-6027
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>  Labels: yarn-5355-merge-blocker
> Attachments: YARN-6027-YARN-5355.0001.patch, 
> YARN-6027-YARN-5355.0002.patch, YARN-6027-YARN-5355.0003.patch, 
> YARN-6027-YARN-5355.0004.patch
>
>
> In YARN-5585 , fromId is supported for retrieving entities. We need similar 
> filter for flows/flowRun apps and flow run and flow as well. 
> Along with supporting fromId, this JIRA should also discuss following points
> * Should we throw an exception for entities/entity retrieval if duplicates 
> found?
> * TimelieEntity :
> ** Should equals method also check for idPrefix?
> ** Does idPrefix is part of identifiers?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-6027) Support fromid(offset) filter for /flows API

2017-02-23 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15880860#comment-15880860
 ] 

Sangjin Lee commented on YARN-6027:
---

Thanks [~rohithsharma] for the updated patch. I'll review it today.

> Support fromid(offset) filter for /flows API
> 
>
> Key: YARN-6027
> URL: https://issues.apache.org/jira/browse/YARN-6027
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>  Labels: yarn-5355-merge-blocker
> Attachments: YARN-6027-YARN-5355.0001.patch, 
> YARN-6027-YARN-5355.0002.patch, YARN-6027-YARN-5355.0003.patch, 
> YARN-6027-YARN-5355.0004.patch
>
>
> In YARN-5585 , fromId is supported for retrieving entities. We need similar 
> filter for flows/flowRun apps and flow run and flow as well. 
> Along with supporting fromId, this JIRA should also discuss following points
> * Should we throw an exception for entities/entity retrieval if duplicates 
> found?
> * TimelieEntity :
> ** Should equals method also check for idPrefix?
> ** Does idPrefix is part of identifiers?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-4985) Refactor the coprocessor code & other definition classes into independent packages

2017-02-22 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879335#comment-15879335
 ] 

Sangjin Lee commented on YARN-4985:
---

Yes, I agree.

> Refactor the coprocessor code & other definition classes into independent 
> packages
> --
>
> Key: YARN-4985
> URL: https://issues.apache.org/jira/browse/YARN-4985
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Vrushali C
>Assignee: Haibo Chen
>  Labels: YARN-5355
> Attachments: YARN-4985-YARN-5355.poc.patch, 
> YARN-4985-YARN-5355.prelim.patch
>
>
> As part of the coprocessor deployment, we have realized that it will be much 
> cleaner to have the coprocessor code sit in a package which does not depend 
> on hadoop-yarn-server classes. It only needs hbase and other util classes.
> These util classes and tag definition related classes can be refactored into 
> their own independent "definition" class package so that making changes to 
> coprocessor code, upgrading hbase, deploying hbase on a different hadoop 
> version cluster etc all becomes operationally much easier and less error 
> prone to having different library jars etc.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-4985) Refactor the coprocessor code & other definition classes into independent packages

2017-02-22 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879210#comment-15879210
 ] 

Sangjin Lee commented on YARN-4985:
---

I went over the coprocessor code in some detail, and largely confirmed what you 
mentioned above. There are a number of classes that the coprocessor code 
depends on that the hbase "client" code also depends on; e.g. FlowRunColumn.

The only code that can probably moved entirely into hbase-server is those 
HBaseTimelineStorageUtils methods that are exercised by the coprocessor. The 
only callers are the coprocessor in that case.

There might be a way to make maven create a combined jar file for the 
coprocessor purposes, but I can't think of a straightforward way to do this. 
It'd probably involve a pretty hacky way to pull that off, and I'm not sure if 
the benefit justifies the work...

> Refactor the coprocessor code & other definition classes into independent 
> packages
> --
>
> Key: YARN-4985
> URL: https://issues.apache.org/jira/browse/YARN-4985
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Vrushali C
>Assignee: Haibo Chen
>  Labels: YARN-5355
> Attachments: YARN-4985-YARN-5355.poc.patch, 
> YARN-4985-YARN-5355.prelim.patch
>
>
> As part of the coprocessor deployment, we have realized that it will be much 
> cleaner to have the coprocessor code sit in a package which does not depend 
> on hadoop-yarn-server classes. It only needs hbase and other util classes.
> These util classes and tag definition related classes can be refactored into 
> their own independent "definition" class package so that making changes to 
> coprocessor code, upgrading hbase, deploying hbase on a different hadoop 
> version cluster etc all becomes operationally much easier and less error 
> prone to having different library jars etc.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-4985) Refactor the coprocessor code & other definition classes into independent packages

2017-02-22 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15878989#comment-15878989
 ] 

Sangjin Lee commented on YARN-4985:
---

Let me look through the code and see what kind of dependencies are emanating 
from the coprocessor code. I'll come back with some findings. 

What is the implication of not doing this refactoring? Is the original problem 
of being able to build Hadoop and HBase from source resolved without this, or 
does it depend on this?

> Refactor the coprocessor code & other definition classes into independent 
> packages
> --
>
> Key: YARN-4985
> URL: https://issues.apache.org/jira/browse/YARN-4985
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Vrushali C
>Assignee: Haibo Chen
>  Labels: YARN-5355
> Attachments: YARN-4985-YARN-5355.poc.patch, 
> YARN-4985-YARN-5355.prelim.patch
>
>
> As part of the coprocessor deployment, we have realized that it will be much 
> cleaner to have the coprocessor code sit in a package which does not depend 
> on hadoop-yarn-server classes. It only needs hbase and other util classes.
> These util classes and tag definition related classes can be refactored into 
> their own independent "definition" class package so that making changes to 
> coprocessor code, upgrading hbase, deploying hbase on a different hadoop 
> version cluster etc all becomes operationally much easier and less error 
> prone to having different library jars etc.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-4985) Refactor the coprocessor code & other definition classes into independent packages

2017-02-21 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15877293#comment-15877293
 ] 

Sangjin Lee commented on YARN-4985:
---

Thanks for the update [~haibochen].

bq. With this new module, we do still have the coprocessor installation issue. 
I am totally speculating here. Is it possible to configure maven so that it 
will combine hbase-schema and hbase-server into one jar, as a workaround?

It might be doable, but it may involve a fairly advance maven trick to pull 
that off. I haven't thought about what it might take.

Going back to the original question, were you able to see if there is any way 
we can limit the dependency coming from the coprocessor code? If that is 
possible, that is the best case scenario.

> Refactor the coprocessor code & other definition classes into independent 
> packages
> --
>
> Key: YARN-4985
> URL: https://issues.apache.org/jira/browse/YARN-4985
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Vrushali C
>Assignee: Haibo Chen
>  Labels: YARN-5355
> Attachments: YARN-4985-YARN-5355.poc.patch, 
> YARN-4985-YARN-5355.prelim.patch
>
>
> As part of the coprocessor deployment, we have realized that it will be much 
> cleaner to have the coprocessor code sit in a package which does not depend 
> on hadoop-yarn-server classes. It only needs hbase and other util classes.
> These util classes and tag definition related classes can be refactored into 
> their own independent "definition" class package so that making changes to 
> coprocessor code, upgrading hbase, deploying hbase on a different hadoop 
> version cluster etc all becomes operationally much easier and less error 
> prone to having different library jars etc.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-4675) Reorganize TimelineClient and TimelineClientImpl into separate classes for ATSv1.x and ATSv2

2017-02-16 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871096#comment-15871096
 ] 

Sangjin Lee commented on YARN-4675:
---

Thanks Naga. Committed it to YARN-5355 and YARN-5355-branch-2.

> Reorganize TimelineClient and TimelineClientImpl into separate classes for 
> ATSv1.x and ATSv2
> 
>
> Key: YARN-4675
> URL: https://issues.apache.org/jira/browse/YARN-4675
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
>  Labels: YARN-5355, yarn-5355-merge-blocker
> Fix For: 3.0.0-alpha3
>
> Attachments: YARN-4675.v2.002.patch, YARN-4675.v2.003.patch, 
> YARN-4675.v2.004.patch, YARN-4675.v2.005.patch, YARN-4675.v2.006.patch, 
> YARN-4675.v2.007.patch, YARN-4675.v2.008.patch, YARN-4675.v2.009.patch, 
> YARN-4675.v2.010.patch, YARN-4675.v2.011.patch, 
> YARN-4675-YARN-2928.v1.001.patch, YARN-4675-YARN-5355.v2.011.patch
>
>
> We need to reorganize TimeClientImpl into TimeClientV1Impl ,  
> TimeClientV2Impl and if required a base class, so that its clear which part 
> of the code belongs to which version and thus better maintainable.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-4675) Reorganize TimelineClient and TimelineClientImpl into separate classes for ATSv1.x and ATSv2

2017-02-16 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870590#comment-15870590
 ] 

Sangjin Lee commented on YARN-4675:
---

I went ahead and committed it to trunk. I'll wait for the YARN-5355 branch 
patch to commit it there as well.

Thanks [~Naganarasimha] for your contribution, and [~varun_saxena] and 
[~gtCarrera9] for your review!

> Reorganize TimelineClient and TimelineClientImpl into separate classes for 
> ATSv1.x and ATSv2
> 
>
> Key: YARN-4675
> URL: https://issues.apache.org/jira/browse/YARN-4675
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
>  Labels: YARN-5355, yarn-5355-merge-blocker
> Attachments: YARN-4675.v2.002.patch, YARN-4675.v2.003.patch, 
> YARN-4675.v2.004.patch, YARN-4675.v2.005.patch, YARN-4675.v2.006.patch, 
> YARN-4675.v2.007.patch, YARN-4675.v2.008.patch, YARN-4675.v2.009.patch, 
> YARN-4675.v2.010.patch, YARN-4675.v2.011.patch, 
> YARN-4675-YARN-2928.v1.001.patch
>
>
> We need to reorganize TimeClientImpl into TimeClientV1Impl ,  
> TimeClientV2Impl and if required a base class, so that its clear which part 
> of the code belongs to which version and thus better maintainable.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-4675) Reorganize TimelineClient and TimelineClientImpl into separate classes for ATSv1.x and ATSv2

2017-02-15 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869215#comment-15869215
 ] 

Sangjin Lee commented on YARN-4675:
---

The latest patch LGTM (v.11). I'll commit it soon. [~Naganarasimha], could you 
also post a patch for YARN-5355 if the trunk version does not apply cleanly?

> Reorganize TimelineClient and TimelineClientImpl into separate classes for 
> ATSv1.x and ATSv2
> 
>
> Key: YARN-4675
> URL: https://issues.apache.org/jira/browse/YARN-4675
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
>  Labels: YARN-5355, yarn-5355-merge-blocker
> Attachments: YARN-4675.v2.002.patch, YARN-4675.v2.003.patch, 
> YARN-4675.v2.004.patch, YARN-4675.v2.005.patch, YARN-4675.v2.006.patch, 
> YARN-4675.v2.007.patch, YARN-4675.v2.008.patch, YARN-4675.v2.009.patch, 
> YARN-4675.v2.010.patch, YARN-4675.v2.011.patch, 
> YARN-4675-YARN-2928.v1.001.patch
>
>
> We need to reorganize TimeClientImpl into TimeClientV1Impl ,  
> TimeClientV2Impl and if required a base class, so that its clear which part 
> of the code belongs to which version and thus better maintainable.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-6027) Support fromid(offset) filter for /flows API

2017-02-15 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15868739#comment-15868739
 ] 

Sangjin Lee commented on YARN-6027:
---

On a related note, how would the client provide the value for fromId? Does it 
need to compose the fromId, or can we add a UID for flow activity in the info 
and they can just grab and send it as fromId? I thought we were doing something 
like that for other pagination?

If it is the latter, then can we not use FLOW_UID for this?

> Support fromid(offset) filter for /flows API
> 
>
> Key: YARN-6027
> URL: https://issues.apache.org/jira/browse/YARN-6027
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>  Labels: yarn-5355-merge-blocker
> Attachments: YARN-6027-YARN-5355.0001.patch, 
> YARN-6027-YARN-5355.0002.patch
>
>
> In YARN-5585 , fromId is supported for retrieving entities. We need similar 
> filter for flows/flowRun apps and flow run and flow as well. 
> Along with supporting fromId, this JIRA should also discuss following points
> * Should we throw an exception for entities/entity retrieval if duplicates 
> found?
> * TimelieEntity :
> ** Should equals method also check for idPrefix?
> ** Does idPrefix is part of identifiers?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-4675) Reorganize TimelineClient and TimelineClientImpl into separate classes for ATSv1.x and ATSv2

2017-02-15 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15868665#comment-15868665
 ] 

Sangjin Lee commented on YARN-4675:
---

It's kind of hard to parse but the javadoc error is related to the patch:
{noformat}
[ERROR] 
/testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AMRMClient.java:698:
 error: unexpected text
[ERROR] * @throws YarnException, when this method is invoked even when ATS V2 
is not
[ERROR] ^
{noformat}
It might be the comma after YarnException that's the problem.

> Reorganize TimelineClient and TimelineClientImpl into separate classes for 
> ATSv1.x and ATSv2
> 
>
> Key: YARN-4675
> URL: https://issues.apache.org/jira/browse/YARN-4675
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
>  Labels: YARN-5355, yarn-5355-merge-blocker
> Attachments: YARN-4675.v2.002.patch, YARN-4675.v2.003.patch, 
> YARN-4675.v2.004.patch, YARN-4675.v2.005.patch, YARN-4675.v2.006.patch, 
> YARN-4675.v2.007.patch, YARN-4675.v2.008.patch, YARN-4675.v2.009.patch, 
> YARN-4675.v2.010.patch, YARN-4675-YARN-2928.v1.001.patch
>
>
> We need to reorganize TimeClientImpl into TimeClientV1Impl ,  
> TimeClientV2Impl and if required a base class, so that its clear which part 
> of the code belongs to which version and thus better maintainable.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-4675) Reorganize TimelineClient and TimelineClientImpl into separate classes for ATSv1.x and ATSv2

2017-02-15 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15868554#comment-15868554
 ] 

Sangjin Lee commented on YARN-4675:
---

The latest version looks good to me (v.10). I'll wait for the jenkins before 
proceeding.

> Reorganize TimelineClient and TimelineClientImpl into separate classes for 
> ATSv1.x and ATSv2
> 
>
> Key: YARN-4675
> URL: https://issues.apache.org/jira/browse/YARN-4675
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
>  Labels: YARN-5355, yarn-5355-merge-blocker
> Attachments: YARN-4675.v2.002.patch, YARN-4675.v2.003.patch, 
> YARN-4675.v2.004.patch, YARN-4675.v2.005.patch, YARN-4675.v2.006.patch, 
> YARN-4675.v2.007.patch, YARN-4675.v2.008.patch, YARN-4675.v2.009.patch, 
> YARN-4675.v2.010.patch, YARN-4675-YARN-2928.v1.001.patch
>
>
> We need to reorganize TimeClientImpl into TimeClientV1Impl ,  
> TimeClientV2Impl and if required a base class, so that its clear which part 
> of the code belongs to which version and thus better maintainable.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-4675) Reorganize TimelineClient and TimelineClientImpl into separate classes for ATSv1.x and ATSv2

2017-02-13 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864400#comment-15864400
 ] 

Sangjin Lee commented on YARN-4675:
---

Thanks for the updated patch [~Naganarasimha]!

If it is not too much, could you kindly prepare the YARN-5355 patch in addition 
to the trunk patch so that we can commit this to both trunk and YARN-5355? This 
is a fairly major API change and we'd like to have that in YARN-5355 as early 
as possible.

A couple more (all minor) comments:
- AMRMClient.registerTimelineV2Client(): should we throw an exception if this 
method is called and timeline service v.2 is not enabled? The case would likely 
be a code bug, and we probably need a stronger failure to catch this
- There still seem to be a few checkstyle issues that are related with the 
patch and they seem fixable. Could you please look into them?

Thanks Naga!

> Reorganize TimelineClient and TimelineClientImpl into separate classes for 
> ATSv1.x and ATSv2
> 
>
> Key: YARN-4675
> URL: https://issues.apache.org/jira/browse/YARN-4675
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
>  Labels: YARN-5355, yarn-5355-merge-blocker
> Attachments: YARN-4675.v2.002.patch, YARN-4675.v2.003.patch, 
> YARN-4675.v2.004.patch, YARN-4675.v2.005.patch, YARN-4675.v2.006.patch, 
> YARN-4675.v2.007.patch, YARN-4675.v2.008.patch, 
> YARN-4675-YARN-2928.v1.001.patch
>
>
> We need to reorganize TimeClientImpl into TimeClientV1Impl ,  
> TimeClientV2Impl and if required a base class, so that its clear which part 
> of the code belongs to which version and thus better maintainable.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-6170) TimelineReaderServer should wait to join with HttpServer2

2017-02-11 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-6170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15862542#comment-15862542
 ] 

Sangjin Lee commented on YARN-6170:
---

Thanks!

> TimelineReaderServer should wait to join with HttpServer2
> -
>
> Key: YARN-6170
> URL: https://issues.apache.org/jira/browse/YARN-6170
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelinereader
>Affects Versions: 3.0.0-alpha2, YARN-5355
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Minor
> Fix For: YARN-5355-branch-2, 3.0.0-alpha3
>
> Attachments: YARN-6170.01.patch
>
>
> While I was backporting YARN-5355-branch-2 to a 2.6.0-based code branch, I 
> noticed that the timeline reader daemon would promptly shut down upon start. 
> It turns out that in the 2.6.0 code line at least there are only daemon 
> threads left once the main method returns. That causes the JVM to shut down.
> The right pattern to start an embedded jetty web server is to call 
> {{Server.start()}} followed by {{Server.join()}}. That way, the server stays 
> up reliably no matter what other threads get created.
> It works on YARN-5355 only because there *happens* to be one other non-daemon 
> thread. We should add the {{join()}} call to be always correct.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-6170) TimelineReaderServer should wait to join with HttpServer2

2017-02-10 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-6170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15862063#comment-15862063
 ] 

Sangjin Lee commented on YARN-6170:
---

Thanks Varun! It would be great if we can merge this to YARN-5355-branch-2 in 
addition to trunk.

> TimelineReaderServer should wait to join with HttpServer2
> -
>
> Key: YARN-6170
> URL: https://issues.apache.org/jira/browse/YARN-6170
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelinereader
>Affects Versions: 3.0.0-alpha2, YARN-5355
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Minor
> Attachments: YARN-6170.01.patch
>
>
> While I was backporting YARN-5355-branch-2 to a 2.6.0-based code branch, I 
> noticed that the timeline reader daemon would promptly shut down upon start. 
> It turns out that in the 2.6.0 code line at least there are only daemon 
> threads left once the main method returns. That causes the JVM to shut down.
> The right pattern to start an embedded jetty web server is to call 
> {{Server.start()}} followed by {{Server.join()}}. That way, the server stays 
> up reliably no matter what other threads get created.
> It works on YARN-5355 only because there *happens* to be one other non-daemon 
> thread. We should add the {{join()}} call to be always correct.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-6170) TimelineReaderServer should wait to join with HttpServer2

2017-02-10 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-6170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15861841#comment-15861841
 ] 

Sangjin Lee commented on YARN-6170:
---

I'd like to add that this doesn't necessarily mean that we do not want to 
consider changes in {{HttpServer2}} itself as suggested in HADOOP-11749.

> TimelineReaderServer should wait to join with HttpServer2
> -
>
> Key: YARN-6170
> URL: https://issues.apache.org/jira/browse/YARN-6170
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelinereader
>Affects Versions: 3.0.0-alpha2, YARN-5355
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Minor
> Attachments: YARN-6170.01.patch
>
>
> While I was backporting YARN-5355-branch-2 to a 2.6.0-based code branch, I 
> noticed that the timeline reader daemon would promptly shut down upon start. 
> It turns out that in the 2.6.0 code line at least there are only daemon 
> threads left once the main method returns. That causes the JVM to shut down.
> The right pattern to start an embedded jetty web server is to call 
> {{Server.start()}} followed by {{Server.join()}}. That way, the server stays 
> up reliably no matter what other threads get created.
> It works on YARN-5355 only because there *happens* to be one other non-daemon 
> thread. We should add the {{join()}} call to be always correct.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-6027) Improve /flows API for more flexible filters fromid, collapse, userid

2017-02-10 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15861739#comment-15861739
 ] 

Sangjin Lee commented on YARN-6027:
---

Absolutely. Thanks for the reminder.

The idea of supporting the "collapse" here is to combine flow activities from 
different dates for the same flow and the user so that they are returned as a 
single flow activity record by the reader REST server. While we recognized that 
this may make rendering certain UI easier on the UI implementation standpoint, 
we saw a few problems with that as well.

First, this goes against the design intent on flow activities as "daily 
activities" of flows as a fair amount of thought went into designing that table 
to represent daily activities. We're not saying since the schema is this way 
the reader has to return the data that way too to satisfy the schema. We tried 
to design the schema to model truly useful concepts and that way we could 
accomplish things like filter pushdowns. In that sense, we need to see if we 
can support various use cases still based on that intent and design.

Another point was that if we were to introduce something like collapse we would 
ideally need to do this in a consistent manner across many/most endpoints. And 
that definitely means a large scope of introducing collapse if we do. So we 
agreed that we would recognize that as a separate discussion and still move 
forward with implementing pagination.

Let me know if I missed anything major.

> Improve /flows API for more flexible filters fromid, collapse, userid
> -
>
> Key: YARN-6027
> URL: https://issues.apache.org/jira/browse/YARN-6027
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>  Labels: yarn-5355-merge-blocker
> Attachments: YARN-6027-YARN-5355.0001.patch
>
>
> In YARN-5585 , fromId is supported for retrieving entities. We need similar 
> filter for flows/flowRun apps and flow run and flow as well. 
> Along with supporting fromId, this JIRA should also discuss following points
> * Should we throw an exception for entities/entity retrieval if duplicates 
> found?
> * TimelieEntity :
> ** Should equals method also check for idPrefix?
> ** Does idPrefix is part of identifiers?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-6170) TimelineReaderServer should wait to join with HttpServer2

2017-02-10 Thread Sangjin Lee (JIRA)

 [ 
https://issues.apache.org/jira/browse/YARN-6170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee updated YARN-6170:
--
Attachment: YARN-6170.01.patch

v.1 patch posted.

> TimelineReaderServer should wait to join with HttpServer2
> -
>
> Key: YARN-6170
> URL: https://issues.apache.org/jira/browse/YARN-6170
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelinereader
>Affects Versions: 3.0.0-alpha2, YARN-5355
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Minor
> Attachments: YARN-6170.01.patch
>
>
> While I was backporting YARN-5355-branch-2 to a 2.6.0-based code branch, I 
> noticed that the timeline reader daemon would promptly shut down upon start. 
> It turns out that in the 2.6.0 code line at least there are only daemon 
> threads left once the main method returns. That causes the JVM to shut down.
> The right pattern to start an embedded jetty web server is to call 
> {{Server.start()}} followed by {{Server.join()}}. That way, the server stays 
> up reliably no matter what other threads get created.
> It works on YARN-5355 only because there *happens* to be one other non-daemon 
> thread. We should add the {{join()}} call to be always correct.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Comment Edited] (YARN-6170) TimelineReaderServer should wait to join with HttpServer2

2017-02-10 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-6170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15861691#comment-15861691
 ] 

Sangjin Lee edited comment on YARN-6170 at 2/10/17 7:06 PM:


Thanks for the info Varun. I found HADOOP-11749.

I can definitely see an argument that {{HttpServer2.start()}} should ensure it 
stays up. However, it might be somewhat awkward to do it there. I'm not too 
sure if I like the idea of turning some threads it owns into non-daemon to do 
it. The right way to do it is still to join, but you cannot do that (inline) in 
the {{start()}} method.

{{HttpServer2}} does offer a {{join()}} method (probably for this purpose). I'm 
going to put up a patch that utilizes this. Let me know what you think.


was (Author: sjlee0):
Thanks for the info Varun. I found HADOOP-11479.

I can definitely see an argument that {{HttpServer2.start()}} should ensure it 
stays up. However, it might be somewhat awkward to do it there. I'm not too 
sure if I like the idea of turning some threads it owns into non-daemon to do 
it. The right way to do it is still to join, but you cannot do that (inline) in 
the {{start()}} method.

{{HttpServer2}} does offer a {{join()}} method (probably for this purpose). I'm 
going to put up a patch that utilizes this. Let me know what you think.

> TimelineReaderServer should wait to join with HttpServer2
> -
>
> Key: YARN-6170
> URL: https://issues.apache.org/jira/browse/YARN-6170
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelinereader
>Affects Versions: 3.0.0-alpha2, YARN-5355
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Minor
>
> While I was backporting YARN-5355-branch-2 to a 2.6.0-based code branch, I 
> noticed that the timeline reader daemon would promptly shut down upon start. 
> It turns out that in the 2.6.0 code line at least there are only daemon 
> threads left once the main method returns. That causes the JVM to shut down.
> The right pattern to start an embedded jetty web server is to call 
> {{Server.start()}} followed by {{Server.join()}}. That way, the server stays 
> up reliably no matter what other threads get created.
> It works on YARN-5355 only because there *happens* to be one other non-daemon 
> thread. We should add the {{join()}} call to be always correct.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-6170) TimelineReaderServer should wait to join with HttpServer2

2017-02-10 Thread Sangjin Lee (JIRA)

 [ 
https://issues.apache.org/jira/browse/YARN-6170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee updated YARN-6170:
--
Affects Version/s: 3.0.0-alpha2

> TimelineReaderServer should wait to join with HttpServer2
> -
>
> Key: YARN-6170
> URL: https://issues.apache.org/jira/browse/YARN-6170
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelinereader
>Affects Versions: 3.0.0-alpha2, YARN-5355
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Minor
>
> While I was backporting YARN-5355-branch-2 to a 2.6.0-based code branch, I 
> noticed that the timeline reader daemon would promptly shut down upon start. 
> It turns out that in the 2.6.0 code line at least there are only daemon 
> threads left once the main method returns. That causes the JVM to shut down.
> The right pattern to start an embedded jetty web server is to call 
> {{Server.start()}} followed by {{Server.join()}}. That way, the server stays 
> up reliably no matter what other threads get created.
> It works on YARN-5355 only because there *happens* to be one other non-daemon 
> thread. We should add the {{join()}} call to be always correct.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-6170) TimelineReaderServer should wait to join with HttpServer2

2017-02-10 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-6170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15861691#comment-15861691
 ] 

Sangjin Lee commented on YARN-6170:
---

Thanks for the info Varun. I found HADOOP-11479.

I can definitely see an argument that {{HttpServer2.start()}} should ensure it 
stays up. However, it might be somewhat awkward to do it there. I'm not too 
sure if I like the idea of turning some threads it owns into non-daemon to do 
it. The right way to do it is still to join, but you cannot do that (inline) in 
the {{start()}} method.

{{HttpServer2}} does offer a {{join()}} method (probably for this purpose). I'm 
going to put up a patch that utilizes this. Let me know what you think.

> TimelineReaderServer should wait to join with HttpServer2
> -
>
> Key: YARN-6170
> URL: https://issues.apache.org/jira/browse/YARN-6170
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelinereader
>Affects Versions: YARN-5355
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Minor
>
> While I was backporting YARN-5355-branch-2 to a 2.6.0-based code branch, I 
> noticed that the timeline reader daemon would promptly shut down upon start. 
> It turns out that in the 2.6.0 code line at least there are only daemon 
> threads left once the main method returns. That causes the JVM to shut down.
> The right pattern to start an embedded jetty web server is to call 
> {{Server.start()}} followed by {{Server.join()}}. That way, the server stays 
> up reliably no matter what other threads get created.
> It works on YARN-5355 only because there *happens* to be one other non-daemon 
> thread. We should add the {{join()}} call to be always correct.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-6170) TimelineReaderServer should wait to join with HttpServer2

2017-02-10 Thread Sangjin Lee (JIRA)

 [ 
https://issues.apache.org/jira/browse/YARN-6170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee updated YARN-6170:
--
Labels:   (was: yarn-5355-merge-blocker)

> TimelineReaderServer should wait to join with HttpServer2
> -
>
> Key: YARN-6170
> URL: https://issues.apache.org/jira/browse/YARN-6170
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelinereader
>Affects Versions: YARN-5355
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Minor
>
> While I was backporting YARN-5355-branch-2 to a 2.6.0-based code branch, I 
> noticed that the timeline reader daemon would promptly shut down upon start. 
> It turns out that in the 2.6.0 code line at least there are only daemon 
> threads left once the main method returns. That causes the JVM to shut down.
> The right pattern to start an embedded jetty web server is to call 
> {{Server.start()}} followed by {{Server.join()}}. That way, the server stays 
> up reliably no matter what other threads get created.
> It works on YARN-5355 only because there *happens* to be one other non-daemon 
> thread. We should add the {{join()}} call to be always correct.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Created] (YARN-6170) TimelineReaderServer should wait to join with HttpServer2

2017-02-09 Thread Sangjin Lee (JIRA)
Sangjin Lee created YARN-6170:
-

 Summary: TimelineReaderServer should wait to join with HttpServer2
 Key: YARN-6170
 URL: https://issues.apache.org/jira/browse/YARN-6170
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelinereader
Affects Versions: YARN-5355
Reporter: Sangjin Lee
Assignee: Sangjin Lee
Priority: Minor


While I was backporting YARN-5355-branch-2 to a 2.6.0-based code branch, I 
noticed that the timeline reader daemon would promptly shut down upon start. It 
turns out that in the 2.6.0 code line at least there are only daemon threads 
left once the main method returns. That causes the JVM to shut down.

The right pattern to start an embedded jetty web server is to call 
{{Server.start()}} followed by {{Server.join()}}. That way, the server stays up 
reliably no matter what other threads get created.

It works on YARN-5355 only because there *happens* to be one other non-daemon 
thread. We should add the {{join()}} call to be always correct.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-4675) Reorganize TimelineClient and TimelineClientImpl into separate classes for ATSv1.x and ATSv2

2017-02-09 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860164#comment-15860164
 ] 

Sangjin Lee commented on YARN-4675:
---

We should be good to go once you addressed Varun's comments. Should this go to 
our feature branch (YARN-5355) or to trunk directly? I would think our feature 
branch should be the target? If so, could you kindly regenerate the patch 
against the branch?

> Reorganize TimelineClient and TimelineClientImpl into separate classes for 
> ATSv1.x and ATSv2
> 
>
> Key: YARN-4675
> URL: https://issues.apache.org/jira/browse/YARN-4675
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
>  Labels: YARN-5355, yarn-5355-merge-blocker
> Attachments: YARN-4675.v2.002.patch, YARN-4675.v2.003.patch, 
> YARN-4675.v2.004.patch, YARN-4675.v2.005.patch, YARN-4675.v2.006.patch, 
> YARN-4675.v2.007.patch, YARN-4675-YARN-2928.v1.001.patch
>
>
> We need to reorganize TimeClientImpl into TimeClientV1Impl ,  
> TimeClientV2Impl and if required a base class, so that its clear which part 
> of the code belongs to which version and thus better maintainable.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-4985) Refactor the coprocessor code & other definition classes into independent packages

2017-02-09 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860136#comment-15860136
 ] 

Sangjin Lee commented on YARN-4985:
---

Thanks for refreshing my memory on this Haibo. Yes, I recall that was the main 
issue. To tease out client and server correctly, we'd need a separate (common) 
module both client and server depends on. The only way we could avoid it is if 
all the fine-grained downstream dependencies can be isolated and moved into the 
hbase-server module. Do you know if it is feasible?

If that is not feasible, maybe this refactoring would be more problematic than 
beneficial. What would we lose if we didn't do this refactoring? Also, if we 
didn't do this refactoring and kept thins as is (just timelineservice-hbase), 
do we still have the dependency issues in terms of installing the coprocessor 
(we need to follow all transitive dependencies between classes)? If so, that 
needs to be addressed no matter what.

> Refactor the coprocessor code & other definition classes into independent 
> packages
> --
>
> Key: YARN-4985
> URL: https://issues.apache.org/jira/browse/YARN-4985
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Vrushali C
>Assignee: Haibo Chen
>  Labels: YARN-5355
> Attachments: YARN-4985-YARN-5355.prelim.patch
>
>
> As part of the coprocessor deployment, we have realized that it will be much 
> cleaner to have the coprocessor code sit in a package which does not depend 
> on hadoop-yarn-server classes. It only needs hbase and other util classes.
> These util classes and tag definition related classes can be refactored into 
> their own independent "definition" class package so that making changes to 
> coprocessor code, upgrading hbase, deploying hbase on a different hadoop 
> version cluster etc all becomes operationally much easier and less error 
> prone to having different library jars etc.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-6027) Improve /flows API for more flexible filters fromid, collapse, userid

2017-02-09 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860111#comment-15860111
 ] 

Sangjin Lee commented on YARN-6027:
---

We discussed this offline during our status call. Just to record what we 
agreed, can we separate the aspect of the collapse filter from this JIRA and 
focus on fromId and userId? I don't think there is an agreement on the collapse 
part, and we can still make progress with the other two filters separately. 
Thanks!

> Improve /flows API for more flexible filters fromid, collapse, userid
> -
>
> Key: YARN-6027
> URL: https://issues.apache.org/jira/browse/YARN-6027
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>  Labels: yarn-5355-merge-blocker
> Attachments: YARN-6027-YARN-5355.0001.patch
>
>
> In YARN-5585 , fromId is supported for retrieving entities. We need similar 
> filter for flows/flowRun apps and flow run and flow as well. 
> Along with supporting fromId, this JIRA should also discuss following points
> * Should we throw an exception for entities/entity retrieval if duplicates 
> found?
> * TimelieEntity :
> ** Should equals method also check for idPrefix?
> ** Does idPrefix is part of identifiers?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-4985) Refactor the coprocessor code & other definition classes into independent packages

2017-02-08 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15859055#comment-15859055
 ] 

Sangjin Lee commented on YARN-4985:
---

[~haibochen]

Is it possible to create the hbase-server module so that it does *not* depend 
on the hbase-client module? That's what I meant by watching out for 
dependencies in the hbase-server code. It would be ideal if the hbase-server 
code does not have to depend on the hbase-client code. It might mean splitting 
the tests and putting some in the client and others in the server.

> Refactor the coprocessor code & other definition classes into independent 
> packages
> --
>
> Key: YARN-4985
> URL: https://issues.apache.org/jira/browse/YARN-4985
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Vrushali C
>Assignee: Haibo Chen
>  Labels: YARN-5355
> Attachments: YARN-4985-YARN-5355.prelim.patch
>
>
> As part of the coprocessor deployment, we have realized that it will be much 
> cleaner to have the coprocessor code sit in a package which does not depend 
> on hadoop-yarn-server classes. It only needs hbase and other util classes.
> These util classes and tag definition related classes can be refactored into 
> their own independent "definition" class package so that making changes to 
> coprocessor code, upgrading hbase, deploying hbase on a different hadoop 
> version cluster etc all becomes operationally much easier and less error 
> prone to having different library jars etc.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-6140) start time key in NM leveldb store should be removed when container is removed

2017-02-03 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-6140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852013#comment-15852013
 ] 

Sangjin Lee commented on YARN-6140:
---

One interesting thing is that in the current YARN-5355 branch the test does not 
fail, whereas a 2.6.0-based test does. The only major difference I can glean 
from it is the old code use {{DB}} whereas the new code uses {{WriteBatch}}. 
Still, I think it is right to remove the key.

> start time key in NM leveldb store should be removed when container is removed
> --
>
> Key: YARN-6140
> URL: https://issues.apache.org/jira/browse/YARN-6140
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: nodemanager
>Affects Versions: YARN-5355
>Reporter: Sangjin Lee
>Assignee: Ajith S
>  Labels: yarn-5355-merge-blocker
>
> It appears that the start time key is not removed when the container is 
> removed. The key was introduced in YARN-5792.
> I found this while backporting the YARN-5355-branch-2 branch to our internal 
> branch loosely based on 2.6.0. The {{TestNMLeveldbStateStoreService}} test 
> was failing because of this.
> I'm not sure why we didn't see this earlier.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-6140) start time key in NM leveldb store should be removed when container is removed

2017-02-03 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-6140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15851801#comment-15851801
 ] 

Sangjin Lee commented on YARN-6140:
---

Please feel free!

> start time key in NM leveldb store should be removed when container is removed
> --
>
> Key: YARN-6140
> URL: https://issues.apache.org/jira/browse/YARN-6140
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: nodemanager
>Affects Versions: YARN-5355
>Reporter: Sangjin Lee
>Assignee: Ajith S
>  Labels: yarn-5355-merge-blocker
>
> It appears that the start time key is not removed when the container is 
> removed. The key was introduced in YARN-5792.
> I found this while backporting the YARN-5355-branch-2 branch to our internal 
> branch loosely based on 2.6.0. The {{TestNMLeveldbStateStoreService}} test 
> was failing because of this.
> I'm not sure why we didn't see this earlier.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Created] (YARN-6140) start time key in NM leveldb store should be removed when container is removed

2017-02-03 Thread Sangjin Lee (JIRA)
Sangjin Lee created YARN-6140:
-

 Summary: start time key in NM leveldb store should be removed when 
container is removed
 Key: YARN-6140
 URL: https://issues.apache.org/jira/browse/YARN-6140
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: nodemanager
Affects Versions: YARN-5355
Reporter: Sangjin Lee


It appears that the start time key is not removed when the container is 
removed. The key was introduced in YARN-5792.

I found this while backporting the YARN-5355-branch-2 branch to our internal 
branch loosely based on 2.6.0. The {{TestNMLeveldbStateStoreService}} test was 
failing because of this.

I'm not sure why we didn't see this earlier.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-4985) Refactor the coprocessor code & other definition classes into independent packages

2017-01-27 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15843111#comment-15843111
 ] 

Sangjin Lee commented on YARN-4985:
---

At the end of the day, {{TimelineSchemaCreator}} is a standalone process. 
Either {{bin/hbase}} or {{bin/hadoop}} would be able to launch the schema 
creator. It doesn't exercise any hbase server code. In that sense, it is 
probably closer to the client module than the server module.

> Refactor the coprocessor code & other definition classes into independent 
> packages
> --
>
> Key: YARN-4985
> URL: https://issues.apache.org/jira/browse/YARN-4985
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Vrushali C
>Assignee: Haibo Chen
>  Labels: YARN-5355
>
> As part of the coprocessor deployment, we have realized that it will be much 
> cleaner to have the coprocessor code sit in a package which does not depend 
> on hadoop-yarn-server classes. It only needs hbase and other util classes.
> These util classes and tag definition related classes can be refactored into 
> their own independent "definition" class package so that making changes to 
> coprocessor code, upgrading hbase, deploying hbase on a different hadoop 
> version cluster etc all becomes operationally much easier and less error 
> prone to having different library jars etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-3637) Handle localization sym-linking correctly at the YARN level

2017-01-25 Thread Sangjin Lee (JIRA)

 [ 
https://issues.apache.org/jira/browse/YARN-3637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee updated YARN-3637:
--
Hadoop Flags: Reviewed

> Handle localization sym-linking correctly at the YARN level
> ---
>
> Key: YARN-3637
> URL: https://issues.apache.org/jira/browse/YARN-3637
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: YARN-3637-trunk.001.patch, YARN-3637-trunk.002.patch, 
> YARN-3637-trunk.003.patch
>
>
> The shared cache needs to handle resource sym-linking at the YARN layer. 
> Currently, we let the application layer (i.e. mapreduce) handle this, but it 
> is probably better for all applications if it is handled transparently.
> Here is the scenario:
> Imagine two separate jars (with unique checksums) that have the same name 
> job.jar.
> They are stored in the shared cache as two separate resources:
> checksum1/job.jar
> checksum2/job.jar
> A new application tries to use both of these resources, but internally refers 
> to them as different names:
> foo.jar maps to checksum1
> bar.jar maps to checksum2
> When the shared cache returns the path to the resources, both resources are 
> named the same (i.e. job.jar). Because of this, when the resources are 
> localized one of them clobbers the other. This is because both symlinks in 
> the container_id directory are the same name (i.e. job.jar) even though they 
> point to two separate resource directories.
> Originally we tackled this in the MapReduce client by using the fragment 
> portion of the resource url. This, however, seems like something that should 
> be solved at the YARN layer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Comment Edited] (YARN-6117) SharedCacheManager does not start up

2017-01-25 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-6117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15838828#comment-15838828
 ] 

Sangjin Lee edited comment on YARN-6117 at 1/25/17 11:39 PM:
-

Ported it to branch-2 as well (2.9.0). Since the shared cache is not fully 
implemented in previous versions, I don't think we need to backport to those 
versions.


was (Author: sjlee0):
Ported it to branch-2 as well (2.9.0).

> SharedCacheManager does not start up
> 
>
> Key: YARN-6117
> URL: https://issues.apache.org/jira/browse/YARN-6117
> Project: Hadoop YARN
>  Issue Type: Bug
>Affects Versions: 2.7.3, 3.0.0-alpha2
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: YARN-6117-trunk.001.patch
>
>
> The webapp directory for the SharedCacheManager is missing and the SCM fails 
> to start up with the following:
> {noformat}
> 2017-01-22 00:14:25,162 INFO org.apache.hadoop.service.AbstractService: 
> Service SharedCacheManager failed in state STARTED; cause: 
> org.apache.hadoop.yarn.webapp.WebAppException: Error starting http server
> org.apache.hadoop.yarn.webapp.WebAppException: Error starting http server
> at 
> org.apache.hadoop.yarn.webapp.WebApps$Builder.build(WebApps.java:330)
> at 
> org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:377)
> at 
> org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:373)
> at 
> org.apache.hadoop.yarn.server.sharedcachemanager.webapp.SCMWebServer.serviceStart(SCMWebServer.java:65)
> at 
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at 
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
> at 
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at 
> org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.main(SharedCacheManager.java:157)
> Caused by: java.io.FileNotFoundException: webapps/sharedcache not found in 
> CLASSPATH
> at 
> org.apache.hadoop.http.HttpServer2.getWebAppsPath(HttpServer2.java:972)
> at org.apache.hadoop.http.HttpServer2.(HttpServer2.java:478)
> at org.apache.hadoop.http.HttpServer2.(HttpServer2.java:117)
> at 
> org.apache.hadoop.http.HttpServer2$Builder.build(HttpServer2.java:392)
> at 
> org.apache.hadoop.yarn.webapp.WebApps$Builder.build(WebApps.java:291)
> ... 7 more
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-6117) SharedCacheManager does not start up

2017-01-25 Thread Sangjin Lee (JIRA)

 [ 
https://issues.apache.org/jira/browse/YARN-6117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee updated YARN-6117:
--
Fix Version/s: 2.9.0

Ported it to branch-2 as well (2.9.0).

> SharedCacheManager does not start up
> 
>
> Key: YARN-6117
> URL: https://issues.apache.org/jira/browse/YARN-6117
> Project: Hadoop YARN
>  Issue Type: Bug
>Affects Versions: 2.7.3, 3.0.0-alpha2
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: YARN-6117-trunk.001.patch
>
>
> The webapp directory for the SharedCacheManager is missing and the SCM fails 
> to start up with the following:
> {noformat}
> 2017-01-22 00:14:25,162 INFO org.apache.hadoop.service.AbstractService: 
> Service SharedCacheManager failed in state STARTED; cause: 
> org.apache.hadoop.yarn.webapp.WebAppException: Error starting http server
> org.apache.hadoop.yarn.webapp.WebAppException: Error starting http server
> at 
> org.apache.hadoop.yarn.webapp.WebApps$Builder.build(WebApps.java:330)
> at 
> org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:377)
> at 
> org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:373)
> at 
> org.apache.hadoop.yarn.server.sharedcachemanager.webapp.SCMWebServer.serviceStart(SCMWebServer.java:65)
> at 
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at 
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
> at 
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at 
> org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.main(SharedCacheManager.java:157)
> Caused by: java.io.FileNotFoundException: webapps/sharedcache not found in 
> CLASSPATH
> at 
> org.apache.hadoop.http.HttpServer2.getWebAppsPath(HttpServer2.java:972)
> at org.apache.hadoop.http.HttpServer2.(HttpServer2.java:478)
> at org.apache.hadoop.http.HttpServer2.(HttpServer2.java:117)
> at 
> org.apache.hadoop.http.HttpServer2$Builder.build(HttpServer2.java:392)
> at 
> org.apache.hadoop.yarn.webapp.WebApps$Builder.build(WebApps.java:291)
> ... 7 more
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-3637) Handle localization sym-linking correctly at the YARN level

2017-01-24 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15837209#comment-15837209
 ] 

Sangjin Lee commented on YARN-3637:
---

I'll commit this patch tomorrow unless there is an objection. [~ctrezzo], to 
which branches should this be committed?

> Handle localization sym-linking correctly at the YARN level
> ---
>
> Key: YARN-3637
> URL: https://issues.apache.org/jira/browse/YARN-3637
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
> Attachments: YARN-3637-trunk.001.patch, YARN-3637-trunk.002.patch, 
> YARN-3637-trunk.003.patch
>
>
> The shared cache needs to handle resource sym-linking at the YARN layer. 
> Currently, we let the application layer (i.e. mapreduce) handle this, but it 
> is probably better for all applications if it is handled transparently.
> Here is the scenario:
> Imagine two separate jars (with unique checksums) that have the same name 
> job.jar.
> They are stored in the shared cache as two separate resources:
> checksum1/job.jar
> checksum2/job.jar
> A new application tries to use both of these resources, but internally refers 
> to them as different names:
> foo.jar maps to checksum1
> bar.jar maps to checksum2
> When the shared cache returns the path to the resources, both resources are 
> named the same (i.e. job.jar). Because of this, when the resources are 
> localized one of them clobbers the other. This is because both symlinks in 
> the container_id directory are the same name (i.e. job.jar) even though they 
> point to two separate resource directories.
> Originally we tackled this in the MapReduce client by using the fragment 
> portion of the resource url. This, however, seems like something that should 
> be solved at the YARN layer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-3637) Handle localization sym-linking correctly at the YARN level

2017-01-20 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15832689#comment-15832689
 ] 

Sangjin Lee commented on YARN-3637:
---

Thanks [~ctrezzo] for your patch!

It looks good for the most part. One small suggestion would be to handle the 
case where the resource name was the same as the file name returned by the 
shared cache manager. I suspect that would be > 90% of the cases. In that case, 
we may want to add the redundant name twice. And this may matter if we're 
dealing with hundreds or thousands of localizable resources.

> Handle localization sym-linking correctly at the YARN level
> ---
>
> Key: YARN-3637
> URL: https://issues.apache.org/jira/browse/YARN-3637
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
> Attachments: YARN-3637-trunk.001.patch, YARN-3637-trunk.002.patch
>
>
> The shared cache needs to handle resource sym-linking at the YARN layer. 
> Currently, we let the application layer (i.e. mapreduce) handle this, but it 
> is probably better for all applications if it is handled transparently.
> Here is the scenario:
> Imagine two separate jars (with unique checksums) that have the same name 
> job.jar.
> They are stored in the shared cache as two separate resources:
> checksum1/job.jar
> checksum2/job.jar
> A new application tries to use both of these resources, but internally refers 
> to them as different names:
> foo.jar maps to checksum1
> bar.jar maps to checksum2
> When the shared cache returns the path to the resources, both resources are 
> named the same (i.e. job.jar). Because of this, when the resources are 
> localized one of them clobbers the other. This is because both symlinks in 
> the container_id directory are the same name (i.e. job.jar) even though they 
> point to two separate resource directories.
> Originally we tackled this in the MapReduce client by using the fragment 
> portion of the resource url. This, however, seems like something that should 
> be solved at the YARN layer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-4675) Reorganize TimeClientImpl into TimeClientV1Impl and TimeClientV2Impl

2017-01-20 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15832483#comment-15832483
 ] 

Sangjin Lee commented on YARN-4675:
---

Thanks [~Naganarasimha]! I went over the latest patch, and have some comments. 
Quite a few comments overlap with [~gtCarrera9]'s comments.

(TimelineClient.java)
- l.57: agree with [~gtCarrera9] that we want to make the constructor protected

(TimelineV2Client.java)
- I would move {{contextAppId}} to {{TimelineV2ClientImpl}}. It is used solely 
internally by {{TimelineV2ClientImpl}} and it should be defined there.
- As a follow-up, we should drop the {{appId}} argument from the 
{{TimelineV2Client}} constructor
- As a follow-up, we can drop the {{setContextAppId}} and {{getContextAppId}} 
methods
- l.37: The blurb "Create a timeline client" sounds strange. We should remove 
that sentence once it's moved.
- l.58: make this constructor protected too
- l.78,96: nit: let's use imports now that there is no ambiguity

(TimelineClientImpl.java)
- we should do a strong check of the timeline service version in 
{{serviceInit()}} not to accept an invalid version
- l.89: this should be removed
- we should remove {{pollTimelineServiceAddress()}}
- we should remove {{initConnConfigurator()}}
- we should remove {{initSslConnConfigurator()}}
- we should remove {{serviceRetryInterval}}
- we should remove {{DEFAULT_TIMEOUT_CONN_CONFIGURATOR}}
- we should remove {{setTimeouts()}}
- we should remove {{DEFAULT_SOCKET_TIMEOUT}}
- {{TimelineClientConnectionRetry}} and {{TimelineJerseyRetryFilter}} are 
duplicated with {{TimelineServiceHelper}}; we should determine where we need 
them and remove duplication; I would remove them from {{TimelineServiceHelper}} 
unless we think we will use them later for v.2
- l.562: we should drop the {{boolean v2}} argument
- we don't need a public {{setTimelineServiceAddress()}} method in v.1.x; I 
would simply inline it
- in {{putEntities()}}, we can remove the check if we have it in 
{{serviceInit()}}; were we always checking for == 1.5 by the way? So we don't 
support 1.0 any more?

(TimelineServiceHelper.java)
- we should remove {{TimelineClientConnectionRetry}} and 
{{TimelineJerseyRetryFilter}}

(TimelineV2ClientImpl.java)
- l.88: we should remove {{connectionRetry}}
- l.168: we should drop the {{boolean v2}} argument

(JobHistoryEventHandler.java)
- i believe we need to touch {{TestJobHistoryEventHandler}} too (the 
{{JHEvenHandlerForTest}} class)
- l.332, 458: super nit: space before the curly brace

(TestJobHistoryEventHandler.java)
- is this all about testing timeline service v.1? note that 
{{mockAppContext()}} now returns a v.1 client only


> Reorganize TimeClientImpl into TimeClientV1Impl and TimeClientV2Impl
> 
>
> Key: YARN-4675
> URL: https://issues.apache.org/jira/browse/YARN-4675
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
>  Labels: YARN-5355, yarn-5355-merge-blocker
> Attachments: YARN-4675.v2.002.patch, YARN-4675.v2.003.patch, 
> YARN-4675-YARN-2928.v1.001.patch
>
>
> We need to reorganize TimeClientImpl into TimeClientV1Impl ,  
> TimeClientV2Impl and if required a base class, so that its clear which part 
> of the code belongs to which version and thus better maintainable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-5928) Move ATSv2 HBase backend code into a new module that is only dependent at runtime by yarn servers

2017-01-19 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-5928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831180#comment-15831180
 ] 

Sangjin Lee commented on YARN-5928:
---

Since this is targeted for 3.0.0-alpha2, I'm merging this to 
branch-3.0.0-alpha2 as well as trunk.

cc [~andrew.wang]

> Move ATSv2 HBase backend code into a new module that is only dependent at 
> runtime by yarn servers
> -
>
> Key: YARN-5928
> URL: https://issues.apache.org/jira/browse/YARN-5928
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Affects Versions: 3.0.0-alpha1
>Reporter: Haibo Chen
>Assignee: Haibo Chen
> Attachments: YARN-5928.01.patch, YARN-5928.02.patch, 
> YARN-5928.06.patch, YARN-5928-YARN-5355.02.patch, 
> YARN-5928-YARN-5355.03.patch, YARN-5928-YARN-5355.04.patch, 
> YARN-5928-YARN-5355.04.patch, YARN-5928-YARN-5355.05.patch, 
> YARN-5928-YARN-5355.06.patch, YARN-5928-YARN-5355.07.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-5928) Move ATSv2 HBase backend code into a new module that is only dependent at runtime by yarn servers

2017-01-19 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-5928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831175#comment-15831175
 ] 

Sangjin Lee commented on YARN-5928:
---

The latest patches LGTM. Will commit shortly.

> Move ATSv2 HBase backend code into a new module that is only dependent at 
> runtime by yarn servers
> -
>
> Key: YARN-5928
> URL: https://issues.apache.org/jira/browse/YARN-5928
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Affects Versions: 3.0.0-alpha1
>Reporter: Haibo Chen
>Assignee: Haibo Chen
> Attachments: YARN-5928.01.patch, YARN-5928.02.patch, 
> YARN-5928.06.patch, YARN-5928-YARN-5355.02.patch, 
> YARN-5928-YARN-5355.03.patch, YARN-5928-YARN-5355.04.patch, 
> YARN-5928-YARN-5355.04.patch, YARN-5928-YARN-5355.05.patch, 
> YARN-5928-YARN-5355.06.patch, YARN-5928-YARN-5355.07.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-5928) Move ATSv2 HBase backend code into a new module that is only dependent at runtime by yarn servers

2017-01-19 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-5928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831135#comment-15831135
 ] 

Sangjin Lee commented on YARN-5928:
---

Thanks [~haibochen]!

Actually only the former argument needs to be updated. Thus, instead of
{noformat}
226 hadoop fs -put 
hadoop-yarn-server-timelineservice-hbase-3.0.0-alpha1-SNAPSHOT.jar
227
/hbase/coprocessor/hadoop-yarn-server-timelineservice-hbase.jar
{noformat}

it should be
{noformat}
226 hadoop fs -put 
hadoop-yarn-server-timelineservice-hbase-3.0.0-alpha1-SNAPSHOT.jar
227/hbase/coprocessor/hadoop-yarn-server-timelineservice.jar
{noformat}

If you don't object, I'll update that line and commit it once jenkins returns 
with a clean build.

> Move ATSv2 HBase backend code into a new module that is only dependent at 
> runtime by yarn servers
> -
>
> Key: YARN-5928
> URL: https://issues.apache.org/jira/browse/YARN-5928
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Affects Versions: 3.0.0-alpha1
>Reporter: Haibo Chen
>Assignee: Haibo Chen
> Attachments: YARN-5928.01.patch, YARN-5928.02.patch, 
> YARN-5928.06.patch, YARN-5928-YARN-5355.02.patch, 
> YARN-5928-YARN-5355.03.patch, YARN-5928-YARN-5355.04.patch, 
> YARN-5928-YARN-5355.04.patch, YARN-5928-YARN-5355.05.patch, 
> YARN-5928-YARN-5355.06.patch, YARN-5928-YARN-5355.07.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-5928) Move ATSv2 HBase backend code into a new module that is only dependent at runtime by yarn servers

2017-01-19 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-5928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830794#comment-15830794
 ] 

Sangjin Lee commented on YARN-5928:
---

[~haibochen], now that YARN-6094 has been committed, do you mind looking over 
the changes to see if anything needs to be updated (e.g. documentation)? I'm 
not sure if anything should change, but wanted to double check. Once we're 
clean, I can go ahead and commit it.

> Move ATSv2 HBase backend code into a new module that is only dependent at 
> runtime by yarn servers
> -
>
> Key: YARN-5928
> URL: https://issues.apache.org/jira/browse/YARN-5928
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Affects Versions: 3.0.0-alpha1
>Reporter: Haibo Chen
>Assignee: Haibo Chen
> Attachments: YARN-5928.01.patch, YARN-5928.02.patch, 
> YARN-5928.06.patch, YARN-5928-YARN-5355.02.patch, 
> YARN-5928-YARN-5355.03.patch, YARN-5928-YARN-5355.04.patch, 
> YARN-5928-YARN-5355.04.patch, YARN-5928-YARN-5355.05.patch, 
> YARN-5928-YARN-5355.06.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-6094) Update the coprocessor to be a dynamically loaded one

2017-01-19 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-6094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830782#comment-15830782
 ] 

Sangjin Lee commented on YARN-6094:
---

Sounds good. Committing shortly.

> Update the coprocessor to be a dynamically loaded one
> -
>
> Key: YARN-6094
> URL: https://issues.apache.org/jira/browse/YARN-6094
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Vrushali C
>Assignee: Vrushali C
>  Labels: yarn-5355-merge-blocker
> Attachments: YARN-6094.001.patch, YARN-6094-YARN-5355.001.patch, 
> YARN-6094-YARN-5355.002.patch, YARN-6094-YARN-5355.003.patch, 
> YARN-6094-YARN-5355.004.patch
>
>
> The timeline service v2 code base on yarn now uses hbase 1.2.4 after 
> YARN-5976. 
> With this version of hbase, system classes (starting with org.apache.hadoop) 
> can be loaded as table level coprocessors. Hence we should update the 
> timeline service coprocessor to be a dynamically loaded one instead of static 
> loading. 
> This involves code changes as well as documentation updates for deployment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-5304) Ship single node HBase config option with single startup command

2017-01-19 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-5304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830747#comment-15830747
 ] 

Sangjin Lee commented on YARN-5304:
---

>From the client perspective, we need to look at some timeout configuration 
>values. If I recall, the default values are too long in that a put may not 
>fail for 30 minutes or that long.

> Ship single node HBase config option with single startup command
> 
>
> Key: YARN-5304
> URL: https://issues.apache.org/jira/browse/YARN-5304
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Joep Rottinghuis
>Assignee: Vrushali C
>  Labels: YARN-5355, yarn-5355-merge-blocker
>
> For small to medium Hadoop deployments we should make it dead-simple to use 
> the timeline service v2. We should have a single command to launch and stop 
> the timelineservice back-end for the default HBase implementation.
> A default config with all the values should be packaged that launches all the 
> needed daemons (on the RM node) with a single command with all the 
> recommended settings.
> Having a timeline admin command, perhaps an init command might be needed, or 
> perhaps the timeline service can even auto-detect that and create tables, 
> deploy needed coprocessors etc.
> The overall purpose is to ensure nobody needs to be an HBase expert to get 
> this going. For those cluster operators with HBase experience, they can 
> choose their own more sophisticated deployment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-5304) Ship single node HBase config option with single startup command

2017-01-19 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-5304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830702#comment-15830702
 ] 

Sangjin Lee commented on YARN-5304:
---

We should review and set some of the recommended properties in the property xml 
file that should be created as part of this JIRA. For example, we should 
explicitly set {{hfile.format.version}} to 3 (see YARN-6094).

If it is not too much, it would be great if we can add all the necessary 
defaults as part of this JIRA. Let me know what you think.


> Ship single node HBase config option with single startup command
> 
>
> Key: YARN-5304
> URL: https://issues.apache.org/jira/browse/YARN-5304
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Joep Rottinghuis
>Assignee: Vrushali C
>  Labels: YARN-5355, yarn-5355-merge-blocker
>
> For small to medium Hadoop deployments we should make it dead-simple to use 
> the timeline service v2. We should have a single command to launch and stop 
> the timelineservice back-end for the default HBase implementation.
> A default config with all the values should be packaged that launches all the 
> needed daemons (on the RM node) with a single command with all the 
> recommended settings.
> Having a timeline admin command, perhaps an init command might be needed, or 
> perhaps the timeline service can even auto-detect that and create tables, 
> deploy needed coprocessors etc.
> The overall purpose is to ensure nobody needs to be an HBase expert to get 
> this going. For those cluster operators with HBase experience, they can 
> choose their own more sophisticated deployment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-6094) Update the coprocessor to be a dynamically loaded one

2017-01-19 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-6094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830692#comment-15830692
 ] 

Sangjin Lee commented on YARN-6094:
---

I see that hfile.format.version = 3 is actually being used in multiple places 
in the unit tests. But it also appears that the default value in 
{{hbase-default.xml}} is 3. It also states
{noformat}
  
  hfile.format.version
  3
  The HFile format version to use for new files.
  Version 3 adds support for tags in hfiles (See 
http://hbase.apache.org/book.html#hbase.tags).
  Distributed Log Replay requires that tags are enabled. Also see the 
configuration
  'hbase.replication.rpc.codec'.
  
  
{noformat}

It seems to me that the timeline service requires this to be 3 because we're 
relying on tags, right? Then should we add this requirement to the 
documentation? If needed, I suppose doing this as part of the documentation 
update JIRA is an option.

> Update the coprocessor to be a dynamically loaded one
> -
>
> Key: YARN-6094
> URL: https://issues.apache.org/jira/browse/YARN-6094
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Vrushali C
>Assignee: Vrushali C
>  Labels: yarn-5355-merge-blocker
> Attachments: YARN-6094.001.patch, YARN-6094-YARN-5355.001.patch, 
> YARN-6094-YARN-5355.002.patch, YARN-6094-YARN-5355.003.patch, 
> YARN-6094-YARN-5355.004.patch
>
>
> The timeline service v2 code base on yarn now uses hbase 1.2.4 after 
> YARN-5976. 
> With this version of hbase, system classes (starting with org.apache.hadoop) 
> can be loaded as table level coprocessors. Hence we should update the 
> timeline service coprocessor to be a dynamically loaded one instead of static 
> loading. 
> This involves code changes as well as documentation updates for deployment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-6094) Update the coprocessor to be a dynamically loaded one

2017-01-19 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-6094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830680#comment-15830680
 ] 

Sangjin Lee commented on YARN-6094:
---

I went over the latest patch, and noticed one thing (sorry should have seen 
them earlier).

In {{TestHBaseTimelineStorageSchema}} I see the following line has been added:
{code}
conf.setInt("hfile.format.version", 3);
{code}
What does this signify? Why is this change needed?


> Update the coprocessor to be a dynamically loaded one
> -
>
> Key: YARN-6094
> URL: https://issues.apache.org/jira/browse/YARN-6094
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Vrushali C
>Assignee: Vrushali C
>  Labels: yarn-5355-merge-blocker
> Attachments: YARN-6094.001.patch, YARN-6094-YARN-5355.001.patch, 
> YARN-6094-YARN-5355.002.patch, YARN-6094-YARN-5355.003.patch, 
> YARN-6094-YARN-5355.004.patch
>
>
> The timeline service v2 code base on yarn now uses hbase 1.2.4 after 
> YARN-5976. 
> With this version of hbase, system classes (starting with org.apache.hadoop) 
> can be loaded as table level coprocessors. Hence we should update the 
> timeline service coprocessor to be a dynamically loaded one instead of static 
> loading. 
> This involves code changes as well as documentation updates for deployment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-4985) Refactor the coprocessor code & other definition classes into independent packages

2017-01-19 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830428#comment-15830428
 ] 

Sangjin Lee commented on YARN-4985:
---

As part of this, it would also be good to see if it is feasible to get rid of 
the hbase-tests module and relocate the tests into the client and the server 
modules respectively. Once client and server are separated, the hbase-tests 
module becomes awkward.

Another part about the "server" module: since it should run in the hbase 
runtime, the pom should prescribe the hbase version and the appropriate hadoop 
version, rather than using the trunk hadoop version, for its dependencies. In 
other words, the pom for the server module should strongly resemble the current 
hbase-tests module.

The schema creator is an interesting case. I think a case can be made for it to 
be consider either "server" or "client". If it becomes too much work to locate 
it in the server module, I don't think it's a bad idea to keep it in the client 
module.

> Refactor the coprocessor code & other definition classes into independent 
> packages
> --
>
> Key: YARN-4985
> URL: https://issues.apache.org/jira/browse/YARN-4985
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Vrushali C
>Assignee: Vrushali C
>  Labels: YARN-5355
>
> As part of the coprocessor deployment, we have realized that it will be much 
> cleaner to have the coprocessor code sit in a package which does not depend 
> on hadoop-yarn-server classes. It only needs hbase and other util classes.
> These util classes and tag definition related classes can be refactored into 
> their own independent "definition" class package so that making changes to 
> coprocessor code, upgrading hbase, deploying hbase on a different hadoop 
> version cluster etc all becomes operationally much easier and less error 
> prone to having different library jars etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-5928) Move ATSv2 HBase backend code into a new module that is only dependent at runtime by yarn servers

2017-01-18 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-5928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15829155#comment-15829155
 ] 

Sangjin Lee commented on YARN-5928:
---

As mentioned in YARN-6064, the build on the branch is broken and that's causing 
the jenkins issues. I'll kick off the jenkins build again once that's fixed.

> Move ATSv2 HBase backend code into a new module that is only dependent at 
> runtime by yarn servers
> -
>
> Key: YARN-5928
> URL: https://issues.apache.org/jira/browse/YARN-5928
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Affects Versions: 3.0.0-alpha1
>Reporter: Haibo Chen
>Assignee: Haibo Chen
> Attachments: YARN-5928.01.patch, YARN-5928.02.patch, 
> YARN-5928.06.patch, YARN-5928-YARN-5355.02.patch, 
> YARN-5928-YARN-5355.03.patch, YARN-5928-YARN-5355.04.patch, 
> YARN-5928-YARN-5355.04.patch, YARN-5928-YARN-5355.05.patch, 
> YARN-5928-YARN-5355.06.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-6094) Update the coprocessor to be a dynamically loaded one

2017-01-18 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-6094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15829123#comment-15829123
 ] 

Sangjin Lee commented on YARN-6094:
---

The build failures are caused by an existing issue on the branch.

The latest patch looks good for the most part. A couple of minor things:

- We should add the new config property along with its default value in 
{{yarn-default.xml}}
- We should declare the default value in {{YarnConfiguration.java}} next to the 
key, instead of {{FlowRunTable.java}}

> Update the coprocessor to be a dynamically loaded one
> -
>
> Key: YARN-6094
> URL: https://issues.apache.org/jira/browse/YARN-6094
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Vrushali C
>Assignee: Vrushali C
>  Labels: yarn-5355-merge-blocker
> Attachments: YARN-6094.001.patch, YARN-6094-YARN-5355.001.patch, 
> YARN-6094-YARN-5355.002.patch, YARN-6094-YARN-5355.003.patch
>
>
> The timeline service v2 code base on yarn now uses hbase 1.2.4 after 
> YARN-5976. 
> With this version of hbase, system classes (starting with org.apache.hadoop) 
> can be loaded as table level coprocessors. Hence we should update the 
> timeline service coprocessor to be a dynamically loaded one instead of static 
> loading. 
> This involves code changes as well as documentation updates for deployment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-6064) Support fromId for flowRuns and flow/flowRun apps REST API's

2017-01-18 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-6064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15829088#comment-15829088
 ] 

Sangjin Lee commented on YARN-6064:
---

I think this commit broke the build on our branch. As part of YARN-5378, the 
{{AppToFlowRowKey}} constructor changed and so did the code in 
{{AbstractTimelineStorageReader}} that uses it. However, what's committed does 
not reflect the change.

[~varun_saxena], could you please check and do an addendum patch? Let me know 
if you need help.

Cc [~vrushalic], [~haibochen]: this explains the jenkins failures on your JIRAs.

> Support fromId for flowRuns and flow/flowRun apps REST API's
> 
>
> Key: YARN-6064
> URL: https://issues.apache.org/jira/browse/YARN-6064
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelinereader
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>  Labels: yarn-5355-merge-blocker
> Fix For: YARN-5355, YARN-5355-branch-2
>
> Attachments: YARN-6064-YARN-5355.0001.patch, 
> YARN-6064-YARN-5355.0002.patch, YARN-6064-YARN-5355.0003.patch, 
> YARN-6064-YARN-5355.0004.patch
>
>
> Splitting out JIRA YARN-6027 for pagination support for flowRuns, flow apps 
> and flow run apps. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-5928) Move ATSv2 HBase backend code into a new module that is only dependent at runtime by yarn servers

2017-01-18 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-5928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15829047#comment-15829047
 ] 

Sangjin Lee commented on YARN-5928:
---

That is failing too. This is the compilation error:
{noformat}
[INFO] -
[ERROR] COMPILATION ERROR : 
[INFO] -
[ERROR] 
/testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/reader/ApplicationEntityReader.java:[377,42]
 error: constructor AppToFlowRowKey in class AppToFlowRowKey cannot be applied 
to given types;
[ERROR]   required: String
  found: String,String
  reason: actual and formal argument lists differ in length
/testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/reader/ApplicationEntityReader.java:[380,12]
 error: method lookupFlowContext in class AbstractTimelineStorageReader cannot 
be applied to given types;
[INFO] 2 errors 
{noformat}
It is as if the latest version of {{AbstractTimelineStorageReader}} was 
compiled against an old version of {{AppToFlowRowKey}}. The funny thing is that 
both classes are in the same module!

> Move ATSv2 HBase backend code into a new module that is only dependent at 
> runtime by yarn servers
> -
>
> Key: YARN-5928
> URL: https://issues.apache.org/jira/browse/YARN-5928
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Affects Versions: 3.0.0-alpha1
>Reporter: Haibo Chen
>Assignee: Haibo Chen
> Attachments: YARN-5928.01.patch, YARN-5928.02.patch, 
> YARN-5928.06.patch, YARN-5928-YARN-5355.02.patch, 
> YARN-5928-YARN-5355.03.patch, YARN-5928-YARN-5355.04.patch, 
> YARN-5928-YARN-5355.04.patch, YARN-5928-YARN-5355.05.patch, 
> YARN-5928-YARN-5355.06.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



  1   2   3   4   5   6   7   8   9   10   >