[jira] [Commented] (YARN-1564) add some basic workflow YARN services

2016-01-08 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-1564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15089711#comment-15089711
 ] 

Hadoop QA commented on YARN-1564:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 14 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 
51s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 45s 
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 37s 
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 5s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
48s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 52s 
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 4s 
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 43s 
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 43s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 44s 
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 22m 8s {color} 
| {color:red} root-jdk1.7.0_91 with JDK v1.7.0_91 generated 4 new issues (was 
724, now 724). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m 44s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 16s 
{color} | {color:red} Patch generated 118 new checkstyle issues in 
hadoop-common-project/hadoop-common (total was 9, now 126). {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 2s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
14s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s 
{color} | {color:red} The patch has 35 line(s) that end in whitespace. Use git 
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s 
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 1s 
{color} | {color:red} hadoop-common-project/hadoop-common introduced 1 new 
FindBugs issues. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 3m 12s 
{color} | {color:red} hadoop-common-project_hadoop-common-jdk1.8.0_66 with JDK 
v1.8.0_66 generated 1 new issues (was 1, now 2). {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 52s 
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 4m 38s 
{color} | {color:red} hadoop-common-project_hadoop-common-jdk1.7.0_91 with JDK 
v1.7.0_91 generated 1 new issues (was 13, now 14). {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s 
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 38s 
{color} | {color:green} hadoop-common in the patch passed with JDK v1.8.0_66. 
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 37s {color} 
| {color:red} hadoop-common in the patch failed with JDK v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green}

[jira] [Commented] (YARN-1564) add some basic workflow YARN services

2016-01-08 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-1564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15089577#comment-15089577
 ] 

ASF GitHub Bot commented on YARN-1564:
--

GitHub user steveloughran opened a pull request:

https://github.com/apache/hadoop/pull/65

YARN-1564 add some basic workflow YARN services

YARN-1564 add some basic workflow YARN services

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/steveloughran/hadoop 
stevel/YARN-1564-workflow-services

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hadoop/pull/65.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #65


commit d9f0172abfd904bad80e9e245fc792560a874834
Author: Steve Loughran 
Date:   2014-06-03T15:46:54Z

YARN-1564 - Patch-001

commit b0101e65cb1e6ed4171212f78c6d75ac855ccb5c
Author: Steve Loughran 
Date:   2014-12-04T15:30:52Z

YARN-1564 sync up with slider tweaks, primarily the disabling of tests on 
windows which only work if the relevant external commands are on the path

commit 8f262578c67edc4e6cbdef3e9bc338dabcd2ecf4
Author: Steve Loughran 
Date:   2015-05-05T12:56:14Z

YARN-1564 review and update workflow services, including making original 
CompositeService a ServiceParent; moving to Java 7 and SLF4J everywhere

commit 38df053d60467b587c0c7ec03f3368a84bf1cfea
Author: Steve Loughran 
Date:   2015-05-06T14:06:43Z

YARN-1564 -pick up enhancements/fixes from Slider-0.70-incubating versions 
of these classes

commit bf608f92a1c791c28fabf84dc2cca97daf35e1cf
Author: Steve Loughran 
Date:   2016-01-08T17:35:52Z

YARN-1564 turn off the findbugs warnings that are very much wrong




> add some basic workflow YARN services
> -
>
> Key: YARN-1564
> URL: https://issues.apache.org/jira/browse/YARN-1564
> Project: Hadoop YARN
>  Issue Type: New Feature
>  Components: api
>Affects Versions: 2.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: YARN-1564-001.patch, YARN-1564-002.patch
>
>   Original Estimate: 24h
>  Time Spent: 48h
>  Remaining Estimate: 0h
>
> I've been using some alternative composite services to help build workflows 
> of process execution in a YARN AM.
> They and their tests could be moved in YARN for the use by others -this would 
> make it easier to build aggregate services in an AM



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-1564) add some basic workflow YARN services

2015-05-07 Thread Steve Loughran (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-1564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14532354#comment-14532354
 ] 

Steve Loughran commented on YARN-1564:
--

Sync on AtomicBool warnings are spurious; code is simply using the (final) 
objects as something to wait/notify off to indicate status updates within the 
atomic values themselves. Nothing wrong with that —avoids having a separate 
object, and makes it obvious what things are waiting on. Presumably the check 
is there for people who don't know what they are doing.

 add some basic workflow YARN services
 -

 Key: YARN-1564
 URL: https://issues.apache.org/jira/browse/YARN-1564
 Project: Hadoop YARN
  Issue Type: New Feature
  Components: api
Affects Versions: 2.4.0
Reporter: Steve Loughran
Assignee: Steve Loughran
Priority: Minor
 Attachments: YARN-1564-001.patch, YARN-1564-002.patch

   Original Estimate: 24h
  Time Spent: 48h
  Remaining Estimate: 0h

 I've been using some alternative composite services to help build workflows 
 of process execution in a YARN AM.
 They and their tests could be moved in YARN for the use by others -this would 
 make it easier to build aggregate services in an AM



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-1564) add some basic workflow YARN services

2015-05-06 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-1564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14530806#comment-14530806
 ] 

Hadoop QA commented on YARN-1564:
-

\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch |  14m 38s | Pre-patch trunk compilation is 
healthy. |
| {color:green}+1{color} | @author |   0m  0s | The patch does not contain any 
@author tags. |
| {color:green}+1{color} | tests included |   0m  0s | The patch appears to 
include 14 new or modified test files. |
| {color:green}+1{color} | javac |   7m 33s | There were no new javac warning 
messages. |
| {color:red}-1{color} | javadoc |   9m 36s | The applied patch generated  1  
additional warning messages. |
| {color:green}+1{color} | release audit |   0m 22s | The applied patch does 
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle |   1m  5s | The applied patch generated  
116 new checkstyle issues (total was 9, now 125). |
| {color:green}+1{color} | whitespace |   0m  1s | The patch has no lines that 
end in whitespace. |
| {color:green}+1{color} | install |   1m 34s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse |   0m 32s | The patch built with 
eclipse:eclipse. |
| {color:red}-1{color} | findbugs |   1m 45s | The patch appears to introduce 5 
new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | common tests |  24m 46s | Tests passed in 
hadoop-common. |
| | |  62m  1s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-common |
|  |  Inconsistent synchronization of 
org.apache.hadoop.service.workflow.LongLivedProcess.recentLineLimit; locked 50% 
of time  Unsynchronized access at LongLivedProcess.java:50% of time  
Unsynchronized access at LongLivedProcess.java:[line 138] |
|  |  Found reliance on default encoding in 
org.apache.hadoop.service.workflow.LongLivedProcess$ProcessStreamReader.run():in
 org.apache.hadoop.service.workflow.LongLivedProcess$ProcessStreamReader.run(): 
new java.io.InputStreamReader(InputStream)  At LongLivedProcess.java:[line 521] 
|
|  |  Exception is caught when Exception is not thrown in 
org.apache.hadoop.service.workflow.LongLivedProcess$ProcessStreamReader.run()  
At LongLivedProcess.java:is not thrown in 
org.apache.hadoop.service.workflow.LongLivedProcess$ProcessStreamReader.run()  
At LongLivedProcess.java:[line 557] |
|  |  Synchronization performed on java.util.concurrent.atomic.AtomicBoolean in 
org.apache.hadoop.service.workflow.WorkflowForkedProcessService.completed(int)  
At 
WorkflowForkedProcessService.java:org.apache.hadoop.service.workflow.WorkflowForkedProcessService.completed(int)
  At WorkflowForkedProcessService.java:[line 265] |
|  |  Synchronization performed on java.util.concurrent.atomic.AtomicBoolean in 
org.apache.hadoop.service.workflow.WorkflowForkedProcessService.run()  At 
WorkflowForkedProcessService.java:org.apache.hadoop.service.workflow.WorkflowForkedProcessService.run()
  At WorkflowForkedProcessService.java:[line 231] |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL | 
http://issues.apache.org/jira/secure/attachment/12730854/YARN-1564-002.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / a583a40 |
| javadoc | 
https://builds.apache.org/job/PreCommit-YARN-Build/7736/artifact/patchprocess/diffJavadocWarnings.txt
 |
| checkstyle |  
https://builds.apache.org/job/PreCommit-YARN-Build/7736/artifact/patchprocess/diffcheckstylehadoop-common.txt
 |
| Findbugs warnings | 
https://builds.apache.org/job/PreCommit-YARN-Build/7736/artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html
 |
| hadoop-common test log | 
https://builds.apache.org/job/PreCommit-YARN-Build/7736/artifact/patchprocess/testrun_hadoop-common.txt
 |
| Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/7736/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP 
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/7736/console |


This message was automatically generated.

 add some basic workflow YARN services
 -

 Key: YARN-1564
 URL: https://issues.apache.org/jira/browse/YARN-1564
 Project: Hadoop YARN
  Issue Type: New Feature
  Components: api
Affects Versions: 2.4.0
Reporter: Steve Loughran
Assignee: Steve Loughran
Priority: Minor
 Attachments: YARN-1564-001.patch, YARN-1564-002.patch

   Original Estimate: 24h
  Time Spent: 48h
  Remaining Estimate: 0h

 I've been using some alternative composite services to help build workflows 
 of process execution in a YARN AM.
 They and their tests could be

[jira] [Commented] (YARN-1564) add some basic workflow YARN services

2015-05-04 Thread Zhijie Shen (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-1564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14526963#comment-14526963
 ] 

Zhijie Shen commented on YARN-1564:
---

YARN-2928 is going to support flow as the first class citizen.It will be great 
if we can coordinate on this between app management and monitoring.

 add some basic workflow YARN services
 -

 Key: YARN-1564
 URL: https://issues.apache.org/jira/browse/YARN-1564
 Project: Hadoop YARN
  Issue Type: New Feature
  Components: api
Affects Versions: 2.4.0
Reporter: Steve Loughran
Assignee: Steve Loughran
Priority: Minor
 Attachments: YARN-1564-001.patch

   Original Estimate: 24h
  Time Spent: 48h
  Remaining Estimate: 0h

 I've been using some alternative composite services to help build workflows 
 of process execution in a YARN AM.
 They and their tests could be moved in YARN for the use by others -this would 
 make it easier to build aggregate services in an AM



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-1564) add some basic workflow YARN services

2015-05-04 Thread Steve Loughran (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-1564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14527257#comment-14527257
 ] 

Steve Loughran commented on YARN-1564:
--

They're mostly unrelated; here are some composite services that extend the 
single CompositeService with
# ability of other bits of code to add children
# policy that if one child stops/fails then the service itself does
# sequential container that starts one after the other
# one that execs things

It's things we use in Slider to actually construct the internals of our app; AM 
comes up  creates a composite to hold all its children, can exec things 
locally/async to do some work. And we subclass some for other features; the app 
itself becomes an aggregation of services with a unified lifecycle. (e.g IPC 
service, Web yarn service, Metrics  Monitoring, TimelineClient)

Where ATS integration would benefit is in
(a) potentially having a service listen to lifecycle events and automatically 
publish them
(b) the YARN-679 entry point, which will provide a single, robust, entry point 
to instantiate anything that implements Service, and, if they add the entry 
point, better passdown of commands


 add some basic workflow YARN services
 -

 Key: YARN-1564
 URL: https://issues.apache.org/jira/browse/YARN-1564
 Project: Hadoop YARN
  Issue Type: New Feature
  Components: api
Affects Versions: 2.4.0
Reporter: Steve Loughran
Assignee: Steve Loughran
Priority: Minor
 Attachments: YARN-1564-001.patch

   Original Estimate: 24h
  Time Spent: 48h
  Remaining Estimate: 0h

 I've been using some alternative composite services to help build workflows 
 of process execution in a YARN AM.
 They and their tests could be moved in YARN for the use by others -this would 
 make it easier to build aggregate services in an AM



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-1564) add some basic workflow YARN services

2015-05-02 Thread Steve Loughran (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-1564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14525243#comment-14525243
 ] 

Steve Loughran commented on YARN-1564:
--

I should look at the tests for the execution service; we have a fork of these 
in slider and they were failing on windows unless the installation had the 
right path setup with all the cygwin binaries (ls and the like).

 add some basic workflow YARN services
 -

 Key: YARN-1564
 URL: https://issues.apache.org/jira/browse/YARN-1564
 Project: Hadoop YARN
  Issue Type: New Feature
  Components: api
Affects Versions: 2.4.0
Reporter: Steve Loughran
Assignee: Steve Loughran
Priority: Minor
 Attachments: YARN-1564-001.patch

   Original Estimate: 24h
  Time Spent: 48h
  Remaining Estimate: 0h

 I've been using some alternative composite services to help build workflows 
 of process execution in a YARN AM.
 They and their tests could be moved in YARN for the use by others -this would 
 make it easier to build aggregate services in an AM



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-1564) add some basic workflow YARN services

2015-05-01 Thread Jian He (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-1564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14524347#comment-14524347
 ] 

Jian He commented on YARN-1564:
---

I can review without rebasing since mostly are new code, don't spend effort on 
rebasing it. 

 add some basic workflow YARN services
 -

 Key: YARN-1564
 URL: https://issues.apache.org/jira/browse/YARN-1564
 Project: Hadoop YARN
  Issue Type: New Feature
  Components: api
Affects Versions: 2.4.0
Reporter: Steve Loughran
Assignee: Steve Loughran
Priority: Minor
 Attachments: YARN-1564-001.patch

   Original Estimate: 24h
  Time Spent: 48h
  Remaining Estimate: 0h

 I've been using some alternative composite services to help build workflows 
 of process execution in a YARN AM.
 They and their tests could be moved in YARN for the use by others -this would 
 make it easier to build aggregate services in an AM



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-1564) add some basic workflow YARN services

2015-05-01 Thread Jian He (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-1564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523846#comment-14523846
 ] 

Jian He commented on YARN-1564:
---

[~ste...@apache.org], do we still want this ? seems most of the code is in 
common, we may move this to common ?

 add some basic workflow YARN services
 -

 Key: YARN-1564
 URL: https://issues.apache.org/jira/browse/YARN-1564
 Project: Hadoop YARN
  Issue Type: New Feature
  Components: api
Affects Versions: 2.4.0
Reporter: Steve Loughran
Assignee: Steve Loughran
Priority: Minor
 Attachments: YARN-1564-001.patch

   Original Estimate: 24h
  Time Spent: 48h
  Remaining Estimate: 0h

 I've been using some alternative composite services to help build workflows 
 of process execution in a YARN AM.
 They and their tests could be moved in YARN for the use by others -this would 
 make it easier to build aggregate services in an AM



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-1564) add some basic workflow YARN services

2015-05-01 Thread Steve Loughran (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-1564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523978#comment-14523978
 ] 

Steve Loughran commented on YARN-1564:
--

# we do still need this
# the code is in common as that is where the core AbstractService code lives
# the patches aren't in sync with trunk -nobody was reviewing them.

If you can promise to review them this month I will update the patch. If 
however, nobody in the YARN team is willing to review them, then I'll move this 
to common and see if I can find someone who is prepared to do the viewing

 add some basic workflow YARN services
 -

 Key: YARN-1564
 URL: https://issues.apache.org/jira/browse/YARN-1564
 Project: Hadoop YARN
  Issue Type: New Feature
  Components: api
Affects Versions: 2.4.0
Reporter: Steve Loughran
Assignee: Steve Loughran
Priority: Minor
 Attachments: YARN-1564-001.patch

   Original Estimate: 24h
  Time Spent: 48h
  Remaining Estimate: 0h

 I've been using some alternative composite services to help build workflows 
 of process execution in a YARN AM.
 They and their tests could be moved in YARN for the use by others -this would 
 make it easier to build aggregate services in an AM



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-1564) add some basic workflow YARN services

2014-06-16 Thread Tsuyoshi OZAWA (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-1564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14032234#comment-14032234
 ] 

Tsuyoshi OZAWA commented on YARN-1564:
--

Resubmitted for kicking Jenkins CI.

 add some basic workflow YARN services
 -

 Key: YARN-1564
 URL: https://issues.apache.org/jira/browse/YARN-1564
 Project: Hadoop YARN
  Issue Type: New Feature
  Components: api
Affects Versions: 2.4.0
Reporter: Steve Loughran
Assignee: Steve Loughran
Priority: Minor
 Attachments: YARN-1564-001.patch

   Original Estimate: 24h
  Time Spent: 48h
  Remaining Estimate: 0h

 I've been using some alternative composite services to help build workflows 
 of process execution in a YARN AM.
 They and their tests could be moved in YARN for the use by others -this would 
 make it easier to build aggregate services in an AM



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (YARN-1564) add some basic workflow YARN services

2014-01-06 Thread Steve Loughran (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-1564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13862988#comment-13862988
 ] 

Steve Loughran commented on YARN-1564:
--

Services are all in 
[[https://github.com/hortonworks/hoya/tree/develop/hoya-core/src/main/java/org/apache/hadoop/hoya/yarn/service]]

* 
[[Parent|https://github.com/hortonworks/hoya/blob/develop/hoya-core/src/main/java/org/apache/hadoop/hoya/yarn/service/Parent.java]]:
 interface that makes {{addService}} public and adds an option to list 
services. This could be retrofitted to {{CompositeService}}
* 
[[CompoundService|https://github.com/hortonworks/hoya/blob/develop/hoya-core/src/main/java/org/apache/hadoop/hoya/yarn/service/CompoundService.java]
 : subclass of {{CompositeService}} that finishes when all of its children have 
*successfully* completed, or when any of its children fail.
* 
[[SequenceService|https://github.com/hortonworks/hoya/blob/develop/hoya-core/src/main/java/org/apache/hadoop/hoya/yarn/service/SequenceService.java]]
  subclass of {{CompositeService}} that executes its children in sequence, 
starting one when the previous one successfully finishes. Again, failures are 
propagated up immediately.
* 
[[EventNotifyingService|https://github.com/hortonworks/hoya/blob/develop/hoya-core/src/main/java/org/apache/hadoop/hoya/yarn/service/EventNotifyingService.java]]
 service which triggers a callback to a supplied interface implementation when 
started -or after a specified delay from the start time.
** 
[[ForkedProcessService.java|https://github.com/hortonworks/hoya/blob/develop/hoya-core/src/main/java/org/apache/hadoop/hoya/yarn/service/ForkedProcessService.java]]:
 more complex, this forks a potentially long lived application (via 
[[RunLongLivedApp|https://github.com/hortonworks/hoya/blob/develop/hoya-core/src/main/java/org/apache/hadoop/hoya/exec/RunLongLivedApp.java]]),
 completing the service when that process finishes.

The set allows you build up sequences of operations, as well as actions to run 
in parallel, and to notify a parent service or other object when they complete 
or fail.

There's tests for the simple ones (Sequence, compound, events) in groovy; 
nothing yet for forked processes.


 add some basic workflow YARN services
 -

 Key: YARN-1564
 URL: https://issues.apache.org/jira/browse/YARN-1564
 Project: Hadoop YARN
  Issue Type: New Feature
  Components: api
Affects Versions: 2.2.0
Reporter: Steve Loughran
Assignee: Steve Loughran
Priority: Minor
   Original Estimate: 24h
  Remaining Estimate: 24h

 I've been using some alternative composite services to help build workflows 
 of process execution in a YARN AM.
 They and their tests could be moved in YARN for the use by others -this would 
 make it easier to build aggregate services in an AM



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)