[jira] [Commented] (YARN-8411) Enable stopped system services to be started during RM start
[ https://issues.apache.org/jira/browse/YARN-8411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16512976#comment-16512976 ] Billie Rinaldi commented on YARN-8411: -- Thanks, [~eyang]! > Enable stopped system services to be started during RM start > > > Key: YARN-8411 > URL: https://issues.apache.org/jira/browse/YARN-8411 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Billie Rinaldi >Assignee: Billie Rinaldi >Priority: Critical > Fix For: 3.2.0, 3.1.1 > > Attachments: YARN-8411.01.patch, YARN-8411.02.patch > > > With YARN-8048, the RM can launch system services using the YARN service > framework. If the service app is in a stopped state, user intervention is > required to delete the service so that the RM can launch it again when the RM > is restarted. It would be an improvement for the RM to be able to start a > stopped system service. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8411) Enable stopped system services to be started during RM start
[ https://issues.apache.org/jira/browse/YARN-8411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16511799#comment-16511799 ] Hudson commented on YARN-8411: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14423 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14423/]) YARN-8411. Restart stopped system service during RM start. (eyang: rev 69b05968974994c6e22d6562a67b9392d1700094) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-api/src/main/java/org/apache/hadoop/yarn/service/client/SystemServiceManagerImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-api/src/main/java/org/apache/hadoop/yarn/service/webapp/ApiServer.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-api/src/test/java/org/apache/hadoop/yarn/service/ServiceClientTest.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-api/src/test/java/org/apache/hadoop/yarn/service/client/TestSystemServiceManagerImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/client/ServiceClient.java > Enable stopped system services to be started during RM start > > > Key: YARN-8411 > URL: https://issues.apache.org/jira/browse/YARN-8411 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Billie Rinaldi >Assignee: Billie Rinaldi >Priority: Critical > Fix For: 3.2.0, 3.1.1 > > Attachments: YARN-8411.01.patch, YARN-8411.02.patch > > > With YARN-8048, the RM can launch system services using the YARN service > framework. If the service app is in a stopped state, user intervention is > required to delete the service so that the RM can launch it again when the RM > is restarted. It would be an improvement for the RM to be able to start a > stopped system service. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8411) Enable stopped system services to be started during RM start
[ https://issues.apache.org/jira/browse/YARN-8411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16511170#comment-16511170 ] Billie Rinaldi commented on YARN-8411: -- Thanks for taking a look, [~rohithsharma]! Yes, start should have the same effect as a launch operation. If the service is already running, it will get an error. If it's not running, it will start running. Specifically, in ServiceClient#verifyNoLiveAppInRM (used by both start and launch operations), it uses YarnClient to find out whether there are any live yarn-service applications tagged with the specified service name, and if there are it throws a YarnException. > Enable stopped system services to be started during RM start > > > Key: YARN-8411 > URL: https://issues.apache.org/jira/browse/YARN-8411 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Billie Rinaldi >Assignee: Billie Rinaldi >Priority: Critical > Attachments: YARN-8411.01.patch, YARN-8411.02.patch > > > With YARN-8048, the RM can launch system services using the YARN service > framework. If the service app is in a stopped state, user intervention is > required to delete the service so that the RM can launch it again when the RM > is restarted. It would be an improvement for the RM to be able to start a > stopped system service. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8411) Enable stopped system services to be started during RM start
[ https://issues.apache.org/jira/browse/YARN-8411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16510593#comment-16510593 ] Rohith Sharma K S commented on YARN-8411: - thanks [~billie.rinaldi] for the patch. It looks good to me as well. I have one generic doubt on start operations, does service start identifies if service is already running? Because in system services services are long running and in case of RM restart or HA switch, by default we are going to start services. > Enable stopped system services to be started during RM start > > > Key: YARN-8411 > URL: https://issues.apache.org/jira/browse/YARN-8411 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Billie Rinaldi >Assignee: Billie Rinaldi >Priority: Critical > Attachments: YARN-8411.01.patch, YARN-8411.02.patch > > > With YARN-8048, the RM can launch system services using the YARN service > framework. If the service app is in a stopped state, user intervention is > required to delete the service so that the RM can launch it again when the RM > is restarted. It would be an improvement for the RM to be able to start a > stopped system service. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8411) Enable stopped system services to be started during RM start
[ https://issues.apache.org/jira/browse/YARN-8411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16510497#comment-16510497 ] Eric Yang commented on YARN-8411: - +1 looks good to me. I will commit patch 002 tomorrow if no objections. > Enable stopped system services to be started during RM start > > > Key: YARN-8411 > URL: https://issues.apache.org/jira/browse/YARN-8411 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Billie Rinaldi >Assignee: Billie Rinaldi >Priority: Critical > Attachments: YARN-8411.01.patch, YARN-8411.02.patch > > > With YARN-8048, the RM can launch system services using the YARN service > framework. If the service app is in a stopped state, user intervention is > required to delete the service so that the RM can launch it again when the RM > is restarted. It would be an improvement for the RM to be able to start a > stopped system service. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8411) Enable stopped system services to be started during RM start
[
https://issues.apache.org/jira/browse/YARN-8411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16510436#comment-16510436
]
genericqa commented on YARN-8411:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
35s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 12s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 18s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m
55s{color} | {color:green} hadoop-yarn-services-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
33s{color} | {color:green} hadoop-yarn-services-api in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 73m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd |
| JIRA Issue | YARN-8411 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12927549/YARN-8411.02.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 5c4cbc57a9eb 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / aeaf9fe |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/21019/testReport/ |
| Max. process+thread count | 758 (vs. ulimit of 1
[jira] [Commented] (YARN-8411) Enable stopped system services to be started during RM start
[ https://issues.apache.org/jira/browse/YARN-8411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16510323#comment-16510323 ] Billie Rinaldi commented on YARN-8411: -- Thanks for the review, [~eyang]! The new patch attempts to distinguish an exception due to the service already existing, and in that case logs an informative message. If it receives a different type of exception, it will log it with the "Got exception" message and will not go on to try to start the service. > Enable stopped system services to be started during RM start > > > Key: YARN-8411 > URL: https://issues.apache.org/jira/browse/YARN-8411 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Billie Rinaldi >Assignee: Billie Rinaldi >Priority: Critical > Attachments: YARN-8411.01.patch, YARN-8411.02.patch > > > With YARN-8048, the RM can launch system services using the YARN service > framework. If the service app is in a stopped state, user intervention is > required to delete the service so that the RM can launch it again when the RM > is restarted. It would be an improvement for the RM to be able to start a > stopped system service. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8411) Enable stopped system services to be started during RM start
[
https://issues.apache.org/jira/browse/YARN-8411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16510177#comment-16510177
]
Eric Yang commented on YARN-8411:
-
[~billie.rinaldi] Thank you for the patch, the approach looks good. One minor
nitpick:
{code}
try {
serviceClient.actionBuild(service);
} catch (Exception e) {
LOG.info("Got exception saving {}, will attempt to start " +
"service", service.getName());
}
ApplicationId applicationId = serviceClient
.actionStartAndGetId(service.getName());
return applicationId;
{code}
The information logging looks like an error. It will be more informative, to
catch the already exist exception, and change the message to:
{code}
"Service {} already exist, will attempt to start service"
{code}
> Enable stopped system services to be started during RM start
>
>
> Key: YARN-8411
> URL: https://issues.apache.org/jira/browse/YARN-8411
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Billie Rinaldi
>Assignee: Billie Rinaldi
>Priority: Critical
> Attachments: YARN-8411.01.patch
>
>
> With YARN-8048, the RM can launch system services using the YARN service
> framework. If the service app is in a stopped state, user intervention is
> required to delete the service so that the RM can launch it again when the RM
> is restarted. It would be an improvement for the RM to be able to start a
> stopped system service.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8411) Enable stopped system services to be started during RM start
[
https://issues.apache.org/jira/browse/YARN-8411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16509131#comment-16509131
]
genericqa commented on YARN-8411:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
32s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 0s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 22s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m
37s{color} | {color:green} hadoop-yarn-services-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
30s{color} | {color:green} hadoop-yarn-services-api in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 71m 44s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd |
| JIRA Issue | YARN-8411 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12927422/YARN-8411.01.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 392bd258ce79 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 2b2f672 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/21008/testReport/ |
| Max. process+thread count | 758 (vs. ulimit of 1
[jira] [Commented] (YARN-8411) Enable stopped system services to be started during RM start
[ https://issues.apache.org/jira/browse/YARN-8411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16508503#comment-16508503 ] Billie Rinaldi commented on YARN-8411: -- One way to implement this would be to have the RM perform two operations, a save followed by a start. In the service framework, save plus start is the same as a launch operation. Then if the save failed because the service already existed, the RM could go on to perform the start operation. > Enable stopped system services to be started during RM start > > > Key: YARN-8411 > URL: https://issues.apache.org/jira/browse/YARN-8411 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Billie Rinaldi >Priority: Major > Fix For: 3.2.0 > > > With YARN-8048, the RM can launch system services using the YARN service > framework. If the service app is in a stopped state, user intervention is > required to delete the service so that the RM can launch it again when the RM > is restarted. It would be an improvement for the RM to be able to start a > stopped system service. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
