[jira] [Commented] (YARN-8636) TimelineSchemaCreator command not working
[
https://issues.apache.org/jira/browse/YARN-8636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574318#comment-16574318
]
Rohith Sharma K S commented on YARN-8636:
-
There is discrepancy in the documentation which need to be updated post
YARN-7190.
> TimelineSchemaCreator command not working
> -
>
> Key: YARN-8636
> URL: https://issues.apache.org/jira/browse/YARN-8636
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Abhishek Modi
>Priority: Minor
>
> {code:java}
> nodemanager/bin> ./hadoop
> org.apache.yarn.timelineservice.storage.TimelineSchemaCreator -create
> Error: Could not find or load main class
> org.apache.yarn.timelineservice.storage.TimelineSchemaCreator
> {code}
> share/hadoop/yarn/timelineservice/ is not part of class path
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8559) Expose mutable-conf scheduler's configuration in RM /scheduler-conf endpoint
[ https://issues.apache.org/jira/browse/YARN-8559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574238#comment-16574238 ] Weiwei Yang commented on YARN-8559: --- Thanks [~leftnoteasy], [~jhung], [~suma.shivaprasad], [~sunilg] to help review the patch. Let me commit this shortly, thanks. > Expose mutable-conf scheduler's configuration in RM /scheduler-conf endpoint > > > Key: YARN-8559 > URL: https://issues.apache.org/jira/browse/YARN-8559 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Reporter: Anna Savarin >Assignee: Weiwei Yang >Priority: Major > Attachments: YARN-8559.001.patch, YARN-8559.002.patch, > YARN-8559.003.patch, YARN-8559.004.patch > > > All Hadoop services provide a set of common endpoints (/stacks, /logLevel, > /metrics, /jmx, /conf). In the case of the Resource Manager, part of the > configuration comes from the scheduler being used. Currently, these > configuration key/values are not exposed through the /conf endpoint, thereby > revealing an incomplete configuration picture. > Make an improvement and expose the scheduling configuration info through the > RM's /conf endpoint. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-8638) Allow linux container runtimes to be pluggable
[
https://issues.apache.org/jira/browse/YARN-8638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Craig Condit reassigned YARN-8638:
--
Assignee: Craig Condit
> Allow linux container runtimes to be pluggable
> --
>
> Key: YARN-8638
> URL: https://issues.apache.org/jira/browse/YARN-8638
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
>Affects Versions: 3.2.0
>Reporter: Craig Condit
>Assignee: Craig Condit
>Priority: Minor
>
> YARN currently supports three different Linux container runtimes (default,
> docker, and javasandbox). However, it would be relatively straightforward to
> support arbitrary runtime implementations. This would enable easier
> experimentation with new and emerging runtime technologies (runc, containerd,
> etc.) without requiring a rebuild and redeployment of Hadoop.
> This could be accomplished via a simple configuration change:
> {code:xml}
>
> yarn.nodemanager.runtime.linux.allowed-runtimes
> default,docker,experimental
>
>
>
> yarn.nodemanager.runtime.linux.experimental.class
> com.somecompany.yarn.runtime.ExperimentalLinuxContainerRuntime
> {code}
>
> In this example, {{yarn.nodemanager.runtime.linux.allowed-runtimes}} would
> now allow arbitrary values. Additionally,
> {{yarn.nodemanager.runtime.linux.\{RUNTIME_KEY}.class}} would indicate the
> {{LinuxContainerRuntime}} implementation to instantiate. A no-argument
> constructor should be sufficient, as {{LinuxContainerRuntime}} already
> provides an {{initialize()}} method.
> {{DockerLinuxContainerRuntime.isDockerContainerRequested(Map
> env)}} and {{JavaSandboxLinuxContainerRuntime.isSandboxContainerRequested()}}
> could be generalized to {{isRuntimeRequested(Map env)}} and
> added to the {{LinuxContainerRuntime}} interface. This would allow
> {{DelegatingLinuxContainerRuntime}} to select an appropriate runtime based on
> whether that runtime claimed ownership of the current container execution.
> For backwards compatibility, the existing values (default,docker,javasandbox)
> would continue to be supported as-is. Under the current logic, the evaluation
> order is javasandbox, docker, default (with default being chosen if no other
> candidates are available). Under the new evaluation logic, pluggable runtimes
> would be evaluated after docker and before default, in the order in which
> they are defined in the allowed-runtimes list. This will change no behavior
> on current clusters (as there would be no pluggable runtimes defined), and
> preserves behavior with respect to ordering of existing runtimes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8331) Race condition in NM container launched after done
[
https://issues.apache.org/jira/browse/YARN-8331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574190#comment-16574190
]
Pradeep Ambati commented on YARN-8331:
--
Thanks for reviewing the patch, [~jlowe]!
I have addressed the issues you pointed in the new patch.
> Race condition in NM container launched after done
> --
>
> Key: YARN-8331
> URL: https://issues.apache.org/jira/browse/YARN-8331
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.1.0, 2.9.1, 3.0.2
>Reporter: Yang Wang
>Assignee: Pradeep Ambati
>Priority: Major
> Attachments: YARN-8331.001.patch, YARN-8331.002.patch
>
>
> When a container is launching, in ContainerLaunch#launchContainer, state is
> SCHEDULED,
> kill event was sent to this container, state : SCHEDULED->KILLING->DONE
> Then ContainerLaunch send CONTAINER_LAUNCHED event and start the container
> processes. These absent container processes will not be cleaned up anymore.
>
> {code:java}
> 2018-05-21 13:11:56,114 INFO [Thread-11] nodemanager.NMAuditLogger
> (NMAuditLogger.java:logSuccess(94)) - USER=nobody OPERATION=Start Container
> Request TARGET=ContainerManageImpl RESULT=SUCCESS
> APPID=application_0_CONTAINERID=container_0__01_00
> 2018-05-21 13:11:56,114 INFO [NM ContainerManager dispatcher]
> application.ApplicationImpl (ApplicationImpl.java:handle(632)) - Application
> application_0_ transitioned from NEW to INITING
> 2018-05-21 13:11:56,114 INFO [NM ContainerManager dispatcher]
> application.ApplicationImpl (ApplicationImpl.java:transition(446)) - Adding
> container_0__01_00 to application application_0_
> 2018-05-21 13:11:56,118 INFO [NM ContainerManager dispatcher]
> application.ApplicationImpl (ApplicationImpl.java:handle(632)) - Application
> application_0_ transitioned from INITING to RUNNING
> 2018-05-21 13:11:56,119 INFO [NM ContainerManager dispatcher]
> container.ContainerImpl (ContainerImpl.java:handle(2111)) - Container
> container_0__01_00 transitioned from NEW to SCHEDULED
> 2018-05-21 13:11:56,119 INFO [NM ContainerManager dispatcher]
> containermanager.AuxServices (AuxServices.java:handle(220)) - Got event
> CONTAINER_INIT for appId application_0_
> 2018-05-21 13:11:56,119 INFO [NM ContainerManager dispatcher]
> scheduler.ContainerScheduler (ContainerScheduler.java:startContainer(504)) -
> Starting container [container_0__01_00]
> 2018-05-21 13:11:56,226 INFO [NM ContainerManager dispatcher]
> container.ContainerImpl (ContainerImpl.java:handle(2111)) - Container
> container_0__01_00 transitioned from SCHEDULED to KILLING
> 2018-05-21 13:11:56,227 INFO [NM ContainerManager dispatcher]
> containermanager.TestContainerManager
> (BaseContainerManagerTest.java:delete(287)) - Psuedo delete: user - nobody,
> type - FILE
> 2018-05-21 13:11:56,227 INFO [NM ContainerManager dispatcher]
> nodemanager.NMAuditLogger (NMAuditLogger.java:logSuccess(94)) - USER=nobody
> OPERATION=Container Finished - Killed TARGET=ContainerImpl
> RESULT=SUCCESS APPID=application_0_
> CONTAINERID=container_0__01_00
> 2018-05-21 13:11:56,238 INFO [NM ContainerManager dispatcher]
> container.ContainerImpl (ContainerImpl.java:handle(2111)) - Container
> container_0__01_00 transitioned from KILLING to DONE
> 2018-05-21 13:11:56,238 INFO [NM ContainerManager dispatcher]
> application.ApplicationImpl (ApplicationImpl.java:transition(489)) - Removing
> container_0__01_00 from application application_0_
> 2018-05-21 13:11:56,239 INFO [NM ContainerManager dispatcher]
> monitor.ContainersMonitorImpl
> (ContainersMonitorImpl.java:onStopMonitoringContainer(932)) - Stopping
> resource-monitoring for container_0__01_00
> 2018-05-21 13:11:56,239 INFO [NM ContainerManager dispatcher]
> containermanager.AuxServices (AuxServices.java:handle(220)) - Got event
> CONTAINER_STOP for appId application_0_
> 2018-05-21 13:11:56,274 WARN [NM ContainerManager dispatcher]
> container.ContainerImpl (ContainerImpl.java:handle(2106)) - Can't handle this
> event at current state: Current: [DONE], eventType: [CONTAINER_LAUNCHED],
> container: [container_0__01_00]
> org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event:
> CONTAINER_LAUNCHED at DONE
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
> at
>
[jira] [Commented] (YARN-8638) Allow linux container runtimes to be pluggable
[
https://issues.apache.org/jira/browse/YARN-8638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574191#comment-16574191
]
Allen Wittenauer commented on YARN-8638:
This sort of exemplifies the misuse of the word 'linux' when 'unix' or 'posix'
would have been better. With the exception of cgroups (which is relatively
new), there is very little that doesn't work on a variety of platforms.
With this proposed change, it might be time to rename the class entirely and
just add 'linux' as an alias. This would open the door up specifically for
things like FreeBSD jails, Solaris Zones, quite a few other sandboxing
technologies, and more.
> Allow linux container runtimes to be pluggable
> --
>
> Key: YARN-8638
> URL: https://issues.apache.org/jira/browse/YARN-8638
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
>Affects Versions: 3.2.0
>Reporter: Craig Condit
>Priority: Minor
>
> YARN currently supports three different Linux container runtimes (default,
> docker, and javasandbox). However, it would be relatively straightforward to
> support arbitrary runtime implementations. This would enable easier
> experimentation with new and emerging runtime technologies (runc, containerd,
> etc.) without requiring a rebuild and redeployment of Hadoop.
> This could be accomplished via a simple configuration change:
> {code:xml}
>
> yarn.nodemanager.runtime.linux.allowed-runtimes
> default,docker,experimental
>
>
>
> yarn.nodemanager.runtime.linux.experimental.class
> com.somecompany.yarn.runtime.ExperimentalLinuxContainerRuntime
> {code}
>
> In this example, {{yarn.nodemanager.runtime.linux.allowed-runtimes}} would
> now allow arbitrary values. Additionally,
> {{yarn.nodemanager.runtime.linux.\{RUNTIME_KEY}.class}} would indicate the
> {{LinuxContainerRuntime}} implementation to instantiate. A no-argument
> constructor should be sufficient, as {{LinuxContainerRuntime}} already
> provides an {{initialize()}} method.
> {{DockerLinuxContainerRuntime.isDockerContainerRequested(Map
> env)}} and {{JavaSandboxLinuxContainerRuntime.isSandboxContainerRequested()}}
> could be generalized to {{isRuntimeRequested(Map env)}} and
> added to the {{LinuxContainerRuntime}} interface. This would allow
> {{DelegatingLinuxContainerRuntime}} to select an appropriate runtime based on
> whether that runtime claimed ownership of the current container execution.
> For backwards compatibility, the existing values (default,docker,javasandbox)
> would continue to be supported as-is. Under the current logic, the evaluation
> order is javasandbox, docker, default (with default being chosen if no other
> candidates are available). Under the new evaluation logic, pluggable runtimes
> would be evaluated after docker and before default, in the order in which
> they are defined in the allowed-runtimes list. This will change no behavior
> on current clusters (as there would be no pluggable runtimes defined), and
> preserves behavior with respect to ordering of existing runtimes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8523) Interactive docker shell
[ https://issues.apache.org/jira/browse/YARN-8523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574174#comment-16574174 ] Eric Yang commented on YARN-8523: - [~Zian Chen] In today's meetup, [~jlowe] suggested to bind docker exec to a named pipe or socket. This will allow node manager restart while persisting docker exec state, and reconnect to the named pipe to resume docker exec session. This is one approach to workaround node manager restart while maintains a live session to docker exec. However, resuming docker exec session may introduce limitation where docker exec is stuck, and no way to start new sessions. Step one is get xtermjs connect to web socket serve by node manager that launches docker exec. Step 2 can refine the session mapping logic to create/resume named pipe to docker exec. I think it would be reasonable for step 2 to be created as a separate JIRA to contain the scope of the work. > Interactive docker shell > > > Key: YARN-8523 > URL: https://issues.apache.org/jira/browse/YARN-8523 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Yang >Priority: Major > Labels: Docker > > Some application might require interactive unix commands executions to carry > out operations. Container-executor can interface with docker exec to debug > or analyze docker containers while the application is running. It would be > nice to support an API to invoke docker exec to perform unix commands and > report back the output to application master. Application master can > distribute and aggregate execution of the commands to record in application > master log file. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-8638) Allow linux container runtimes to be pluggable
Craig Condit created YARN-8638:
--
Summary: Allow linux container runtimes to be pluggable
Key: YARN-8638
URL: https://issues.apache.org/jira/browse/YARN-8638
Project: Hadoop YARN
Issue Type: Improvement
Components: nodemanager
Affects Versions: 3.2.0
Reporter: Craig Condit
YARN currently supports three different Linux container runtimes (default,
docker, and javasandbox). However, it would be relatively straightforward to
support arbitrary runtime implementations. This would enable easier
experimentation with new and emerging runtime technologies (runc, containerd,
etc.) without requiring a rebuild and redeployment of Hadoop.
This could be accomplished via a simple configuration change:
{code:xml}
yarn.nodemanager.runtime.linux.allowed-runtimes
default,docker,experimental
yarn.nodemanager.runtime.linux.experimental.class
com.somecompany.yarn.runtime.ExperimentalLinuxContainerRuntime
{code}
In this example, {{yarn.nodemanager.runtime.linux.allowed-runtimes}} would now
allow arbitrary values. Additionally,
{{yarn.nodemanager.runtime.linux.\{RUNTIME_KEY}.class}} would indicate the
{{LinuxContainerRuntime}} implementation to instantiate. A no-argument
constructor should be sufficient, as {{LinuxContainerRuntime}} already provides
an {{initialize()}} method.
{{DockerLinuxContainerRuntime.isDockerContainerRequested(Map
env)}} and {{JavaSandboxLinuxContainerRuntime.isSandboxContainerRequested()}}
could be generalized to {{isRuntimeRequested(Map env)}} and
added to the {{LinuxContainerRuntime}} interface. This would allow
{{DelegatingLinuxContainerRuntime}} to select an appropriate runtime based on
whether that runtime claimed ownership of the current container execution.
For backwards compatibility, the existing values (default,docker,javasandbox)
would continue to be supported as-is. Under the current logic, the evaluation
order is javasandbox, docker, default (with default being chosen if no other
candidates are available). Under the new evaluation logic, pluggable runtimes
would be evaluated after docker and before default, in the order in which they
are defined in the allowed-runtimes list. This will change no behavior on
current clusters (as there would be no pluggable runtimes defined), and
preserves behavior with respect to ordering of existing runtimes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8637) [GPG] Add FederationStateStore getAppInfo API for GlobalPolicyGenerator
[ https://issues.apache.org/jira/browse/YARN-8637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574087#comment-16574087 ] Subru Krishnan commented on YARN-8637: -- Thanks [~botong] for the patch. lt looks mostly good, have a minor comments: * Set the default to either in-memory or ZK (essentially keep it consistent with current *FederationStateStore* configs) as SQL dependency shouldn't be expected out of the box. Also, don't see the need to add a value in yarn-default. * Implementation for ZK is missing. * Please add a test for the in-memory impl as well. * I feel we should some other name than *ApplicationsInfo* (and corresponding getters/setter) as that looks too close to *AppInfo*? I am worried it may cause some confusion. > [GPG] Add FederationStateStore getAppInfo API for GlobalPolicyGenerator > --- > > Key: YARN-8637 > URL: https://issues.apache.org/jira/browse/YARN-8637 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Botong Huang >Assignee: Botong Huang >Priority: Major > Attachments: YARN-8637-YARN-7402.v1.patch > > > The core api for FederationStateStore is provided in _FederationStateStore_. > In this patch, we are added a _FederationGPGStateStore_ api just for GPG. > Specifically, we are adding the API to get full application info from > statestore with the starting timestamp of the app entry, so that the > _ApplicationCleaner_ (YARN-7599) in GPG can delete and cleanup old entries in > the table. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8551) Build Common module for MaWo application
[ https://issues.apache.org/jira/browse/YARN-8551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yesha Vora updated YARN-8551: - Attachment: YARN-8551.001.patch > Build Common module for MaWo application > > > Key: YARN-8551 > URL: https://issues.apache.org/jira/browse/YARN-8551 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Yesha Vora >Assignee: Yesha Vora >Priority: Major > Attachments: YARN-8551.001.patch > > > Build Common module for MaWo application. > This module should include defination of Task. A Task should contain > * TaskID > * Task Command > * Task Environment > * Task Timeout > * Task Type > ** Simple Task > *** Its a single Task > ** Composite Task > *** Its a composition of multiple simple tasks > ** Teardown Task > *** Its a last task to be executed after a job is finished > ** Null Task > *** Its a null task -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8637) [GPG] Add FederationStateStore getAppInfo API for GlobalPolicyGenerator
[
https://issues.apache.org/jira/browse/YARN-8637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574052#comment-16574052
]
genericqa commented on YARN-8637:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
35s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} YARN-7402 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
14s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
55s{color} | {color:green} YARN-7402 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
26s{color} | {color:green} YARN-7402 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} YARN-7402 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
12s{color} | {color:green} YARN-7402 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 14s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
46s{color} | {color:green} YARN-7402 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
53s{color} | {color:green} YARN-7402 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 7m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 6s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
45s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
12s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
18s{color} | {color:green} hadoop-yarn-server-common in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
38s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 95m 58s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | YARN-8637 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12934883/YARN-8637-YARN-7402.v1.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle xml cc |
| uname | Linux 63d5784ed43e 3.13.0-153-generic
[jira] [Commented] (YARN-8298) Yarn Service Upgrade: Support express upgrade of a service
[ https://issues.apache.org/jira/browse/YARN-8298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574051#comment-16574051 ] Eric Yang commented on YARN-8298: - [~csingh] While developing this patch, it was intended for the patch to run with Hadoop 3.1.2 CLI, and issue upgrade command to ServiceAM started by Hadoop 3.1.1. This is the reason that orchestration logic runs in ServiceClient as part of resource manager instead of ServiceAM. From architectural point of view, it would be best for upgrade logic to run in ServiceAM. Hadoop 3.1.1 ServiceAM does not have express upgrade logic. Therefore, express upgrade is triggered using ServiceState that can be understood by older ServiceAM. For architectural correctness, it is in our interest to introduce a ServiceState that can be understood by ServiceAM as express upgrade. This helps us to move orchestration logic from ServiceClient to ServiceMaster. RM can be relief from upgrade logic burden. This would be an incompatible change to ServiceAM to gain new express upgrade abilities. Therefore, I remove 3.1.2 from targeted version. > Yarn Service Upgrade: Support express upgrade of a service > -- > > Key: YARN-8298 > URL: https://issues.apache.org/jira/browse/YARN-8298 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.1.1 >Reporter: Chandni Singh >Assignee: Chandni Singh >Priority: Major > Attachments: YARN-8298.001.patch, YARN-8298.002.patch > > > Currently service upgrade involves 2 steps > * initiate upgrade by providing new spec > * trigger upgrade of each instance/component > > We need to add the ability to upgrade the service in one shot: > # Aborting the upgrade will not be supported > # Upgrade finalization will be done automatically. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8298) Yarn Service Upgrade: Support express upgrade of a service
[ https://issues.apache.org/jira/browse/YARN-8298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-8298: Target Version/s: 3.2.0 (was: 3.2.0, 3.1.2) > Yarn Service Upgrade: Support express upgrade of a service > -- > > Key: YARN-8298 > URL: https://issues.apache.org/jira/browse/YARN-8298 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.1.1 >Reporter: Chandni Singh >Assignee: Chandni Singh >Priority: Major > Attachments: YARN-8298.001.patch, YARN-8298.002.patch > > > Currently service upgrade involves 2 steps > * initiate upgrade by providing new spec > * trigger upgrade of each instance/component > > We need to add the ability to upgrade the service in one shot: > # Aborting the upgrade will not be supported > # Upgrade finalization will be done automatically. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8588) Logging improvements for better debuggability
[ https://issues.apache.org/jira/browse/YARN-8588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574042#comment-16574042 ] Wangda Tan commented on YARN-8588: -- +1, LGTM. thanks [~suma.shivaprasad] > Logging improvements for better debuggability > - > > Key: YARN-8588 > URL: https://issues.apache.org/jira/browse/YARN-8588 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad >Priority: Major > Attachments: YARN-8588.1.patch, YARN-8588.2.patch > > > Capacity allocations decided in GuaranteedCapacityOvertimePolicy are > available via AutoCreatedLeafQueueConfig. However this class lacks a toString > and some other DEBUG level logs are needed for better debuggability -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-8298) Yarn Service Upgrade: Support express upgrade of a service
[ https://issues.apache.org/jira/browse/YARN-8298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang reassigned YARN-8298: --- Assignee: Chandni Singh (was: Eric Yang) > Yarn Service Upgrade: Support express upgrade of a service > -- > > Key: YARN-8298 > URL: https://issues.apache.org/jira/browse/YARN-8298 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.1.1 >Reporter: Chandni Singh >Assignee: Chandni Singh >Priority: Major > Attachments: YARN-8298.001.patch, YARN-8298.002.patch > > > Currently service upgrade involves 2 steps > * initiate upgrade by providing new spec > * trigger upgrade of each instance/component > > We need to add the ability to upgrade the service in one shot: > # Aborting the upgrade will not be supported > # Upgrade finalization will be done automatically. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8331) Race condition in NM container launched after done
[
https://issues.apache.org/jira/browse/YARN-8331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574013#comment-16574013
]
genericqa commented on YARN-8331:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 37s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 52s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 18m
34s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 71m 50s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | YARN-8331 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12934881/YARN-8331.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 0058bc650599 4.4.0-130-generic #156-Ubuntu SMP Thu Jun 14
08:53:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 3214cd7 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/21544/testReport/ |
| Max. process+thread count | 409 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/21544/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Race condition in NM container launched after done
>
[jira] [Commented] (YARN-8509) Total pending resource calculation in preemption should use user-limit factor instead of minimum-user-limit-percent
[
https://issues.apache.org/jira/browse/YARN-8509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573997#comment-16573997
]
Zian Chen commented on YARN-8509:
-
Hi Eric, thanks for the comments. Discussed with Wangda, the patch uploaded
before is not correct due to misunderstand of the original problem.
I have changed the Jira title. The intention of this Jira is to fix calculation
of pending resource consider user-limit in preemption scenario. Currently,
pending resource calculation in preemption uses the calculation algorithm in
scheduling which is this one,
{code:java}
user_limit = min(max(current_capacity)/ #active_users, current_capacity *
user_limit_percent), queue_capacity * user_limit_factor)
{code}
this is good for scheduling cause we want to make sure users can get at least
"minimum-user-limit-percent" of resource to use, which is more like a lower
bound of user-limit. However we should not capture total pending resource a
leaf queue can get by minimum-user-limit-percent, instead, we want to use
user-limit-factor which is the upper bound to capture pending resource in
preemption. Cause if we use minimum-user-limit-percent to capture pending
resource, resource under-utilization will happen in preemption scenario. Thus,
we suggest the pending resource calculation for preemption should use this
formula.
{code:java}
total_pending(partition,queue) = min {Q_max(partition) - Q_used(partition), Σ
(min {
User.ulf(partition) - User.used(partition), User.pending(partition})}
{code}
Let me give an example,
{code:java}
Root
/ | \ \
a b c d
30 30 30 10
1) Only one node (n1) in the cluster, it has 100G.
2) app1 submit to queue-a, asks for 10G used, 6G pending.
3) app2 submit to queue-b, asks for 40G used, 30G pending.
4) app3 submit to queue-c, asks for 50G used, 30G pending.
{code}
Here we only have one user, and user-limit-factor for queues are
||Queue name|| minimum-user-limit-percent ||user-limit-factor||
| a| 1| 1.0 f|
| b| 1| 2.0 f|
| c| 1| 2.0 f|
| d| 1| 2.0 f|
With old calculation, user-limit for queue-a is 30G, which can let app1 has 6G
pending, but user-limit for queue-b becomes 40G, which makes headroom become
zero after subtract 40G used, the 30G pending resource been asked can not be
accepted, same thing with queue-c too.
However if we see this test case in preemption point of view, we should allow
queue-b and queue-c take more pending resources. Because even though queue-a
has 30G guaranteed configured, it's under utilization. And by pending resource
captured by the old algorithm, queue-b and queue-c can not take available
resource through preemption which make the cluster resource not used
effectively.
To summarize, since user-limit-factor maintains the hard-limit of how much
resource can be used by a user, we should calculate pending resource consider
user-limit-factor instead of minimum-user-limit-percent.
Could you share your opinion on this, [~eepayne]?
> Total pending resource calculation in preemption should use user-limit factor
> instead of minimum-user-limit-percent
> ---
>
> Key: YARN-8509
> URL: https://issues.apache.org/jira/browse/YARN-8509
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: Zian Chen
>Assignee: Zian Chen
>Priority: Major
> Attachments: YARN-8509.001.patch, YARN-8509.002.patch,
> YARN-8509.003.patch
>
>
> In LeafQueue#getTotalPendingResourcesConsideringUserLimit, we calculate total
> pending resource based on user-limit percent and user-limit factor which will
> cap pending resource for each user to the minimum of user-limit pending and
> actual pending. This will prevent queue from taking more pending resource to
> achieve queue balance after all queue satisfied with its ideal allocation.
>
> We need to change the logic to let queue pending can go beyond userlimit.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8568) Replace the deprecated zk-address property in the HA config example in ResourceManagerHA.md
[ https://issues.apache.org/jira/browse/YARN-8568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573985#comment-16573985 ] Hudson commented on YARN-8568: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14732 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14732/]) YARN-8568. Replace the deprecated zk-address property in the HA config (rkanter: rev 8478732bb28e9e71061d6b4a043a3a1b5c688902) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceManagerHA.md > Replace the deprecated zk-address property in the HA config example in > ResourceManagerHA.md > --- > > Key: YARN-8568 > URL: https://issues.apache.org/jira/browse/YARN-8568 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Affects Versions: 3.0.x >Reporter: Antal Bálint Steinbach >Assignee: Antal Bálint Steinbach >Priority: Minor > Fix For: 3.2.0, 3.0.3, 3.1.2 > > Attachments: YARN-8568.001.patch > > > yarn.resourcemanager.zk-address is deprecated. Instead, use hadoop.zk.address > In the example, "yarn.resourcemanager.zk-address" is used which is > deprecated. In the description, the property name is correct > "hadoop.zk.address". -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7599) [GPG] ApplicationCleaner in Global Policy Generator
[ https://issues.apache.org/jira/browse/YARN-7599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573959#comment-16573959 ] Botong Huang commented on YARN-7599: YARN-8637 created to separate the statestore API extension part out of this Jira > [GPG] ApplicationCleaner in Global Policy Generator > --- > > Key: YARN-7599 > URL: https://issues.apache.org/jira/browse/YARN-7599 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Botong Huang >Assignee: Botong Huang >Priority: Minor > Labels: federation, gpg > Attachments: YARN-7599-YARN-7402.v1.patch, > YARN-7599-YARN-7402.v2.patch > > > In Federation, we need a cleanup service for StateStore as well as Yarn > Registry. For the former, we need to remove old application records. For the > latter, failed and killed applications might leave records in the Yarn > Registry (see YARN-6128). We plan to do both cleanup work in > ApplicationCleaner in GPG -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8637) [GPG] Add FederationStateStore getAppInfo API for GlobalPolicyGenerator
[ https://issues.apache.org/jira/browse/YARN-8637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Botong Huang updated YARN-8637: --- Attachment: YARN-8637-YARN-7402.v1.patch > [GPG] Add FederationStateStore getAppInfo API for GlobalPolicyGenerator > --- > > Key: YARN-8637 > URL: https://issues.apache.org/jira/browse/YARN-8637 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Botong Huang >Assignee: Botong Huang >Priority: Major > Attachments: YARN-8637-YARN-7402.v1.patch > > > The core api for FederationStateStore is provided in _FederationStateStore_. > In this patch, we are added a _FederationGPGStateStore_ api just for GPG. > Specifically, we are adding the API to get full application info from > statestore with the starting timestamp of the app entry, so that the > _ApplicationCleaner_ (YARN-7599) in GPG can delete and cleanup old entries in > the table. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8523) Interactive docker shell
[ https://issues.apache.org/jira/browse/YARN-8523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573946#comment-16573946 ] Zian Chen commented on YARN-8523: - [~eyang], thanks for raising this feature. This is very useful for live debug of container diagnosis. we can add a series of interactive commands to let user debug more effectively, like tail -f container log, container resource usage, etc. For handling nodemanager restart scenario, we can register a event listener to listen restart or shutdown signal of node manager web socket and respond in xterm js terminal accordingly, (like print out NM restart/shutdown message to user, etc) and do reconnect retries several times after typical nm restart interval. Again, if NM meet any unexpected issue which can not resume its service, that's something we can not solve on this interactive docker shell by itself and we should just give user reasonable alert message to inform the current situation (like retry failed with timeout, please check NM log to get more information, etc). I think pass command through NM web socket and reuse container-executor security check would be a good prototype we can build first without have too much burden on handling root daemon by carving another secure channel. > Interactive docker shell > > > Key: YARN-8523 > URL: https://issues.apache.org/jira/browse/YARN-8523 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Yang >Priority: Major > Labels: Docker > > Some application might require interactive unix commands executions to carry > out operations. Container-executor can interface with docker exec to debug > or analyze docker containers while the application is running. It would be > nice to support an API to invoke docker exec to perform unix commands and > report back the output to application master. Application master can > distribute and aggregate execution of the commands to record in application > master log file. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8523) Interactive docker shell
[ https://issues.apache.org/jira/browse/YARN-8523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573946#comment-16573946 ] Zian Chen edited comment on YARN-8523 at 8/8/18 10:07 PM: -- [~eyang], thanks for raising this feature. This is very useful for live debug of container diagnosis. we can add a series of interactive commands to let user debug more effectively, like tail -f container log, container resource usage, etc. For handling nodemanager restart scenario, we can register a event listener to listen restart or shutdown signal of node manager web socket and respond in xterm js terminal accordingly, (like print out NM restart/shutdown message to user, etc) and do reconnect retries several times after typical nm restart interval. Again, if NM meet any unexpected issue which can not resume its service, that's something we can not solve on this interactive docker shell by itself and we should just give user reasonable alert message to inform the current situation (like retry failed with timeout, please check NM log to get more information, etc). I think pass command through NM web socket and reuse container-executor security check would be a good prototype we can build first without have too much burden on handling root daemon by carving another secure channel. was (Author: zian chen): [~eyang], thanks for raising this feature. This is very useful for live debug of container diagnosis. we can add a series of interactive commands to let user debug more effectively, like tail -f container log, container resource usage, etc. For handling nodemanager restart scenario, we can register a event listener to listen restart or shutdown signal of node manager web socket and respond in xterm js terminal accordingly, (like print out NM restart/shutdown message to user, etc) and do reconnect retries several times after typical nm restart interval. Again, if NM meet any unexpected issue which can not resume its service, that's something we can not solve on this interactive docker shell by itself and we should just give user reasonable alert message to inform the current situation (like retry failed with timeout, please check NM log to get more information, etc). I think pass command through NM web socket and reuse container-executor security check would be a good prototype we can build first without have too much burden on handling root daemon by carving another secure channel. > Interactive docker shell > > > Key: YARN-8523 > URL: https://issues.apache.org/jira/browse/YARN-8523 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Yang >Priority: Major > Labels: Docker > > Some application might require interactive unix commands executions to carry > out operations. Container-executor can interface with docker exec to debug > or analyze docker containers while the application is running. It would be > nice to support an API to invoke docker exec to perform unix commands and > report back the output to application master. Application master can > distribute and aggregate execution of the commands to record in application > master log file. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8568) Replace the deprecated zk-address property in the HA config example in ResourceManagerHA.md
[ https://issues.apache.org/jira/browse/YARN-8568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573945#comment-16573945 ] Robert Kanter commented on YARN-8568: - +1 LGTM > Replace the deprecated zk-address property in the HA config example in > ResourceManagerHA.md > --- > > Key: YARN-8568 > URL: https://issues.apache.org/jira/browse/YARN-8568 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Affects Versions: 3.0.x >Reporter: Antal Bálint Steinbach >Assignee: Antal Bálint Steinbach >Priority: Minor > Attachments: YARN-8568.001.patch > > > yarn.resourcemanager.zk-address is deprecated. Instead, use hadoop.zk.address > In the example, "yarn.resourcemanager.zk-address" is used which is > deprecated. In the description, the property name is correct > "hadoop.zk.address". -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8331) Race condition in NM container launched after done
[
https://issues.apache.org/jira/browse/YARN-8331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pradeep Ambati updated YARN-8331:
-
Attachment: YARN-8331.002.patch
> Race condition in NM container launched after done
> --
>
> Key: YARN-8331
> URL: https://issues.apache.org/jira/browse/YARN-8331
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.1.0, 2.9.1, 3.0.2
>Reporter: Yang Wang
>Assignee: Pradeep Ambati
>Priority: Major
> Attachments: YARN-8331.001.patch, YARN-8331.002.patch
>
>
> When a container is launching, in ContainerLaunch#launchContainer, state is
> SCHEDULED,
> kill event was sent to this container, state : SCHEDULED->KILLING->DONE
> Then ContainerLaunch send CONTAINER_LAUNCHED event and start the container
> processes. These absent container processes will not be cleaned up anymore.
>
> {code:java}
> 2018-05-21 13:11:56,114 INFO [Thread-11] nodemanager.NMAuditLogger
> (NMAuditLogger.java:logSuccess(94)) - USER=nobody OPERATION=Start Container
> Request TARGET=ContainerManageImpl RESULT=SUCCESS
> APPID=application_0_CONTAINERID=container_0__01_00
> 2018-05-21 13:11:56,114 INFO [NM ContainerManager dispatcher]
> application.ApplicationImpl (ApplicationImpl.java:handle(632)) - Application
> application_0_ transitioned from NEW to INITING
> 2018-05-21 13:11:56,114 INFO [NM ContainerManager dispatcher]
> application.ApplicationImpl (ApplicationImpl.java:transition(446)) - Adding
> container_0__01_00 to application application_0_
> 2018-05-21 13:11:56,118 INFO [NM ContainerManager dispatcher]
> application.ApplicationImpl (ApplicationImpl.java:handle(632)) - Application
> application_0_ transitioned from INITING to RUNNING
> 2018-05-21 13:11:56,119 INFO [NM ContainerManager dispatcher]
> container.ContainerImpl (ContainerImpl.java:handle(2111)) - Container
> container_0__01_00 transitioned from NEW to SCHEDULED
> 2018-05-21 13:11:56,119 INFO [NM ContainerManager dispatcher]
> containermanager.AuxServices (AuxServices.java:handle(220)) - Got event
> CONTAINER_INIT for appId application_0_
> 2018-05-21 13:11:56,119 INFO [NM ContainerManager dispatcher]
> scheduler.ContainerScheduler (ContainerScheduler.java:startContainer(504)) -
> Starting container [container_0__01_00]
> 2018-05-21 13:11:56,226 INFO [NM ContainerManager dispatcher]
> container.ContainerImpl (ContainerImpl.java:handle(2111)) - Container
> container_0__01_00 transitioned from SCHEDULED to KILLING
> 2018-05-21 13:11:56,227 INFO [NM ContainerManager dispatcher]
> containermanager.TestContainerManager
> (BaseContainerManagerTest.java:delete(287)) - Psuedo delete: user - nobody,
> type - FILE
> 2018-05-21 13:11:56,227 INFO [NM ContainerManager dispatcher]
> nodemanager.NMAuditLogger (NMAuditLogger.java:logSuccess(94)) - USER=nobody
> OPERATION=Container Finished - Killed TARGET=ContainerImpl
> RESULT=SUCCESS APPID=application_0_
> CONTAINERID=container_0__01_00
> 2018-05-21 13:11:56,238 INFO [NM ContainerManager dispatcher]
> container.ContainerImpl (ContainerImpl.java:handle(2111)) - Container
> container_0__01_00 transitioned from KILLING to DONE
> 2018-05-21 13:11:56,238 INFO [NM ContainerManager dispatcher]
> application.ApplicationImpl (ApplicationImpl.java:transition(489)) - Removing
> container_0__01_00 from application application_0_
> 2018-05-21 13:11:56,239 INFO [NM ContainerManager dispatcher]
> monitor.ContainersMonitorImpl
> (ContainersMonitorImpl.java:onStopMonitoringContainer(932)) - Stopping
> resource-monitoring for container_0__01_00
> 2018-05-21 13:11:56,239 INFO [NM ContainerManager dispatcher]
> containermanager.AuxServices (AuxServices.java:handle(220)) - Got event
> CONTAINER_STOP for appId application_0_
> 2018-05-21 13:11:56,274 WARN [NM ContainerManager dispatcher]
> container.ContainerImpl (ContainerImpl.java:handle(2106)) - Can't handle this
> event at current state: Current: [DONE], eventType: [CONTAINER_LAUNCHED],
> container: [container_0__01_00]
> org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event:
> CONTAINER_LAUNCHED at DONE
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl.handle(ContainerImpl.java:2104)
> at
>
[jira] [Commented] (YARN-7599) [GPG] ApplicationCleaner in Global Policy Generator
[
https://issues.apache.org/jira/browse/YARN-7599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573920#comment-16573920
]
genericqa commented on YARN-7599:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} YARN-7599 does not apply to YARN-7402. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-7599 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12912643/YARN-7599-YARN-7402.v2.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/21543/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> [GPG] ApplicationCleaner in Global Policy Generator
> ---
>
> Key: YARN-7599
> URL: https://issues.apache.org/jira/browse/YARN-7599
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Botong Huang
>Assignee: Botong Huang
>Priority: Minor
> Labels: federation, gpg
> Attachments: YARN-7599-YARN-7402.v1.patch,
> YARN-7599-YARN-7402.v2.patch
>
>
> In Federation, we need a cleanup service for StateStore as well as Yarn
> Registry. For the former, we need to remove old application records. For the
> latter, failed and killed applications might leave records in the Yarn
> Registry (see YARN-6128). We plan to do both cleanup work in
> ApplicationCleaner in GPG
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8160) Yarn Service Upgrade: Support upgrade of service that use docker containers
[
https://issues.apache.org/jira/browse/YARN-8160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chandni Singh updated YARN-8160:
Attachment: YARN-8610.001.patch
> Yarn Service Upgrade: Support upgrade of service that use docker containers
>
>
> Key: YARN-8160
> URL: https://issues.apache.org/jira/browse/YARN-8160
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Chandni Singh
>Assignee: Chandni Singh
>Priority: Major
> Labels: Docker
> Attachments: YARN-8610.001.patch,
> container_e02_1533231998644_0009_01_03.nm.log
>
>
> Ability to upgrade dockerized yarn native services.
> Ref: YARN-5637
> *Background*
> Container upgrade is supported by the NM via {{reInitializeContainer}} api.
> {{reInitializeContainer}} does *NOT* change the ContainerId of the upgraded
> container.
> NM performs the following steps during {{reInitializeContainer}}:
> - kills the existing process
> - cleans up the container
> - launches another container with the new {{ContainerLaunchContext}}
> NOTE: {{ContainerLaunchContext}} holds all the information that needs to
> upgrade the container.
> With {{reInitializeContainer}}, the following does *NOT* change
> - container ID. This is not created by NM. It is provided to it and here RM
> is not creating another container allocation.
> - {{localizedResources}} this stays the same if the upgrade does *NOT*
> require additional resources IIUC.
>
> The following changes with {{reInitializeContainer}}
> - the working directory of the upgraded container changes. It is *NOT* a
> relaunch.
> *Changes required in the case of docker container*
> - {{reInitializeContainer}} seems to not be working with Docker containers.
> Investigate and fix this.
> - [Future change] Add an additional api to NM to pull the images and modify
> {{reInitializeContainer}} to trigger docker container launch without pulling
> the image first which could be based on a flag.
> -- When the service upgrade is initialized, we can provide the user with
> an option to just pull the images on the NMs.
> -- When a component instance is upgrade, it calls the
> {{reInitializeContainer}} with the flag pull-image set to false, since the NM
> will have already pulled the images.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8160) Yarn Service Upgrade: Support upgrade of service that use docker containers
[
https://issues.apache.org/jira/browse/YARN-8160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573888#comment-16573888
]
Chandni Singh edited comment on YARN-8160 at 8/8/18 9:30 PM:
-
The reapContainer and cleaning of container files during cleanup can be
mutually exclusive to the run. Attached is the patch for it. I am testing it.
was (Author: csingh):
The cleaning of container files during cleanup can be mutually exclusive to the
run. Attached is the patch for it.
> Yarn Service Upgrade: Support upgrade of service that use docker containers
>
>
> Key: YARN-8160
> URL: https://issues.apache.org/jira/browse/YARN-8160
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Chandni Singh
>Assignee: Chandni Singh
>Priority: Major
> Labels: Docker
> Attachments: YARN-8610.001.patch,
> container_e02_1533231998644_0009_01_03.nm.log
>
>
> Ability to upgrade dockerized yarn native services.
> Ref: YARN-5637
> *Background*
> Container upgrade is supported by the NM via {{reInitializeContainer}} api.
> {{reInitializeContainer}} does *NOT* change the ContainerId of the upgraded
> container.
> NM performs the following steps during {{reInitializeContainer}}:
> - kills the existing process
> - cleans up the container
> - launches another container with the new {{ContainerLaunchContext}}
> NOTE: {{ContainerLaunchContext}} holds all the information that needs to
> upgrade the container.
> With {{reInitializeContainer}}, the following does *NOT* change
> - container ID. This is not created by NM. It is provided to it and here RM
> is not creating another container allocation.
> - {{localizedResources}} this stays the same if the upgrade does *NOT*
> require additional resources IIUC.
>
> The following changes with {{reInitializeContainer}}
> - the working directory of the upgraded container changes. It is *NOT* a
> relaunch.
> *Changes required in the case of docker container*
> - {{reInitializeContainer}} seems to not be working with Docker containers.
> Investigate and fix this.
> - [Future change] Add an additional api to NM to pull the images and modify
> {{reInitializeContainer}} to trigger docker container launch without pulling
> the image first which could be based on a flag.
> -- When the service upgrade is initialized, we can provide the user with
> an option to just pull the images on the NMs.
> -- When a component instance is upgrade, it calls the
> {{reInitializeContainer}} with the flag pull-image set to false, since the NM
> will have already pulled the images.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8160) Yarn Service Upgrade: Support upgrade of service that use docker containers
[
https://issues.apache.org/jira/browse/YARN-8160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chandni Singh updated YARN-8160:
Attachment: (was: YARN-8160.001.patch)
> Yarn Service Upgrade: Support upgrade of service that use docker containers
>
>
> Key: YARN-8160
> URL: https://issues.apache.org/jira/browse/YARN-8160
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Chandni Singh
>Assignee: Chandni Singh
>Priority: Major
> Labels: Docker
> Attachments: container_e02_1533231998644_0009_01_03.nm.log
>
>
> Ability to upgrade dockerized yarn native services.
> Ref: YARN-5637
> *Background*
> Container upgrade is supported by the NM via {{reInitializeContainer}} api.
> {{reInitializeContainer}} does *NOT* change the ContainerId of the upgraded
> container.
> NM performs the following steps during {{reInitializeContainer}}:
> - kills the existing process
> - cleans up the container
> - launches another container with the new {{ContainerLaunchContext}}
> NOTE: {{ContainerLaunchContext}} holds all the information that needs to
> upgrade the container.
> With {{reInitializeContainer}}, the following does *NOT* change
> - container ID. This is not created by NM. It is provided to it and here RM
> is not creating another container allocation.
> - {{localizedResources}} this stays the same if the upgrade does *NOT*
> require additional resources IIUC.
>
> The following changes with {{reInitializeContainer}}
> - the working directory of the upgraded container changes. It is *NOT* a
> relaunch.
> *Changes required in the case of docker container*
> - {{reInitializeContainer}} seems to not be working with Docker containers.
> Investigate and fix this.
> - [Future change] Add an additional api to NM to pull the images and modify
> {{reInitializeContainer}} to trigger docker container launch without pulling
> the image first which could be based on a flag.
> -- When the service upgrade is initialized, we can provide the user with
> an option to just pull the images on the NMs.
> -- When a component instance is upgrade, it calls the
> {{reInitializeContainer}} with the flag pull-image set to false, since the NM
> will have already pulled the images.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8637) [GPG] Add FederationStateStore getAppInfo API for GlobalPolicyGenerator
[ https://issues.apache.org/jira/browse/YARN-8637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Botong Huang updated YARN-8637: --- Summary: [GPG] Add FederationStateStore getAppInfo API for GlobalPolicyGenerator (was: Add FederationStateStore getAppInfo API for GlobalPolicyGenerator) > [GPG] Add FederationStateStore getAppInfo API for GlobalPolicyGenerator > --- > > Key: YARN-8637 > URL: https://issues.apache.org/jira/browse/YARN-8637 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Botong Huang >Assignee: Botong Huang >Priority: Major > > The core api for FederationStateStore is provided in _FederationStateStore_. > In this patch, we are added a _FederationGPGStateStore_ api just for GPG. > Specifically, we are adding the API to get full application info from > statestore with the starting timestamp of the app entry, so that the > _ApplicationCleaner_ (YARN-7599) in GPG can delete and cleanup old entries in > the table. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8637) Add FederationStateStore getAppInfo API for GlobalPolicyGenerator
[ https://issues.apache.org/jira/browse/YARN-8637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Botong Huang updated YARN-8637: --- Issue Type: Sub-task (was: Task) Parent: YARN-7402 > Add FederationStateStore getAppInfo API for GlobalPolicyGenerator > - > > Key: YARN-8637 > URL: https://issues.apache.org/jira/browse/YARN-8637 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Botong Huang >Assignee: Botong Huang >Priority: Major > > The core api for FederationStateStore is provided in _FederationStateStore_. > In this patch, we are added a _FederationGPGStateStore_ api just for GPG. > Specifically, we are adding the API to get full application info from > statestore with the starting timestamp of the app entry, so that the > _ApplicationCleaner_ (YARN-7599) in GPG can delete and cleanup old entries in > the table. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-8637) Add FederationStateStore getAppInfo API for GlobalPolicyGenerator
Botong Huang created YARN-8637: -- Summary: Add FederationStateStore getAppInfo API for GlobalPolicyGenerator Key: YARN-8637 URL: https://issues.apache.org/jira/browse/YARN-8637 Project: Hadoop YARN Issue Type: Task Reporter: Botong Huang Assignee: Botong Huang The core api for FederationStateStore is provided in _FederationStateStore_. In this patch, we are added a _FederationGPGStateStore_ api just for GPG. Specifically, we are adding the API to get full application info from statestore with the starting timestamp of the app entry, so that the _ApplicationCleaner_ (YARN-7599) in GPG can delete and cleanup old entries in the table. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8160) Yarn Service Upgrade: Support upgrade of service that use docker containers
[
https://issues.apache.org/jira/browse/YARN-8160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573888#comment-16573888
]
Chandni Singh edited comment on YARN-8160 at 8/8/18 9:09 PM:
-
The cleaning of container files during cleanup can be mutually exclusive to the
run. Attached is the patch for it.
was (Author: csingh):
The clean of container files during container cleanup can be mutually exclusive
to the run. Attached is the patch for it.
> Yarn Service Upgrade: Support upgrade of service that use docker containers
>
>
> Key: YARN-8160
> URL: https://issues.apache.org/jira/browse/YARN-8160
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Chandni Singh
>Assignee: Chandni Singh
>Priority: Major
> Labels: Docker
> Attachments: YARN-8160.001.patch,
> container_e02_1533231998644_0009_01_03.nm.log
>
>
> Ability to upgrade dockerized yarn native services.
> Ref: YARN-5637
> *Background*
> Container upgrade is supported by the NM via {{reInitializeContainer}} api.
> {{reInitializeContainer}} does *NOT* change the ContainerId of the upgraded
> container.
> NM performs the following steps during {{reInitializeContainer}}:
> - kills the existing process
> - cleans up the container
> - launches another container with the new {{ContainerLaunchContext}}
> NOTE: {{ContainerLaunchContext}} holds all the information that needs to
> upgrade the container.
> With {{reInitializeContainer}}, the following does *NOT* change
> - container ID. This is not created by NM. It is provided to it and here RM
> is not creating another container allocation.
> - {{localizedResources}} this stays the same if the upgrade does *NOT*
> require additional resources IIUC.
>
> The following changes with {{reInitializeContainer}}
> - the working directory of the upgraded container changes. It is *NOT* a
> relaunch.
> *Changes required in the case of docker container*
> - {{reInitializeContainer}} seems to not be working with Docker containers.
> Investigate and fix this.
> - [Future change] Add an additional api to NM to pull the images and modify
> {{reInitializeContainer}} to trigger docker container launch without pulling
> the image first which could be based on a flag.
> -- When the service upgrade is initialized, we can provide the user with
> an option to just pull the images on the NMs.
> -- When a component instance is upgrade, it calls the
> {{reInitializeContainer}} with the flag pull-image set to false, since the NM
> will have already pulled the images.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8160) Yarn Service Upgrade: Support upgrade of service that use docker containers
[
https://issues.apache.org/jira/browse/YARN-8160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chandni Singh updated YARN-8160:
Attachment: YARN-8160.001.patch
> Yarn Service Upgrade: Support upgrade of service that use docker containers
>
>
> Key: YARN-8160
> URL: https://issues.apache.org/jira/browse/YARN-8160
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Chandni Singh
>Assignee: Chandni Singh
>Priority: Major
> Labels: Docker
> Attachments: YARN-8160.001.patch,
> container_e02_1533231998644_0009_01_03.nm.log
>
>
> Ability to upgrade dockerized yarn native services.
> Ref: YARN-5637
> *Background*
> Container upgrade is supported by the NM via {{reInitializeContainer}} api.
> {{reInitializeContainer}} does *NOT* change the ContainerId of the upgraded
> container.
> NM performs the following steps during {{reInitializeContainer}}:
> - kills the existing process
> - cleans up the container
> - launches another container with the new {{ContainerLaunchContext}}
> NOTE: {{ContainerLaunchContext}} holds all the information that needs to
> upgrade the container.
> With {{reInitializeContainer}}, the following does *NOT* change
> - container ID. This is not created by NM. It is provided to it and here RM
> is not creating another container allocation.
> - {{localizedResources}} this stays the same if the upgrade does *NOT*
> require additional resources IIUC.
>
> The following changes with {{reInitializeContainer}}
> - the working directory of the upgraded container changes. It is *NOT* a
> relaunch.
> *Changes required in the case of docker container*
> - {{reInitializeContainer}} seems to not be working with Docker containers.
> Investigate and fix this.
> - [Future change] Add an additional api to NM to pull the images and modify
> {{reInitializeContainer}} to trigger docker container launch without pulling
> the image first which could be based on a flag.
> -- When the service upgrade is initialized, we can provide the user with
> an option to just pull the images on the NMs.
> -- When a component instance is upgrade, it calls the
> {{reInitializeContainer}} with the flag pull-image set to false, since the NM
> will have already pulled the images.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8160) Yarn Service Upgrade: Support upgrade of service that use docker containers
[
https://issues.apache.org/jira/browse/YARN-8160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573888#comment-16573888
]
Chandni Singh commented on YARN-8160:
-
The clean of container files during container cleanup can be mutually exclusive
to the run. Attached is the patch for it.
> Yarn Service Upgrade: Support upgrade of service that use docker containers
>
>
> Key: YARN-8160
> URL: https://issues.apache.org/jira/browse/YARN-8160
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Chandni Singh
>Assignee: Chandni Singh
>Priority: Major
> Labels: Docker
> Attachments: YARN-8160.001.patch,
> container_e02_1533231998644_0009_01_03.nm.log
>
>
> Ability to upgrade dockerized yarn native services.
> Ref: YARN-5637
> *Background*
> Container upgrade is supported by the NM via {{reInitializeContainer}} api.
> {{reInitializeContainer}} does *NOT* change the ContainerId of the upgraded
> container.
> NM performs the following steps during {{reInitializeContainer}}:
> - kills the existing process
> - cleans up the container
> - launches another container with the new {{ContainerLaunchContext}}
> NOTE: {{ContainerLaunchContext}} holds all the information that needs to
> upgrade the container.
> With {{reInitializeContainer}}, the following does *NOT* change
> - container ID. This is not created by NM. It is provided to it and here RM
> is not creating another container allocation.
> - {{localizedResources}} this stays the same if the upgrade does *NOT*
> require additional resources IIUC.
>
> The following changes with {{reInitializeContainer}}
> - the working directory of the upgraded container changes. It is *NOT* a
> relaunch.
> *Changes required in the case of docker container*
> - {{reInitializeContainer}} seems to not be working with Docker containers.
> Investigate and fix this.
> - [Future change] Add an additional api to NM to pull the images and modify
> {{reInitializeContainer}} to trigger docker container launch without pulling
> the image first which could be based on a flag.
> -- When the service upgrade is initialized, we can provide the user with
> an option to just pull the images on the NMs.
> -- When a component instance is upgrade, it calls the
> {{reInitializeContainer}} with the flag pull-image set to false, since the NM
> will have already pulled the images.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8509) Total pending resource calculation in preemption should use user-limit factor instead of minimum-user-limit-percent
[ https://issues.apache.org/jira/browse/YARN-8509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zian Chen updated YARN-8509: Summary: Total pending resource calculation in preemption should use user-limit factor instead of minimum-user-limit-percent (was: Fix UserLimit calculation for preemption to balance scenario after queue satisfied ) > Total pending resource calculation in preemption should use user-limit factor > instead of minimum-user-limit-percent > --- > > Key: YARN-8509 > URL: https://issues.apache.org/jira/browse/YARN-8509 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Zian Chen >Assignee: Zian Chen >Priority: Major > Attachments: YARN-8509.001.patch, YARN-8509.002.patch, > YARN-8509.003.patch > > > In LeafQueue#getTotalPendingResourcesConsideringUserLimit, we calculate total > pending resource based on user-limit percent and user-limit factor which will > cap pending resource for each user to the minimum of user-limit pending and > actual pending. This will prevent queue from taking more pending resource to > achieve queue balance after all queue satisfied with its ideal allocation. > > We need to change the logic to let queue pending can go beyond userlimit. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8298) Yarn Service Upgrade: Support express upgrade of a service
[
https://issues.apache.org/jira/browse/YARN-8298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573865#comment-16573865
]
genericqa commented on YARN-8298:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
5s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 30m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 43s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
5s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
26s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 6s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
7s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
59s{color} | {color:red} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common
generated 1 new + 4189 unchanged - 0 fixed = 4190 total (was 4189) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
20s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 24m 43s{color}
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 12m
27s{color} | {color:green} hadoop-yarn-services-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
51s{color} | {color:green} hadoop-yarn-services-api in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}139m 15s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.client.cli.TestYarnCLI |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | YARN-8298 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12934856/YARN-8298.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
[jira] [Updated] (YARN-8588) Logging improvements for better debuggability
[ https://issues.apache.org/jira/browse/YARN-8588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-8588: - Target Version/s: 3.2.0, 3.1.2 > Logging improvements for better debuggability > - > > Key: YARN-8588 > URL: https://issues.apache.org/jira/browse/YARN-8588 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad >Priority: Major > Attachments: YARN-8588.1.patch, YARN-8588.2.patch > > > Capacity allocations decided in GuaranteedCapacityOvertimePolicy are > available via AutoCreatedLeafQueueConfig. However this class lacks a toString > and some other DEBUG level logs are needed for better debuggability -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8634) Implement yarn cmd for killing all applications in the cluster
[
https://issues.apache.org/jira/browse/YARN-8634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573861#comment-16573861
]
Eric Payne commented on YARN-8634:
--
[~BilwaST], I tam also unsure whether or not this feature is necessary.
Why wouldn't something like the following work:
{code:java}
yarn application -list | \
egrep -v '^Total|Application-Id' | \
sed 's/ .*$//' | \
sort -nr | \
while read an
do
yarn application -kill $an
done
{code}
> Implement yarn cmd for killing all applications in the cluster
> --
>
> Key: YARN-8634
> URL: https://issues.apache.org/jira/browse/YARN-8634
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Bilwa S T
>Assignee: Bilwa S T
>Priority: Major
> Attachments: YARN-8634-001.patch
>
>
> If we want to kill all running applications then currently we need to give
> all application Id's. So instead we can have a command which can kill all
> applications.
> Command would be *yarn application -killall*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8582) Documentation for AM HTTPS Support
[ https://issues.apache.org/jira/browse/YARN-8582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated YARN-8582: Attachment: YARN-8582.001.patch > Documentation for AM HTTPS Support > -- > > Key: YARN-8582 > URL: https://issues.apache.org/jira/browse/YARN-8582 > Project: Hadoop YARN > Issue Type: Sub-task > Components: docs >Reporter: Robert Kanter >Assignee: Robert Kanter >Priority: Major > Attachments: YARN-8582.001.patch > > > Documentation for YARN-6586. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8588) Logging improvements for better debuggability
[
https://issues.apache.org/jira/browse/YARN-8588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573843#comment-16573843
]
genericqa commented on YARN-8588:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 6s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 27s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 64m
25s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}116m 10s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | YARN-8588 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12933972/YARN-8588.2.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 6d9bb11da962 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 9499df7 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/21542/testReport/ |
| Max. process+thread count | 951 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/21542/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT
[jira] [Assigned] (YARN-6456) Allow administrators to set a single ContainerRuntime for all containers
[
https://issues.apache.org/jira/browse/YARN-6456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang reassigned YARN-6456:
---
Assignee: Craig Condit
> Allow administrators to set a single ContainerRuntime for all containers
>
>
> Key: YARN-6456
> URL: https://issues.apache.org/jira/browse/YARN-6456
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Miklos Szegedi
>Assignee: Craig Condit
>Priority: Major
> Labels: Docker
>
>
> With LCE, there are multiple ContainerRuntimes available for handling
> different types of containers; default, docker, java sandbox. Admins should
> have the ability to override the user decision and set a single global
> ContainerRuntime to be used for all containers.
> Original Description:
> {quote}One reason to use Docker containers is to be able to isolate different
> workloads, even, if they run as the same user.
> I have noticed some issues in the current design:
> 1. DockerLinuxContainerRuntime mounts containerLocalDirs
> {{nm-local-dir/usercache/user/appcache/application_1491598755372_0011/}} and
> userLocalDirs {{nm-local-dir/usercache/user/}}, so that a container can see
> and modify the files of another container. I think the application file cache
> directory should be enough for the container to run in most of the cases.
> 2. The whole cgroups directory is mounted. Would the container directory be
> enough?
> 3. There is no way to enforce exclusive use of Docker for all containers.
> There should be an option that it is not the user but the admin that requires
> to use Docker.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-6495) check docker container's exit code when writing to cgroup task files
[
https://issues.apache.org/jira/browse/YARN-6495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang reassigned YARN-6495:
---
Assignee: Jim Brennan (was: Jaeboo Jeong)
> check docker container's exit code when writing to cgroup task files
>
>
> Key: YARN-6495
> URL: https://issues.apache.org/jira/browse/YARN-6495
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Jaeboo Jeong
>Assignee: Jim Brennan
>Priority: Major
> Labels: Docker
> Attachments: YARN-6495.001.patch, YARN-6495.002.patch
>
>
> If I execute simple command like date on docker container, the application
> failed to complete successfully.
> for example,
> {code}
> $ yarn jar
> $HADOOP_HOME/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar
> -shell_env YARN_CONTAINER_RUNTIME_TYPE=docker -shell_env
> YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=hadoop-docker -shell_command "date" -jar
> $HADOOP_HOME/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar
> -num_containers 1 -timeout 360
> …
> 17/04/12 00:16:40 INFO distributedshell.Client: Application did finished
> unsuccessfully. YarnState=FINISHED, DSFinalStatus=FAILED. Breaking monitoring
> loop
> 17/04/12 00:16:40 ERROR distributedshell.Client: Application failed to
> complete successfully
> {code}
> The error log is like below.
> {code}
> ...
> Failed to write pid to file
> /cgroup_parent/cpu/hadoop-yarn/container_/tasks - No such process
> ...
> {code}
> When writing pid to cgroup tasks, container-executor doesn’t check docker
> container’s status.
> If the container finished very quickly, we can’t write pid to cgroup tasks,
> and it is not problem.
> So container-executor needs to check docker container’s exit code during
> writing pid to cgroup tasks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8298) Yarn Service Upgrade: Support express upgrade of a service
[
https://issues.apache.org/jira/browse/YARN-8298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573812#comment-16573812
]
genericqa commented on YARN-8298:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
29s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 28m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 25s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
31s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
31s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 16s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
5s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
48s{color} | {color:red} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common
generated 1 new + 4189 unchanged - 0 fixed = 4190 total (was 4189) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
15s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 25m 8s{color}
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 12m
25s{color} | {color:green} hadoop-yarn-services-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
47s{color} | {color:green} hadoop-yarn-services-api in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
45s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}136m 55s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.client.cli.TestYarnCLI |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | YARN-8298 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12934850/YARN-8298.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
[jira] [Commented] (YARN-8298) Yarn Service Upgrade: Support express upgrade of a service
[ https://issues.apache.org/jira/browse/YARN-8298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573775#comment-16573775 ] Eric Yang commented on YARN-8298: - [~suma.shivaprasad] Good point, I will remove the unused method. > Yarn Service Upgrade: Support express upgrade of a service > -- > > Key: YARN-8298 > URL: https://issues.apache.org/jira/browse/YARN-8298 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.1.1 >Reporter: Chandni Singh >Assignee: Eric Yang >Priority: Major > Attachments: YARN-8298.001.patch, YARN-8298.002.patch > > > Currently service upgrade involves 2 steps > * initiate upgrade by providing new spec > * trigger upgrade of each instance/component > > We need to add the ability to upgrade the service in one shot: > # Aborting the upgrade will not be supported > # Upgrade finalization will be done automatically. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8298) Yarn Service Upgrade: Support express upgrade of a service
[ https://issues.apache.org/jira/browse/YARN-8298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573773#comment-16573773 ] Eric Yang commented on YARN-8298: - [~csingh] Good point, I am accidentally depending on UPGRADING_AUTO_FINALIZE to trigger express upgrade. That part doesn't need to change. I will include your suggestion in the next patch. > Yarn Service Upgrade: Support express upgrade of a service > -- > > Key: YARN-8298 > URL: https://issues.apache.org/jira/browse/YARN-8298 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.1.1 >Reporter: Chandni Singh >Assignee: Eric Yang >Priority: Major > Attachments: YARN-8298.001.patch, YARN-8298.002.patch > > > Currently service upgrade involves 2 steps > * initiate upgrade by providing new spec > * trigger upgrade of each instance/component > > We need to add the ability to upgrade the service in one shot: > # Aborting the upgrade will not be supported > # Upgrade finalization will be done automatically. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8298) Yarn Service Upgrade: Support express upgrade of a service
[
https://issues.apache.org/jira/browse/YARN-8298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573712#comment-16573712
]
Chandni Singh edited comment on YARN-8298 at 8/8/18 7:31 PM:
-
[~eyang] Thanks for working on this.
Why does {{ServiceState.UPGRADING_AUTO_FINALIZE}} trigger express upgrade?
Please refer to the design document in YARN-7512.
{{ServiceState.UPGRADING_AUTO_FINALIZE}} is to trigger automatic finalization
of service upgrade. It isn't specific to express upgrade of the service.
It will be useful to not club ServiceState.UPGRADING_AUTO_FINALIZE with express
upgrade. They have different meanings. A user may want auto-finalization
without express upgrade. There isn't any technical complexity in supporting
both independently as well.
If you require a flag to trigger express upgrade, why not add a query param to
update service endpoint?
If the ServiceState = {{ServiceState.UPGRADING}} or
{{ServiceState.UPGRADING_AUTO_FINALIZE}} and
the query param expressUpgrade = true, then perform express upgrade.
was (Author: csingh):
[~eyang] Thanks for working on this.
Why does {{ServiceState.UPGRADING_AUTO_FINALIZE}} trigger express upgrade?
Please refer to the design document in YARN-7512.
{{ServiceState.UPGRADING_AUTO_FINALIZE}} is to trigger automatic finalization
of service upgrade. It isn't for express upgrade of the service.
It will be useful to not club ServiceState.UPGRADING_AUTO_FINALIZE with express
upgrade. They have different meanings. A user may want auto-finalization
without express upgrade. There isn't any technical complexity in supporting
both independently as well.
If you require a flag to trigger express upgrade, why not add a query param to
update service endpoint?
If the ServiceState = {{ServiceState.UPGRADING}} or
{{ServiceState.UPGRADING_AUTO_FINALIZE}} and
the query param expressUpgrade = true, then perform express upgrade.
> Yarn Service Upgrade: Support express upgrade of a service
> --
>
> Key: YARN-8298
> URL: https://issues.apache.org/jira/browse/YARN-8298
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 3.1.1
>Reporter: Chandni Singh
>Assignee: Eric Yang
>Priority: Major
> Attachments: YARN-8298.001.patch, YARN-8298.002.patch
>
>
> Currently service upgrade involves 2 steps
> * initiate upgrade by providing new spec
> * trigger upgrade of each instance/component
>
> We need to add the ability to upgrade the service in one shot:
> # Aborting the upgrade will not be supported
> # Upgrade finalization will be done automatically.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8520) Document best practice for user management
[
https://issues.apache.org/jira/browse/YARN-8520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573743#comment-16573743
]
genericqa commented on YARN-8520:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
36m 45s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 4s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 49m 18s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | YARN-8520 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12934857/YARN-8520.004.patch |
| Optional Tests | asflicense mvnsite |
| uname | Linux 940ce57c905c 4.4.0-130-generic #156-Ubuntu SMP Thu Jun 14
08:53:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 9499df7 |
| maven | version: Apache Maven 3.3.9 |
| Max. process+thread count | 408 (vs. ulimit of 1) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/21541/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Document best practice for user management
> --
>
> Key: YARN-8520
> URL: https://issues.apache.org/jira/browse/YARN-8520
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: documentation, yarn
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Labels: Docker
> Attachments: YARN-8520.001.patch, YARN-8520.002.patch,
> YARN-8520.003.patch, YARN-8520.004.patch
>
>
> Docker container must have consistent username and groups with host operating
> system when external mount points are exposed to docker container. This
> prevents malicious or unauthorized impersonation to occur. This task is to
> document the best practice to ensure user and group membership are consistent
> across docker containers.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8298) Yarn Service Upgrade: Support express upgrade of a service
[ https://issues.apache.org/jira/browse/YARN-8298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chandni Singh updated YARN-8298: Description: Currently service upgrade involves 2 steps * initiate upgrade by providing new spec * trigger upgrade of each instance/component We need to add the ability to upgrade the service in one shot: # Aborting the upgrade will not be supported # Upgrade finalization will be done automatically. was: Currently service upgrade involves 2 steps * initiate upgrade by providing new spec * trigger upgrade of each instance/component We need to add the ability to upgrade a component in shot which accepts the spec of the component. However there are couple of limitations when upgrading in this way: # Aborting the upgrade will not be supported # Upgrade finalization will be done automatically. > Yarn Service Upgrade: Support express upgrade of a service > -- > > Key: YARN-8298 > URL: https://issues.apache.org/jira/browse/YARN-8298 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.1.1 >Reporter: Chandni Singh >Assignee: Eric Yang >Priority: Major > Attachments: YARN-8298.001.patch, YARN-8298.002.patch > > > Currently service upgrade involves 2 steps > * initiate upgrade by providing new spec > * trigger upgrade of each instance/component > > We need to add the ability to upgrade the service in one shot: > # Aborting the upgrade will not be supported > # Upgrade finalization will be done automatically. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8298) Yarn Service Upgrade: Support express upgrade of a service
[
https://issues.apache.org/jira/browse/YARN-8298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573712#comment-16573712
]
Chandni Singh edited comment on YARN-8298 at 8/8/18 7:03 PM:
-
[~eyang] Thanks for working on this.
Why does {{ServiceState.UPGRADING_AUTO_FINALIZE}} trigger express upgrade?
Please refer to the design document in YARN-7512.
{{ServiceState.UPGRADING_AUTO_FINALIZE}} is to trigger automatic finalization
of service upgrade. It isn't for express upgrade of the service.
It will be useful to not club ServiceState.UPGRADING_AUTO_FINALIZE with express
upgrade. They have different meanings. A user may want auto-finalization
without express upgrade. There isn't any technical complexity in supporting
both independently as well.
If you require a flag to trigger express upgrade, why not add a query param to
update service endpoint?
If the ServiceState = {{ServiceState.UPGRADING}} or
{{ServiceState.UPGRADING_AUTO_FINALIZE}} and
the query param expressUpgrade = true, then perform express upgrade.
was (Author: csingh):
[~eyang] Thanks for working on this.
Why does {{ServiceState.UPGRADING_AUTO_FINALIZE}} trigger express upgrade?
Please refer to the design document in YARN-7512.
{{ServiceState.UPGRADING_AUTO_FINALIZE}} is to trigger automatic finalization
of service upgrade. It isn't for express upgrade of the service.
It will be useful to not club ServiceState.UPGRADING_AUTO_FINALIZE with express
upgrade. They have different meanings. A user may want auto-finalization
without express upgrade. There isn't any complexity in supporting this
technically as will
If you require a flag to trigger express upgrade, why not add a query param to
update service endpoint?
If the ServiceState = {{ServiceState.UPGRADING}} or
{{ServiceState.UPGRADING_AUTO_FINALIZE}} and
the query param expressUpgrade = true, then perform express upgrade.
> Yarn Service Upgrade: Support express upgrade of a service
> --
>
> Key: YARN-8298
> URL: https://issues.apache.org/jira/browse/YARN-8298
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 3.1.1
>Reporter: Chandni Singh
>Assignee: Eric Yang
>Priority: Major
> Attachments: YARN-8298.001.patch, YARN-8298.002.patch
>
>
> Currently service upgrade involves 2 steps
> * initiate upgrade by providing new spec
> * trigger upgrade of each instance/component
>
> We need to add the ability to upgrade a component in shot which accepts the
> spec of the component. However there are couple of limitations when upgrading
> in this way:
> # Aborting the upgrade will not be supported
> # Upgrade finalization will be done automatically.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8298) Yarn Service Upgrade: Support express upgrade of a service
[
https://issues.apache.org/jira/browse/YARN-8298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573712#comment-16573712
]
Chandni Singh edited comment on YARN-8298 at 8/8/18 7:01 PM:
-
[~eyang] Thanks for working on this.
Why does {{ServiceState.UPGRADING_AUTO_FINALIZE}} trigger express upgrade?
Please refer to the design document in YARN-7512.
{{ServiceState.UPGRADING_AUTO_FINALIZE}} is to trigger automatic finalization
of service upgrade. It isn't for express upgrade of the service.
It will be useful to not club ServiceState.UPGRADING_AUTO_FINALIZE with express
upgrade. They have different meanings. A user may want auto-finalization
without express upgrade. There isn't any complexity in supporting this
technically as will
If you require a flag to trigger express upgrade, why not add a query param to
update service endpoint?
If the ServiceState = {{ServiceState.UPGRADING}} or
{{ServiceState.UPGRADING_AUTO_FINALIZE}} and
the query param expressUpgrade = true, then perform express upgrade.
was (Author: csingh):
[~eyang]
Why does {{ServiceState.UPGRADING_AUTO_FINALIZE}} trigger express upgrade?
Please refer to the design document in YARN-7512.
{{ServiceState.UPGRADING_AUTO_FINALIZE}} is to trigger automatic finalization
of service upgrade. It isn't for express upgrade of the service.
It will be useful to not club ServiceState.UPGRADING_AUTO_FINALIZE with express
upgrade. They have different meanings. A user may want auto-finalization
without express upgrade. There isn't any complexity in supporting this
technically as will
If you require a flag to trigger express upgrade, why not add a query param to
update service endpoint?
If the ServiceState = {{ServiceState.UPGRADING}} or
{{ServiceState.UPGRADING_AUTO_FINALIZE}} and
the query param expressUpgrade = true, then perform express upgrade.
> Yarn Service Upgrade: Support express upgrade of a service
> --
>
> Key: YARN-8298
> URL: https://issues.apache.org/jira/browse/YARN-8298
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 3.1.1
>Reporter: Chandni Singh
>Assignee: Eric Yang
>Priority: Major
> Attachments: YARN-8298.001.patch, YARN-8298.002.patch
>
>
> Currently service upgrade involves 2 steps
> * initiate upgrade by providing new spec
> * trigger upgrade of each instance/component
>
> We need to add the ability to upgrade a component in shot which accepts the
> spec of the component. However there are couple of limitations when upgrading
> in this way:
> # Aborting the upgrade will not be supported
> # Upgrade finalization will be done automatically.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8298) Yarn Service Upgrade: Support express upgrade of a service
[
https://issues.apache.org/jira/browse/YARN-8298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573712#comment-16573712
]
Chandni Singh commented on YARN-8298:
-
[~eyang]
Why does {{ServiceState.UPGRADING_AUTO_FINALIZE}} trigger express upgrade?
Please refer to the design document in YARN-7512.
{{ServiceState.UPGRADING_AUTO_FINALIZE}} is to trigger automatic finalization
of service upgrade. It isn't for express upgrade of the service.
It will be useful to not club ServiceState.UPGRADING_AUTO_FINALIZE with express
upgrade. They have different meanings. A user may want auto-finalization
without express upgrade. There isn't any complexity in supporting this
technically as will
If you require a flag to trigger express upgrade, why not add a query param to
update service endpoint?
If the ServiceState = {{ServiceState.UPGRADING}} or
{{ServiceState.UPGRADING_AUTO_FINALIZE}} and
the query param expressUpgrade = true, then perform express upgrade.
> Yarn Service Upgrade: Support express upgrade of a service
> --
>
> Key: YARN-8298
> URL: https://issues.apache.org/jira/browse/YARN-8298
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 3.1.1
>Reporter: Chandni Singh
>Assignee: Eric Yang
>Priority: Major
> Attachments: YARN-8298.001.patch, YARN-8298.002.patch
>
>
> Currently service upgrade involves 2 steps
> * initiate upgrade by providing new spec
> * trigger upgrade of each instance/component
>
> We need to add the ability to upgrade a component in shot which accepts the
> spec of the component. However there are couple of limitations when upgrading
> in this way:
> # Aborting the upgrade will not be supported
> # Upgrade finalization will be done automatically.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8298) Yarn Service Upgrade: Support express upgrade of a service
[ https://issues.apache.org/jira/browse/YARN-8298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chandni Singh updated YARN-8298: Summary: Yarn Service Upgrade: Support express upgrade of a service (was: Yarn Service Upgrade: Support fast component upgrades which accepts component spec) > Yarn Service Upgrade: Support express upgrade of a service > -- > > Key: YARN-8298 > URL: https://issues.apache.org/jira/browse/YARN-8298 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.1.1 >Reporter: Chandni Singh >Assignee: Chandni Singh >Priority: Major > Attachments: YARN-8298.001.patch, YARN-8298.002.patch > > > Currently service upgrade involves 2 steps > * initiate upgrade by providing new spec > * trigger upgrade of each instance/component > > We need to add the ability to upgrade a component in shot which accepts the > spec of the component. However there are couple of limitations when upgrading > in this way: > # Aborting the upgrade will not be supported > # Upgrade finalization will be done automatically. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-8298) Yarn Service Upgrade: Support express upgrade of a service
[ https://issues.apache.org/jira/browse/YARN-8298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chandni Singh reassigned YARN-8298: --- Assignee: Eric Yang (was: Chandni Singh) > Yarn Service Upgrade: Support express upgrade of a service > -- > > Key: YARN-8298 > URL: https://issues.apache.org/jira/browse/YARN-8298 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.1.1 >Reporter: Chandni Singh >Assignee: Eric Yang >Priority: Major > Attachments: YARN-8298.001.patch, YARN-8298.002.patch > > > Currently service upgrade involves 2 steps > * initiate upgrade by providing new spec > * trigger upgrade of each instance/component > > We need to add the ability to upgrade a component in shot which accepts the > spec of the component. However there are couple of limitations when upgrading > in this way: > # Aborting the upgrade will not be supported > # Upgrade finalization will be done automatically. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8298) Yarn Service Upgrade: Support fast component upgrades which accepts component spec
[ https://issues.apache.org/jira/browse/YARN-8298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573685#comment-16573685 ] Suma Shivaprasad commented on YARN-8298: Thanks for the patch [~eyang] validateAndResolveCompsStable does not seem to be used in the code - Should it be replaced with validateAndResolveCompsUpgrade?. Is there a reason why we want to allow only READY instances to be upgraded? > Yarn Service Upgrade: Support fast component upgrades which accepts component > spec > -- > > Key: YARN-8298 > URL: https://issues.apache.org/jira/browse/YARN-8298 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.1.1 >Reporter: Chandni Singh >Assignee: Chandni Singh >Priority: Major > Attachments: YARN-8298.001.patch, YARN-8298.002.patch > > > Currently service upgrade involves 2 steps > * initiate upgrade by providing new spec > * trigger upgrade of each instance/component > > We need to add the ability to upgrade a component in shot which accepts the > spec of the component. However there are couple of limitations when upgrading > in this way: > # Aborting the upgrade will not be supported > # Upgrade finalization will be done automatically. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8520) Document best practice for user management
[ https://issues.apache.org/jira/browse/YARN-8520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-8520: Attachment: YARN-8520.004.patch > Document best practice for user management > -- > > Key: YARN-8520 > URL: https://issues.apache.org/jira/browse/YARN-8520 > Project: Hadoop YARN > Issue Type: Sub-task > Components: documentation, yarn >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Labels: Docker > Attachments: YARN-8520.001.patch, YARN-8520.002.patch, > YARN-8520.003.patch, YARN-8520.004.patch > > > Docker container must have consistent username and groups with host operating > system when external mount points are exposed to docker container. This > prevents malicious or unauthorized impersonation to occur. This task is to > document the best practice to ensure user and group membership are consistent > across docker containers. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8298) Yarn Service Upgrade: Support fast component upgrades which accepts component spec
[ https://issues.apache.org/jira/browse/YARN-8298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573629#comment-16573629 ] Eric Yang commented on YARN-8298: - Patch 2 added test case for dependency resolver. > Yarn Service Upgrade: Support fast component upgrades which accepts component > spec > -- > > Key: YARN-8298 > URL: https://issues.apache.org/jira/browse/YARN-8298 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.1.1 >Reporter: Chandni Singh >Assignee: Chandni Singh >Priority: Major > Attachments: YARN-8298.001.patch, YARN-8298.002.patch > > > Currently service upgrade involves 2 steps > * initiate upgrade by providing new spec > * trigger upgrade of each instance/component > > We need to add the ability to upgrade a component in shot which accepts the > spec of the component. However there are couple of limitations when upgrading > in this way: > # Aborting the upgrade will not be supported > # Upgrade finalization will be done automatically. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8298) Yarn Service Upgrade: Support fast component upgrades which accepts component spec
[ https://issues.apache.org/jira/browse/YARN-8298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-8298: Attachment: YARN-8298.002.patch > Yarn Service Upgrade: Support fast component upgrades which accepts component > spec > -- > > Key: YARN-8298 > URL: https://issues.apache.org/jira/browse/YARN-8298 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.1.1 >Reporter: Chandni Singh >Assignee: Chandni Singh >Priority: Major > Attachments: YARN-8298.001.patch, YARN-8298.002.patch > > > Currently service upgrade involves 2 steps > * initiate upgrade by providing new spec > * trigger upgrade of each instance/component > > We need to add the ability to upgrade a component in shot which accepts the > spec of the component. However there are couple of limitations when upgrading > in this way: > # Aborting the upgrade will not be supported > # Upgrade finalization will be done automatically. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8549) Adding a NoOp timeline writer and reader plugin classes for ATSv2
[ https://issues.apache.org/jira/browse/YARN-8549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573599#comment-16573599 ] Suma Shivaprasad commented on YARN-8549: Thanks for the patch [~prabham]. Can we replace "return null" to return an empty TimeLineResponse in NoOpTimelineWriterImpl and fix checkstyle/whitespace issues. > Adding a NoOp timeline writer and reader plugin classes for ATSv2 > - > > Key: YARN-8549 > URL: https://issues.apache.org/jira/browse/YARN-8549 > Project: Hadoop YARN > Issue Type: Sub-task > Components: ATSv2, timelineclient, timelineserver >Reporter: Prabha Manepalli >Assignee: Prabha Manepalli >Priority: Minor > Attachments: YARN-8549-branch-2.03.patch, > YARN-8549-branch-2.04.patch, YARN-8549.v1.patch, YARN-8549.v2.patch > > > Stub implementation for TimeLineReader and TimeLineWriter classes. > These are useful for functional testing of writer and reader path for ATSv2 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8298) Yarn Service Upgrade: Support fast component upgrades which accepts component spec
[ https://issues.apache.org/jira/browse/YARN-8298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-8298: Attachment: YARN-8298.001.patch > Yarn Service Upgrade: Support fast component upgrades which accepts component > spec > -- > > Key: YARN-8298 > URL: https://issues.apache.org/jira/browse/YARN-8298 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Chandni Singh >Assignee: Chandni Singh >Priority: Major > Attachments: YARN-8298.001.patch > > > Currently service upgrade involves 2 steps > * initiate upgrade by providing new spec > * trigger upgrade of each instance/component > > We need to add the ability to upgrade a component in shot which accepts the > spec of the component. However there are couple of limitations when upgrading > in this way: > # Aborting the upgrade will not be supported > # Upgrade finalization will be done automatically. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8601) Print ExecutionType in Container report CLI
[ https://issues.apache.org/jira/browse/YARN-8601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573559#comment-16573559 ] Hudson commented on YARN-8601: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14728 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14728/]) YARN-8601. Print ExecutionType in Container report CLI. Contributed by (bibinchundatt: rev ff06bd1be83a2a6d2ee39cb002e91499720a7243) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestYarnCLI.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/ApplicationCLI.java > Print ExecutionType in Container report CLI > --- > > Key: YARN-8601 > URL: https://issues.apache.org/jira/browse/YARN-8601 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Bilwa S T >Assignee: Bilwa S T >Priority: Minor > Fix For: 3.2.0, 3.1.2 > > Attachments: YARN-8601-001.patch > > > Currently in container status CLI execution type is not printed. so it can be > added . -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8623) Update Docker examples to use image which exists
[ https://issues.apache.org/jira/browse/YARN-8623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573536#comment-16573536 ] Eric Yang commented on YARN-8623: - For the unknown uid 1004 issue, the docker image might want to configure to use host level user to enforce user security. For more details of how to accomplish this, see: YARN-8520 > Update Docker examples to use image which exists > > > Key: YARN-8623 > URL: https://issues.apache.org/jira/browse/YARN-8623 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Craig Condit >Priority: Minor > Labels: Docker > > The example Docker image given in the documentation > (images/hadoop-docker:latest) does not exist. We could change > images/hadoop-docker:latest to apache/hadoop-runner:latest, which does exist. > We'd need to do a quick sanity test to see if the image works with YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8623) Update Docker examples to use image which exists
[
https://issues.apache.org/jira/browse/YARN-8623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573529#comment-16573529
]
Eric Yang commented on YARN-8623:
-
[~ccondit-target] The observation is a compounded issues of incomplete setup.
YARN added its own security check to make sure that only a real sudoer can run
privileged container. In order to run privileged container,
container-executor.cfg must contain setting to allow image from Apache
repository can run as privileged container:
{code}
docker.trusted.registries=apache
{code}
The launching user must also be a real sudoer in the host level to allow the
privileged container from launching. To enable ENTRYPOINT, user must pass in
environment variable: YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE=true
for apache/hadoop-runner to run as intended. This is probably too much trouble
for first time trying the software. This can be simplified to use
centos:latest image for illustration purpose.
> Update Docker examples to use image which exists
>
>
> Key: YARN-8623
> URL: https://issues.apache.org/jira/browse/YARN-8623
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Craig Condit
>Priority: Minor
> Labels: Docker
>
> The example Docker image given in the documentation
> (images/hadoop-docker:latest) does not exist. We could change
> images/hadoop-docker:latest to apache/hadoop-runner:latest, which does exist.
> We'd need to do a quick sanity test to see if the image works with YARN.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8623) Update Docker examples to use image which exists
[
https://issues.apache.org/jira/browse/YARN-8623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573478#comment-16573478
]
Craig Condit commented on YARN-8623:
I've been looking into this one a bit, and unfortunately,
apache/hadoop-runner:latest doesn't seem to be a viable option. That image
contains an ENTRYPOINT which attempts to sudo (without a password), resulting
in the container failing to run when the launching UID != 1000 (the hadoop user
within the container). This can be seen via docker logs as:
{code:java}
sudo: unknown uid 1004: who are you?{code}
I see two potential paths forward:
# Update the documentation to be more generic (maybe your-docker-image:latest
instead of images/hadoop-docker:latest), and refer to the existing
documentation on preparing an image for use with Hadoop.
# Create (and maintain going forward) an official Apache Hadoop docker image
pre-configured to work out of the box, or at least as a starting point for
user-created images.
[~shaneku...@gmail.com], thoughts?
> Update Docker examples to use image which exists
>
>
> Key: YARN-8623
> URL: https://issues.apache.org/jira/browse/YARN-8623
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Craig Condit
>Priority: Minor
> Labels: Docker
>
> The example Docker image given in the documentation
> (images/hadoop-docker:latest) does not exist. We could change
> images/hadoop-docker:latest to apache/hadoop-runner:latest, which does exist.
> We'd need to do a quick sanity test to see if the image works with YARN.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8623) Update Docker examples to use image which exists
[ https://issues.apache.org/jira/browse/YARN-8623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Craig Condit updated YARN-8623: --- Description: The example Docker image given in the documentation (images/hadoop-docker:latest) does not exist. We could change images/hadoop-docker:latest to apache/hadoop-runner:latest, which does exist. We'd need to do a quick sanity test to see if the image works with YARN. (was: The example Docker image given in the documentation (images/hadoop-docker:latest) does not exist. We could change images/hadoop-docker:latest to apache/hadoop-runner:trunk, which does exist. We'd need to do a quick sanity test to see if the image works with YARN.) > Update Docker examples to use image which exists > > > Key: YARN-8623 > URL: https://issues.apache.org/jira/browse/YARN-8623 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Craig Condit >Priority: Minor > Labels: Docker > > The example Docker image given in the documentation > (images/hadoop-docker:latest) does not exist. We could change > images/hadoop-docker:latest to apache/hadoop-runner:latest, which does exist. > We'd need to do a quick sanity test to see if the image works with YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8331) Race condition in NM container launched after done
[
https://issues.apache.org/jira/browse/YARN-8331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573432#comment-16573432
]
Jason Lowe commented on YARN-8331:
--
Thanks for the patch, [~pradeepambati]!
In ContainersLauncher the exit code of a container that never launched should
not be SUCCESS. It should be treated the same as it is done in
ContainerLaunch#validateContainerState when a container is killed before it is
launched, and the error message should match as well since it's the same type
of scenario.
Now that ContainersLauncher is sending the killed on request event even when it
doesn't think the container is being launched or running at all, do we need to
add some event handling for CONTAINER_KILLED_ON_REQUEST to more states
following KILLING? I'm thinking it is now possible for ContainerImpl to
receive multiple of these which the state machine does not expect. For example
consider the following scenario:
# Container in SCHEDULED state waiting for LAUNCHED event
# Container receives KILL event and sends kill to launcher
# In the meantime container proceeds to launch a very quick process and
LAUNCHED is sent to the container. Event is ignored because the container is
in KILLING state.
# Process runs extremely fast, so launcher sends CONTAINER_EXITED_WITH_SUCCESS
event to the container and clears container from ContainersLauncher.
# Container receives exit success event in the KILLING state and moves to the
EXITED_WITH_SUCCESS state.
# Launcher now finally receives the request to kill the container, but the
container is no longer being tracked. KILLED_ON_REQUEST is sent to the
container as part of this patch.
# Container receives KILLED_ON_REQUEST event in the EXITED_WITH_SUCCESS state
which is an invalid transition. Similarly if it had progressed to the DONE
state it also would be an invalid state transition.
I think we need to add some no-op transitions in states that can occur after
KILLING if KILLED_ON_REQUEST is received to cover these race conditions.
The event field of LocalizationCleanupMatcher is not used and is not needed.
Checking for an instance of ContainerLocalizationCleanupEvent means we are also
verifying the event type is CLEANUP_CONTAINER_RESOURCES. Alternatively
LocalizationCleanupMatcher could check for instanceof
ContainerLocalizationEvent and verify the event is CLEANUP_CONTAINER_RESOURCES.
> Race condition in NM container launched after done
> --
>
> Key: YARN-8331
> URL: https://issues.apache.org/jira/browse/YARN-8331
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.1.0, 2.9.1, 3.0.2
>Reporter: Yang Wang
>Assignee: Pradeep Ambati
>Priority: Major
> Attachments: YARN-8331.001.patch
>
>
> When a container is launching, in ContainerLaunch#launchContainer, state is
> SCHEDULED,
> kill event was sent to this container, state : SCHEDULED->KILLING->DONE
> Then ContainerLaunch send CONTAINER_LAUNCHED event and start the container
> processes. These absent container processes will not be cleaned up anymore.
>
> {code:java}
> 2018-05-21 13:11:56,114 INFO [Thread-11] nodemanager.NMAuditLogger
> (NMAuditLogger.java:logSuccess(94)) - USER=nobody OPERATION=Start Container
> Request TARGET=ContainerManageImpl RESULT=SUCCESS
> APPID=application_0_CONTAINERID=container_0__01_00
> 2018-05-21 13:11:56,114 INFO [NM ContainerManager dispatcher]
> application.ApplicationImpl (ApplicationImpl.java:handle(632)) - Application
> application_0_ transitioned from NEW to INITING
> 2018-05-21 13:11:56,114 INFO [NM ContainerManager dispatcher]
> application.ApplicationImpl (ApplicationImpl.java:transition(446)) - Adding
> container_0__01_00 to application application_0_
> 2018-05-21 13:11:56,118 INFO [NM ContainerManager dispatcher]
> application.ApplicationImpl (ApplicationImpl.java:handle(632)) - Application
> application_0_ transitioned from INITING to RUNNING
> 2018-05-21 13:11:56,119 INFO [NM ContainerManager dispatcher]
> container.ContainerImpl (ContainerImpl.java:handle(2111)) - Container
> container_0__01_00 transitioned from NEW to SCHEDULED
> 2018-05-21 13:11:56,119 INFO [NM ContainerManager dispatcher]
> containermanager.AuxServices (AuxServices.java:handle(220)) - Got event
> CONTAINER_INIT for appId application_0_
> 2018-05-21 13:11:56,119 INFO [NM ContainerManager dispatcher]
> scheduler.ContainerScheduler (ContainerScheduler.java:startContainer(504)) -
> Starting container [container_0__01_00]
> 2018-05-21 13:11:56,226 INFO [NM ContainerManager dispatcher]
> container.ContainerImpl (ContainerImpl.java:handle(2111)) - Container
> container_0__01_00 transitioned from SCHEDULED to KILLING
> 2018-05-21 13:11:56,227 INFO [NM
[jira] [Commented] (YARN-8633) Update DataTables version in yarn-common in line with JQuery 3 upgrade
[ https://issues.apache.org/jira/browse/YARN-8633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573329#comment-16573329 ] Hudson commented on YARN-8633: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14726 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14726/]) Revert "YARN-8633. Update DataTables version in yarn-common in line with (sunilg: rev 5b898c176ffb41e6fa3c605feb8ed3fcb60a5fe8) * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.10.7/images/sort_both.png * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.9.4/images/back_enabled.jpg * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.10.7/css/demo_page.css * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.9.4/images/sort_both.png * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.10.7/images/sort_desc.png * (edit) LICENSE.txt * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.9.4/images/sort_desc_disabled.png * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.10.7/css/demo_table.css * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.10.7/images/forward_enabled.jpg * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.9.4/images/back_disabled.jpg * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.9.4/images/forward_enabled.jpg * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/view/JQueryUI.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.9.4/images/forward_disabled.jpg * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.9.4/images/sort_desc.png * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.10.7/images/sort_desc_disabled.png * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.9.4/js/jquery.dataTables.min.js * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.9.4/images/Sorting icons.psd * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.10.7/images/Sorting icons.psd * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.10.7/css/jui-dt.css * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.10.7/js/jquery.dataTables.min.js * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.10.7/images/sort_asc_disabled.png * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.9.4/images/sort_asc_disabled.png * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.10.7/images/back_disabled.jpg * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/pom.xml * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.10.7/images/sort_asc.png * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.9.4/css/jui-dt.css * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.9.4/images/favicon.ico * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.10.7/images/forward_disabled.jpg * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.9.4/images/sort_asc.png * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.9.4/css/demo_page.css * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.9.4/css/demo_table.css * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.10.7/images/back_enabled.jpg * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.10.7/images/favicon.ico > Update DataTables version in yarn-common in line with JQuery 3 upgrade > -- > > Key: YARN-8633 > URL: https://issues.apache.org/jira/browse/YARN-8633 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Akhil PB >Assignee: Akhil PB >Priority: Major >
[jira] [Commented] (YARN-8633) Update DataTables version in yarn-common in line with JQuery 3 upgrade
[ https://issues.apache.org/jira/browse/YARN-8633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573294#comment-16573294 ] Hudson commented on YARN-8633: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14725 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14725/]) YARN-8633. Update DataTables version in yarn-common in line with JQuery (sunilg: rev 64901abdfac72c22f6b002ff45b1107174e82207) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/view/JQueryUI.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.10.7/css/demo_page.css * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.10.7/images/back_enabled.jpg * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.9.4/images/favicon.ico * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.9.4/images/sort_asc.png * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.10.7/images/sort_both.png * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.10.7/images/sort_desc.png * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.10.7/images/Sorting icons.psd * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.10.7/images/sort_asc_disabled.png * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.9.4/images/forward_disabled.jpg * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.9.4/images/Sorting icons.psd * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.10.7/js/jquery.dataTables.min.js * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.9.4/js/jquery.dataTables.min.js * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.9.4/images/back_enabled.jpg * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.9.4/images/sort_both.png * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/pom.xml * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.10.7/css/demo_table.css * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.10.7/images/back_disabled.jpg * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.9.4/css/demo_page.css * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.9.4/images/sort_desc.png * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.9.4/images/back_disabled.jpg * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.10.7/images/favicon.ico * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.10.7/images/forward_enabled.jpg * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.10.7/css/jui-dt.css * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.9.4/images/sort_desc_disabled.png * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.10.7/images/forward_disabled.jpg * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.9.4/images/sort_asc_disabled.png * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.10.7/images/sort_desc_disabled.png * (edit) LICENSE.txt * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.10.7/images/sort_asc.png * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.9.4/images/forward_enabled.jpg * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.9.4/css/demo_table.css * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.9.4/css/jui-dt.css > Update DataTables version in yarn-common in line with JQuery 3 upgrade > -- > > Key: YARN-8633 > URL: https://issues.apache.org/jira/browse/YARN-8633 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Akhil PB >Assignee: Akhil PB >Priority: Major >
[jira] [Commented] (YARN-8633) Update DataTables version in yarn-common in line with JQuery 3 upgrade
[ https://issues.apache.org/jira/browse/YARN-8633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573283#comment-16573283 ] Sunil Govindan commented on YARN-8633: -- [~msingh] cud u pls help to see whether this test failures are related? > Update DataTables version in yarn-common in line with JQuery 3 upgrade > -- > > Key: YARN-8633 > URL: https://issues.apache.org/jira/browse/YARN-8633 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Akhil PB >Assignee: Akhil PB >Priority: Major > Attachments: YARN-8633.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8633) Update DataTables version in yarn-common in line with JQuery 3 upgrade
[ https://issues.apache.org/jira/browse/YARN-8633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573281#comment-16573281 ] Sunil Govindan commented on YARN-8633: -- Accidentally pushed this. I have reverted now. [~akhilpb], i thought the test case were not failing locally. But in my linux machine, its failing still. Cud u pls check. Thank you > Update DataTables version in yarn-common in line with JQuery 3 upgrade > -- > > Key: YARN-8633 > URL: https://issues.apache.org/jira/browse/YARN-8633 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Akhil PB >Assignee: Akhil PB >Priority: Major > Attachments: YARN-8633.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8633) Update DataTables version in yarn-common in line with JQuery 3 upgrade
[ https://issues.apache.org/jira/browse/YARN-8633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated YARN-8633: - Summary: Update DataTables version in yarn-common in line with JQuery 3 upgrade (was: Update JQuery version references in yarn-common) > Update DataTables version in yarn-common in line with JQuery 3 upgrade > -- > > Key: YARN-8633 > URL: https://issues.apache.org/jira/browse/YARN-8633 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Akhil PB >Assignee: Akhil PB >Priority: Major > Attachments: YARN-8633.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8633) Update JQuery version references in yarn-common
[ https://issues.apache.org/jira/browse/YARN-8633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573273#comment-16573273 ] Sunil Govindan commented on YARN-8633: -- Tested this locally. Old UI pages are loading good. Committing shortly. > Update JQuery version references in yarn-common > --- > > Key: YARN-8633 > URL: https://issues.apache.org/jira/browse/YARN-8633 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Akhil PB >Assignee: Akhil PB >Priority: Major > Attachments: YARN-8633.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8609) NM oom because of large container statuses
[ https://issues.apache.org/jira/browse/YARN-8609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573232#comment-16573232 ] Jason Lowe commented on YARN-8609: -- This JIRA does mention all those things, and now it points to YARN-3998 as the fix (I just linked the two JIRAs). If we resolve it as fixed with a patch that only truncates individual diagnostic messages, that will not prevent an OOM if something adds a ton of separate diagnostic messages to a container. It would be a partial fix to the OOM while YARN-3998 is a complete fix. > NM oom because of large container statuses > -- > > Key: YARN-8609 > URL: https://issues.apache.org/jira/browse/YARN-8609 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Reporter: Xianghao Lu >Priority: Major > Attachments: YARN-8609.001.patch, contain_status.jpg, oom.jpeg > > > Sometimes, NodeManger will send large container statuses to ResourceManager > when NodeManger start with recovering, as a result , NodeManger will be > failed to start because of oom. > In my case, the large container statuses size is 135M, which contain 11 > container statuses, and I find the diagnostics of 5 containers are very > large(27M), so, I truncate the container diagnostics as the patch. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8636) TimelineSchemaCreator command not working
[
https://issues.apache.org/jira/browse/YARN-8636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573132#comment-16573132
]
Abhishek Modi commented on YARN-8636:
-
[~bibinchundatt] Yes all headers are lost.
But when I checked in branch-2 code headers are there. One change that I
noticed is the extension. In trunk we are using .md extension while in branch-2
.md.vm extension is being used. Though I am not sure whether that is causing
this.
> TimelineSchemaCreator command not working
> -
>
> Key: YARN-8636
> URL: https://issues.apache.org/jira/browse/YARN-8636
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Abhishek Modi
>Priority: Minor
>
> {code:java}
> nodemanager/bin> ./hadoop
> org.apache.yarn.timelineservice.storage.TimelineSchemaCreator -create
> Error: Could not find or load main class
> org.apache.yarn.timelineservice.storage.TimelineSchemaCreator
> {code}
> share/hadoop/yarn/timelineservice/ is not part of class path
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-4778) Support specifying resources for task containers in SLS
[ https://issues.apache.org/jira/browse/YARN-4778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572670#comment-16572670 ] Ananyo Rao edited comment on YARN-4778 at 8/8/18 11:59 AM: --- [~leftnoteasy] [~jianhe] Currently, resources for task containers are not specified in the json file created from jhist using rumen. Is there a way to do this? We are running hadoop 2.6. Is it available in later versions of hadoop? was (Author: vadagaananyo): [~leftnoteasy] is there a way to acquire and set specific container resources for MR jobs? Currently the json file doesn't hold these information. Could you suggest a way for acquiring them as well? > Support specifying resources for task containers in SLS > --- > > Key: YARN-4778 > URL: https://issues.apache.org/jira/browse/YARN-4778 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Major > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-4778.1.patch > > > Currently, SLS doesn't support specify resources for task containers, it uses > a global default value for all containers. > Instead, we should be able to specify different resources for task containers > in sls-job.conf. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8636) TimelineSchemaCreator command not working
[
https://issues.apache.org/jira/browse/YARN-8636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573073#comment-16573073
]
Bibin A Chundatt edited comment on YARN-8636 at 8/8/18 11:40 AM:
-
[~abmodi]
https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html#Enable_the_coprocessor
Seems all the headers are lost too.. Could you cross check the same.
was (Author: bibinchundatt):
[~abmodi]
https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html#Enable_the_coprocessor
Seems all the headers are lost too.. Just cross check the same.
> TimelineSchemaCreator command not working
> -
>
> Key: YARN-8636
> URL: https://issues.apache.org/jira/browse/YARN-8636
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Abhishek Modi
>Priority: Minor
>
> {code:java}
> nodemanager/bin> ./hadoop
> org.apache.yarn.timelineservice.storage.TimelineSchemaCreator -create
> Error: Could not find or load main class
> org.apache.yarn.timelineservice.storage.TimelineSchemaCreator
> {code}
> share/hadoop/yarn/timelineservice/ is not part of class path
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8636) TimelineSchemaCreator command not working
[
https://issues.apache.org/jira/browse/YARN-8636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573073#comment-16573073
]
Bibin A Chundatt commented on YARN-8636:
[~abmodi]
https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html#Enable_the_coprocessor
Seems all the headers are lost too.. Just cross check the same.
> TimelineSchemaCreator command not working
> -
>
> Key: YARN-8636
> URL: https://issues.apache.org/jira/browse/YARN-8636
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Abhishek Modi
>Priority: Minor
>
> {code:java}
> nodemanager/bin> ./hadoop
> org.apache.yarn.timelineservice.storage.TimelineSchemaCreator -create
> Error: Could not find or load main class
> org.apache.yarn.timelineservice.storage.TimelineSchemaCreator
> {code}
> share/hadoop/yarn/timelineservice/ is not part of class path
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8636) TimelineSchemaCreator command not working
[
https://issues.apache.org/jira/browse/YARN-8636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-8636:
---
Priority: Minor (was: Major)
> TimelineSchemaCreator command not working
> -
>
> Key: YARN-8636
> URL: https://issues.apache.org/jira/browse/YARN-8636
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Abhishek Modi
>Priority: Minor
>
> {code:java}
> nodemanager/bin> ./hadoop
> org.apache.yarn.timelineservice.storage.TimelineSchemaCreator -create
> Error: Could not find or load main class
> org.apache.yarn.timelineservice.storage.TimelineSchemaCreator
> {code}
> share/hadoop/yarn/timelineservice/ is not part of class path
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8636) TimelineSchemaCreator command not working
[
https://issues.apache.org/jira/browse/YARN-8636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573046#comment-16573046
]
Bibin A Chundatt commented on YARN-8636:
Seems YARN-7378 got committed to only 2.9.0
> TimelineSchemaCreator command not working
> -
>
> Key: YARN-8636
> URL: https://issues.apache.org/jira/browse/YARN-8636
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Abhishek Modi
>Priority: Major
>
> {code:java}
> nodemanager/bin> ./hadoop
> org.apache.yarn.timelineservice.storage.TimelineSchemaCreator -create
> Error: Could not find or load main class
> org.apache.yarn.timelineservice.storage.TimelineSchemaCreator
> {code}
> share/hadoop/yarn/timelineservice/ is not part of class path
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8636) TimelineSchemaCreator command not working
[
https://issues.apache.org/jira/browse/YARN-8636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573047#comment-16573047
]
Abhishek Modi commented on YARN-8636:
-
[~bibinchundatt] sorry I missed checking the latest documentation. I think in
this case, we should add timeline service folder to the Hadoop classpath.
[~rohithsharma] or [~vrushalic] can confirm.
> TimelineSchemaCreator command not working
> -
>
> Key: YARN-8636
> URL: https://issues.apache.org/jira/browse/YARN-8636
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Abhishek Modi
>Priority: Major
>
> {code:java}
> nodemanager/bin> ./hadoop
> org.apache.yarn.timelineservice.storage.TimelineSchemaCreator -create
> Error: Could not find or load main class
> org.apache.yarn.timelineservice.storage.TimelineSchemaCreator
> {code}
> share/hadoop/yarn/timelineservice/ is not part of class path
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8636) TimelineSchemaCreator command not working
[
https://issues.apache.org/jira/browse/YARN-8636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573039#comment-16573039
]
Bibin A Chundatt commented on YARN-8636:
[~abmodi]
I referred following documentation
https://hadoop.apache.org/docs/r3.1.0/hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html#Timeline_Service_v.2_REST_API
> TimelineSchemaCreator command not working
> -
>
> Key: YARN-8636
> URL: https://issues.apache.org/jira/browse/YARN-8636
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Abhishek Modi
>Priority: Major
>
> {code:java}
> nodemanager/bin> ./hadoop
> org.apache.yarn.timelineservice.storage.TimelineSchemaCreator -create
> Error: Could not find or load main class
> org.apache.yarn.timelineservice.storage.TimelineSchemaCreator
> {code}
> share/hadoop/yarn/timelineservice/ is not part of class path
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8636) TimelineSchemaCreator command not working
[
https://issues.apache.org/jira/browse/YARN-8636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573032#comment-16573032
]
Abhishek Modi commented on YARN-8636:
-
[~bibinchundatt] As per the docs and my understanding this command needs to be
run with hbase:
bin/hbase
org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator
-create
And the location of hadoop-yarn-server-timelineservice-hbase-2.9.1.jar needs to
be added in classpath of hbase for running this command.
I referred to
[https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html#Enable_the_coprocessor]
and was able to run this command successfully.
> TimelineSchemaCreator command not working
> -
>
> Key: YARN-8636
> URL: https://issues.apache.org/jira/browse/YARN-8636
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Priority: Major
>
> {code:java}
> nodemanager/bin> ./hadoop
> org.apache.yarn.timelineservice.storage.TimelineSchemaCreator -create
> Error: Could not find or load main class
> org.apache.yarn.timelineservice.storage.TimelineSchemaCreator
> {code}
> share/hadoop/yarn/timelineservice/ is not part of class path
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-8636) TimelineSchemaCreator command not working
[
https://issues.apache.org/jira/browse/YARN-8636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Modi reassigned YARN-8636:
---
Assignee: Abhishek Modi
> TimelineSchemaCreator command not working
> -
>
> Key: YARN-8636
> URL: https://issues.apache.org/jira/browse/YARN-8636
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Abhishek Modi
>Priority: Major
>
> {code:java}
> nodemanager/bin> ./hadoop
> org.apache.yarn.timelineservice.storage.TimelineSchemaCreator -create
> Error: Could not find or load main class
> org.apache.yarn.timelineservice.storage.TimelineSchemaCreator
> {code}
> share/hadoop/yarn/timelineservice/ is not part of class path
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8636) TimelineSchemaCreator command not working
[
https://issues.apache.org/jira/browse/YARN-8636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572959#comment-16572959
]
Bibin A Chundatt edited comment on YARN-8636 at 8/8/18 9:56 AM:
cc:[~rohithsharma]
was (Author: bibinchundatt):
cc: [~rohit12sh]
> TimelineSchemaCreator command not working
> -
>
> Key: YARN-8636
> URL: https://issues.apache.org/jira/browse/YARN-8636
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Priority: Major
>
> {code:java}
> nodemanager/bin> ./hadoop
> org.apache.yarn.timelineservice.storage.TimelineSchemaCreator -create
> Error: Could not find or load main class
> org.apache.yarn.timelineservice.storage.TimelineSchemaCreator
> {code}
> share/hadoop/yarn/timelineservice/ is not part of class path
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8636) TimelineSchemaCreator command not working
[
https://issues.apache.org/jira/browse/YARN-8636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572959#comment-16572959
]
Bibin A Chundatt commented on YARN-8636:
cc: [~rohit12sh]
> TimelineSchemaCreator command not working
> -
>
> Key: YARN-8636
> URL: https://issues.apache.org/jira/browse/YARN-8636
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Priority: Major
>
> {code:java}
> nodemanager/bin> ./hadoop
> org.apache.yarn.timelineservice.storage.TimelineSchemaCreator -create
> Error: Could not find or load main class
> org.apache.yarn.timelineservice.storage.TimelineSchemaCreator
> {code}
> share/hadoop/yarn/timelineservice/ is not part of class path
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-8636) TimelineSchemaCreator command not working
Bibin A Chundatt created YARN-8636:
--
Summary: TimelineSchemaCreator command not working
Key: YARN-8636
URL: https://issues.apache.org/jira/browse/YARN-8636
Project: Hadoop YARN
Issue Type: Bug
Reporter: Bibin A Chundatt
{code:java}
nodemanager/bin> ./hadoop
org.apache.yarn.timelineservice.storage.TimelineSchemaCreator -create
Error: Could not find or load main class
org.apache.yarn.timelineservice.storage.TimelineSchemaCreator
{code}
share/hadoop/yarn/timelineservice/ is not part of class path
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-8635) Container fails to start if umask is 077
Bibin A Chundatt created YARN-8635:
--
Summary: Container fails to start if umask is 077
Key: YARN-8635
URL: https://issues.apache.org/jira/browse/YARN-8635
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 3.1.0
Reporter: Bibin A Chundatt
{code}
java.io.IOException: Application application_1533652359071_0001 initialization
failed (exitCode=255) with output: main : command provided 0
main : run as user is mapred
main : requested yarn user is mapred
Path
/opt/HA/OSBR310/nmlocal/usercache/mapred/appcache/application_1533652359071_0001
has permission 700 but needs permission 750.
Did not create any app directories
at
org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.startLocalizer(LinuxContainerExecutor.java:411)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1229)
Caused by:
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationException:
ExitCodeException exitCode=255:
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:180)
at
org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.startLocalizer(LinuxContainerExecutor.java:402)
... 1 more
Caused by: ExitCodeException exitCode=255:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:1009)
at org.apache.hadoop.util.Shell.run(Shell.java:902)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1227)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:152)
... 2 more
2018-08-08 17:43:26,918 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
Container container_e04_1533652359071_0001_01_27 transitioned from
LOCALIZING to LOCALIZATION_FAILED
2018-08-08 17:43:26,916 WARN
org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exit code
from container container_e04_1533652359071_0001_01_31 startLocalizer is :
255
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationException:
ExitCodeException exitCode=255:
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:180)
at
org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.startLocalizer(LinuxContainerExecutor.java:402)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1229)
Caused by: ExitCodeException exitCode=255:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:1009)
at org.apache.hadoop.util.Shell.run(Shell.java:902)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1227)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:152)
... 2 more
2018-08-08 17:43:26,923 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
Localizer failed for containe
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8634) Implement yarn cmd for killing all applications in the cluster
[
https://issues.apache.org/jira/browse/YARN-8634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572913#comment-16572913
]
Bibin A Chundatt commented on YARN-8634:
{code}
More than cluster level application kill ,will it be better in have kill per
queue. And if "root" queue is specified will kill all apps in cluster
{code}
This option will be flexible enough to handle that scenario too
> Implement yarn cmd for killing all applications in the cluster
> --
>
> Key: YARN-8634
> URL: https://issues.apache.org/jira/browse/YARN-8634
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Bilwa S T
>Assignee: Bilwa S T
>Priority: Major
> Attachments: YARN-8634-001.patch
>
>
> If we want to kill all running applications then currently we need to give
> all application Id's. So instead we can have a command which can kill all
> applications.
> Command would be *yarn application -killall*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8568) Replace the deprecated zk-address property in the HA config example in ResourceManagerHA.md
[ https://issues.apache.org/jira/browse/YARN-8568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572867#comment-16572867 ] Antal Bálint Steinbach commented on YARN-8568: -- Hi [~rkanter] , I was using _hadoop.zk.address configuration_ in _yarn-site.xml during_ testing RM HA and it was working. > Replace the deprecated zk-address property in the HA config example in > ResourceManagerHA.md > --- > > Key: YARN-8568 > URL: https://issues.apache.org/jira/browse/YARN-8568 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Affects Versions: 3.0.x >Reporter: Antal Bálint Steinbach >Assignee: Antal Bálint Steinbach >Priority: Minor > Attachments: YARN-8568.001.patch > > > yarn.resourcemanager.zk-address is deprecated. Instead, use hadoop.zk.address > In the example, "yarn.resourcemanager.zk-address" is used which is > deprecated. In the description, the property name is correct > "hadoop.zk.address". -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8634) Implement yarn cmd for killing all applications in the cluster
[ https://issues.apache.org/jira/browse/YARN-8634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572851#comment-16572851 ] Bilwa S T commented on YARN-8634: - [~bibinchundatt] Scenario can be for some maintenance work if admin wants to stop all the long running job. this command maybe required > Implement yarn cmd for killing all applications in the cluster > -- > > Key: YARN-8634 > URL: https://issues.apache.org/jira/browse/YARN-8634 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Bilwa S T >Assignee: Bilwa S T >Priority: Major > Attachments: YARN-8634-001.patch > > > If we want to kill all running applications then currently we need to give > all application Id's. So instead we can have a command which can kill all > applications. > Command would be *yarn application -killall* -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8634) Implement yarn cmd for killing all applications in the cluster
[
https://issues.apache.org/jira/browse/YARN-8634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572813#comment-16572813
]
Bibin A Chundatt commented on YARN-8634:
[~BilwaST]
Thank you raising this issue. Scenario does make sense to me in testing cluster.
But in production scenario may be not . Existing may be enough.
{code}
-killKills the application. Set of
applications can be provided
separated with space
{code}
More than cluster level application kill ,will it be better in have kill per
queue. And if "root" queue is specified will kill all apps in cluster.
cc: [~sunil.gov...@gmail.com] : Thoughts??
> Implement yarn cmd for killing all applications in the cluster
> --
>
> Key: YARN-8634
> URL: https://issues.apache.org/jira/browse/YARN-8634
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Bilwa S T
>Assignee: Bilwa S T
>Priority: Major
> Attachments: YARN-8634-001.patch
>
>
> If we want to kill all running applications then currently we need to give
> all application Id's. So instead we can have a command which can kill all
> applications.
> Command would be *yarn application -killall*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8634) Implement yarn cmd for killing all applications in the cluster
[
https://issues.apache.org/jira/browse/YARN-8634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572811#comment-16572811
]
genericqa commented on YARN-8634:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
28s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 28m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 7s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
23s{color} | {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
23s{color} | {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 23s{color}
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
25s{color} | {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 5 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 4m
28s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
21s{color} | {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 25s{color}
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 50m 48s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | YARN-8634 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12934764/YARN-8634-001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux d1a2c3f86ab1 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 6677717 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC1 |
| mvninstall |
https://builds.apache.org/job/PreCommit-YARN-Build/21538/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
|
| compile |
https://builds.apache.org/job/PreCommit-YARN-Build/21538/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
|
| javac |
https://builds.apache.org/job/PreCommit-YARN-Build/21538/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
|
| mvnsite |
[jira] [Updated] (YARN-8634) Implement yarn cmd for killing all applications in the cluster
[ https://issues.apache.org/jira/browse/YARN-8634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bilwa S T updated YARN-8634: Attachment: YARN-8634-001.patch > Implement yarn cmd for killing all applications in the cluster > -- > > Key: YARN-8634 > URL: https://issues.apache.org/jira/browse/YARN-8634 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Bilwa S T >Assignee: Bilwa S T >Priority: Major > Attachments: YARN-8634-001.patch > > > If we want to kill all running applications then currently we need to give > all application Id's. So instead we can have a command which can kill all > applications. > Command would be *yarn application -killall* -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-8634) Implement yarn cmd for killing all applications in the cluster
[ https://issues.apache.org/jira/browse/YARN-8634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bilwa S T reassigned YARN-8634: --- Assignee: Bilwa S T > Implement yarn cmd for killing all applications in the cluster > -- > > Key: YARN-8634 > URL: https://issues.apache.org/jira/browse/YARN-8634 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Bilwa S T >Assignee: Bilwa S T >Priority: Major > > If we want to kill all running applications then currently we need to give > all application Id's. So instead we can have a command which can kill > applications. > Command would be *yarn application -killall* -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8634) Implement yarn cmd for killing all applications in the cluster
[ https://issues.apache.org/jira/browse/YARN-8634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bilwa S T updated YARN-8634: Description: If we want to kill all running applications then currently we need to give all application Id's. So instead we can have a command which can kill all applications. Command would be *yarn application -killall* was: If we want to kill all running applications then currently we need to give all application Id's. So instead we can have a command which can kill applications. Command would be *yarn application -killall* > Implement yarn cmd for killing all applications in the cluster > -- > > Key: YARN-8634 > URL: https://issues.apache.org/jira/browse/YARN-8634 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Bilwa S T >Assignee: Bilwa S T >Priority: Major > > If we want to kill all running applications then currently we need to give > all application Id's. So instead we can have a command which can kill all > applications. > Command would be *yarn application -killall* -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-8634) Implement yarn cmd for killing all applications in the cluster
Bilwa S T created YARN-8634: --- Summary: Implement yarn cmd for killing all applications in the cluster Key: YARN-8634 URL: https://issues.apache.org/jira/browse/YARN-8634 Project: Hadoop YARN Issue Type: Improvement Reporter: Bilwa S T If we want to kill all running applications then currently we need to give all application Id's. So instead we can have a command which can kill applications. Command would be *yarn application -killall* -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
