[jira] [Created] (YARN-1971) WindowsLocalWrapperScriptBuilder does not check for errors in generated script
Remus Rusanu created YARN-1971:
--
Summary: WindowsLocalWrapperScriptBuilder does not check for
errors in generated script
Key: YARN-1971
URL: https://issues.apache.org/jira/browse/YARN-1971
Project: Hadoop YARN
Issue Type: Bug
Reporter: Remus Rusanu
Assignee: Remus Rusanu
Priority: Minor
Similar to YARN-1865. The
DefaultContainerExecutor.WindowsLocalWrapperScriptBuilder builds a shell script
that contains commands that potentially may fail:
{code}
pout.println(@echo + containerIdStr ++ normalizedPidFile +.tmp);
pout.println(@move /Y + normalizedPidFile + .tmp + normalizedPidFile);
{code}
These can fail due to access permissions, disc out of space, bad hardware,
cosmic rays etc etc. There should be proper error checking to ease
troubleshooting.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (YARN-1972) Implement secure Windows Container Executor
Remus Rusanu created YARN-1972: -- Summary: Implement secure Windows Container Executor Key: YARN-1972 URL: https://issues.apache.org/jira/browse/YARN-1972 Project: Hadoop YARN Issue Type: Improvement Reporter: Remus Rusanu Assignee: Remus Rusanu This work item represents the Java side changes required to implement a secure windows container executor, based on the YARN-1063 changes on native/winutils side. Necessary changes include leveraging the winutils task createas to launch the container process as the required user and a secure localizer (launch localization as a separate process running as the container user). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1876) Document the REST APIs of timeline and generic history services
[ https://issues.apache.org/jira/browse/YARN-1876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13976561#comment-13976561 ] Steve Loughran commented on YARN-1876: -- +1 doc patch only, Document the REST APIs of timeline and generic history services --- Key: YARN-1876 URL: https://issues.apache.org/jira/browse/YARN-1876 Project: Hadoop YARN Issue Type: Improvement Reporter: Zhijie Shen Assignee: Zhijie Shen Labels: documentaion Attachments: YARN-1876.1.patch, YARN-1876.2.patch, YARN-1876.3.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1063) Winutils needs ability to create task as domain user
[
https://issues.apache.org/jira/browse/YARN-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13976628#comment-13976628
]
Hadoop QA commented on YARN-1063:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12600298/YARN-1063.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3606//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3606//console
This message is automatically generated.
Winutils needs ability to create task as domain user
Key: YARN-1063
URL: https://issues.apache.org/jira/browse/YARN-1063
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager
Affects Versions: trunk-win
Environment: Windows
Reporter: Kyle Leckie
Labels: security
Fix For: trunk-win
Attachments: YARN-1063.patch
h1. Summary:
Securing a Hadoop cluster requires constructing some form of security
boundary around the processes executed in YARN containers. Isolation based on
Windows user isolation seems most feasible. This approach is similar to the
approach taken by the existing LinuxContainerExecutor. The current patch to
winutils.exe adds the ability to create a process as a domain user.
h1. Alternative Methods considered:
h2. Process rights limited by security token restriction:
On Windows access decisions are made by examining the security token of a
process. It is possible to spawn a process with a restricted security token.
Any of the rights granted by SIDs of the default token may be restricted. It
is possible to see this in action by examining the security tone of a
sandboxed process launch be a web browser. Typically the launched process
will have a fully restricted token and need to access machine resources
through a dedicated broker process that enforces a custom security policy.
This broker process mechanism would break compatibility with the typical
Hadoop container process. The Container process must be able to utilize
standard function calls for disk and network IO. I performed some work
looking at ways to ACL the local files to the specific launched without
granting rights to other processes launched on the same machine but found
this to be an overly complex solution.
h2. Relying on APP containers:
Recent versions of windows have the ability to launch processes within an
isolated container. Application containers are supported for execution of
WinRT based executables. This method was ruled out due to the lack of
official support for standard windows APIs. At some point in the future
windows may support functionality similar to BSD jails or Linux containers,
at that point support for containers should be added.
h1. Create As User Feature Description:
h2. Usage:
A new sub command was added to the set of task commands. Here is the syntax:
winutils task createAsUser [TASKNAME] [USERNAME] [COMMAND_LINE]
Some notes:
* The username specified is in the format of user@domain
* The machine executing this command must be joined to the domain of the user
specified
* The domain controller must allow the account executing the command access
to the user information. For this join the account to the predefined group
labeled Pre-Windows 2000 Compatible Access
* The account running the command must have several rights on the local
machine. These can be managed manually using secpol.msc:
** Act as part of the operating system - SE_TCB_NAME
** Replace a process-level token - SE_ASSIGNPRIMARYTOKEN_NAME
** Adjust memory quotas for a process -
[jira] [Commented] (YARN-1970) Prepare YARN codebase for JUnit 4.11.
[ https://issues.apache.org/jira/browse/YARN-1970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13976805#comment-13976805 ] Hudson commented on YARN-1970: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1740 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1740/]) YARN-1970. Prepare YARN codebase for JUnit 4.11. Contributed by Chris Nauroth. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1589001) * /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/test/java/org/apache/hadoop/yarn/sls/utils/TestSLSUtils.java * /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/test/java/org/apache/hadoop/yarn/sls/web/TestSLSWebApp.java * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/ProtocolHATestBase.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestApplicationMasterServiceOnHA.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestResourceTrackerOnHA.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMHA.java Prepare YARN codebase for JUnit 4.11. - Key: YARN-1970 URL: https://issues.apache.org/jira/browse/YARN-1970 Project: Hadoop YARN Issue Type: Test Affects Versions: 3.0.0, 2.4.0 Reporter: Chris Nauroth Assignee: Chris Nauroth Priority: Minor Fix For: 3.0.0, 2.5.0 Attachments: YARN-1970.1.patch HADOOP-10503 upgrades the entire Hadoop repo to use JUnit 4.11. Some of the YARN code needs some minor updates to fix deprecation warnings and test isolation problems before the upgrade. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1970) Prepare YARN codebase for JUnit 4.11.
[ https://issues.apache.org/jira/browse/YARN-1970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13976887#comment-13976887 ] Hudson commented on YARN-1970: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1765 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1765/]) YARN-1970. Prepare YARN codebase for JUnit 4.11. Contributed by Chris Nauroth. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1589001) * /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/test/java/org/apache/hadoop/yarn/sls/utils/TestSLSUtils.java * /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/test/java/org/apache/hadoop/yarn/sls/web/TestSLSWebApp.java * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/ProtocolHATestBase.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestApplicationMasterServiceOnHA.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestResourceTrackerOnHA.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMHA.java Prepare YARN codebase for JUnit 4.11. - Key: YARN-1970 URL: https://issues.apache.org/jira/browse/YARN-1970 Project: Hadoop YARN Issue Type: Test Affects Versions: 3.0.0, 2.4.0 Reporter: Chris Nauroth Assignee: Chris Nauroth Priority: Minor Fix For: 3.0.0, 2.5.0 Attachments: YARN-1970.1.patch HADOOP-10503 upgrades the entire Hadoop repo to use JUnit 4.11. Some of the YARN code needs some minor updates to fix deprecation warnings and test isolation problems before the upgrade. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-1916) Leveldb timeline store applies secondary filters incorrectly
[ https://issues.apache.org/jira/browse/YARN-1916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-1916: -- Affects Version/s: 2.4.0 Leveldb timeline store applies secondary filters incorrectly Key: YARN-1916 URL: https://issues.apache.org/jira/browse/YARN-1916 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.4.0 Reporter: Billie Rinaldi Assignee: Billie Rinaldi Attachments: YARN-1916.1.patch When applying a secondary filter (fieldname:fieldvalue) in a get entities query, LeveldbTimelineStore retrieves entities that do not have the specified fieldname, in addition to correctly retrieving entities that have the fieldname with the specified fieldvalue. It should not return entities that do not have the fieldname. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-1916) Leveldb timeline store applies secondary filters incorrectly
[ https://issues.apache.org/jira/browse/YARN-1916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-1916: -- Target Version/s: 2.4.1 Leveldb timeline store applies secondary filters incorrectly Key: YARN-1916 URL: https://issues.apache.org/jira/browse/YARN-1916 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.4.0 Reporter: Billie Rinaldi Assignee: Billie Rinaldi Attachments: YARN-1916.1.patch When applying a secondary filter (fieldname:fieldvalue) in a get entities query, LeveldbTimelineStore retrieves entities that do not have the specified fieldname, in addition to correctly retrieving entities that have the fieldname with the specified fieldvalue. It should not return entities that do not have the fieldname. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-126) yarn rmadmin help message contains reference to hadoop cli and JT
[ https://issues.apache.org/jira/browse/YARN-126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13977236#comment-13977236 ] Chen He commented on YARN-126: -- RETARGETED TO 3.0 yarn rmadmin help message contains reference to hadoop cli and JT - Key: YARN-126 URL: https://issues.apache.org/jira/browse/YARN-126 Project: Hadoop YARN Issue Type: Bug Components: client Affects Versions: 2.0.3-alpha Reporter: Thomas Graves Assignee: Rémy SAISSY Labels: usability Attachments: YARN-126.patch has option to specify a job tracker and the last line for general command line syntax had bin/hadoop command [genericOptions] [commandOptions] ran yarn rmadmin to get usage: RMAdmin Usage: java RMAdmin [-refreshQueues] [-refreshNodes] [-refreshUserToGroupsMappings] [-refreshSuperUserGroupsConfiguration] [-refreshAdminAcls] [-refreshServiceAcl] [-help [cmd]] Generic options supported are -conf configuration file specify an application configuration file -D property=valueuse value for given property -fs local|namenode:port specify a namenode -jt local|jobtracker:portspecify a job tracker -files comma separated list of filesspecify comma separated files to be copied to the map reduce cluster -libjars comma separated list of jarsspecify comma separated jar files to include in the classpath. -archives comma separated list of archivesspecify comma separated archives to be unarchived on the compute machines. The general command line syntax is bin/hadoop command [genericOptions] [commandOptions] -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-126) yarn rmadmin help message contains reference to hadoop cli and JT
[ https://issues.apache.org/jira/browse/YARN-126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen He updated YARN-126: - Target Version/s: 3.0.0 (was: 3.0.0, 0.23.11) yarn rmadmin help message contains reference to hadoop cli and JT - Key: YARN-126 URL: https://issues.apache.org/jira/browse/YARN-126 Project: Hadoop YARN Issue Type: Bug Components: client Affects Versions: 2.0.3-alpha Reporter: Thomas Graves Assignee: Rémy SAISSY Labels: usability Attachments: YARN-126.patch has option to specify a job tracker and the last line for general command line syntax had bin/hadoop command [genericOptions] [commandOptions] ran yarn rmadmin to get usage: RMAdmin Usage: java RMAdmin [-refreshQueues] [-refreshNodes] [-refreshUserToGroupsMappings] [-refreshSuperUserGroupsConfiguration] [-refreshAdminAcls] [-refreshServiceAcl] [-help [cmd]] Generic options supported are -conf configuration file specify an application configuration file -D property=valueuse value for given property -fs local|namenode:port specify a namenode -jt local|jobtracker:portspecify a job tracker -files comma separated list of filesspecify comma separated files to be copied to the map reduce cluster -libjars comma separated list of jarsspecify comma separated jar files to include in the classpath. -archives comma separated list of archivesspecify comma separated archives to be unarchived on the compute machines. The general command line syntax is bin/hadoop command [genericOptions] [commandOptions] -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1964) Support Docker containers in YARN
[ https://issues.apache.org/jira/browse/YARN-1964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13977304#comment-13977304 ] Abin Shahab commented on YARN-1964: --- I have a patch for a docker container executor, and it takes containers and docker run CLI options. Will post it for review. Support Docker containers in YARN - Key: YARN-1964 URL: https://issues.apache.org/jira/browse/YARN-1964 Project: Hadoop YARN Issue Type: New Feature Reporter: Arun C Murthy Assignee: Arun C Murthy Docker (https://www.docker.io/) is, increasingly, a very popular container technology. In context of YARN, the support for Docker will provide a very elegant solution to allow applications to *package* their software into a Docker container (entire Linux file system incl. custom versions of perl, python etc.) and use it as a blueprint to launch all their YARN containers with requisite software environment. This provides both consistency (all YARN containers will have the same software environment) and isolation (no interference with whatever is installed on the physical machine). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1879) Mark Idempotent/AtMostOnce annotations to ApplicationMasterProtocol
[ https://issues.apache.org/jira/browse/YARN-1879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13977424#comment-13977424 ] Jian He commented on YARN-1879: --- The patch needs update against trunk. one nit retrycache-retry-cache, other than that, looks good to me. Mark Idempotent/AtMostOnce annotations to ApplicationMasterProtocol --- Key: YARN-1879 URL: https://issues.apache.org/jira/browse/YARN-1879 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Jian He Assignee: Tsuyoshi OZAWA Priority: Critical Attachments: YARN-1879.1.patch, YARN-1879.1.patch, YARN-1879.2-wip.patch, YARN-1879.2.patch, YARN-1879.3.patch, YARN-1879.4.patch, YARN-1879.5.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1842) InvalidApplicationMasterRequestException raised during AM-requested shutdown
[
https://issues.apache.org/jira/browse/YARN-1842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13977446#comment-13977446
]
Brian Murphy commented on YARN-1842:
Hey there,
We are seeing this bug occur while shutting down Samza containers as well. We
are running Hadoop 2.3.0 on Ubuntu 12.10. The container hangs indefinitely in
the KILLING state.
Here is the stack trace:
{code}
2014-04-22 20:25:08 SamzaAppMaster$ [ERROR] Error occured in amClient's callback
org.apache.samza.SamzaException: Received a reboot signal from the RM, so
throwing an exception to reboot the AM.
at
org.apache.samza.job.yarn.SamzaAppMasterLifecycle.onReboot(SamzaAppMasterLifecycle.scala:59)
at
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onShutdownRequest$1.apply(SamzaAppMaster.scala:136)
at
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onShutdownRequest$1.apply(SamzaAppMaster.scala:136)
at scala.collection.immutable.List.foreach(List.scala:318)
at
org.apache.samza.job.yarn.SamzaAppMaster$.onShutdownRequest(SamzaAppMaster.scala:136)
at
org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:285)
2014-04-22 20:25:09 ELContextCleaner [INFO] javax.el.BeanELResolver purged
2014-04-22 20:25:09 ContextHandler [INFO] stopped
o.e.j.w.WebAppContext{/,jar:file:/mnt/data/hadoop/yarn/usercache/brian/appcache/application_1397507485520_0040/filecache/10/samza-job-package-0.7.0-dist.tar.gz/lib/samza-yarn_2.10-0.7.0.jar!/scalate}
2014-04-22 20:25:10 ELContextCleaner [INFO] javax.el.BeanELResolver purged
2014-04-22 20:25:10 ContextHandler [INFO] stopped
o.e.j.w.WebAppContext{/,jar:file:/mnt/data/hadoop/yarn/usercache/brian/appcache/application_1397507485520_0040/filecache/10/samza-job-package-0.7.0-dist.tar.gz/lib/samza-yarn_2.10-0.7.0.jar!/scalate}
2014-04-22 20:25:10 SamzaAppMasterLifecycle [INFO] Shutting down.
2014-04-22 20:25:10 SamzaAppMaster$ [WARN] Listener
org.apache.samza.job.yarn.SamzaAppMasterLifecycle@3c9ead34 failed to shutdown.
org.apache.hadoop.yarn.exceptions.InvalidApplicationMasterRequestException:
Application doesn't exist in cache appattempt_1397507485520_0040_01
at
org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.throwApplicationDoesNotExistInCacheException(ApplicationMasterService.java:329)
at
org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.finishApplicationMaster(ApplicationMasterService.java:288)
at
org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.finishApplicationMaster(ApplicationMasterProtocolPBServiceImpl.java:75)
at
org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:97)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1962)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1958)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1956)
{code}
InvalidApplicationMasterRequestException raised during AM-requested shutdown
Key: YARN-1842
URL: https://issues.apache.org/jira/browse/YARN-1842
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Affects Versions: 2.3.0
Reporter: Steve Loughran
Priority: Minor
Attachments: hoyalogs.tar.gz
Report of the RM raising a stack trace
[https://gist.github.com/matyix/9596735] during AM-initiated shutdown. The AM
could just swallow this and exit, but it could be a sign of a race condition
YARN-side, or maybe just in the RM client code/AM dual signalling the
shutdown.
I haven't replicated this myself; maybe the stack will help track down the
problem. Otherwise: what is the policy YARN apps should adopt for AM's
handling errors on shutdown? go straight to an exit(-1)?
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (YARN-1964) Support Docker containers in YARN
[ https://issues.apache.org/jira/browse/YARN-1964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated YARN-1964: Assignee: (was: Arun C Murthy) Support Docker containers in YARN - Key: YARN-1964 URL: https://issues.apache.org/jira/browse/YARN-1964 Project: Hadoop YARN Issue Type: New Feature Reporter: Arun C Murthy Docker (https://www.docker.io/) is, increasingly, a very popular container technology. In context of YARN, the support for Docker will provide a very elegant solution to allow applications to *package* their software into a Docker container (entire Linux file system incl. custom versions of perl, python etc.) and use it as a blueprint to launch all their YARN containers with requisite software environment. This provides both consistency (all YARN containers will have the same software environment) and isolation (no interference with whatever is installed on the physical machine). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-1964) Support Docker containers in YARN
[ https://issues.apache.org/jira/browse/YARN-1964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated YARN-1964: Assignee: Abin Shahab Support Docker containers in YARN - Key: YARN-1964 URL: https://issues.apache.org/jira/browse/YARN-1964 Project: Hadoop YARN Issue Type: New Feature Reporter: Arun C Murthy Assignee: Abin Shahab Docker (https://www.docker.io/) is, increasingly, a very popular container technology. In context of YARN, the support for Docker will provide a very elegant solution to allow applications to *package* their software into a Docker container (entire Linux file system incl. custom versions of perl, python etc.) and use it as a blueprint to launch all their YARN containers with requisite software environment. This provides both consistency (all YARN containers will have the same software environment) and isolation (no interference with whatever is installed on the physical machine). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1964) Support Docker containers in YARN
[ https://issues.apache.org/jira/browse/YARN-1964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13977489#comment-13977489 ] Arun C Murthy commented on YARN-1964: - [~ashahab] - That's great to hear, awesome! Thanks for taking this up! Support Docker containers in YARN - Key: YARN-1964 URL: https://issues.apache.org/jira/browse/YARN-1964 Project: Hadoop YARN Issue Type: New Feature Reporter: Arun C Murthy Assignee: Abin Shahab Docker (https://www.docker.io/) is, increasingly, a very popular container technology. In context of YARN, the support for Docker will provide a very elegant solution to allow applications to *package* their software into a Docker container (entire Linux file system incl. custom versions of perl, python etc.) and use it as a blueprint to launch all their YARN containers with requisite software environment. This provides both consistency (all YARN containers will have the same software environment) and isolation (no interference with whatever is installed on the physical machine). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-1879) Mark Idempotent/AtMostOnce annotations to ApplicationMasterProtocol
[ https://issues.apache.org/jira/browse/YARN-1879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi OZAWA updated YARN-1879: - Attachment: YARN-1879.6.patch Mark Idempotent/AtMostOnce annotations to ApplicationMasterProtocol --- Key: YARN-1879 URL: https://issues.apache.org/jira/browse/YARN-1879 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Jian He Assignee: Tsuyoshi OZAWA Priority: Critical Attachments: YARN-1879.1.patch, YARN-1879.1.patch, YARN-1879.2-wip.patch, YARN-1879.2.patch, YARN-1879.3.patch, YARN-1879.4.patch, YARN-1879.5.patch, YARN-1879.6.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1879) Mark Idempotent/AtMostOnce annotations to ApplicationMasterProtocol
[ https://issues.apache.org/jira/browse/YARN-1879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13977525#comment-13977525 ] Tsuyoshi OZAWA commented on YARN-1879: -- Updated a patch on trunk and changed a configuration key name from retrycache to retry-cache. Mark Idempotent/AtMostOnce annotations to ApplicationMasterProtocol --- Key: YARN-1879 URL: https://issues.apache.org/jira/browse/YARN-1879 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Jian He Assignee: Tsuyoshi OZAWA Priority: Critical Attachments: YARN-1879.1.patch, YARN-1879.1.patch, YARN-1879.2-wip.patch, YARN-1879.2.patch, YARN-1879.3.patch, YARN-1879.4.patch, YARN-1879.5.patch, YARN-1879.6.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1879) Mark Idempotent/AtMostOnce annotations to ApplicationMasterProtocol
[ https://issues.apache.org/jira/browse/YARN-1879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13977560#comment-13977560 ] Vinod Kumar Vavilapalli commented on YARN-1879: --- Apologies for coming in late. I'd like to take a look at this patch. Please hold off the commit for a day or two. Thanks. Mark Idempotent/AtMostOnce annotations to ApplicationMasterProtocol --- Key: YARN-1879 URL: https://issues.apache.org/jira/browse/YARN-1879 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Jian He Assignee: Tsuyoshi OZAWA Priority: Critical Attachments: YARN-1879.1.patch, YARN-1879.1.patch, YARN-1879.2-wip.patch, YARN-1879.2.patch, YARN-1879.3.patch, YARN-1879.4.patch, YARN-1879.5.patch, YARN-1879.6.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1879) Mark Idempotent/AtMostOnce annotations to ApplicationMasterProtocol
[
https://issues.apache.org/jira/browse/YARN-1879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13977588#comment-13977588
]
Hadoop QA commented on YARN-1879:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12641349/YARN-1879.6.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3607//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3607//console
This message is automatically generated.
Mark Idempotent/AtMostOnce annotations to ApplicationMasterProtocol
---
Key: YARN-1879
URL: https://issues.apache.org/jira/browse/YARN-1879
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Jian He
Assignee: Tsuyoshi OZAWA
Priority: Critical
Attachments: YARN-1879.1.patch, YARN-1879.1.patch,
YARN-1879.2-wip.patch, YARN-1879.2.patch, YARN-1879.3.patch,
YARN-1879.4.patch, YARN-1879.5.patch, YARN-1879.6.patch
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1842) InvalidApplicationMasterRequestException raised during AM-requested shutdown
[ https://issues.apache.org/jira/browse/YARN-1842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13977601#comment-13977601 ] Vinod Kumar Vavilapalli commented on YARN-1842: --- Hey folks, can someone who can reproduce it with ease upload both RM as well as the AM logs? InvalidApplicationMasterRequestException raised during AM-requested shutdown Key: YARN-1842 URL: https://issues.apache.org/jira/browse/YARN-1842 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.3.0 Reporter: Steve Loughran Priority: Minor Attachments: hoyalogs.tar.gz Report of the RM raising a stack trace [https://gist.github.com/matyix/9596735] during AM-initiated shutdown. The AM could just swallow this and exit, but it could be a sign of a race condition YARN-side, or maybe just in the RM client code/AM dual signalling the shutdown. I haven't replicated this myself; maybe the stack will help track down the problem. Otherwise: what is the policy YARN apps should adopt for AM's handling errors on shutdown? go straight to an exit(-1)? -- This message was sent by Atlassian JIRA (v6.2#6252)
