Apache Hadoop qbt Report: trunk+JDK9 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java9-linux-x86/30/ [Jun 25, 2018 1:15:31 AM] (wwei) YARN-8443. Total #VCores in cluster metrics is wrong when [Jun 25, 2018 1:38:57 PM] (stevel) HADOOP-14396. Add builder interface to FileContext. Contributed by Lei [Jun 25, 2018 4:50:27 PM] (inigoiri) HADOOP-15458. TestLocalFileSystem#testFSOutputStreamBuilder fails on [Jun 25, 2018 5:38:03 PM] (rohithsharmaks) YARN-8457. Compilation is broken with -Pyarn-ui. [Jun 25, 2018 8:05:22 PM] (aengineer) HDDS-191. Queue SCMCommands via EventQueue in SCM. Contributed by Elek, [Jun 25, 2018 8:59:41 PM] (mackrorysd) HADOOP-15423. Merge fileCache and dirCache into ine single cache in [Jun 25, 2018 10:36:45 PM] (todd) HADOOP-15550. Avoid static initialization of ObjectMappers [Jun 25, 2018 10:47:54 PM] (miklos.szegedi) YARN-8438. TestContainer.testKillOnNew flaky on trunk. Contributed by [Jun 26, 2018 4:12:05 AM] (Bharat) HDDS-192:Create new SCMCommand to request a replication of a container. [Jun 26, 2018 6:25:15 PM] (eyang) YARN-8214. Change default RegistryDNS port. Contributed by [Jun 26, 2018 9:34:57 PM] (eyang) YARN-8108. Added option to disable loading existing filters to prevent [Jun 26, 2018 10:21:35 PM] (miklos.szegedi) YARN-8461. Support strict memory control on individual container with [Jun 27, 2018 2:25:57 AM] (wangda) YARN-8423. GPU does not get released even though the application gets [Jun 27, 2018 2:27:17 AM] (wangda) YARN-8464. Async scheduling thread could be interrupted when there are [Jun 27, 2018 5:35:15 PM] (sunilg) YARN-8401. [UI2] new ui is not accessible with out internet connection. [Jun 27, 2018 7:39:15 PM] (xyao) HDDS-194. Remove NodePoolManager and node pool handling from SCM. [Jun 27, 2018 8:25:45 PM] (xyao) Revert "HDDS-194. Remove NodePoolManager and node pool handling from [Jun 27, 2018 8:28:00 PM] (xyao) HDDS-194. Remove NodePoolManager and node pool handling from SCM. [Jun 27, 2018 8:35:30 PM] (xyao) HDDS-186. Create under replicated queue. Contributed by Ajay Kumar. [Jun 27, 2018 8:56:45 PM] (xyao) HDDS-170. Fix TestBlockDeletingService#testBlockDeletionTimeout. [Jun 27, 2018 9:15:15 PM] (aengineer) HDDS-94. Change ozone datanode command to start the standalone datanode [Jun 27, 2018 9:18:25 PM] (aengineer) HDDS-193. Make Datanode heartbeat dispatcher in SCM event based. [Jun 28, 2018 5:37:22 AM] (aajisaka) HADOOP-15495. Upgrade commons-lang version to 3.7 in [Jun 28, 2018 5:58:40 AM] (aajisaka) HADOOP-14313. Replace/improve Hadoop's byte[] comparator. Contributed by [Jun 28, 2018 6:39:33 AM] (aengineer) HDDS-195. Create generic CommandWatcher utility. Contributed by Elek, [Jun 28, 2018 4:21:56 PM] (Bharat) HDFS-13705:The native ISA-L library loading failure should be made [Jun 28, 2018 4:39:49 PM] (eyang) YARN-8409. Fixed NPE in ActiveStandbyElectorBasedElectorService. [Jun 28, 2018 5:23:31 PM] (sunilg) YARN-8379. Improve balancing resources in already satisfied queues by [Jun 28, 2018 10:41:39 PM] (nanda) HDDS-185: TestCloseContainerByPipeline#testCloseContainerViaRatis fail [Jun 28, 2018 11:07:16 PM] (nanda) HDDS-178: DN should update transactionId on block delete. Contributed by [Jun 29, 2018 4:56:13 PM] (inigoiri) HDFS-13707. [PROVIDED Storage] Fix failing integration tests in [Jun 29, 2018 5:02:53 PM] (sunilg) YARN-8455. Add basic ACL check for all ATS v2 REST APIs. Contributed by [Jun 29, 2018 5:21:25 PM] (rohithsharmaks) YARN-8469. [UI2] URL needs to be trimmed to handle index.html [Jun 29, 2018 6:06:28 PM] (jlowe) YARN-8451. Multiple NM heartbeat thread created when a slow NM resync [Jun 29, 2018 6:18:32 PM] (ericp) HADOOP-15548: Randomize local dirs. Contributed by Jim Brennan. [Jun 29, 2018 6:47:30 PM] (gifuma) YARN-8481. AMRMProxyPolicies should accept heartbeat response from -1 overall The following subsystems voted -1: compile findbugs mvninstall mvnsite pathlen shadedclient unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace Specific tests: Failed junit tests : hadoop.security.authentication.util.TestKerberosName hadoop.security.authentication.server.TestAltKerberosAuthenticationHandler hadoop.security.authentication.client.TestKerberosAuthenticator mvninstall: https://builds.apache.org/job/hadoop-qbt-trunk-java9-linux-x86/30/artifact/out/patch-mvninstall-root.txt [120K] compile: https://builds.apache.org/job/hadoop-qbt-trunk-java9-linux-x86/30/artifact/out/patch-compile-root.txt [56K] cc: https://builds.apache.org/job/hadoop-qbt-trunk-java9-linux-x86/30/artifact/out/patch-compile-root.txt [56K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java9-linux-x86/30/artifact/out/patch-compile-root.txt [56K] checkstyle: https://builds.apache.org/job/hadoop-qbt
[jira] [Created] (YARN-8485) Priviledged container app launch is failing intermittently
Yesha Vora created YARN-8485:
Summary: Priviledged container app launch is failing intermittently
Key: YARN-8485
URL: https://issues.apache.org/jira/browse/YARN-8485
Project: Hadoop YARN
Issue Type: Bug
Components: yarn-native-services
Reporter: Yesha Vora
Privileged application fails intermittently
{code:java}
yarn jar
/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell-*.jar
-shell_command "sleep 30" -num_containers 1 -shell_env
YARN_CONTAINER_RUNTIME_TYPE=docker -shell_env
YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=xxx -shell_env
YARN_CONTAINER_RUNTIME_DOCKER_RUN_PRIVILEGED_CONTAINER=true -jar
/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell-*.jar{code}
Here, container launch fails with 'Privileged containers are disabled' even
though Docker privilege container is enabled in the cluster
{code:java|title=nm log}
2018-06-28 21:21:15,647 INFO runtime.DockerLinuxContainerRuntime
(DockerLinuxContainerRuntime.java:allowPrivilegedContainerExecution(664)) - All
checks pass. Launching privileged container for :
container_e01_1530220647587_0001_01_02
2018-06-28 21:21:15,665 WARN nodemanager.LinuxContainerExecutor
(LinuxContainerExecutor.java:handleExitCode(593)) - Exit code from container
container_e01_1530220647587_0001_01_02 is : 29
2018-06-28 21:21:15,666 WARN nodemanager.LinuxContainerExecutor
(LinuxContainerExecutor.java:handleExitCode(599)) - Exception from
container-launch with container ID: container_e01_1530220647587_0001_01_02
and exit code: 29
org.apache.hadoop.yarn.server.nodemanager.containermanager.runtime.ContainerExecutionException:
Launch container failed
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(DockerLinuxContainerRuntime.java:958)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DelegatingLinuxContainerRuntime.launchContainer(DelegatingLinuxContainerRuntime.java:141)
at
org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.handleLaunchForLaunchType(LinuxContainerExecutor.java:564)
at
org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:479)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:494)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:306)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:103)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2018-06-28 21:21:15,668 INFO nodemanager.ContainerExecutor
(ContainerExecutor.java:logOutput(541)) - Exception from container-launch.
2018-06-28 21:21:15,668 INFO nodemanager.ContainerExecutor
(ContainerExecutor.java:logOutput(541)) - Container id:
container_e01_1530220647587_0001_01_02
2018-06-28 21:21:15,668 INFO nodemanager.ContainerExecutor
(ContainerExecutor.java:logOutput(541)) - Exit code: 29
2018-06-28 21:21:15,668 INFO nodemanager.ContainerExecutor
(ContainerExecutor.java:logOutput(541)) - Exception message: Launch container
failed
2018-06-28 21:21:15,668 INFO nodemanager.ContainerExecutor
(ContainerExecutor.java:logOutput(541)) - Shell error output: check privileges
failed for user: hrt_qa, error code: 0
2018-06-28 21:21:15,668 INFO nodemanager.ContainerExecutor
(ContainerExecutor.java:logOutput(541)) - Privileged containers are disabled
for user: hrt_qa
2018-06-28 21:21:15,668 INFO nodemanager.ContainerExecutor
(ContainerExecutor.java:logOutput(541)) - Error constructing docker command,
docker error code=11, error message='Privileged containers are disabled'
2018-06-28 21:21:15,668 INFO nodemanager.ContainerExecutor
(ContainerExecutor.java:logOutput(541)) -
2018-06-28 21:21:15,669 INFO nodemanager.ContainerExecutor
(ContainerExecutor.java:logOutput(541)) - Shell output: main : command provided
4
2018-06-28 21:21:15,669 INFO nodemanager.ContainerExecutor
(ContainerExecutor.java:logOutput(541)) - main : run as user is hrt_qa
2018-06-28 21:21:15,669 INFO nodemanager.ContainerExecutor
(ContainerExecutor.java:logOutput(541)) - main : requested yarn user is hrt_qa
2018-06-28 21:21:15,669 INFO nodemanager.ContainerExecutor
(ContainerExecutor.java:logOutput(541)) - Creating script paths...
2018-06-28 21:21:15,669 INFO nodemanager.ContainerExecutor
(ContainerExecutor.java:logOutput(541)) - Creating local dirs...
2018-06-28 21:21:
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/827/ [Jun 29, 2018 4:56:13 PM] (inigoiri) HDFS-13707. [PROVIDED Storage] Fix failing integration tests in [Jun 29, 2018 5:02:53 PM] (sunilg) YARN-8455. Add basic ACL check for all ATS v2 REST APIs. Contributed by [Jun 29, 2018 5:21:25 PM] (rohithsharmaks) YARN-8469. [UI2] URL needs to be trimmed to handle index.html [Jun 29, 2018 6:06:28 PM] (jlowe) YARN-8451. Multiple NM heartbeat thread created when a slow NM resync [Jun 29, 2018 6:18:32 PM] (ericp) HADOOP-15548: Randomize local dirs. Contributed by Jim Brennan. [Jun 29, 2018 6:47:30 PM] (gifuma) YARN-8481. AMRMProxyPolicies should accept heartbeat response from -1 overall The following subsystems voted -1: asflicense findbugs pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed CTEST tests : test_test_libhdfs_threaded_hdfs_static test_libhdfs_threaded_hdfspp_test_shim_static Failed junit tests : hadoop.hdfs.web.TestWebHdfsTimeouts hadoop.fs.contract.router.web.TestRouterWebHDFSContractAppend hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageEntities hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageDomain hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRunCompaction hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageSchema hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRun hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageApps hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowActivity hadoop.yarn.applications.distributedshell.TestDistributedShell hadoop.mapred.TestMRTimelineEventHandling hadoop.yarn.sls.TestSLSStreamAMSynth cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/827/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/827/artifact/out/diff-compile-javac-root.txt [360K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/827/artifact/out/diff-checkstyle-root.txt [4.0K] pathlen: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/827/artifact/out/pathlen.txt [12K] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/827/artifact/out/diff-patch-pylint.txt [24K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/827/artifact/out/diff-patch-shellcheck.txt [20K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/827/artifact/out/diff-patch-shelldocs.txt [16K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/827/artifact/out/whitespace-eol.txt [9.4M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/827/artifact/out/whitespace-tabs.txt [1.1M] xml: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/827/artifact/out/xml.txt [4.0K] findbugs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/827/artifact/out/branch-findbugs-hadoop-hdds_client.txt [56K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/827/artifact/out/branch-findbugs-hadoop-hdds_container-service.txt [48K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/827/artifact/out/branch-findbugs-hadoop-hdds_server-scm.txt [56K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/827/artifact/out/branch-findbugs-hadoop-hdds_tools.txt [12K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/827/artifact/out/branch-findbugs-hadoop-ozone_client.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/827/artifact/out/branch-findbugs-hadoop-ozone_common.txt [24K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/827/artifact/out/branch-findbugs-hadoop-ozone_objectstore-service.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/827/artifact/out/branch-findbugs-hadoop-ozone_ozone-manager.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/827/artifact/out/branch-findbugs-hadoop-ozone_ozonefs.txt [8.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/827/artifact/out/branch-findbugs-hadoop-ozone_tools.txt [4.0K] javadoc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/827
