Apache Hadoop qbt Report: trunk+JDK8 on Windows/x64
For more details, see https://builds.apache.org/job/hadoop-trunk-win/419/ [Mar 26, 2018 4:42:51 PM] (inigoiri) HDFS-13204. RBF: Optimize name service safe mode icon. Contributed by [Mar 26, 2018 5:21:35 PM] (eyang) YARN-8043. Added the exception message for failed launches running [Mar 26, 2018 5:45:29 PM] (xyao) HADOOP-15339. Support additional key/value propereties in JMX bean [Mar 26, 2018 6:16:06 PM] (wangda) YARN-8062. yarn rmadmin -getGroups returns group from which the user has [Mar 26, 2018 6:19:15 PM] (wangda) YARN-8068. Application Priority field causes NPE in app timeline publish [Mar 26, 2018 6:20:16 PM] (wangda) YARN-8072. RM log is getting flooded with [Mar 26, 2018 8:05:15 PM] (mackrorysd) HADOOP-15299. Bump Jackson 2 version to Jackson 2.9.x. [Mar 26, 2018 9:30:11 PM] (haibochen) YARN-7794. SLSRunner is not loading timeline service jars, causing [Mar 26, 2018 9:55:53 PM] (rkanter) MAPREDUCE-6441. Improve temporary directory name generation in [Mar 26, 2018 10:46:31 PM] (eyang) YARN-8018. Added support for initiating yarn service upgrade. [Mar 26, 2018 10:59:32 PM] (xiao) HADOOP-15313. TestKMS should close providers. -1 overall The following subsystems voted -1: compile mvninstall unit The following subsystems voted -1 but were configured to be filtered/ignored: cc javac The following subsystems are considered long running: (runtime bigger than 1h 00m 00s) unit Specific tests: Failed junit tests : hadoop.crypto.TestCryptoStreamsWithOpensslAesCtrCryptoCodec hadoop.fs.contract.rawlocal.TestRawlocalContractAppend hadoop.fs.TestFileUtil hadoop.fs.TestFsShellCopy hadoop.fs.TestFsShellList hadoop.fs.TestLocalFileSystem hadoop.fs.TestRawLocalFileSystemContract hadoop.fs.TestTrash hadoop.http.TestHttpServer hadoop.http.TestHttpServerLogs hadoop.io.nativeio.TestNativeIO hadoop.ipc.TestIPC hadoop.ipc.TestSocketFactory hadoop.metrics2.impl.TestStatsDMetrics hadoop.metrics2.sink.TestRollingFileSystemSinkWithLocal hadoop.security.TestSecurityUtil hadoop.security.TestShellBasedUnixGroupsMapping hadoop.security.token.TestDtUtilShell hadoop.util.TestNativeCodeLoader hadoop.util.TestNodeHealthScriptRunner hadoop.fs.TestResolveHdfsSymlink hadoop.hdfs.client.impl.TestBlockReaderLocalLegacy hadoop.hdfs.crypto.TestHdfsCryptoStreams hadoop.hdfs.qjournal.client.TestQuorumJournalManager hadoop.hdfs.qjournal.server.TestJournalNode hadoop.hdfs.qjournal.server.TestJournalNodeSync hadoop.hdfs.security.TestDelegationTokenForProxyUser hadoop.hdfs.server.balancer.TestBalancer hadoop.hdfs.server.blockmanagement.TestNameNodePrunesMissingStorages hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistLockedMemory hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistPolicy hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyWriter hadoop.hdfs.server.datanode.fsdataset.impl.TestProvidedImpl hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation hadoop.hdfs.server.datanode.fsdataset.impl.TestWriteToReplica hadoop.hdfs.server.datanode.TestBlockPoolSliceStorage hadoop.hdfs.server.datanode.TestBlockRecovery hadoop.hdfs.server.datanode.TestBlockScanner hadoop.hdfs.server.datanode.TestDataNodeFaultInjector hadoop.hdfs.server.datanode.TestDataNodeMetrics hadoop.hdfs.server.datanode.TestDataNodeUUID hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure hadoop.hdfs.server.datanode.TestDirectoryScanner hadoop.hdfs.server.datanode.TestHSync hadoop.hdfs.server.datanode.web.TestDatanodeHttpXFrame hadoop.hdfs.server.diskbalancer.command.TestDiskBalancerCommand hadoop.hdfs.server.diskbalancer.TestDiskBalancerRPC hadoop.hdfs.server.mover.TestMover hadoop.hdfs.server.mover.TestStorageMover hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA hadoop.hdfs.server.namenode.ha.TestDNFencing hadoop.hdfs.server.namenode.ha.TestHAAppend hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics hadoop.hdfs.server.namenode.snapshot.TestINodeFileUnderConstructionWithSnapshot hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots hadoop.hdfs.server.
[jira] [Created] (YARN-8082) Include LocalizedResource size information in the NM download log for localization
Kuhu Shukla created YARN-8082: - Summary: Include LocalizedResource size information in the NM download log for localization Key: YARN-8082 URL: https://issues.apache.org/jira/browse/YARN-8082 Project: Hadoop YARN Issue Type: Bug Affects Versions: 3.0.0 Reporter: Kuhu Shukla Assignee: Kuhu Shukla The size of the resource that finished downloading helps with debugging localization delays and failures. A close approximate local size of the resource is available in the LocalizedResource object which can be used to address this minor change. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-8081) Add support to upgrade a component
Chandni Singh created YARN-8081: --- Summary: Add support to upgrade a component Key: YARN-8081 URL: https://issues.apache.org/jira/browse/YARN-8081 Project: Hadoop YARN Issue Type: Sub-task Reporter: Chandni Singh Assignee: Chandni Singh -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-8044) Determine the appropriate default ContainerRetryPolicy
[
https://issues.apache.org/jira/browse/YARN-8044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shane Kumpf resolved YARN-8044.
---
Resolution: Won't Fix
> Determine the appropriate default ContainerRetryPolicy
> --
>
> Key: YARN-8044
> URL: https://issues.apache.org/jira/browse/YARN-8044
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Shane Kumpf
>Priority: Major
>
> {{AbstractLauncher}} sets the retry policy to {{RETRY_ON_ALL_ERRORS}}, which
> may be too inclusive. Some error codes, such as -1, should likely result in a
> hard fail.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-8080) YARN native service should support component restart policy
Wangda Tan created YARN-8080: Summary: YARN native service should support component restart policy Key: YARN-8080 URL: https://issues.apache.org/jira/browse/YARN-8080 Project: Hadoop YARN Issue Type: Task Reporter: Wangda Tan Assignee: Wangda Tan Attachments: YARN-8080.001.patch Existing native service assumes the service is long running and never finishes. Containers will be restarted even if exit code == 0. To support boarder use cases, we need to allow restart policy of component specified by users. Propose to have following policies: 1) Always: containers always restarted by framework regardless of container exit status. This is existing/default behavior. 2) Never: Do not restart containers in any cases after container finishes: To support job-like workload (for example Tensorflow training job). If a task exit with code == 0, we should not restart the task. This can be used by services which is not restart/recovery-able. 3) On-failure: Similar to above, only restart task with exitcode != 0. Behaviors after component *instance* finalize (Succeeded or Failed when restart_policy != ALWAYS): 1) For single component, single instance: complete service. 2) For single component, multiple instance: other running instances from the same component won't be affected by the finalized component instance. Service will be terminated once all instances finalized. 3) For multiple components: Service will be terminated once all components finalized. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-8079) YARN native service should respect source file of ConfigFile inside Service/Component spec
Wangda Tan created YARN-8079:
Summary: YARN native service should respect source file of
ConfigFile inside Service/Component spec
Key: YARN-8079
URL: https://issues.apache.org/jira/browse/YARN-8079
Project: Hadoop YARN
Issue Type: Bug
Reporter: Wangda Tan
Assignee: Wangda Tan
Currently, {{srcFile}} is not respected. {{ProviderUtils}} doesn't properly
read srcFile, instead it always construct {{remoteFile}} by using componentDir
and fileName of {{destFile}}:
{code}
Path remoteFile = new Path(compInstanceDir, fileName);
{code}
To me it is a common use case which services have some files existed in HDFS
and need to be localized when components get launched. (For example, if we want
to serve a Tensorflow model, we need to localize Tensorflow model (typically
not huge, less than GB) to local disk. Otherwise launched docker container has
to access HDFS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-8037) CGroupsResourceCalculator logs excessive warnings on container relaunch
[
https://issues.apache.org/jira/browse/YARN-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shane Kumpf resolved YARN-8037.
---
Resolution: Not A Problem
> CGroupsResourceCalculator logs excessive warnings on container relaunch
> ---
>
> Key: YARN-8037
> URL: https://issues.apache.org/jira/browse/YARN-8037
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Shane Kumpf
>Priority: Major
>
> When a container is relaunched, the old process no longer exists. When using
> the {{CGroupsResourceCalculator}} this results in the warning and exception
> below being logged every second until the relaunch occurs, which is excessive
> and filling up the logs.
> {code:java}
> 2018-03-16 14:30:33,438 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.CGroupsResourceCalculator:
> Failed to parse 12844
> org.apache.hadoop.yarn.exceptions.YarnException: The process vanished in the
> interim 12844
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.CGroupsResourceCalculator.processFile(CGroupsResourceCalculator.java:336)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.CGroupsResourceCalculator.readTotalProcessJiffies(CGroupsResourceCalculator.java:252)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.CGroupsResourceCalculator.updateProcessTree(CGroupsResourceCalculator.java:181)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.CombinedResourceCalculator.updateProcessTree(CombinedResourceCalculator.java:52)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl$MonitoringThread.run(ContainersMonitorImpl.java:457)
> Caused by: java.io.FileNotFoundException:
> /sys/fs/cgroup/cpu,cpuacct/hadoop-yarn/container_e01_1521209613260_0002_01_02/cpuacct.stat
> (No such file or directory)
> at java.io.FileInputStream.open0(Native Method)
> at java.io.FileInputStream.open(FileInputStream.java:195)
> at java.io.FileInputStream.(FileInputStream.java:138)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.CGroupsResourceCalculator.processFile(CGroupsResourceCalculator.java:320)
> ... 4 more
> 2018-03-16 14:30:33,438 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.CGroupsResourceCalculator:
> Failed to parse cgroups
> /sys/fs/cgroup/memory/hadoop-yarn/container_e01_1521209613260_0002_01_02/memory.memsw.usage_in_bytes
> org.apache.hadoop.yarn.exceptions.YarnException: The process vanished in the
> interim 12844
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.CGroupsResourceCalculator.processFile(CGroupsResourceCalculator.java:336)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.CGroupsResourceCalculator.getMemorySize(CGroupsResourceCalculator.java:238)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.CGroupsResourceCalculator.updateProcessTree(CGroupsResourceCalculator.java:187)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.CombinedResourceCalculator.updateProcessTree(CombinedResourceCalculator.java:52)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl$MonitoringThread.run(ContainersMonitorImpl.java:457)
> Caused by: java.io.FileNotFoundException:
> /sys/fs/cgroup/memory/hadoop-yarn/container_e01_1521209613260_0002_01_02/memory.usage_in_bytes
> (No such file or directory)
> at java.io.FileInputStream.open0(Native Method)
> at java.io.FileInputStream.open(FileInputStream.java:195)
> at java.io.FileInputStream.(FileInputStream.java:138)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.CGroupsResourceCalculator.processFile(CGroupsResourceCalculator.java:320)
> ... 4 more{code}
> We should consider moving the exception to debug to reduce the noise at a
> minimum. Alternatively, it may make sense to stop the existing
> {{MonitoringThread}} during relaunch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/733/ [Mar 26, 2018 10:33:07 AM] (yqlin) HDFS-13291. RBF: Implement available space based OrderResolver. [Mar 26, 2018 4:42:51 PM] (inigoiri) HDFS-13204. RBF: Optimize name service safe mode icon. Contributed by [Mar 26, 2018 5:21:35 PM] (eyang) YARN-8043. Added the exception message for failed launches running [Mar 26, 2018 5:45:29 PM] (xyao) HADOOP-15339. Support additional key/value propereties in JMX bean [Mar 26, 2018 6:16:06 PM] (wangda) YARN-8062. yarn rmadmin -getGroups returns group from which the user has [Mar 26, 2018 6:19:15 PM] (wangda) YARN-8068. Application Priority field causes NPE in app timeline publish [Mar 26, 2018 6:20:16 PM] (wangda) YARN-8072. RM log is getting flooded with [Mar 26, 2018 8:05:15 PM] (mackrorysd) HADOOP-15299. Bump Jackson 2 version to Jackson 2.9.x. [Mar 26, 2018 9:30:11 PM] (haibochen) YARN-7794. SLSRunner is not loading timeline service jars, causing [Mar 26, 2018 9:55:53 PM] (rkanter) MAPREDUCE-6441. Improve temporary directory name generation in [Mar 26, 2018 10:46:31 PM] (eyang) YARN-8018. Added support for initiating yarn service upgrade. [Mar 26, 2018 10:59:32 PM] (xiao) HADOOP-15313. TestKMS should close providers. -1 overall The following subsystems voted -1: findbugs unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: FindBugs : module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api org.apache.hadoop.yarn.api.records.Resource.getResources() may expose internal representation by returning Resource.resources At Resource.java:by returning Resource.resources At Resource.java:[line 234] Failed junit tests : hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure hadoop.hdfs.server.namenode.TestReencryptionWithKMS hadoop.hdfs.web.TestWebHdfsTimeouts hadoop.yarn.server.nodemanager.webapp.TestContainerLogsPage cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/733/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/733/artifact/out/diff-compile-javac-root.txt [288K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/733/artifact/out/diff-checkstyle-root.txt [17M] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/733/artifact/out/diff-patch-pylint.txt [24K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/733/artifact/out/diff-patch-shellcheck.txt [20K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/733/artifact/out/diff-patch-shelldocs.txt [12K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/733/artifact/out/whitespace-eol.txt [9.2M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/733/artifact/out/whitespace-tabs.txt [1.1M] xml: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/733/artifact/out/xml.txt [4.0K] findbugs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/733/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html [8.0K] javadoc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/733/artifact/out/diff-javadoc-javadoc-root.txt [760K] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/733/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [416K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/733/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt [48K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/733/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt [84K] Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-8078) TestDistributedShell#testDSShellWithoutDomainV2 fails on trunk
Weiwei Yang created YARN-8078: - Summary: TestDistributedShell#testDSShellWithoutDomainV2 fails on trunk Key: YARN-8078 URL: https://issues.apache.org/jira/browse/YARN-8078 Project: Hadoop YARN Issue Type: Test Reporter: Weiwei Yang -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
