[jira] [Created] (YARN-10738) When multi thread scheduling with multi node, we should shuffle to prevent hot accessing nodes.
Qi Zhu created YARN-10738: - Summary: When multi thread scheduling with multi node, we should shuffle to prevent hot accessing nodes. Key: YARN-10738 URL: https://issues.apache.org/jira/browse/YARN-10738 Project: Hadoop YARN Issue Type: Improvement Reporter: Qi Zhu Assignee: Qi Zhu -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-10737) Fix typos in CapacityScheduler#schedule.
Qi Zhu created YARN-10737: - Summary: Fix typos in CapacityScheduler#schedule. Key: YARN-10737 URL: https://issues.apache.org/jira/browse/YARN-10737 Project: Hadoop YARN Issue Type: Improvement Reporter: Qi Zhu Assignee: Qi Zhu -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/477/ [Apr 13, 2021 3:44:31 AM] (noreply) Revert "HDFS-15423 RBF: WebHDFS create shouldn't choose DN from all sub-clusters (#2605)" (#2900) [Apr 13, 2021 8:08:49 AM] (noreply) HADOOP-17630. [JDK 15] TestPrintableString fails due to Unicode 13.0 support. (#2890) [Apr 13, 2021 3:54:45 PM] (noreply) HDFS-15971. Make mkstemp cross platform (#2898) [Apr 13, 2021 5:58:42 PM] (Ayush Saxena) Revert "HDFS-15884. RBF: Remove unused method getCreateLocation in RouterRpcServer (#2754). Contributed by tomscut." -1 overall The following subsystems voted -1: blanks pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml Failed junit tests : hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys hadoop.hdfs.server.namenode.ha.TestPipelinesFailover hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots hadoop.hdfs.server.datanode.TestDirectoryScanner hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks hadoop.mapreduce.v2.hs.TestJobHistoryParsing hadoop.tools.fedbalance.procedure.TestBalanceProcedureScheduler hadoop.tools.fedbalance.TestDistCpProcedure hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination hadoop.hdfs.server.federation.router.TestRouterRpc hadoop.tools.dynamometer.TestDynamometerInfra hadoop.tools.dynamometer.TestDynamometerInfra cc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/477/artifact/out/results-compile-cc-root.txt [96K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/477/artifact/out/results-compile-javac-root.txt [368K] blanks: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/477/artifact/out/blanks-eol.txt [13M] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/477/artifact/out/blanks-tabs.txt [2.0M] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/477/artifact/out/results-checkstyle-root.txt [16M] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/477/artifact/out/results-pathlen.txt [16K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/477/artifact/out/results-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/477/artifact/out/results-shellcheck.txt [28K] xml: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/477/artifact/out/xml.txt [24K] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/477/artifact/out/results-javadoc-javadoc-root.txt [1.1M] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/477/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [496K] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/477/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-hs.txt [16K] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/477/artifact/out/patch-unit-hadoop-tools_hadoop-federation-balance.txt [12K] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/477/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt [60K] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/477/artifact/out/patch-unit-hadoop-tools_hadoop-dynamometer_hadoop-dynamometer-infra.txt [8.0K] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/477/artifact/out/patch-unit-hadoop-tools_hadoop-dynamometer.txt [24K] Powered by Apache Yetus 0.14.0-SNAPSHOT
[jira] [Resolved] (YARN-10733) TimelineService Hbase tests are failing with timeout error on branch-2.10
[
https://issues.apache.org/jira/browse/YARN-10733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jim Brennan resolved YARN-10733.
Fix Version/s: 2.10.2
Resolution: Fixed
Thanks [~ahussein], I have committed this to branch-2.10.
> TimelineService Hbase tests are failing with timeout error on branch-2.10
> -
>
> Key: YARN-10733
> URL: https://issues.apache.org/jira/browse/YARN-10733
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test, timelineserver, yarn
>Affects Versions: 2.10.0
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.10.2
>
> Attachments: 2021-04-12T12-40-21_403-jvmRun1.dump,
> 2021-04-12T12-40-58_857.dumpstream,
> org.apache.hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRunCompaction-output.txt.zip
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> {code:bash}
> 03:54:41 [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:2.22.2:test (default-test) on
> project hadoop-yarn-server-timelineservice-hbase-tests: There was a timeout
> or other error in the fork -> [Help 1]
> 03:54:41 [ERROR]
> 03:54:41 [ERROR] To see the full stack trace of the errors, re-run Maven with
> the -e switch.
> 03:54:41 [ERROR] Re-run Maven using the -X switch to enable full debug
> logging.
> 03:54:41 [ERROR]
> 03:54:41 [ERROR] For more information about the errors and possible
> solutions, please read the following articles:
> 03:54:41 [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> 03:54:41 [ERROR]
> 03:54:41 [ERROR] After correcting the problems, you can resume the build with
> the command
> 03:54:41 [ERROR] mvn -rf
> :hadoop-yarn-server-timelineservice-hbase-tests
> {code}
> Failure of the tests is due to test unit
> {{TestHBaseStorageFlowRunCompaction}} getting stuck.
> Upon checking the surefire reports, I found several Class no Found Exceptions.
> {code:bash}
> Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/fs/CanUnbuffer
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:66)
> at
> org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:698)
> at
> org.apache.hadoop.hbase.regionserver.HStore.validateStoreFile(HStore.java:1895)
> at
> org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:1009)
> at
> org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.flushCache(HStore.java:2523)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2638)
> ... 33 more
> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.CanUnbuffer
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 51 more
> {code}
> and
> {code:bash}
> Caused by: java.lang.NoClassDefFoundError: Could not initialize class
> org.apache.hadoop.hbase.regionserver.StoreFileInfo
> at
> org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:698)
> at
> org.apache.hadoop.hbase.regionserver.HStore.validateStoreFile(HStore.java:1895)
> at
> org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:1009)
> at
> org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.flushCache(HStore.java:2523)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2638)
> ... 10 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail:
[jira] [Created] (YARN-10736) Fix GetApplicationsRequest JavaDoc
Miklos Gergely created YARN-10736: - Summary: Fix GetApplicationsRequest JavaDoc Key: YARN-10736 URL: https://issues.apache.org/jira/browse/YARN-10736 Project: Hadoop YARN Issue Type: Bug Reporter: Miklos Gergely getName and setName javadoc comments are mixed up -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-10735) Unmanaged AM is won't populate AMRMToken to ApplicationReport in secure cluster
Wang, Xinglong created YARN-10735:
-
Summary: Unmanaged AM is won't populate AMRMToken to
ApplicationReport in secure cluster
Key: YARN-10735
URL: https://issues.apache.org/jira/browse/YARN-10735
Project: Hadoop YARN
Issue Type: Bug
Reporter: Wang, Xinglong
Assignee: Wang, Xinglong
With kerberos enabled, NPE will be reported when launching UnmanagedAMLauncher.
It is due to there is no AMRMToken is returned in ApplicationReport. After a
while investigation, it turns out that RMAppImpl has a bad if condition inside
createAndGetApplicationReport
{code:java}
21/04/14 02:46:01 INFO unmanagedamlauncher.UnmanagedAMLauncher: Initializing
Client
21/04/14 02:46:02 INFO unmanagedamlauncher.UnmanagedAMLauncher: Starting Client
21/04/14 02:46:02 INFO client.AHSProxy: Connecting to Application History
server at /0.0.0.0:10200
21/04/14 02:46:02 INFO unmanagedamlauncher.UnmanagedAMLauncher: Setting up
application submission context for ASM
21/04/14 02:46:02 INFO client.ConfiguredRMFailoverProxyProvider: Failing over
to rm2
21/04/14 02:46:02 INFO unmanagedamlauncher.UnmanagedAMLauncher: Setting
unmanaged AM
21/04/14 02:46:02 INFO unmanagedamlauncher.UnmanagedAMLauncher: Submitting
application to ASM
21/04/14 02:46:03 INFO impl.YarnClientImpl: Submitted application
application_1618393442264_0002
21/04/14 02:46:04 INFO unmanagedamlauncher.UnmanagedAMLauncher: Got application
report from ASM for, appId=2,
appAttemptId=appattempt_1618393442264_0002_01, clientToAMToken=Token {
kind: YARN_CLIENT_TOKEN, service: }, appDiagnostics=AM container is launched,
waiting for AM container to Register with RM, appMasterHost=N/A,
appQueue=hdmi-default, appMasterRpcPort=-1, appStartTime=1618393562917,
yarnAppState=ACCEPTED, distributedFinalState=UNDEFINED, appTrackingUrl=N/A,
appUser=b_carmel
21/04/14 02:46:04 INFO unmanagedamlauncher.UnmanagedAMLauncher: Launching AM
with application attempt id appattempt_1618393442264_0002_01
21/04/14 02:46:04 FATAL unmanagedamlauncher.UnmanagedAMLauncher: Error running
Client
java.lang.NullPointerException
at
org.apache.hadoop.yarn.applications.unmanagedamlauncher.UnmanagedAMLauncher.launchAM(UnmanagedAMLauncher.java:186)
at
org.apache.hadoop.yarn.applications.unmanagedamlauncher.UnmanagedAMLauncher.run(UnmanagedAMLauncher.java:354)
at
org.apache.hadoop.yarn.applications.unmanagedamlauncher.UnmanagedAMLauncher.main(UnmanagedAMLauncher.java:111)
{code}
{code:java}
public ApplicationReport createAndGetApplicationReport(String clientUserName,
boolean allowAccess) {
..
if (currentAttempt != null &&
currentAttempt.getAppAttemptState() == RMAppAttemptState.LAUNCHED) {
if (getApplicationSubmissionContext().getUnmanagedAM() &&
clientUserName != null && getUser().equals(clientUserName)) {
Token token = currentAttempt.getAMRMToken();
if (token != null) {
amrmToken = BuilderUtils.newAMRMToken(token.getIdentifier(),
token.getKind().toString(), token.getPassword(),
token.getService().toString());
}
}
}
{code}
clientUserName is fullName of a kerberos principle like a...@domain.com whereas
getUser() will return the username recorded in RMAppImpl which is short name.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/268/ [Apr 13, 2021 12:42:40 AM] (noreply) HADOOP-17601. Upgrade Jackson databind in branch-2.10 to 2.9.10.7. (#2835) -1 overall The following subsystems voted -1: asflicense hadolint mvnsite pathlen spotbugs unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.fs.TestFileUtil hadoop.crypto.key.kms.server.TestKMS hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.contrib.bkjournal.TestBookKeeperJournalManager hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain hadoop.hdfs.server.datanode.TestDirectoryScanner hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys hadoop.hdfs.server.datanode.TestBlockRecovery hadoop.hdfs.server.namenode.TestNameNodeHttpServerXFrame hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.contrib.bkjournal.TestBookKeeperJournalManager hadoop.hdfs.server.federation.router.TestRouterQuota hadoop.hdfs.server.federation.router.TestRouterNamenodeHeartbeat hadoop.hdfs.server.federation.resolver.order.TestLocalResolver hadoop.hdfs.server.federation.resolver.TestMultipleDestinationResolver hadoop.yarn.server.nodemanager.TestNodeStatusUpdater hadoop.yarn.server.resourcemanager.TestClientRMService hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker hadoop.mapreduce.task.reduce.TestFetcher hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter hadoop.yarn.sls.TestSLSRunner hadoop.resourceestimator.solver.impl.TestLpSolver hadoop.resourceestimator.service.TestResourceEstimatorService cc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/268/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/268/artifact/out/diff-compile-javac-root.txt [476K] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/268/artifact/out/diff-checkstyle-root.txt [16M] hadolint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/268/artifact/out/diff-patch-hadolint.txt [4.0K] mvnsite: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/268/artifact/out/patch-mvnsite-root.txt [564K] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/268/artifact/out/pathlen.txt [12K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/268/artifact/out/diff-patch-pylint.txt [48K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/268/artifact/out/diff-patch-shellcheck.txt [56K] shelldocs: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/268/artifact/out/diff-patch-shelldocs.txt [48K] whitespace: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/268/artifact/out/whitespace-eol.txt [12M] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/268/artifact/out/whitespace-tabs.txt [1.3M] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/268/artifact/out/diff-javadoc-javadoc-root.txt [20K] spotbugs: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/268/artifact/out/branch-spotbugs-hadoop-build-tools.txt [60K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/268/artifact/out/branch-spotbugs-hadoop-project.txt [24K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/268/artifact/out/branch-spotbugs-hadoop-common-project_hadoop-annotations.txt [24K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/268/artifact/out/branch-spotbugs-hadoop-project-dist.txt [24K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/268/artifact/out/branch-spotbugs-hadoop-assemblies.txt [24K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/268/artifact/out/branch-spotbugs-hadoop-maven-plugins.txt [28K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/268/artifact/out/branch-spotbugs-hadoop-common-project_hadoop-minikdc.txt [24K]
