[jira] [Created] (YARN-9051) Integrate multiple CustomResourceTypesConfigurationProvider implementations into one
Szilard Nemeth created YARN-9051:
Summary: Integrate multiple
CustomResourceTypesConfigurationProvider implementations into one
Key: YARN-9051
URL: https://issues.apache.org/jira/browse/YARN-9051
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Szilard Nemeth
Assignee: Szilard Nemeth
CustomResourceTypesConfigurationProvider (extends LocalConfigurationProvider)
has 5 implementations on trunk nowadays.
These could be integrated into 1 common class.
Also,
{{org.apache.hadoop.yarn.util.resource.TestResourceUtils#addNewTypesToResources}}
has similar functionality so this can be considered as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/966/
[Nov 22, 2018 4:55:22 AM] (sunilg) YARN-8964. [UI2] YARN ui2 should use
clusters/{cluster name} for all
[Nov 22, 2018 5:16:10 AM] (msingh) HDDS-814.
dfs.ratis.leader.election.minimum.timeout.duration should not
[Nov 22, 2018 8:52:29 AM] (wwei) YARN-8984. AMRMClient#OutstandingSchedRequests
leaks when AllocationTags
[Nov 22, 2018 9:32:07 AM] (shashikant) HDDS-866. Handle
RaftRetryFailureException in OzoneClient. Contributed
[Nov 22, 2018 11:04:29 AM] (elek) HDDS-841. [JDK11] Fix Javadoc errors in
hadoop-hdds-container-service
[Nov 22, 2018 1:09:14 PM] (shashikant) HDDS-869. Fix log message in
XceiverClientRatis#sendCommandAsync.
[Nov 22, 2018 2:34:30 PM] (rohithsharmaks) YARN-9042. Fix javadoc error in
deviceplugin package. Contributed by
-1 overall
The following subsystems voted -1:
asflicense findbugs hadolint pathlen unit
The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace
The following subsystems are considered long running:
(runtime bigger than 1h 0m 0s)
unit
Specific tests:
Failed junit tests :
hadoop.util.TestDiskCheckerWithDiskIo
hadoop.util.TestReadWriteDiskValidator
hadoop.hdfs.web.TestWebHdfsTimeouts
hadoop.mapreduce.jobhistory.TestEvents
cc:
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/966/artifact/out/diff-compile-cc-root.txt
[4.0K]
javac:
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/966/artifact/out/diff-compile-javac-root.txt
[336K]
checkstyle:
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/966/artifact/out/diff-checkstyle-root.txt
[17M]
hadolint:
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/966/artifact/out/diff-patch-hadolint.txt
[4.0K]
pathlen:
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/966/artifact/out/pathlen.txt
[12K]
pylint:
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/966/artifact/out/diff-patch-pylint.txt
[40K]
shellcheck:
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/966/artifact/out/diff-patch-shellcheck.txt
[68K]
shelldocs:
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/966/artifact/out/diff-patch-shelldocs.txt
[12K]
whitespace:
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/966/artifact/out/whitespace-eol.txt
[9.3M]
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/966/artifact/out/whitespace-tabs.txt
[1.1M]
findbugs:
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/966/artifact/out/branch-findbugs-hadoop-hdds_client.txt
[8.0K]
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/966/artifact/out/branch-findbugs-hadoop-hdds_container-service.txt
[4.0K]
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/966/artifact/out/branch-findbugs-hadoop-hdds_framework.txt
[4.0K]
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/966/artifact/out/branch-findbugs-hadoop-hdds_server-scm.txt
[8.0K]
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/966/artifact/out/branch-findbugs-hadoop-hdds_tools.txt
[4.0K]
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/966/artifact/out/branch-findbugs-hadoop-ozone_client.txt
[8.0K]
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/966/artifact/out/branch-findbugs-hadoop-ozone_common.txt
[4.0K]
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/966/artifact/out/branch-findbugs-hadoop-ozone_objectstore-service.txt
[8.0K]
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/966/artifact/out/branch-findbugs-hadoop-ozone_ozone-manager.txt
[4.0K]
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/966/artifact/out/branch-findbugs-hadoop-ozone_ozonefs.txt
[12K]
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/966/artifact/out/branch-findbugs-hadoop-ozone_s3gateway.txt
[4.0K]
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/966/artifact/out/branch-findbugs-hadoop-ozone_tools.txt
[8.0K]
javadoc:
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/966/artifact/out/diff-javadoc-javadoc-root.txt
[752K]
unit:
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/966/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
[176K]
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/966/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
[324K]
[VOTE] Release Apache Hadoop 3.2.0 - RC0
Hi folks, Thanks to all contributors who helped in this release [1]. I have created first release candidate (RC0) for Apache Hadoop 3.2.0. Artifacts for this RC are available here: http://home.apache.org/~sunilg/hadoop-3.2.0-RC0/ RC tag in git is release-3.2.0-RC0. The maven artifacts are available via repository.apache.org at https://repository.apache.org/content/repositories/orgapachehadoop-1174/ This vote will run 7 days (5 weekdays), ending on Nov 30 at 11:59 pm PST. 3.2.0 contains 1079 [2] fixed JIRA issues since 3.1.0. Below feature additions are the highlights of this release. 1. Node Attributes Support in YARN 2. Hadoop Submarine project for running Deep Learning workloads on YARN 3. Support service upgrade via YARN Service API and CLI 4. HDFS Storage Policy Satisfier 5. Support Windows Azure Storage - Blob file system in Hadoop 6. Phase 3 improvements for S3Guard and Phase 5 improvements S3a 7. Improvements in Router-based HDFS federation Thanks to Wangda, Vinod, Marton for helping me in preparing the release. I have done few testing with my pseudo cluster. My +1 to start. Regards, Sunil [1] https://lists.apache.org/thread.html/68c1745dcb65602aecce6f7e6b7f0af3d974b1bf0048e7823e58b06f@%3Cyarn-dev.hadoop.apache.org%3E [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.2.0) AND fixVersion not in (3.1.0, 3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY fixVersion ASC
[jira] [Created] (YARN-9050) Usability improvements for scheduler activities
Tao Yang created YARN-9050:
--
Summary: Usability improvements for scheduler activities
Key: YARN-9050
URL: https://issues.apache.org/jira/browse/YARN-9050
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Tao Yang
Assignee: Tao Yang
Attachments: image-2018-11-23-16-46-38-138.png
We have did some usability improvements for scheduler activities based YARN3.1
in our cluster as follows:
1. Not available for multi-thread asynchronous scheduling. App and node
activites maybe confused when multiple scheduling threads record activites of
different allocation processes in the same variables like appsAllocation and
recordingNodesAllocation in ActivitiesManager. I think these variables should
be thread-local to make activities clear between multiple threads.
2. Incomplete activites for multi-node lookup machanism, since ActivitiesLogger
will skip recording through {{if (node == null || activitiesManager == null)
return; }} when node is null which represents this allocation is for
multi-nodes. We need support recording activities for multi-node lookup
machanism.
3. Current app activites can not meet requirements of diagnostics, for example,
we can know that node doesn't match request but hard to know why, especially
when using placement constraints, it's difficult to make a detailed diagnosis
manually. So I propose to improve the diagnoses of activites, add diagnosis for
placement constraints check, update insufficient resource diagnosis with
detailed info (like 'insufficient resource names:[memory-mb]') and so on.
4. Add more useful fields for app activities, in some scenarios we need to
distinguish different requests but can't locate requests based on app
activities info, there are some other fields can help to filter what we want
such as allocation tags. We have added containerPriority, allocationRequestId
and allocationTags fields in AppAllocation.
5. Filter app activities by key fields, sometimes the results of app activities
is massive, it's hard to find what we want. We have support filter by
allocation-tags to meet requirements from some apps, more over, we can take
container-priority and allocation-request-id as candidates if necessary.
6. Aggragate app activities by diagnoses. For a single allocation process,
activities still can be massive in a large cluster, we frequently want to know
why request can't be allocated in cluster, it's hard to check every node
manually in a large cluster, so that aggragation for app activities by
diagnoses is neccessary to solve this trouble. We have added groupingType
parameter for app-activities REST API for this, supports grouping by
diagnositics and example like this:
!image-2018-11-23-16-46-38-138.png!
I think we can have a discuss about these points, useful improvements which can
be accepted will be added into the patch. Thanks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
