[jira] [Commented] (YARN-8401) Yarnui2 not working with out internet connection
[
https://issues.apache.org/jira/browse/YARN-8401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16521885#comment-16521885
]
Bibin A Chundatt commented on YARN-8401:
[~sunilg]
Any update required??
> Yarnui2 not working with out internet connection
>
>
> Key: YARN-8401
> URL: https://issues.apache.org/jira/browse/YARN-8401
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Blocker
> Attachments: YARN-8401.001.patch
>
>
> {code}
> 2018-06-06 21:10:58,611 WARN org.eclipse.jetty.webapp.WebAppContext: Failed
> startup of context
> o.e.j.w.WebAppContext@108a46d6{/ui2,file:///opt/HA/310/install/hadoop/resourcemanager/share/hadoop/yarn/webapps/ui2/,null}
> java.net.UnknownHostException: java.sun.com
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:589)
> at java.net.Socket.connect(Socket.java:538)
> at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
> at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
> at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
> at sun.net.www.http.HttpClient.(HttpClient.java:211)
> at sun.net.www.http.HttpClient.New(HttpClient.java:308)
> at sun.net.www.http.HttpClient.New(HttpClient.java:326)
> at
> sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1168)
> at
> sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1104)
> at
> sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:998)
> at
> sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:932)
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1512)
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1440)
> at
> com.sun.org.apache.xerces.internal.impl.XMLEntityManager.setupCurrentEntity(XMLEntityManager.java:646)
> at
> com.sun.org.apache.xerces.internal.impl.XMLEntityManager.startEntity(XMLEntityManager.java:1300)
> at
> com.sun.org.apache.xerces.internal.impl.XMLEntityManager.startDTDEntity(XMLEntityManager.java:1267)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDTDScannerImpl.setInputSource(XMLDTDScannerImpl.java:263)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$DTDDriver.dispatch(XMLDocumentScannerImpl.java:1164)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$DTDDriver.next(XMLDocumentScannerImpl.java:1050)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:964)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)
> at
> com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:117)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
> at
> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)
> at
> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)
> at
> com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
> at
> com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213)
> at
> com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:649)
> at
> com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(SAXParserImpl.java:333)
> at org.eclipse.jetty.xml.XmlParser.parse(XmlParser.java:255)
> at org.eclipse.jetty.webapp.Descriptor.parse(Descriptor.java:54)
> at
> org.eclipse.jetty.webapp.WebDescriptor.parse(WebDescriptor.java:207)
> at org.eclipse.jetty.webapp.MetaData.setWebXml(MetaData.java:189)
> at
> org.eclipse.jetty.webapp.WebXmlConfiguration.preConfigure(WebXmlConfiguration.java:60)
> at
> org.eclipse.jetty.webapp.WebAppContext.preConfigure(WebAppContext.java:485)
> at
> org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:521)
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
[jira] [Commented] (YARN-8434) Nodemanager not registering to active RM in federation
[
https://issues.apache.org/jira/browse/YARN-8434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16521846#comment-16521846
]
Bibin A Chundatt commented on YARN-8434:
Thank you [~subru] for looking in to the issue
I have configured the following in NM as per the documentation
[http://hadoop.apache.org/docs/r3.0.1/hadoop-yarn/hadoop-yarn-site/Federation.html]
{code:java}
yarn.client.failover-proxy-provider
org.apache.hadoop.yarn.server.federation.failover.FederationRMFailoverProxyProvider
{code}
But the same is used for NM-RM communication too. {{ServerRMProxy}} also uses
{{yarn.client.failover-proxy-provider}} for failover .
Seems NM also will try to connect to RM using
{{FederationRMFailoverProxyProvider}}
{code:java}
When HA is enabled, the class to be used by Clients, AMs and
NMs to failover to the Active RM. It should extend
org.apache.hadoop.yarn.client.RMFailoverProxyProvider
yarn.client.failover-proxy-provider
org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider
{code}
> Nodemanager not registering to active RM in federation
> --
>
> Key: YARN-8434
> URL: https://issues.apache.org/jira/browse/YARN-8434
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Blocker
>
> FederationRMFailoverProxyProvider doesn't handle connecting to active RM.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8047) RMWebApp make external class pluggable
[
https://issues.apache.org/jira/browse/YARN-8047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16521836#comment-16521836
]
genericqa commented on YARN-8047:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
34s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
3s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
50s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 5m
21s{color} | {color:red} hadoop-yarn in trunk failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 43s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
53s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 4m
16s{color} | {color:red} hadoop-yarn in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 4m 16s{color}
| {color:red} hadoop-yarn in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
13s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 53s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
45s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
14s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 67m
6s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
35s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}153m 41s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd |
| JIRA Issue | YARN-8047 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12928754/YARN-8047-002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit sha
[jira] [Commented] (YARN-8443) Total #VCores in cluster metrics is wrong when CapacityScheduler reserved some containers
[
https://issues.apache.org/jira/browse/YARN-8443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16521767#comment-16521767
]
Tao Yang commented on YARN-8443:
Thanks [~cheersyang], [~giovanni.fumarola] for reviewing and committing!
> Total #VCores in cluster metrics is wrong when CapacityScheduler reserved
> some containers
> -
>
> Key: YARN-8443
> URL: https://issues.apache.org/jira/browse/YARN-8443
> Project: Hadoop YARN
> Issue Type: Bug
> Components: webapp
>Affects Versions: 2.9.0, 3.1.0, 3.2.0
>Reporter: Tao Yang
>Assignee: Tao Yang
>Priority: Major
> Fix For: 3.2.0, 3.1.1, 2.9.2, 3.0.4
>
> Attachments: YARN-8443.001.patch
>
>
> Cluster metrics on the web UI will give wrong Total Vcores when there is
> reserved container for CapacityScheduler.
> Reference code:
> {code:java|title=ClusterMetricsInfo.java}
> if (rs instanceof CapacityScheduler) {
> CapacityScheduler cs = (CapacityScheduler) rs;
> this.totalMB = availableMB + allocatedMB + reservedMB;
> this.totalVirtualCores =
> availableVirtualCores + allocatedVirtualCores + containersReserved;
>...
> }
> {code}
> The key of this problem is the calculation of totalVirtualCores,
> {{containersReserved}} is the number of reserved containers, not reserved
> VCores. The correct calculation expression of totalVirtualCores should be
> (availableVirtualCores + allocatedVirtualCores + reservedVirtualCores)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8443) Total #VCores in cluster metrics is wrong when CapacityScheduler reserved some containers
[
https://issues.apache.org/jira/browse/YARN-8443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16521765#comment-16521765
]
Weiwei Yang commented on YARN-8443:
---
I've committed this to trunk, branch-3.1, branch-3.0 and branch-2.9. Thanks for
the fix [~Tao Yang] and thanks for the review [~giovanni.fumarola].
> Total #VCores in cluster metrics is wrong when CapacityScheduler reserved
> some containers
> -
>
> Key: YARN-8443
> URL: https://issues.apache.org/jira/browse/YARN-8443
> Project: Hadoop YARN
> Issue Type: Bug
> Components: webapp
>Affects Versions: 2.9.0, 3.1.0, 3.2.0
>Reporter: Tao Yang
>Assignee: Tao Yang
>Priority: Major
> Fix For: 3.2.0, 3.1.1, 2.9.2, 3.0.4
>
> Attachments: YARN-8443.001.patch
>
>
> Cluster metrics on the web UI will give wrong Total Vcores when there is
> reserved container for CapacityScheduler.
> Reference code:
> {code:java|title=ClusterMetricsInfo.java}
> if (rs instanceof CapacityScheduler) {
> CapacityScheduler cs = (CapacityScheduler) rs;
> this.totalMB = availableMB + allocatedMB + reservedMB;
> this.totalVirtualCores =
> availableVirtualCores + allocatedVirtualCores + containersReserved;
>...
> }
> {code}
> The key of this problem is the calculation of totalVirtualCores,
> {{containersReserved}} is the number of reserved containers, not reserved
> VCores. The correct calculation expression of totalVirtualCores should be
> (availableVirtualCores + allocatedVirtualCores + reservedVirtualCores)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8443) Total #VCores in cluster metrics is wrong when CapacityScheduler reserved some containers
[
https://issues.apache.org/jira/browse/YARN-8443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiwei Yang updated YARN-8443:
--
Fix Version/s: 2.9.2
> Total #VCores in cluster metrics is wrong when CapacityScheduler reserved
> some containers
> -
>
> Key: YARN-8443
> URL: https://issues.apache.org/jira/browse/YARN-8443
> Project: Hadoop YARN
> Issue Type: Bug
> Components: webapp
>Affects Versions: 2.9.0, 3.1.0, 3.2.0
>Reporter: Tao Yang
>Assignee: Tao Yang
>Priority: Major
> Fix For: 3.2.0, 3.1.1, 2.9.2, 3.0.4
>
> Attachments: YARN-8443.001.patch
>
>
> Cluster metrics on the web UI will give wrong Total Vcores when there is
> reserved container for CapacityScheduler.
> Reference code:
> {code:java|title=ClusterMetricsInfo.java}
> if (rs instanceof CapacityScheduler) {
> CapacityScheduler cs = (CapacityScheduler) rs;
> this.totalMB = availableMB + allocatedMB + reservedMB;
> this.totalVirtualCores =
> availableVirtualCores + allocatedVirtualCores + containersReserved;
>...
> }
> {code}
> The key of this problem is the calculation of totalVirtualCores,
> {{containersReserved}} is the number of reserved containers, not reserved
> VCores. The correct calculation expression of totalVirtualCores should be
> (availableVirtualCores + allocatedVirtualCores + reservedVirtualCores)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8443) Total #VCores in cluster metrics is wrong when CapacityScheduler reserved some containers
[
https://issues.apache.org/jira/browse/YARN-8443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16521763#comment-16521763
]
Hudson commented on YARN-8443:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14472 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/14472/])
YARN-8443. Total #VCores in cluster metrics is wrong when (wwei: rev
440140cea6718229094a3d2b97b9b9bd28b95d9b)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/ClusterMetricsInfo.java
> Total #VCores in cluster metrics is wrong when CapacityScheduler reserved
> some containers
> -
>
> Key: YARN-8443
> URL: https://issues.apache.org/jira/browse/YARN-8443
> Project: Hadoop YARN
> Issue Type: Bug
> Components: webapp
>Affects Versions: 2.9.0, 3.1.0, 3.2.0
>Reporter: Tao Yang
>Assignee: Tao Yang
>Priority: Major
> Fix For: 3.2.0, 3.1.1, 3.0.4
>
> Attachments: YARN-8443.001.patch
>
>
> Cluster metrics on the web UI will give wrong Total Vcores when there is
> reserved container for CapacityScheduler.
> Reference code:
> {code:java|title=ClusterMetricsInfo.java}
> if (rs instanceof CapacityScheduler) {
> CapacityScheduler cs = (CapacityScheduler) rs;
> this.totalMB = availableMB + allocatedMB + reservedMB;
> this.totalVirtualCores =
> availableVirtualCores + allocatedVirtualCores + containersReserved;
>...
> }
> {code}
> The key of this problem is the calculation of totalVirtualCores,
> {{containersReserved}} is the number of reserved containers, not reserved
> VCores. The correct calculation expression of totalVirtualCores should be
> (availableVirtualCores + allocatedVirtualCores + reservedVirtualCores)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6973) Adding RM Cluster Id in ApplicationReport
[ https://issues.apache.org/jira/browse/YARN-6973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16521761#comment-16521761 ] HuangTao commented on YARN-6973: Now, I am working on this task. > Adding RM Cluster Id in ApplicationReport > - > > Key: YARN-6973 > URL: https://issues.apache.org/jira/browse/YARN-6973 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Giovanni Matteo Fumarola >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8443) Total #VCores in cluster metrics is wrong when CapacityScheduler reserved some containers
[
https://issues.apache.org/jira/browse/YARN-8443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiwei Yang updated YARN-8443:
--
Fix Version/s: 3.0.4
> Total #VCores in cluster metrics is wrong when CapacityScheduler reserved
> some containers
> -
>
> Key: YARN-8443
> URL: https://issues.apache.org/jira/browse/YARN-8443
> Project: Hadoop YARN
> Issue Type: Bug
> Components: webapp
>Affects Versions: 2.9.0, 3.1.0, 3.2.0
>Reporter: Tao Yang
>Assignee: Tao Yang
>Priority: Major
> Fix For: 3.2.0, 3.1.1, 3.0.4
>
> Attachments: YARN-8443.001.patch
>
>
> Cluster metrics on the web UI will give wrong Total Vcores when there is
> reserved container for CapacityScheduler.
> Reference code:
> {code:java|title=ClusterMetricsInfo.java}
> if (rs instanceof CapacityScheduler) {
> CapacityScheduler cs = (CapacityScheduler) rs;
> this.totalMB = availableMB + allocatedMB + reservedMB;
> this.totalVirtualCores =
> availableVirtualCores + allocatedVirtualCores + containersReserved;
>...
> }
> {code}
> The key of this problem is the calculation of totalVirtualCores,
> {{containersReserved}} is the number of reserved containers, not reserved
> VCores. The correct calculation expression of totalVirtualCores should be
> (availableVirtualCores + allocatedVirtualCores + reservedVirtualCores)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8443) Total #VCores in cluster metrics is wrong when CapacityScheduler reserved some containers
[
https://issues.apache.org/jira/browse/YARN-8443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiwei Yang updated YARN-8443:
--
Summary: Total #VCores in cluster metrics is wrong when CapacityScheduler
reserved some containers (was: Cluster metrics have wrong Total VCores when
there is reserved container for CapacityScheduler)
> Total #VCores in cluster metrics is wrong when CapacityScheduler reserved
> some containers
> -
>
> Key: YARN-8443
> URL: https://issues.apache.org/jira/browse/YARN-8443
> Project: Hadoop YARN
> Issue Type: Bug
> Components: webapp
>Affects Versions: 2.9.0, 3.1.0, 3.2.0
>Reporter: Tao Yang
>Assignee: Tao Yang
>Priority: Major
> Attachments: YARN-8443.001.patch
>
>
> Cluster metrics on the web UI will give wrong Total Vcores when there is
> reserved container for CapacityScheduler.
> Reference code:
> {code:java|title=ClusterMetricsInfo.java}
> if (rs instanceof CapacityScheduler) {
> CapacityScheduler cs = (CapacityScheduler) rs;
> this.totalMB = availableMB + allocatedMB + reservedMB;
> this.totalVirtualCores =
> availableVirtualCores + allocatedVirtualCores + containersReserved;
>...
> }
> {code}
> The key of this problem is the calculation of totalVirtualCores,
> {{containersReserved}} is the number of reserved containers, not reserved
> VCores. The correct calculation expression of totalVirtualCores should be
> (availableVirtualCores + allocatedVirtualCores + reservedVirtualCores)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8443) Cluster metrics have wrong Total VCores when there is reserved container for CapacityScheduler
[
https://issues.apache.org/jira/browse/YARN-8443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16521739#comment-16521739
]
Weiwei Yang commented on YARN-8443:
---
Thanks for fixing this [~Tao Yang], +1. I will help to commit this today.
> Cluster metrics have wrong Total VCores when there is reserved container for
> CapacityScheduler
> --
>
> Key: YARN-8443
> URL: https://issues.apache.org/jira/browse/YARN-8443
> Project: Hadoop YARN
> Issue Type: Bug
> Components: webapp
>Affects Versions: 2.9.0, 3.1.0, 3.2.0
>Reporter: Tao Yang
>Assignee: Tao Yang
>Priority: Major
> Attachments: YARN-8443.001.patch
>
>
> Cluster metrics on the web UI will give wrong Total Vcores when there is
> reserved container for CapacityScheduler.
> Reference code:
> {code:java|title=ClusterMetricsInfo.java}
> if (rs instanceof CapacityScheduler) {
> CapacityScheduler cs = (CapacityScheduler) rs;
> this.totalMB = availableMB + allocatedMB + reservedMB;
> this.totalVirtualCores =
> availableVirtualCores + allocatedVirtualCores + containersReserved;
>...
> }
> {code}
> The key of this problem is the calculation of totalVirtualCores,
> {{containersReserved}} is the number of reserved containers, not reserved
> VCores. The correct calculation expression of totalVirtualCores should be
> (availableVirtualCores + allocatedVirtualCores + reservedVirtualCores)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8446) Support of managing multi-dimensional resources
[
https://issues.apache.org/jira/browse/YARN-8446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiwei Yang updated YARN-8446:
--
Description:
To better support long running jobs and services, we need to extend YARN to
support other resources, such as disk, IP, port. Current resource types is not
flexible enough to make this work because it only supports COUNTABLE type which
is single value.
Propose to extend resource types by adding two more general types, such as SET,
MULTIDIMENSIONAL (naming TBD). With schema like
*SET*: a set of values
{noformat}
["10.100.0.1", "10.100.0.2"]
["9981", "9982", "9983"]
{noformat}
*MULTIDIMENSIONAL*: a set of values, each value can be a resource instance with
multiple values.
{noformat}
[ disk1 : { attributes: { "type" : "SATA", "index" : 1 }, "size" : "500gb",
"iops" : "1000" },
disk2 : { attributes: { "type" : "SSD", "index" : 2 }, "size" : "100gb",
"iops" : "1000" } ]
{noformat}
this way, we could support better resource management and isolations. The idea
is to make this as general as possible so we can easily support some other
complex resources.
was:
To better support long running jobs and services, we need to extend YARN to
support other resources, such as disk, IP, port. Current resource types is not
flexible enough to make this work because it only supports COUNTABLE type which
is single value.
Propose to extend resource types by adding two more general types, such as SET,
MULTIDIMENSIONAL (naming TBD). With schema like
*SET*: a set of values
{noformat}
["10.100.0.1", "10.100.0.2"]
["9981", "9982", "9983"]
{noformat}
*MULTIDIMENSIONAL*: a set of values, each value can be a resource instance with
multiple values.
{noformat}
[ disk1 : { attributes: { "type" : "SATA", "index" : 1 }, "size" : "500gb",
"iops" : "1000" },
disk2 : { attributes: { "type" : "SSD", "index" : 2 }, "size" : "100gb",
"iops" : "1000" } ]
{noformat}
this way, we could support better resource management and isolations. The idea
is to make this as general as possible so we can easily support some other
complex resources.
> Support of managing multi-dimensional resources
> ---
>
> Key: YARN-8446
> URL: https://issues.apache.org/jira/browse/YARN-8446
> Project: Hadoop YARN
> Issue Type: New Feature
>Reporter: Weiwei Yang
>Priority: Major
>
> To better support long running jobs and services, we need to extend YARN to
> support other resources, such as disk, IP, port. Current resource types is
> not flexible enough to make this work because it only supports COUNTABLE type
> which is single value.
> Propose to extend resource types by adding two more general types, such as
> SET, MULTIDIMENSIONAL (naming TBD). With schema like
> *SET*: a set of values
> {noformat}
> ["10.100.0.1", "10.100.0.2"]
> ["9981", "9982", "9983"]
> {noformat}
> *MULTIDIMENSIONAL*: a set of values, each value can be a resource instance
> with multiple values.
> {noformat}
> [ disk1 : { attributes: { "type" : "SATA", "index" : 1 }, "size" : "500gb",
> "iops" : "1000" },
> disk2 : { attributes: { "type" : "SSD", "index" : 2 }, "size" : "100gb",
> "iops" : "1000" } ]
> {noformat}
> this way, we could support better resource management and isolations. The
> idea is to make this as general as possible so we can easily support some
> other complex resources.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8184) Too many metrics if containerLocalizer/ResourceLocalizationService uses ReadWriteDiskValidator
[ https://issues.apache.org/jira/browse/YARN-8184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16521436#comment-16521436 ] Hudson commented on YARN-8184: -- FAILURE: Integrated in Jenkins build Hadoop-precommit-ozone-acceptance #20 (See [https://builds.apache.org/job/Hadoop-precommit-ozone-acceptance/20/]) YARN-8184. Too many metrics if (yufei_gu: [https://github.com/apache/hadoop/commit/1cdce86d33d4b73ba6dd4136c966eb7e822b6f36]) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ContainerLocalizer.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java > Too many metrics if containerLocalizer/ResourceLocalizationService uses > ReadWriteDiskValidator > -- > > Key: YARN-8184 > URL: https://issues.apache.org/jira/browse/YARN-8184 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Reporter: Yufei Gu >Assignee: Yufei Gu >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-8184.001.patch, YARN-8184.002.patch > > > ContainerLocalizer or ResourceLocalizationService will use the > ReadWriteDiskValidator as its disk validator when it downloads files if we > configure the yarn.nodemanger.disk-validator to ReadWriteDiskValidator's > name. In that case, ReadWriteDiskValidator will create a metric item for each > directory localized, which will be too many metrics. We should let > ContainerLocalizer only use the basic disk validator. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8326) Yarn 3.0 seems runs slower than Yarn 2.6
[ https://issues.apache.org/jira/browse/YARN-8326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16521437#comment-16521437 ] Hudson commented on YARN-8326: -- FAILURE: Integrated in Jenkins build Hadoop-precommit-ozone-acceptance #20 (See [https://builds.apache.org/job/Hadoop-precommit-ozone-acceptance/20/]) YARN-8326. Removed exit code file check for launched container. (eyang: [https://github.com/apache/hadoop/commit/8a32bc39eb210fca8052c472601e24c2446b4cc2]) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java > Yarn 3.0 seems runs slower than Yarn 2.6 > > > Key: YARN-8326 > URL: https://issues.apache.org/jira/browse/YARN-8326 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Affects Versions: 3.0.0 > Environment: This is the yarn-site.xml for 3.0. > > > > hadoop.registry.dns.bind-port > 5353 > > > hadoop.registry.dns.domain-name > hwx.site > > > hadoop.registry.dns.enabled > true > > > hadoop.registry.dns.zone-mask > 255.255.255.0 > > > hadoop.registry.dns.zone-subnet > 172.17.0.0 > > > manage.include.files > false > > > yarn.acl.enable > false > > > yarn.admin.acl > yarn > > > yarn.client.nodemanager-connect.max-wait-ms > 6 > > > yarn.client.nodemanager-connect.retry-interval-ms > 1 > > > yarn.http.policy > HTTP_ONLY > > > yarn.log-aggregation-enable > false > > > yarn.log-aggregation.retain-seconds > 2592000 > > > yarn.log.server.url > > [http://xx:19888/jobhistory/logs|http://whiny2.fyre.ibm.com:19888/jobhistory/logs] > > > yarn.log.server.web-service.url > > [http://xx:8188/ws/v1/applicationhistory|http://whiny2.fyre.ibm.com:8188/ws/v1/applicationhistory] > > > yarn.node-labels.enabled > false > > > yarn.node-labels.fs-store.retry-policy-spec > 2000, 500 > > > yarn.node-labels.fs-store.root-dir > /system/yarn/node-labels > > > yarn.nodemanager.address > 0.0.0.0:45454 > > > yarn.nodemanager.admin-env > MALLOC_ARENA_MAX=$MALLOC_ARENA_MAX > > > yarn.nodemanager.aux-services > mapreduce_shuffle,spark2_shuffle,timeline_collector > > > yarn.nodemanager.aux-services.mapreduce_shuffle.class > org.apache.hadoop.mapred.ShuffleHandler > > > yarn.nodemanager.aux-services.spark2_shuffle.class > org.apache.spark.network.yarn.YarnShuffleService > > > yarn.nodemanager.aux-services.spark2_shuffle.classpath > /usr/spark2/aux/* > > > yarn.nodemanager.aux-services.spark_shuffle.class > org.apache.spark.network.yarn.YarnShuffleService > > > yarn.nodemanager.aux-services.timeline_collector.class > > org.apache.hadoop.yarn.server.timelineservice.collector.PerNodeTimelineCollectorsAuxService > > > yarn.nodemanager.bind-host > 0.0.0.0 > > > yarn.nodemanager.container-executor.class > > org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor > > > yarn.nodemanager.container-metrics.unregister-delay-ms > 6 > > > yarn.nodemanager.container-monitor.interval-ms > 3000 > > > yarn.nodemanager.delete.debug-delay-sec > 0 > > > > yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage > 90 > > > yarn.nodemanager.disk-health-checker.min-free-space-per-disk-mb > 1000 > > > yarn.nodemanager.disk-health-checker.min-healthy-disks > 0.25 > > > yarn.nodemanager.health-checker.interval-ms > 135000 > > > yarn.nodemanager.health-checker.script.timeout-ms > 6 > > > > yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage > false > > > yarn.nodemanager.linux-container-executor.group > hadoop > > > > yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users > false > > > yarn.nodemanager.local-dirs > /hadoop/yarn/local > > > yarn.nodemanager.log-aggregation.compression-type > gz > > > yarn.nodemanager.log-aggregation.debug-enabled > false > > > yarn.nodemanager.log-aggregation.num-log-files-per-app > 30 > > > > yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds > 3600 > > > yarn.nodemanager.log-dirs > /hadoop/yarn/log > > > yarn.nodemanager.log.retain-seconds > 604800 > > > yarn.nodemanager.pmem-check-enabled > false > > > yarn.nodemanager.recovery.dir > /var/log/hadoop-yarn/nodemanager/recovery-state > > > yarn.nodemanager.recovery.enabled > true > > > yarn.nodemanager.recovery.supervised > true > > > yarn.nodemanager.remote-app-log-dir > /app-logs > > > yarn.nodemanager.remote-app-log-dir-suffix > logs > > > yarn.nodemanager.resource-plugins > > > > yarn.nodemanager.resource-plugins.gpu.allowed-gpu-devices > auto >
[jira] [Commented] (YARN-7449) Split up class TestYarnClient to TestYarnClient and TestYarnClientImpl
[
https://issues.apache.org/jira/browse/YARN-7449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16521422#comment-16521422
]
Hudson commented on YARN-7449:
--
FAILURE: Integrated in Jenkins build Hadoop-precommit-ozone-acceptance #20 (See
[https://builds.apache.org/job/Hadoop-precommit-ozone-acceptance/20/])
YARN-7449. Split up class TestYarnClient to TestYarnClient and (miklos.szegedi:
[https://github.com/apache/hadoop/commit/bbbc7cc426f71ad0fe4174efcd25e5ac3f62b501])
* (add)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestYarnClientImpl.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestYarnClient.java
> Split up class TestYarnClient to TestYarnClient and TestYarnClientImpl
> --
>
> Key: YARN-7449
> URL: https://issues.apache.org/jira/browse/YARN-7449
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: client, yarn
>Reporter: Yufei Gu
>Assignee: Szilard Nemeth
>Priority: Minor
> Labels: newbie, newbie++
> Fix For: 3.2.0
>
> Attachments: YARN-7449-001.patch, YARN-7449.002.patch
>
>
> {{TestYarnClient}} tests both {{YarnClient}} and {{YarnClientImpl}}. We
> should test {{YarnClient}} without thinking of its implementation. That's the
> whole point of {{YarnClient}}. There are bunch of refactors we could do. The
> first thing is to Split up class {{TestYarnClient}} to {{TestYarnClient}} and
> {{TestYarnClientImpl}}. Let {{TestYarnClient}} only tests {{YarnClient}}. All
> implementation related stuff go to {{TestYarnClientImpl}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8437) Build oom-listener fails on older versions
[ https://issues.apache.org/jira/browse/YARN-8437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16521419#comment-16521419 ] Hudson commented on YARN-8437: -- FAILURE: Integrated in Jenkins build Hadoop-precommit-ozone-acceptance #20 (See [https://builds.apache.org/job/Hadoop-precommit-ozone-acceptance/20/]) YARN-8437. Build oom-listener fails on older versions. (Miklos Szegedi (haibochen: [https://github.com/apache/hadoop/commit/4939ffedb151ce1550fcdd7ac04c79d8d0195891]) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/oom-listener/test/oom_listener_test_main.cc * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt > Build oom-listener fails on older versions > -- > > Key: YARN-8437 > URL: https://issues.apache.org/jira/browse/YARN-8437 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-8437.000.patch > > > oom-listener was introduced in YARN-4599. We have seen some build issues on > centos6. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8391) Investigate AllocationFileLoaderService.reloadListener locking issue
[ https://issues.apache.org/jira/browse/YARN-8391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16521420#comment-16521420 ] Hudson commented on YARN-8391: -- FAILURE: Integrated in Jenkins build Hadoop-precommit-ozone-acceptance #20 (See [https://builds.apache.org/job/Hadoop-precommit-ozone-acceptance/20/]) YARN-8391. Investigate AllocationFileLoaderService.reloadListener (miklos.szegedi: [https://github.com/apache/hadoop/commit/9a9e969570f23b627f9571819f388916d8fd7ec9]) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationFileLoaderService.java > Investigate AllocationFileLoaderService.reloadListener locking issue > > > Key: YARN-8391 > URL: https://issues.apache.org/jira/browse/YARN-8391 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Affects Versions: 3.2.0 >Reporter: Haibo Chen >Assignee: Szilard Nemeth >Priority: Critical > Fix For: 3.2.0 > > Attachments: YARN-8191.001.patch, YARN-8391.002.patch > > > Per findbugs report in YARN-8390, there is some inconsistent locking of > reloadListener > > h1. Warnings > Click on a warning row to see full context information. > h2. Multithreaded correctness Warnings > ||Code||Warning|| > |IS|Inconsistent synchronization of > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService.reloadListener; > locked 75% of time| > | |[Bug type IS2_INCONSISTENT_SYNC (click for > details)|https://builds.apache.org/job/PreCommit-YARN-Build/20939/artifact/out/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html#IS2_INCONSISTENT_SYNC] > > In class > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService > Field > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService.reloadListener > Synchronized 75% of the time > Unsynchronized access at AllocationFileLoaderService.java:[line 117] > Synchronized access at AllocationFileLoaderService.java:[line 212] > Synchronized access at AllocationFileLoaderService.java:[line 228] > Synchronized access at AllocationFileLoaderService.java:[line 269]| > h1. Details > h2. IS2_INCONSISTENT_SYNC: Inconsistent synchronization > The fields of this class appear to be accessed inconsistently with respect to > synchronization. This bug report indicates that the bug pattern detector > judged that > * The class contains a mix of locked and unlocked accesses, > * The class is *not* annotated as javax.annotation.concurrent.NotThreadSafe, > * At least one locked access was performed by one of the class's own > methods, and > * The number of unsynchronized field accesses (reads and writes) was no more > than one third of all accesses, with writes being weighed twice as high as > reads > A typical bug matching this bug pattern is forgetting to synchronize one of > the methods in a class that is intended to be thread-safe. > You can select the nodes labeled "Unsynchronized access" to show the code > locations where the detector believed that a field was accessed without > synchronization. > Note that there are various sources of inaccuracy in this detector; for > example, the detector cannot statically detect all situations in which a lock > is held. Also, even when the detector is accurate in distinguishing locked > vs. unlocked accesses, the code in question may still be correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8440) Typo in YarnConfiguration javadoc: "Miniumum request grant-able.."
[ https://issues.apache.org/jira/browse/YARN-8440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16521421#comment-16521421 ] Hudson commented on YARN-8440: -- FAILURE: Integrated in Jenkins build Hadoop-precommit-ozone-acceptance #20 (See [https://builds.apache.org/job/Hadoop-precommit-ozone-acceptance/20/]) YARN-8440. Typo in YarnConfiguration javadoc: "Miniumum request (miklos.szegedi: [https://github.com/apache/hadoop/commit/55432b09810b59ee361d0d4a8958efabb49fab3c]) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java > Typo in YarnConfiguration javadoc: "Miniumum request grant-able.." > -- > > Key: YARN-8440 > URL: https://issues.apache.org/jira/browse/YARN-8440 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Trivial > Fix For: 3.2.0 > > Attachments: YARN-8440.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8445) YARN native service doesn't allow service name equals to component name
[
https://issues.apache.org/jira/browse/YARN-8445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16521429#comment-16521429
]
Hudson commented on YARN-8445:
--
FAILURE: Integrated in Jenkins build Hadoop-precommit-ozone-acceptance #20 (See
[https://builds.apache.org/job/Hadoop-precommit-ozone-acceptance/20/])
YARN-8445. Improved error message for duplicated service and component (eyang:
[https://github.com/apache/hadoop/commit/9f15483c5d7c94251f4c84e0155449188f202779])
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/exceptions/RestApiErrorMessages.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/utils/ServiceApiUtil.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/java/org/apache/hadoop/yarn/service/TestServiceApiUtil.java
> YARN native service doesn't allow service name equals to component name
> ---
>
> Key: YARN-8445
> URL: https://issues.apache.org/jira/browse/YARN-8445
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Chandni Singh
>Assignee: Chandni Singh
>Priority: Major
> Fix For: 3.1.1
>
> Attachments: YARN-8445.001.patch
>
>
> Now YARN service doesn't allow specifying service name equals to component
> name.
> And it causes AM launch fails with msg like:
> {code}
> org.apache.hadoop.metrics2.MetricsException: Metrics source tf-zeppelin
> already exists!
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:152)
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:125)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:229)
> at
> org.apache.hadoop.yarn.service.ServiceMetrics.register(ServiceMetrics.java:75)
> at
> org.apache.hadoop.yarn.service.component.Component.(Component.java:193)
> at
> org.apache.hadoop.yarn.service.ServiceScheduler.createAllComponents(ServiceScheduler.java:552)
> at
> org.apache.hadoop.yarn.service.ServiceScheduler.buildInstance(ServiceScheduler.java:251)
> at
> org.apache.hadoop.yarn.service.ServiceScheduler.serviceInit(ServiceScheduler.java:283)
> at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
> at
> org.apache.hadoop.yarn.service.ServiceMaster.serviceInit(ServiceMaster.java:142)
> at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at org.apache.hadoop.yarn.service.ServiceMaster.main(ServiceMaster.java:338)
> 2018-06-18 06:50:39,473 [main] INFO service.ServiceScheduler - Stopping
> service scheduler
> {code}
> It's better to add this check in validation phase instead of failing AM.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8412) Move ResourceRequest.clone logic everywhere into a proper API
[ https://issues.apache.org/jira/browse/YARN-8412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16521431#comment-16521431 ] Hudson commented on YARN-8412: -- FAILURE: Integrated in Jenkins build Hadoop-precommit-ozone-acceptance #20 (See [https://builds.apache.org/job/Hadoop-precommit-ozone-acceptance/20/]) YARN-8412. Move ResourceRequest.clone logic everywhere into a proper (inigoiri: [https://github.com/apache/hadoop/commit/99948565cb5d5706241d7a8fc591e1617c499e03]) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AMRMClientImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClientOnRMRestart.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ResourceRequest.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/AMRMClientRelayer.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/placement/LocalityAppPlacementAllocator.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/policies/amrmproxy/LocalityMulticastAMRMProxyPolicy.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/scheduler/ResourceRequestSet.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/Application.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/BuilderUtils.java > Move ResourceRequest.clone logic everywhere into a proper API > - > > Key: YARN-8412 > URL: https://issues.apache.org/jira/browse/YARN-8412 > Project: Hadoop YARN > Issue Type: Task >Reporter: Botong Huang >Assignee: Botong Huang >Priority: Minor > Fix For: 2.10.0, 3.2.0 > > Attachments: YARN-8412-branch-2.v2.patch, YARN-8412.v1.patch, > YARN-8412.v2.patch > > > ResourceRequest.clone code is replicated in lots of places, some missing to > copy one field or two due to new fields added over time. This JIRA attempts > to move them into a proper API so that everyone can use this single > implementation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8441) Typo in CSQueueUtils local variable names: queueGuranteedResource
[ https://issues.apache.org/jira/browse/YARN-8441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16521424#comment-16521424 ] Hudson commented on YARN-8441: -- FAILURE: Integrated in Jenkins build Hadoop-precommit-ozone-acceptance #20 (See [https://builds.apache.org/job/Hadoop-precommit-ozone-acceptance/20/]) YARN-8441. Typo in CSQueueUtils local variable names: (miklos.szegedi: [https://github.com/apache/hadoop/commit/46f90581641feec37e285964df983d221bee5e1d]) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueueUtils.java > Typo in CSQueueUtils local variable names: queueGuranteedResource > - > > Key: YARN-8441 > URL: https://issues.apache.org/jira/browse/YARN-8441 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Trivial > Fix For: 3.2.0 > > Attachments: YARN-8441.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8442) Strange characters and missing spaces in FairScheduler documentation
[
https://issues.apache.org/jira/browse/YARN-8442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16521423#comment-16521423
]
Hudson commented on YARN-8442:
--
FAILURE: Integrated in Jenkins build Hadoop-precommit-ozone-acceptance #20 (See
[https://builds.apache.org/job/Hadoop-precommit-ozone-acceptance/20/])
YARN-8442. Strange characters and missing spaces in FairScheduler
(miklos.szegedi:
[https://github.com/apache/hadoop/commit/388fafa004dc405a4e10f4487cff7c5a714af32f])
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/FairScheduler.md
> Strange characters and missing spaces in FairScheduler documentation
>
>
> Key: YARN-8442
> URL: https://issues.apache.org/jira/browse/YARN-8442
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: YARN-8442.001.patch
>
>
> [https://hadoop.apache.org/docs/r3.1.0/hadoop-yarn/hadoop-yarn-site/FairScheduler.html]
> There are several missing spaces and strange characters in:
> Allocation file format / queuePlacementPolicy element / nestedUserQueue
> Quoting the wrong part of the document:
> {code:java}
> This is similar to ‘user’ rule,the difference being in ‘nestedUserQueue’
> rule,user...
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8444) NodeResourceMonitor crashes on bad swapFree value
[
https://issues.apache.org/jira/browse/YARN-8444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16521433#comment-16521433
]
Hudson commented on YARN-8444:
--
FAILURE: Integrated in Jenkins build Hadoop-precommit-ozone-acceptance #20 (See
[https://builds.apache.org/job/Hadoop-precommit-ozone-acceptance/20/])
YARN-8444: NodeResourceMonitor crashes on bad swapFree value. (ericp:
[https://github.com/apache/hadoop/commit/6432128622d64f3f9dd638b9c254c77cdf5408aa])
* (edit)
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestSysInfoLinux.java
* (edit)
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/SysInfoLinux.java
> NodeResourceMonitor crashes on bad swapFree value
> -
>
> Key: YARN-8444
> URL: https://issues.apache.org/jira/browse/YARN-8444
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.8.3, 3.0.2
>Reporter: Jim Brennan
>Assignee: Jim Brennan
>Priority: Major
> Fix For: 2.10.0, 3.2.0, 3.1.1, 2.9.2, 2.8.5, 3.0.4
>
> Attachments: YARN-8444.001.patch
>
>
> Saw this on a node that was running out of memory. Can't have
> NodeResourceMonitor exiting. System was above 99% memory used at the time, so
> this is not a common occurrence, but we should fix since this is a critical
> monitor to the health of the node.
>
> {noformat}
> 2018-06-04 14:28:08,539 [Container Monitor] DEBUG
> ContainersMonitorImpl.audit: Memory usage of ProcessTree 110564 for
> container-id container_e24_1526662705797_129647_01_004791: 2.1 GB of 3.5 GB
> physical memory used; 5.0 GB of 7.3 GB virtual memory used
> 2018-06-04 14:28:10,622 [Node Resource Monitor] ERROR
> yarn.YarnUncaughtExceptionHandler: Thread Thread[Node Resource
> Monitor,5,main] threw an Exception.
> java.lang.NumberFormatException: For input string: "18446744073709551596"
> at
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> at java.lang.Long.parseLong(Long.java:592)
> at java.lang.Long.parseLong(Long.java:631)
> at
> org.apache.hadoop.util.SysInfoLinux.readProcMemInfoFile(SysInfoLinux.java:257)
> at
> org.apache.hadoop.util.SysInfoLinux.getAvailablePhysicalMemorySize(SysInfoLinux.java:591)
> at
> org.apache.hadoop.util.SysInfoLinux.getAvailableVirtualMemorySize(SysInfoLinux.java:601)
> at
> org.apache.hadoop.yarn.util.ResourceCalculatorPlugin.getAvailableVirtualMemorySize(ResourceCalculatorPlugin.java:74)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeResourceMonitorImpl$MonitoringThread.run(NodeResourceMonitorImpl.java:193)
> 2018-06-04 14:28:30,747
> [org.apache.hadoop.util.JvmPauseMonitor$Monitor@226eba67] INFO
> util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of
> approximately 9330ms
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
