[jira] [Commented] (YARN-1578) Fix how to read history file in FileSystemApplicationHistoryStore
[
https://issues.apache.org/jira/browse/YARN-1578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898910#comment-13898910
]
Zhijie Shen commented on YARN-1578:
---
Yes, I know the bug, which is handled separately in YARN-1685. So let's leave
the bug in this ticket. How do you think?
Fix how to read history file in FileSystemApplicationHistoryStore
-
Key: YARN-1578
URL: https://issues.apache.org/jira/browse/YARN-1578
Project: Hadoop YARN
Issue Type: Sub-task
Affects Versions: YARN-321
Reporter: Shinichi Yamashita
Assignee: Shinichi Yamashita
Attachments: YARN-1578-2.patch, YARN-1578-3.patch, YARN-1578.patch,
application_1390978867235_0001, resoucemanager.log, screenshot.png,
screenshot2.pdf
I carried out PiEstimator job at Hadoop cluster which applied YARN-321.
After the job end and when I accessed Web UI of HistoryServer, it displayed
500. And HistoryServer daemon log was output as follows.
{code}
2014-01-09 13:31:12,227 ERROR org.apache.hadoop.yarn.webapp.Dispatcher: error
handling URI:
/applicationhistory/appattempt/appattempt_1389146249925_0008_01
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:153)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
(snip...)
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore.mergeContainerHistoryData(FileSystemApplicationHistoryStore.java:696)
at
org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore.getContainers(FileSystemApplicationHistoryStore.java:429)
at
org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl.getContainers(ApplicationHistoryManagerImpl.java:201)
at
org.apache.hadoop.yarn.server.webapp.AppAttemptBlock.render(AppAttemptBlock.java:110)
(snip...)
{code}
I confirmed that there was container which was not finished from
ApplicationHistory file.
In ResourceManager daemon log, ResourceManager reserved this container, but
did not allocate it.
When FileSystemApplicationHistoryStore reads container information without
finish data in history file, this problem occurs.
In consideration of the case which there is not finish data, we should fix
how to read history file in FileSystemApplicationHistoryStore.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1716) finishedStatus is accessed without proper locking in RMContainerImpl#FinishedTransition#transition()
[
https://issues.apache.org/jira/browse/YARN-1716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898915#comment-13898915
]
Zhijie Shen commented on YARN-1716:
---
Writer lock is not necessary in tranistion classes, all the transition code is
already executed in the writer lock. See:
{code}
public void handle(RMContainerEvent event) {
{code}
finishedStatus is accessed without proper locking in
RMContainerImpl#FinishedTransition#transition()
Key: YARN-1716
URL: https://issues.apache.org/jira/browse/YARN-1716
Project: Hadoop YARN
Issue Type: Bug
Reporter: Ted Yu
Attachments: YARN-1716.1.patch
{code}
container.finishedStatus = finishedEvent.getRemoteContainerStatus();
{code}
The access should be guarded by obtaining readLock
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1697) NodeManager reports negative running containers
[ https://issues.apache.org/jira/browse/YARN-1697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898914#comment-13898914 ] Hudson commented on YARN-1697: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5153 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5153/]) Fix YARN-1697 in CHANGES.txt (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1567553) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt NodeManager reports negative running containers --- Key: YARN-1697 URL: https://issues.apache.org/jira/browse/YARN-1697 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Affects Versions: 2.3.0 Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 2.4.0 Attachments: YARN-1697.patch We're seeing the NodeManager metrics report a negative number of running containers. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1578) Fix how to read history file in FileSystemApplicationHistoryStore
[
https://issues.apache.org/jira/browse/YARN-1578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898932#comment-13898932
]
Shinichi Yamashita commented on YARN-1578:
--
OK, I leave the bug in this ticket.
Fix how to read history file in FileSystemApplicationHistoryStore
-
Key: YARN-1578
URL: https://issues.apache.org/jira/browse/YARN-1578
Project: Hadoop YARN
Issue Type: Sub-task
Affects Versions: YARN-321
Reporter: Shinichi Yamashita
Assignee: Shinichi Yamashita
Attachments: YARN-1578-2.patch, YARN-1578-3.patch, YARN-1578.patch,
application_1390978867235_0001, resoucemanager.log, screenshot.png,
screenshot2.pdf
I carried out PiEstimator job at Hadoop cluster which applied YARN-321.
After the job end and when I accessed Web UI of HistoryServer, it displayed
500. And HistoryServer daemon log was output as follows.
{code}
2014-01-09 13:31:12,227 ERROR org.apache.hadoop.yarn.webapp.Dispatcher: error
handling URI:
/applicationhistory/appattempt/appattempt_1389146249925_0008_01
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:153)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
(snip...)
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore.mergeContainerHistoryData(FileSystemApplicationHistoryStore.java:696)
at
org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore.getContainers(FileSystemApplicationHistoryStore.java:429)
at
org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl.getContainers(ApplicationHistoryManagerImpl.java:201)
at
org.apache.hadoop.yarn.server.webapp.AppAttemptBlock.render(AppAttemptBlock.java:110)
(snip...)
{code}
I confirmed that there was container which was not finished from
ApplicationHistory file.
In ResourceManager daemon log, ResourceManager reserved this container, but
did not allocate it.
When FileSystemApplicationHistoryStore reads container information without
finish data in history file, this problem occurs.
In consideration of the case which there is not finish data, we should fix
how to read history file in FileSystemApplicationHistoryStore.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (YARN-1578) Fix how to read history file in FileSystemApplicationHistoryStore
[
https://issues.apache.org/jira/browse/YARN-1578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shinichi Yamashita updated YARN-1578:
-
Attachment: YARN-1578-4.patch
I attach the new patch file, which reflected some comments.
Fix how to read history file in FileSystemApplicationHistoryStore
-
Key: YARN-1578
URL: https://issues.apache.org/jira/browse/YARN-1578
Project: Hadoop YARN
Issue Type: Sub-task
Affects Versions: YARN-321
Reporter: Shinichi Yamashita
Assignee: Shinichi Yamashita
Attachments: YARN-1578-2.patch, YARN-1578-3.patch, YARN-1578-4.patch,
YARN-1578.patch, application_1390978867235_0001, resoucemanager.log,
screenshot.png, screenshot2.pdf
I carried out PiEstimator job at Hadoop cluster which applied YARN-321.
After the job end and when I accessed Web UI of HistoryServer, it displayed
500. And HistoryServer daemon log was output as follows.
{code}
2014-01-09 13:31:12,227 ERROR org.apache.hadoop.yarn.webapp.Dispatcher: error
handling URI:
/applicationhistory/appattempt/appattempt_1389146249925_0008_01
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:153)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
(snip...)
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore.mergeContainerHistoryData(FileSystemApplicationHistoryStore.java:696)
at
org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore.getContainers(FileSystemApplicationHistoryStore.java:429)
at
org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl.getContainers(ApplicationHistoryManagerImpl.java:201)
at
org.apache.hadoop.yarn.server.webapp.AppAttemptBlock.render(AppAttemptBlock.java:110)
(snip...)
{code}
I confirmed that there was container which was not finished from
ApplicationHistory file.
In ResourceManager daemon log, ResourceManager reserved this container, but
did not allocate it.
When FileSystemApplicationHistoryStore reads container information without
finish data in history file, this problem occurs.
In consideration of the case which there is not finish data, we should fix
how to read history file in FileSystemApplicationHistoryStore.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1578) Fix how to read history file in FileSystemApplicationHistoryStore
[
https://issues.apache.org/jira/browse/YARN-1578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898964#comment-13898964
]
Hadoop QA commented on YARN-1578:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12628457/YARN-1578-4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3083//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3083//console
This message is automatically generated.
Fix how to read history file in FileSystemApplicationHistoryStore
-
Key: YARN-1578
URL: https://issues.apache.org/jira/browse/YARN-1578
Project: Hadoop YARN
Issue Type: Sub-task
Affects Versions: YARN-321
Reporter: Shinichi Yamashita
Assignee: Shinichi Yamashita
Attachments: YARN-1578-2.patch, YARN-1578-3.patch, YARN-1578-4.patch,
YARN-1578.patch, application_1390978867235_0001, resoucemanager.log,
screenshot.png, screenshot2.pdf
I carried out PiEstimator job at Hadoop cluster which applied YARN-321.
After the job end and when I accessed Web UI of HistoryServer, it displayed
500. And HistoryServer daemon log was output as follows.
{code}
2014-01-09 13:31:12,227 ERROR org.apache.hadoop.yarn.webapp.Dispatcher: error
handling URI:
/applicationhistory/appattempt/appattempt_1389146249925_0008_01
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:153)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
(snip...)
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore.mergeContainerHistoryData(FileSystemApplicationHistoryStore.java:696)
at
org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore.getContainers(FileSystemApplicationHistoryStore.java:429)
at
org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl.getContainers(ApplicationHistoryManagerImpl.java:201)
at
org.apache.hadoop.yarn.server.webapp.AppAttemptBlock.render(AppAttemptBlock.java:110)
(snip...)
{code}
I confirmed that there was container which was not finished from
ApplicationHistory file.
In ResourceManager daemon log, ResourceManager reserved this container, but
did not allocate it.
When FileSystemApplicationHistoryStore reads container information without
finish data in history file, this problem occurs.
In consideration of the case which there is not finish data, we should fix
how to read history file in FileSystemApplicationHistoryStore.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1697) NodeManager reports negative running containers
[ https://issues.apache.org/jira/browse/YARN-1697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899004#comment-13899004 ] Hudson commented on YARN-1697: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #479 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/479/]) Fix YARN-1697 in CHANGES.txt (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1567553) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt YARN-1697. NodeManager reports negative running containers (Sandy Ryza) (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1567356) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/metrics/NodeManagerMetrics.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/TestContainer.java NodeManager reports negative running containers --- Key: YARN-1697 URL: https://issues.apache.org/jira/browse/YARN-1697 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Affects Versions: 2.3.0 Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 2.4.0 Attachments: YARN-1697.patch We're seeing the NodeManager metrics report a negative number of running containers. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1498) Common scheduler changes for moving apps between queues
[ https://issues.apache.org/jira/browse/YARN-1498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899012#comment-13899012 ] Hudson commented on YARN-1498: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #479 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/479/]) Move YARN-1496, YARN-1498, YARN-1504, YARN-1499, and YARN-1497 under 2.4 in CHANGES.txt (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1567487) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt Common scheduler changes for moving apps between queues --- Key: YARN-1498 URL: https://issues.apache.org/jira/browse/YARN-1498 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Affects Versions: 2.2.0 Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 3.0.0 Attachments: YARN-1498-1.patch, YARN-1498-addendum.patch, YARN-1498.patch, YARN-1498.patch This JIRA is to track changes that aren't in particular schedulers but that help them support moving apps between queues. In particular, it makes sure that QueueMetrics are properly updated when an app changes queue. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1497) Expose moving apps between queues on the command line
[ https://issues.apache.org/jira/browse/YARN-1497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899011#comment-13899011 ] Hudson commented on YARN-1497: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #479 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/479/]) Move YARN-1496, YARN-1498, YARN-1504, YARN-1499, and YARN-1497 under 2.4 in CHANGES.txt (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1567487) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt Expose moving apps between queues on the command line - Key: YARN-1497 URL: https://issues.apache.org/jira/browse/YARN-1497 Project: Hadoop YARN Issue Type: Sub-task Components: client Affects Versions: 2.2.0 Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 3.0.0 Attachments: YARN-1497-1.patch, YARN-1497.patch, YARN-1497.patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1499) Fair Scheduler changes for moving apps between queues
[ https://issues.apache.org/jira/browse/YARN-1499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899007#comment-13899007 ] Hudson commented on YARN-1499: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #479 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/479/]) Move YARN-1496, YARN-1498, YARN-1504, YARN-1499, and YARN-1497 under 2.4 in CHANGES.txt (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1567487) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt Fair Scheduler changes for moving apps between queues - Key: YARN-1499 URL: https://issues.apache.org/jira/browse/YARN-1499 Project: Hadoop YARN Issue Type: Sub-task Components: scheduler Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 3.0.0 Attachments: YARN-1499-1.patch, YARN-1499-2.patch, YARN-1499.patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1496) Protocol additions to allow moving apps between queues
[ https://issues.apache.org/jira/browse/YARN-1496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899010#comment-13899010 ] Hudson commented on YARN-1496: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #479 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/479/]) Move YARN-1496, YARN-1498, YARN-1504, YARN-1499, and YARN-1497 under 2.4 in CHANGES.txt (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1567487) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt Protocol additions to allow moving apps between queues -- Key: YARN-1496 URL: https://issues.apache.org/jira/browse/YARN-1496 Project: Hadoop YARN Issue Type: Sub-task Components: scheduler Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 3.0.0 Attachments: YARN-1496-1.patch, YARN-1496-2.patch, YARN-1496-3.patch, YARN-1496-4.patch, YARN-1496-5.patch, YARN-1496-6.patch, YARN-1496.patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1504) RM changes for moving apps between queues
[ https://issues.apache.org/jira/browse/YARN-1504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899008#comment-13899008 ] Hudson commented on YARN-1504: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #479 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/479/]) Move YARN-1496, YARN-1498, YARN-1504, YARN-1499, and YARN-1497 under 2.4 in CHANGES.txt (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1567487) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt RM changes for moving apps between queues - Key: YARN-1504 URL: https://issues.apache.org/jira/browse/YARN-1504 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Affects Versions: 2.2.0 Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 3.0.0 Attachments: YARN-1504-1.patch, YARN-1504.patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1719) ATSWebServices produces jersey warnings
[
https://issues.apache.org/jira/browse/YARN-1719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899009#comment-13899009
]
Hudson commented on YARN-1719:
--
SUCCESS: Integrated in Hadoop-Yarn-trunk #479 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/479/])
YARN-1719. Fixed the root path related Jersey warnings produced in
ATSWebServices. Contributed by Billie Rinaldi. (zjshen:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1567498)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/ATSWebServices.java
ATSWebServices produces jersey warnings
---
Key: YARN-1719
URL: https://issues.apache.org/jira/browse/YARN-1719
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Billie Rinaldi
Assignee: Billie Rinaldi
Fix For: 2.4.0
Attachments: YARN-1719.1.patch
These don't appear to affect how the web services work, but the following
warnings are logged:
{noformat}
WARNING: The following warnings have been detected with resource and/or
provider
classes:
WARNING: A sub-resource method, public
org.apache.hadoop.yarn.server.applicati
onhistoryservice.webapp.ATSWebServices$AboutInfo
org.apache.hadoop.yarn.server.a
pplicationhistoryservice.webapp.ATSWebServices.about(javax.servlet.http.HttpServ
letRequest,javax.servlet.http.HttpServletResponse), with URI template, /,
is t
reated as a resource method
WARNING: A sub-resource method, public
org.apache.hadoop.yarn.api.records.appt
imeline.ATSPutErrors
org.apache.hadoop.yarn.server.applicationhistoryservice.web
app.ATSWebServices.postEntities(javax.servlet.http.HttpServletRequest,javax.serv
let.http.HttpServletResponse,org.apache.hadoop.yarn.api.records.apptimeline.ATSE
ntities), with URI template, /, is treated as a resource method
{noformat}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1697) NodeManager reports negative running containers
[ https://issues.apache.org/jira/browse/YARN-1697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899104#comment-13899104 ] Hudson commented on YARN-1697: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1671 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1671/]) Fix YARN-1697 in CHANGES.txt (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1567553) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt YARN-1697. NodeManager reports negative running containers (Sandy Ryza) (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1567356) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/metrics/NodeManagerMetrics.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/TestContainer.java NodeManager reports negative running containers --- Key: YARN-1697 URL: https://issues.apache.org/jira/browse/YARN-1697 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Affects Versions: 2.3.0 Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 2.4.0 Attachments: YARN-1697.patch We're seeing the NodeManager metrics report a negative number of running containers. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1499) Fair Scheduler changes for moving apps between queues
[ https://issues.apache.org/jira/browse/YARN-1499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899107#comment-13899107 ] Hudson commented on YARN-1499: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1671 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1671/]) Move YARN-1496, YARN-1498, YARN-1504, YARN-1499, and YARN-1497 under 2.4 in CHANGES.txt (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1567487) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt Fair Scheduler changes for moving apps between queues - Key: YARN-1499 URL: https://issues.apache.org/jira/browse/YARN-1499 Project: Hadoop YARN Issue Type: Sub-task Components: scheduler Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 3.0.0 Attachments: YARN-1499-1.patch, YARN-1499-2.patch, YARN-1499.patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1719) ATSWebServices produces jersey warnings
[
https://issues.apache.org/jira/browse/YARN-1719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899109#comment-13899109
]
Hudson commented on YARN-1719:
--
SUCCESS: Integrated in Hadoop-Hdfs-trunk #1671 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1671/])
YARN-1719. Fixed the root path related Jersey warnings produced in
ATSWebServices. Contributed by Billie Rinaldi. (zjshen:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1567498)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/ATSWebServices.java
ATSWebServices produces jersey warnings
---
Key: YARN-1719
URL: https://issues.apache.org/jira/browse/YARN-1719
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Billie Rinaldi
Assignee: Billie Rinaldi
Fix For: 2.4.0
Attachments: YARN-1719.1.patch
These don't appear to affect how the web services work, but the following
warnings are logged:
{noformat}
WARNING: The following warnings have been detected with resource and/or
provider
classes:
WARNING: A sub-resource method, public
org.apache.hadoop.yarn.server.applicati
onhistoryservice.webapp.ATSWebServices$AboutInfo
org.apache.hadoop.yarn.server.a
pplicationhistoryservice.webapp.ATSWebServices.about(javax.servlet.http.HttpServ
letRequest,javax.servlet.http.HttpServletResponse), with URI template, /,
is t
reated as a resource method
WARNING: A sub-resource method, public
org.apache.hadoop.yarn.api.records.appt
imeline.ATSPutErrors
org.apache.hadoop.yarn.server.applicationhistoryservice.web
app.ATSWebServices.postEntities(javax.servlet.http.HttpServletRequest,javax.serv
let.http.HttpServletResponse,org.apache.hadoop.yarn.api.records.apptimeline.ATSE
ntities), with URI template, /, is treated as a resource method
{noformat}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1496) Protocol additions to allow moving apps between queues
[ https://issues.apache.org/jira/browse/YARN-1496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899111#comment-13899111 ] Hudson commented on YARN-1496: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1671 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1671/]) Move YARN-1496, YARN-1498, YARN-1504, YARN-1499, and YARN-1497 under 2.4 in CHANGES.txt (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1567487) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt Protocol additions to allow moving apps between queues -- Key: YARN-1496 URL: https://issues.apache.org/jira/browse/YARN-1496 Project: Hadoop YARN Issue Type: Sub-task Components: scheduler Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 3.0.0 Attachments: YARN-1496-1.patch, YARN-1496-2.patch, YARN-1496-3.patch, YARN-1496-4.patch, YARN-1496-5.patch, YARN-1496-6.patch, YARN-1496.patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1504) RM changes for moving apps between queues
[ https://issues.apache.org/jira/browse/YARN-1504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899108#comment-13899108 ] Hudson commented on YARN-1504: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1671 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1671/]) Move YARN-1496, YARN-1498, YARN-1504, YARN-1499, and YARN-1497 under 2.4 in CHANGES.txt (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1567487) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt RM changes for moving apps between queues - Key: YARN-1504 URL: https://issues.apache.org/jira/browse/YARN-1504 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Affects Versions: 2.2.0 Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 3.0.0 Attachments: YARN-1504-1.patch, YARN-1504.patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1497) Expose moving apps between queues on the command line
[ https://issues.apache.org/jira/browse/YARN-1497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899112#comment-13899112 ] Hudson commented on YARN-1497: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1671 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1671/]) Move YARN-1496, YARN-1498, YARN-1504, YARN-1499, and YARN-1497 under 2.4 in CHANGES.txt (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1567487) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt Expose moving apps between queues on the command line - Key: YARN-1497 URL: https://issues.apache.org/jira/browse/YARN-1497 Project: Hadoop YARN Issue Type: Sub-task Components: client Affects Versions: 2.2.0 Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 3.0.0 Attachments: YARN-1497-1.patch, YARN-1497.patch, YARN-1497.patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1498) Common scheduler changes for moving apps between queues
[ https://issues.apache.org/jira/browse/YARN-1498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899113#comment-13899113 ] Hudson commented on YARN-1498: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1671 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1671/]) Move YARN-1496, YARN-1498, YARN-1504, YARN-1499, and YARN-1497 under 2.4 in CHANGES.txt (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1567487) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt Common scheduler changes for moving apps between queues --- Key: YARN-1498 URL: https://issues.apache.org/jira/browse/YARN-1498 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Affects Versions: 2.2.0 Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 3.0.0 Attachments: YARN-1498-1.patch, YARN-1498-addendum.patch, YARN-1498.patch, YARN-1498.patch This JIRA is to track changes that aren't in particular schedulers but that help them support moving apps between queues. In particular, it makes sure that QueueMetrics are properly updated when an app changes queue. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1641) ZK store should attempt a write periodically to ensure it is still Active
[ https://issues.apache.org/jira/browse/YARN-1641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899135#comment-13899135 ] Hudson commented on YARN-1641: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5154 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5154/]) YARN-1641. ZK store should attempt a write periodically to ensure it is still Active. (kasha) (kasha: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1567628) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/ZKRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStore.java ZK store should attempt a write periodically to ensure it is still Active - Key: YARN-1641 URL: https://issues.apache.org/jira/browse/YARN-1641 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Affects Versions: 2.3.0 Reporter: Karthik Kambatla Assignee: Karthik Kambatla Fix For: 2.4.0 Attachments: yarn-1641-1.patch, yarn-1641-2.patch Fencing in ZK store kicks in when the RM tries to write something to the store. If the RM doesn't write anything to the store, it doesn't get fenced and can continue to assume being the Active. By periodically writing a file (say, every RM_ZK_TIMEOUT_MS seconds), we can ensure it gets fenced. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1497) Expose moving apps between queues on the command line
[ https://issues.apache.org/jira/browse/YARN-1497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899160#comment-13899160 ] Hudson commented on YARN-1497: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1696 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1696/]) Move YARN-1496, YARN-1498, YARN-1504, YARN-1499, and YARN-1497 under 2.4 in CHANGES.txt (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1567487) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt Expose moving apps between queues on the command line - Key: YARN-1497 URL: https://issues.apache.org/jira/browse/YARN-1497 Project: Hadoop YARN Issue Type: Sub-task Components: client Affects Versions: 2.2.0 Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 3.0.0 Attachments: YARN-1497-1.patch, YARN-1497.patch, YARN-1497.patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1504) RM changes for moving apps between queues
[ https://issues.apache.org/jira/browse/YARN-1504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899157#comment-13899157 ] Hudson commented on YARN-1504: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1696 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1696/]) Move YARN-1496, YARN-1498, YARN-1504, YARN-1499, and YARN-1497 under 2.4 in CHANGES.txt (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1567487) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt RM changes for moving apps between queues - Key: YARN-1504 URL: https://issues.apache.org/jira/browse/YARN-1504 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Affects Versions: 2.2.0 Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 3.0.0 Attachments: YARN-1504-1.patch, YARN-1504.patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1498) Common scheduler changes for moving apps between queues
[ https://issues.apache.org/jira/browse/YARN-1498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899161#comment-13899161 ] Hudson commented on YARN-1498: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1696 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1696/]) Move YARN-1496, YARN-1498, YARN-1504, YARN-1499, and YARN-1497 under 2.4 in CHANGES.txt (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1567487) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt Common scheduler changes for moving apps between queues --- Key: YARN-1498 URL: https://issues.apache.org/jira/browse/YARN-1498 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Affects Versions: 2.2.0 Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 3.0.0 Attachments: YARN-1498-1.patch, YARN-1498-addendum.patch, YARN-1498.patch, YARN-1498.patch This JIRA is to track changes that aren't in particular schedulers but that help them support moving apps between queues. In particular, it makes sure that QueueMetrics are properly updated when an app changes queue. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1499) Fair Scheduler changes for moving apps between queues
[ https://issues.apache.org/jira/browse/YARN-1499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899156#comment-13899156 ] Hudson commented on YARN-1499: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1696 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1696/]) Move YARN-1496, YARN-1498, YARN-1504, YARN-1499, and YARN-1497 under 2.4 in CHANGES.txt (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1567487) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt Fair Scheduler changes for moving apps between queues - Key: YARN-1499 URL: https://issues.apache.org/jira/browse/YARN-1499 Project: Hadoop YARN Issue Type: Sub-task Components: scheduler Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 3.0.0 Attachments: YARN-1499-1.patch, YARN-1499-2.patch, YARN-1499.patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1697) NodeManager reports negative running containers
[ https://issues.apache.org/jira/browse/YARN-1697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899153#comment-13899153 ] Hudson commented on YARN-1697: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1696 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1696/]) Fix YARN-1697 in CHANGES.txt (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1567553) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt YARN-1697. NodeManager reports negative running containers (Sandy Ryza) (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1567356) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/metrics/NodeManagerMetrics.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/TestContainer.java NodeManager reports negative running containers --- Key: YARN-1697 URL: https://issues.apache.org/jira/browse/YARN-1697 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Affects Versions: 2.3.0 Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 2.4.0 Attachments: YARN-1697.patch We're seeing the NodeManager metrics report a negative number of running containers. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1496) Protocol additions to allow moving apps between queues
[ https://issues.apache.org/jira/browse/YARN-1496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899159#comment-13899159 ] Hudson commented on YARN-1496: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1696 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1696/]) Move YARN-1496, YARN-1498, YARN-1504, YARN-1499, and YARN-1497 under 2.4 in CHANGES.txt (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1567487) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt Protocol additions to allow moving apps between queues -- Key: YARN-1496 URL: https://issues.apache.org/jira/browse/YARN-1496 Project: Hadoop YARN Issue Type: Sub-task Components: scheduler Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 3.0.0 Attachments: YARN-1496-1.patch, YARN-1496-2.patch, YARN-1496-3.patch, YARN-1496-4.patch, YARN-1496-5.patch, YARN-1496-6.patch, YARN-1496.patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1719) ATSWebServices produces jersey warnings
[
https://issues.apache.org/jira/browse/YARN-1719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899158#comment-13899158
]
Hudson commented on YARN-1719:
--
SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1696 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1696/])
YARN-1719. Fixed the root path related Jersey warnings produced in
ATSWebServices. Contributed by Billie Rinaldi. (zjshen:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1567498)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/ATSWebServices.java
ATSWebServices produces jersey warnings
---
Key: YARN-1719
URL: https://issues.apache.org/jira/browse/YARN-1719
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Billie Rinaldi
Assignee: Billie Rinaldi
Fix For: 2.4.0
Attachments: YARN-1719.1.patch
These don't appear to affect how the web services work, but the following
warnings are logged:
{noformat}
WARNING: The following warnings have been detected with resource and/or
provider
classes:
WARNING: A sub-resource method, public
org.apache.hadoop.yarn.server.applicati
onhistoryservice.webapp.ATSWebServices$AboutInfo
org.apache.hadoop.yarn.server.a
pplicationhistoryservice.webapp.ATSWebServices.about(javax.servlet.http.HttpServ
letRequest,javax.servlet.http.HttpServletResponse), with URI template, /,
is t
reated as a resource method
WARNING: A sub-resource method, public
org.apache.hadoop.yarn.api.records.appt
imeline.ATSPutErrors
org.apache.hadoop.yarn.server.applicationhistoryservice.web
app.ATSWebServices.postEntities(javax.servlet.http.HttpServletRequest,javax.serv
let.http.HttpServletResponse,org.apache.hadoop.yarn.api.records.apptimeline.ATSE
ntities), with URI template, /, is treated as a resource method
{noformat}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (YARN-1530) [Umbrella] Store, manage and serve per-framework application-timeline data
[ https://issues.apache.org/jira/browse/YARN-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billie Rinaldi updated YARN-1530: - Attachment: application timeline design-20140210.pdf [Umbrella] Store, manage and serve per-framework application-timeline data -- Key: YARN-1530 URL: https://issues.apache.org/jira/browse/YARN-1530 Project: Hadoop YARN Issue Type: Bug Reporter: Vinod Kumar Vavilapalli Attachments: application timeline design-20140108.pdf, application timeline design-20140116.pdf, application timeline design-20140130.pdf, application timeline design-20140210.pdf This is a sibling JIRA for YARN-321. Today, each application/framework has to do store, and serve per-framework data all by itself as YARN doesn't have a common solution. This JIRA attempts to solve the storage, management and serving of per-framework data from various applications, both running and finished. The aim is to change YARN to collect and store data in a generic manner with plugin points for frameworks to do their own thing w.r.t interpretation and serving. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1613) Fix config name YARN_HISTORY_SERVICE_ENABLED
[ https://issues.apache.org/jira/browse/YARN-1613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-1613: -- Fix Version/s: (was: 3.0.0) 2.4.0 Setting the right fix-version. Fix config name YARN_HISTORY_SERVICE_ENABLED Key: YARN-1613 URL: https://issues.apache.org/jira/browse/YARN-1613 Project: Hadoop YARN Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Akira AJISAKA Labels: newbie Fix For: 2.4.0 Attachments: YARN-1613.patch YARN_HISTORY_SERVICE_ENABLED property name is yarn.ahs..enabled, which is wrong. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1625) mvn apache-rat:check outputs warning message in YARN-321 branch
[
https://issues.apache.org/jira/browse/YARN-1625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated YARN-1625:
--
Fix Version/s: (was: 3.0.0)
2.4.0
Setting the right fix-version.
mvn apache-rat:check outputs warning message in YARN-321 branch
---
Key: YARN-1625
URL: https://issues.apache.org/jira/browse/YARN-1625
Project: Hadoop YARN
Issue Type: Sub-task
Affects Versions: YARN-321
Reporter: Shinichi Yamashita
Assignee: Shinichi Yamashita
Priority: Trivial
Fix For: 2.4.0
Attachments: YARN-1625.patch
When I ran dev-support/test-patch.sh, following message output.
{code}
mvn apache-rat:check -DHadoopPatchProcess /tmp/patchReleaseAuditOutput.txt
21
There appear to be 1 release audit warnings after applying the patch.
{code}
{code}
!?
/home/sinchii/git/YARN-321-test/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/applicationhistory/.keep
Lines that start with ? in the release audit report indicate files that
do not have an Apache license header.
{code}
To avoid release audit warning, it should fix pom.xml.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (YARN-1417) RM may issue expired container tokens to AM while issuing new containers.
[ https://issues.apache.org/jira/browse/YARN-1417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-1417: -- Attachment: YARN-1417.3.patch Patch that sets container token only when the container is pulled by the AM and adds the unit test. RM may issue expired container tokens to AM while issuing new containers. - Key: YARN-1417 URL: https://issues.apache.org/jira/browse/YARN-1417 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Jian He Priority: Blocker Attachments: YARN-1417.2.patch, YARN-1417.3.patch Today we create new container token when we create container in RM as a part of schedule cycle. However that container may get reserved or assigned. If the container gets reserved and remains like that (in reserved state) for more than container token expiry interval then RM will end up issuing container with expired token. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1417) RM may issue expired container tokens to AM while issuing new containers.
[
https://issues.apache.org/jira/browse/YARN-1417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899446#comment-13899446
]
Hadoop QA commented on YARN-1417:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12628531/YARN-1417.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3084//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3084//console
This message is automatically generated.
RM may issue expired container tokens to AM while issuing new containers.
-
Key: YARN-1417
URL: https://issues.apache.org/jira/browse/YARN-1417
Project: Hadoop YARN
Issue Type: Bug
Reporter: Omkar Vinit Joshi
Assignee: Jian He
Priority: Blocker
Attachments: YARN-1417.2.patch, YARN-1417.3.patch
Today we create new container token when we create container in RM as a part
of schedule cycle. However that container may get reserved or assigned. If
the container gets reserved and remains like that (in reserved state) for
more than container token expiry interval then RM will end up issuing
container with expired token.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1553) Do not use HttpConfig.isSecure() in YARN
[
https://issues.apache.org/jira/browse/YARN-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899448#comment-13899448
]
Haohui Mai commented on YARN-1553:
--
bq. In the few places that we used “//“, should we retain the old behavior and
explicitly use the URL with scheme?
This is a general way of handling links for both http and https. Previously
we've used your proposed scheme in the old HDFS UI, it was quite error-prone.
bq. getRMDefaultPortNumber() method should use the new method instead of using
RM_SERVICES_ADDRESS_CONF_KEYS_HTTP directly.
It is a static method so I don't quite see choosing between
{{RM_SERVICES_ADDRESS_CONF_KEYS_HTTP}} /
{{RM_SERVICES_ADDRESS_CONF_KEYS_HTTPS}} are possible. Do you have any idea?
Do not use HttpConfig.isSecure() in YARN
Key: YARN-1553
URL: https://issues.apache.org/jira/browse/YARN-1553
Project: Hadoop YARN
Issue Type: Bug
Reporter: Haohui Mai
Assignee: Haohui Mai
Attachments: YARN-1553.000.patch, YARN-1553.001.patch,
YARN-1553.002.patch, YARN-1553.003.patch, YARN-1553.004.patch,
YARN-1553.005.patch, YARN-1553.006.patch
HDFS-5305 and related jira decide that each individual project will have
their own configuration on http policy. {{HttpConfig.isSecure}} is a global
static method which does not fit the design anymore. The same functionality
should be moved into the YARN code base.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (YARN-1553) Do not use HttpConfig.isSecure() in YARN
[
https://issues.apache.org/jira/browse/YARN-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated YARN-1553:
-
Attachment: YARN-1553.007.patch
The v7 patch addresses Vinod's comments.
Do not use HttpConfig.isSecure() in YARN
Key: YARN-1553
URL: https://issues.apache.org/jira/browse/YARN-1553
Project: Hadoop YARN
Issue Type: Bug
Reporter: Haohui Mai
Assignee: Haohui Mai
Attachments: YARN-1553.000.patch, YARN-1553.001.patch,
YARN-1553.002.patch, YARN-1553.003.patch, YARN-1553.004.patch,
YARN-1553.005.patch, YARN-1553.006.patch, YARN-1553.007.patch
HDFS-5305 and related jira decide that each individual project will have
their own configuration on http policy. {{HttpConfig.isSecure}} is a global
static method which does not fit the design anymore. The same functionality
should be moved into the YARN code base.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1714) Per user and per queue view in YARN RM
[ https://issues.apache.org/jira/browse/YARN-1714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899458#comment-13899458 ] Siqi Li commented on YARN-1714: --- For question 1, yes we do have full text search over the app list. By doing this, we can filter the apps on the server side. So, user will not be bothered by other people's jobs. Also, once we need to add security, it's better to see the apps that you own. 2. I'm not aware of any REST API does the same thing, can you give me some examples? 3. For now, users have to edit url, but we can also provide direct link where people can click to get in this page. Moreover, we can automatically take user name and call this new url directly in the future when turning on security. 4. yes, we should reuse AppsBlock. Per user and per queue view in YARN RM -- Key: YARN-1714 URL: https://issues.apache.org/jira/browse/YARN-1714 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Reporter: Siqi Li Assignee: Siqi Li Priority: Critical Attachments: YARN-1714.v1.patch, YARN-1714.v2.patch, YARN-1714.v3.patch ResourceManager exposes either one or all jobs via WebUI. It would be good to have filter for user so that see only their jobs. Provide rest style url to access only user specified queue or user apps. For instance, http://hadoop-example.com:50030/cluster/user/toto displays apps owned by toto http://hadoop-example.com:50030/cluster/user/toto,glinda displays apps owned by toto and glinda http://hadoop-example.com:50030/cluster/queue/root.queue1 displays apps in root.queue1 http://hadoop-example.com:50030/cluster/queue/root.queue1,root.queue2 displays apps in root.queue1 and root.queue2 -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1553) Do not use HttpConfig.isSecure() in YARN
[
https://issues.apache.org/jira/browse/YARN-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated YARN-1553:
-
Attachment: YARN-1553.008.patch
The v8 patch passes in the configuration in {{getRMDefaultPortNumber()}} so
that it can display the correct error message.
Do not use HttpConfig.isSecure() in YARN
Key: YARN-1553
URL: https://issues.apache.org/jira/browse/YARN-1553
Project: Hadoop YARN
Issue Type: Bug
Reporter: Haohui Mai
Assignee: Haohui Mai
Attachments: YARN-1553.000.patch, YARN-1553.001.patch,
YARN-1553.002.patch, YARN-1553.003.patch, YARN-1553.004.patch,
YARN-1553.005.patch, YARN-1553.006.patch, YARN-1553.007.patch,
YARN-1553.008.patch
HDFS-5305 and related jira decide that each individual project will have
their own configuration on http policy. {{HttpConfig.isSecure}} is a global
static method which does not fit the design anymore. The same functionality
should be moved into the YARN code base.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (YARN-1717) Misc improvements to leveldb timeline store
[ https://issues.apache.org/jira/browse/YARN-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billie Rinaldi updated YARN-1717: - Attachment: YARN-1717.3.patch Removed delete locks -- they're unnecessary in this implementation since deleting can only happen while ATS is offline. Misc improvements to leveldb timeline store --- Key: YARN-1717 URL: https://issues.apache.org/jira/browse/YARN-1717 Project: Hadoop YARN Issue Type: Sub-task Reporter: Billie Rinaldi Assignee: Billie Rinaldi Attachments: YARN-1717.1.patch, YARN-1717.2.patch, YARN-1717.3.patch The leveldb timeline store implementation needs the following: * better documentation of its internal structures * braces for all control flow statements * simple locking to prevent issues related to concurrent writes * internal changes to enable deleting entities ** never overwrite existing primary filter entries ** add hidden reverse pointers to related entities -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1525) Web UI should redirect to active RM when HA is enabled.
[ https://issues.apache.org/jira/browse/YARN-1525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cindy Li updated YARN-1525: --- Attachment: YARN1525.patch KarthikXuan, please take a look at the latest patch. I've removed the IDE generated format changes. Web UI should redirect to active RM when HA is enabled. --- Key: YARN-1525 URL: https://issues.apache.org/jira/browse/YARN-1525 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Cindy Li Attachments: YARN1525.patch, YARN1525.patch.v1, YARN1525.patch.v2, YARN1525.patch.v3, YARN1525.v7.patch, YARN1525.v7.patch, YARN1525.v8.patch, YARN1525.v9.patch When failover happens, web UI should redirect to the current active rm. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1692) ConcurrentModificationException in fair scheduler AppSchedulable
[
https://issues.apache.org/jira/browse/YARN-1692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899506#comment-13899506
]
Lohit Vijayarenu commented on YARN-1692:
+1 on the patch. Can anyone else review this as well.
ConcurrentModificationException in fair scheduler AppSchedulable
Key: YARN-1692
URL: https://issues.apache.org/jira/browse/YARN-1692
Project: Hadoop YARN
Issue Type: Bug
Components: scheduler
Affects Versions: 2.0.5-alpha
Reporter: Sangjin Lee
Assignee: Sangjin Lee
Attachments: yarn-1692.patch
We saw a ConcurrentModificationException thrown in the fair scheduler:
{noformat}
2014-02-07 01:40:01,978 ERROR
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
Exception in fair scheduler UpdateThread
java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextEntry(HashMap.java:926)
at java.util.HashMap$ValueIterator.next(HashMap.java:954)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AppSchedulable.updateDemand(AppSchedulable.java:85)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.updateDemand(FSLeafQueue.java:125)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSParentQueue.updateDemand(FSParentQueue.java:82)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.update(FairScheduler.java:217)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler$UpdateThread.run(FairScheduler.java:195)
at java.lang.Thread.run(Thread.java:724)
{noformat}
The map that gets returned by FSSchedulerApp.getResourceRequests() are
iterated on without proper synchronization.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1717) Misc improvements to leveldb timeline store
[
https://issues.apache.org/jira/browse/YARN-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899512#comment-13899512
]
Hadoop QA commented on YARN-1717:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12628544/YARN-1717.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3085//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3085//console
This message is automatically generated.
Misc improvements to leveldb timeline store
---
Key: YARN-1717
URL: https://issues.apache.org/jira/browse/YARN-1717
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Billie Rinaldi
Assignee: Billie Rinaldi
Attachments: YARN-1717.1.patch, YARN-1717.2.patch, YARN-1717.3.patch
The leveldb timeline store implementation needs the following:
* better documentation of its internal structures
* braces for all control flow statements
* simple locking to prevent issues related to concurrent writes
* internal changes to enable deleting entities
** never overwrite existing primary filter entries
** add hidden reverse pointers to related entities
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1692) ConcurrentModificationException in fair scheduler AppSchedulable
[
https://issues.apache.org/jira/browse/YARN-1692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899530#comment-13899530
]
Karthik Kambatla commented on YARN-1692:
Let me take a look later today.
ConcurrentModificationException in fair scheduler AppSchedulable
Key: YARN-1692
URL: https://issues.apache.org/jira/browse/YARN-1692
Project: Hadoop YARN
Issue Type: Bug
Components: scheduler
Affects Versions: 2.0.5-alpha
Reporter: Sangjin Lee
Assignee: Sangjin Lee
Attachments: yarn-1692.patch
We saw a ConcurrentModificationException thrown in the fair scheduler:
{noformat}
2014-02-07 01:40:01,978 ERROR
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
Exception in fair scheduler UpdateThread
java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextEntry(HashMap.java:926)
at java.util.HashMap$ValueIterator.next(HashMap.java:954)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AppSchedulable.updateDemand(AppSchedulable.java:85)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.updateDemand(FSLeafQueue.java:125)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSParentQueue.updateDemand(FSParentQueue.java:82)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.update(FairScheduler.java:217)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler$UpdateThread.run(FairScheduler.java:195)
at java.lang.Thread.run(Thread.java:724)
{noformat}
The map that gets returned by FSSchedulerApp.getResourceRequests() are
iterated on without proper synchronization.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1158) ResourceManager UI has application stdout missing if application stdout is not in the same directory as AppMaster stdout
[ https://issues.apache.org/jira/browse/YARN-1158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899534#comment-13899534 ] Zhijie Shen commented on YARN-1158: --- Have some investigation in this problem. The root cause is that RM web UI only shows the log of AM container. For the logs of other containers, users need to go to the corresponding NM web UI to see it. The bad thing there's no links to redirect users. As AHS list the containers of the applications as well, and provide the link to the container logs, RM web UI is good to do the same thing to simply the way for users to look for container logs. ResourceManager UI has application stdout missing if application stdout is not in the same directory as AppMaster stdout Key: YARN-1158 URL: https://issues.apache.org/jira/browse/YARN-1158 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Reporter: Tassapol Athiapinya Fix For: 2.3.0 Configure yarn-site.xml's yarn.nodemanager.local-dirs to multiple directories. Turn on log aggregation. Run distributed shell application. If an application writes AppMaster.stdout in one directory and stdout in another directory. Goto ResourceManager web UI. Open up container logs. Only AppMaster.stdout would appear. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1296) schedulerAllocateTimer is accessed without holding samplerLock in ResourceSchedulerWrapper
[
https://issues.apache.org/jira/browse/YARN-1296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899546#comment-13899546
]
Tsuyoshi OZAWA commented on YARN-1296:
--
[~sandyr], do you have any comment about this JIRA?
schedulerAllocateTimer is accessed without holding samplerLock in
ResourceSchedulerWrapper
--
Key: YARN-1296
URL: https://issues.apache.org/jira/browse/YARN-1296
Project: Hadoop YARN
Issue Type: Bug
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Minor
Attachments: yarn-1296-v1.patch
Here is related code:
{code}
public Allocation allocate(ApplicationAttemptId attemptId,
ListResourceRequest resourceRequests,
ListContainerId containerIds,
ListString strings, ListString strings2) {
if (metricsON) {
final Timer.Context context = schedulerAllocateTimer.time();
{code}
samplerLock should be used to guard the access.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1525) Web UI should redirect to active RM when HA is enabled.
[
https://issues.apache.org/jira/browse/YARN-1525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899550#comment-13899550
]
Hadoop QA commented on YARN-1525:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12628549/YARN1525.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The following test timeouts occurred in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.client.api.impl.TestNMClient
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3086//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3086//console
This message is automatically generated.
Web UI should redirect to active RM when HA is enabled.
---
Key: YARN-1525
URL: https://issues.apache.org/jira/browse/YARN-1525
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Xuan Gong
Assignee: Cindy Li
Attachments: YARN1525.patch, YARN1525.patch.v1, YARN1525.patch.v2,
YARN1525.patch.v3, YARN1525.v7.patch, YARN1525.v7.patch, YARN1525.v8.patch,
YARN1525.v9.patch
When failover happens, web UI should redirect to the current active rm.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1718) Fix a couple isTerminals in Fair Scheduler queue placement rules
[ https://issues.apache.org/jira/browse/YARN-1718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899577#comment-13899577 ] Karthik Kambatla commented on YARN-1718: +1 Fix a couple isTerminals in Fair Scheduler queue placement rules - Key: YARN-1718 URL: https://issues.apache.org/jira/browse/YARN-1718 Project: Hadoop YARN Issue Type: Bug Components: scheduler Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-1718.patch SecondaryGroupExistingQueue and Default are incorrect -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Resolved] (YARN-1720) QueuePlacementRule.SecondaryGroupExistingQueue should not be terminal
[ https://issues.apache.org/jira/browse/YARN-1720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla resolved YARN-1720. Resolution: Duplicate QueuePlacementRule.SecondaryGroupExistingQueue should not be terminal - Key: YARN-1720 URL: https://issues.apache.org/jira/browse/YARN-1720 Project: Hadoop YARN Issue Type: Bug Components: scheduler Affects Versions: 2.2.0 Reporter: Aditya Acharya Assignee: Sandy Ryza The SecondaryGroupExistingQueue QueuePlacementRule's isTerminal() method should always return false, not create, because it does not, in fact, create a queue ever. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1296) schedulerAllocateTimer is accessed without holding samplerLock in ResourceSchedulerWrapper
[
https://issues.apache.org/jira/browse/YARN-1296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899578#comment-13899578
]
Sandy Ryza commented on YARN-1296:
--
I'm not super familiar with this code, but [~ywskycn] might be?
schedulerAllocateTimer is accessed without holding samplerLock in
ResourceSchedulerWrapper
--
Key: YARN-1296
URL: https://issues.apache.org/jira/browse/YARN-1296
Project: Hadoop YARN
Issue Type: Bug
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Minor
Attachments: yarn-1296-v1.patch
Here is related code:
{code}
public Allocation allocate(ApplicationAttemptId attemptId,
ListResourceRequest resourceRequests,
ListContainerId containerIds,
ListString strings, ListString strings2) {
if (metricsON) {
final Timer.Context context = schedulerAllocateTimer.time();
{code}
samplerLock should be used to guard the access.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (YARN-1158) ResourceManager UI can not show the logs of worker containers
[ https://issues.apache.org/jira/browse/YARN-1158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-1158: -- Fix Version/s: (was: 2.3.0) Summary: ResourceManager UI can not show the logs of worker containers (was: ResourceManager UI has application stdout missing if application stdout is not in the same directory as AppMaster stdout) ResourceManager UI can not show the logs of worker containers - Key: YARN-1158 URL: https://issues.apache.org/jira/browse/YARN-1158 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Reporter: Tassapol Athiapinya Configure yarn-site.xml's yarn.nodemanager.local-dirs to multiple directories. Turn on log aggregation. Run distributed shell application. If an application writes AppMaster.stdout in one directory and stdout in another directory. Goto ResourceManager web UI. Open up container logs. Only AppMaster.stdout would appear. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1525) Web UI should redirect to active RM when HA is enabled.
[ https://issues.apache.org/jira/browse/YARN-1525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899603#comment-13899603 ] Cindy Li commented on YARN-1525: The test failure org.apache.hadoop.yarn.client.api.impl.TestNMClient is irrelevant. Web UI should redirect to active RM when HA is enabled. --- Key: YARN-1525 URL: https://issues.apache.org/jira/browse/YARN-1525 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Cindy Li Attachments: YARN1525.patch, YARN1525.patch.v1, YARN1525.patch.v2, YARN1525.patch.v3, YARN1525.v7.patch, YARN1525.v7.patch, YARN1525.v8.patch, YARN1525.v9.patch When failover happens, web UI should redirect to the current active rm. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1296) schedulerAllocateTimer is accessed without holding samplerLock in ResourceSchedulerWrapper
[
https://issues.apache.org/jira/browse/YARN-1296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899608#comment-13899608
]
Wei Yan commented on YARN-1296:
---
Sorry I missed this jira before.
I'll look into the patch. thanks.
schedulerAllocateTimer is accessed without holding samplerLock in
ResourceSchedulerWrapper
--
Key: YARN-1296
URL: https://issues.apache.org/jira/browse/YARN-1296
Project: Hadoop YARN
Issue Type: Bug
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Minor
Attachments: yarn-1296-v1.patch
Here is related code:
{code}
public Allocation allocate(ApplicationAttemptId attemptId,
ListResourceRequest resourceRequests,
ListContainerId containerIds,
ListString strings, ListString strings2) {
if (metricsON) {
final Timer.Context context = schedulerAllocateTimer.time();
{code}
samplerLock should be used to guard the access.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1714) Per user and per queue view in YARN RM
[ https://issues.apache.org/jira/browse/YARN-1714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899650#comment-13899650 ] Zhijie Shen commented on YARN-1714: --- bq. 2. I'm not aware of any REST API does the same thing, can you give me some examples? You can have a look at RMWebServices#getApps. IMO, it's good to add all these filters in one round instead of one after the other, unless different filters really differs a lot in the way they're going to be presented. bq. 3. For now, users have to edit url, but we can also provide direct link where people can click to get in this page. Moreover, we can automatically take user name and call this new url directly in the future when turning on security. It's a good idea that in secured mode, the users login to the apps page that only shows their own app list. However, I still think some interactive method should be provided to users to filter apps, instead of editing URL directly. For example, I'm an Admin, and want to watch the applications from another user. Anyway, if your use case doesn't care about interaction on web UI, but the app list, the current REST API should already serve you well. Otherwise, we should make web UI more interactive. Per user and per queue view in YARN RM -- Key: YARN-1714 URL: https://issues.apache.org/jira/browse/YARN-1714 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Reporter: Siqi Li Assignee: Siqi Li Priority: Critical Attachments: YARN-1714.v1.patch, YARN-1714.v2.patch, YARN-1714.v3.patch ResourceManager exposes either one or all jobs via WebUI. It would be good to have filter for user so that see only their jobs. Provide rest style url to access only user specified queue or user apps. For instance, http://hadoop-example.com:50030/cluster/user/toto displays apps owned by toto http://hadoop-example.com:50030/cluster/user/toto,glinda displays apps owned by toto and glinda http://hadoop-example.com:50030/cluster/queue/root.queue1 displays apps in root.queue1 http://hadoop-example.com:50030/cluster/queue/root.queue1,root.queue2 displays apps in root.queue1 and root.queue2 -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1525) Web UI should redirect to active RM when HA is enabled.
[
https://issues.apache.org/jira/browse/YARN-1525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899668#comment-13899668
]
Karthik Kambatla commented on YARN-1525:
Thanks Cindy. Thanks for your patience through the reviews.
Functionally, the patch looks good. Some comments on the cosmetics:
* The following indentation needs to be fixed:
{code}
@Override
public boolean isStandbyMode() {
{code}
* Setting the redirect path in WebApp#isStandbyMode() is a little misleading.
Can we leave isStandbyMode() to only report whether the RM is in standby mode,
and may be getRedirectPath() can take care of constructing the path if
required.
* Rename RMWebApp#findRedirectPath to buildRedirectPath?
* Couple of lines longer than 80 chars
{code}
if (webApp.isStandbyMode() !uri.equals(/ + webApp.name() +
/cluster)) {
String redirectPath = webApp.getRedirectPath();
if (redirectPath != null !redirectPath.isEmpty()) {
String redirectMsg =
This is a standby resource manager, redirecting to the current
active one:
+ redirectPath;
{code}
* TestRMFailover#testRMWebAppFailover
** Rename to testRMWebAppRedirect to capture the behavior better
** If it is not too much trouble, given the test doesn't do any failover, we
should probably move it to TestRMHA?
* Let us add Private-Unstable annotations to RMHAUtils so we can move the
methods or even remove this class if we don't see the code being reused.
I ll be glad to test the updated patch out on a secure cluster for verification.
Web UI should redirect to active RM when HA is enabled.
---
Key: YARN-1525
URL: https://issues.apache.org/jira/browse/YARN-1525
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Xuan Gong
Assignee: Cindy Li
Attachments: YARN1525.patch, YARN1525.patch.v1, YARN1525.patch.v2,
YARN1525.patch.v3, YARN1525.v7.patch, YARN1525.v7.patch, YARN1525.v8.patch,
YARN1525.v9.patch
When failover happens, web UI should redirect to the current active rm.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1531) Update yarn command document
[
https://issues.apache.org/jira/browse/YARN-1531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899676#comment-13899676
]
Karthik Kambatla commented on YARN-1531:
+1. Committing this.
Update yarn command document
Key: YARN-1531
URL: https://issues.apache.org/jira/browse/YARN-1531
Project: Hadoop YARN
Issue Type: Bug
Components: documentation
Reporter: Akira AJISAKA
Assignee: Akira AJISAKA
Labels: documentaion
Attachments: YARN-1531.2.patch, YARN-1531.3.patch, YARN-1531.patch
There are some options which are not written to Yarn Command document.
For example, yarn rmadmin command options are as follows:
{code}
Usage: yarn rmadmin
-refreshQueues
-refreshNodes
-refreshSuperUserGroupsConfiguration
-refreshUserToGroupsMappings
-refreshAdminAcls
-refreshServiceAcl
-getGroups [username]
-help [cmd]
-transitionToActive serviceId
-transitionToStandby serviceId
-failover [--forcefence] [--forceactive] serviceId serviceId
-getServiceState serviceId
-checkHealth serviceId
{code}
But some of the new options such as -getGroups, -transitionToActive, and
-transitionToStandby are not documented.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (YARN-1531) Update yarn command documentation
[
https://issues.apache.org/jira/browse/YARN-1531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated YARN-1531:
---
Summary: Update yarn command documentation (was: Update yarn command
document)
Update yarn command documentation
-
Key: YARN-1531
URL: https://issues.apache.org/jira/browse/YARN-1531
Project: Hadoop YARN
Issue Type: Bug
Components: documentation
Reporter: Akira AJISAKA
Assignee: Akira AJISAKA
Labels: documentaion
Attachments: YARN-1531.2.patch, YARN-1531.3.patch, YARN-1531.patch
There are some options which are not written to Yarn Command document.
For example, yarn rmadmin command options are as follows:
{code}
Usage: yarn rmadmin
-refreshQueues
-refreshNodes
-refreshSuperUserGroupsConfiguration
-refreshUserToGroupsMappings
-refreshAdminAcls
-refreshServiceAcl
-getGroups [username]
-help [cmd]
-transitionToActive serviceId
-transitionToStandby serviceId
-failover [--forcefence] [--forceactive] serviceId serviceId
-getServiceState serviceId
-checkHealth serviceId
{code}
But some of the new options such as -getGroups, -transitionToActive, and
-transitionToStandby are not documented.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (YARN-1531) True up yarn command documentation
[
https://issues.apache.org/jira/browse/YARN-1531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated YARN-1531:
---
Summary: True up yarn command documentation (was: Update yarn command
documentation)
True up yarn command documentation
--
Key: YARN-1531
URL: https://issues.apache.org/jira/browse/YARN-1531
Project: Hadoop YARN
Issue Type: Bug
Components: documentation
Reporter: Akira AJISAKA
Assignee: Akira AJISAKA
Labels: documentaion
Attachments: YARN-1531.2.patch, YARN-1531.3.patch, YARN-1531.patch
There are some options which are not written to Yarn Command document.
For example, yarn rmadmin command options are as follows:
{code}
Usage: yarn rmadmin
-refreshQueues
-refreshNodes
-refreshSuperUserGroupsConfiguration
-refreshUserToGroupsMappings
-refreshAdminAcls
-refreshServiceAcl
-getGroups [username]
-help [cmd]
-transitionToActive serviceId
-transitionToStandby serviceId
-failover [--forcefence] [--forceactive] serviceId serviceId
-getServiceState serviceId
-checkHealth serviceId
{code}
But some of the new options such as -getGroups, -transitionToActive, and
-transitionToStandby are not documented.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1551) Allow user-specified reason for killApplication
[ https://issues.apache.org/jira/browse/YARN-1551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899697#comment-13899697 ] Karthik Kambatla commented on YARN-1551: My previous comment applies to the latest patch as well. Otherwise, +1. [~vinodkv] - do you have any other comments. Otherwise, can I go ahead and commit this later tomorrow? Allow user-specified reason for killApplication --- Key: YARN-1551 URL: https://issues.apache.org/jira/browse/YARN-1551 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 2.4.0 Reporter: Gera Shegalov Assignee: Gera Shegalov Attachments: YARN-1551.v01.patch, YARN-1551.v02.patch, YARN-1551.v03.patch, YARN-1551.v04.patch, YARN-1551.v05.patch This completes MAPREDUCE-5648 -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1692) ConcurrentModificationException in fair scheduler AppSchedulable
[
https://issues.apache.org/jira/browse/YARN-1692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899718#comment-13899718
]
Sandy Ryza commented on YARN-1692:
--
+1
ConcurrentModificationException in fair scheduler AppSchedulable
Key: YARN-1692
URL: https://issues.apache.org/jira/browse/YARN-1692
Project: Hadoop YARN
Issue Type: Bug
Components: scheduler
Affects Versions: 2.0.5-alpha
Reporter: Sangjin Lee
Assignee: Sangjin Lee
Attachments: yarn-1692.patch
We saw a ConcurrentModificationException thrown in the fair scheduler:
{noformat}
2014-02-07 01:40:01,978 ERROR
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
Exception in fair scheduler UpdateThread
java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextEntry(HashMap.java:926)
at java.util.HashMap$ValueIterator.next(HashMap.java:954)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AppSchedulable.updateDemand(AppSchedulable.java:85)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.updateDemand(FSLeafQueue.java:125)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSParentQueue.updateDemand(FSParentQueue.java:82)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.update(FairScheduler.java:217)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler$UpdateThread.run(FairScheduler.java:195)
at java.lang.Thread.run(Thread.java:724)
{noformat}
The map that gets returned by FSSchedulerApp.getResourceRequests() are
iterated on without proper synchronization.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Created] (YARN-1721) When moving app between queues in Fair Scheduler, grab lock on FSSchedulerApp
Sandy Ryza created YARN-1721: Summary: When moving app between queues in Fair Scheduler, grab lock on FSSchedulerApp Key: YARN-1721 URL: https://issues.apache.org/jira/browse/YARN-1721 Project: Hadoop YARN Issue Type: Bug Components: scheduler Reporter: Sandy Ryza Assignee: Sandy Ryza FairScheduler.moveApplication should grab lock on FSSchedulerApp, so that allocate() can't be modifying it at the same time. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1721) When moving app between queues in Fair Scheduler, grab lock on FSSchedulerApp
[ https://issues.apache.org/jira/browse/YARN-1721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899729#comment-13899729 ] Sandy Ryza commented on YARN-1721: -- To avoid deadlocks, we'll need to make sure that if we lock queues in move, we lock them before the apps they run in. When moving app between queues in Fair Scheduler, grab lock on FSSchedulerApp - Key: YARN-1721 URL: https://issues.apache.org/jira/browse/YARN-1721 Project: Hadoop YARN Issue Type: Bug Components: scheduler Reporter: Sandy Ryza Assignee: Sandy Ryza FairScheduler.moveApplication should grab lock on FSSchedulerApp, so that allocate() can't be modifying it at the same time. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1578) Fix how to read history file in FileSystemApplicationHistoryStore
[
https://issues.apache.org/jira/browse/YARN-1578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899783#comment-13899783
]
Zhijie Shen commented on YARN-1578:
---
+1 for the latest patch. Will commit it
Fix how to read history file in FileSystemApplicationHistoryStore
-
Key: YARN-1578
URL: https://issues.apache.org/jira/browse/YARN-1578
Project: Hadoop YARN
Issue Type: Sub-task
Affects Versions: YARN-321
Reporter: Shinichi Yamashita
Assignee: Shinichi Yamashita
Attachments: YARN-1578-2.patch, YARN-1578-3.patch, YARN-1578-4.patch,
YARN-1578.patch, application_1390978867235_0001, resoucemanager.log,
screenshot.png, screenshot2.pdf
I carried out PiEstimator job at Hadoop cluster which applied YARN-321.
After the job end and when I accessed Web UI of HistoryServer, it displayed
500. And HistoryServer daemon log was output as follows.
{code}
2014-01-09 13:31:12,227 ERROR org.apache.hadoop.yarn.webapp.Dispatcher: error
handling URI:
/applicationhistory/appattempt/appattempt_1389146249925_0008_01
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:153)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
(snip...)
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore.mergeContainerHistoryData(FileSystemApplicationHistoryStore.java:696)
at
org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore.getContainers(FileSystemApplicationHistoryStore.java:429)
at
org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl.getContainers(ApplicationHistoryManagerImpl.java:201)
at
org.apache.hadoop.yarn.server.webapp.AppAttemptBlock.render(AppAttemptBlock.java:110)
(snip...)
{code}
I confirmed that there was container which was not finished from
ApplicationHistory file.
In ResourceManager daemon log, ResourceManager reserved this container, but
did not allocate it.
When FileSystemApplicationHistoryStore reads container information without
finish data in history file, this problem occurs.
In consideration of the case which there is not finish data, we should fix
how to read history file in FileSystemApplicationHistoryStore.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (YARN-1666) Make admin refreshNodes work across RM failover
[ https://issues.apache.org/jira/browse/YARN-1666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-1666: Attachment: YARN-1666.4.patch change ConfigurationProvider#getConfigurationInputStream to return InputStream. add RM_CONFIGURATION_FILES in YarnConfiguration which contains hadoop-policy.xml, yarn-site.xml, capacity-scheduler.xml, and core-site.xml. Also, HostsFileReader can take inputStream as paramter Make admin refreshNodes work across RM failover --- Key: YARN-1666 URL: https://issues.apache.org/jira/browse/YARN-1666 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong Attachments: YARN-1666.1.patch, YARN-1666.2.patch, YARN-1666.2.patch, YARN-1666.3.patch, YARN-1666.4.patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1531) True up yarn command documentation
[
https://issues.apache.org/jira/browse/YARN-1531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899785#comment-13899785
]
Hudson commented on YARN-1531:
--
SUCCESS: Integrated in Hadoop-trunk-Commit #5159 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/5159/])
YARN-1531. True up yarn command documentation (Akira Ajisaka via kasha) (kasha:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1567775)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/YarnCommands.apt.vm
True up yarn command documentation
--
Key: YARN-1531
URL: https://issues.apache.org/jira/browse/YARN-1531
Project: Hadoop YARN
Issue Type: Bug
Components: documentation
Reporter: Akira AJISAKA
Assignee: Akira AJISAKA
Labels: documentaion
Fix For: 2.4.0
Attachments: YARN-1531.2.patch, YARN-1531.3.patch, YARN-1531.patch
There are some options which are not written to Yarn Command document.
For example, yarn rmadmin command options are as follows:
{code}
Usage: yarn rmadmin
-refreshQueues
-refreshNodes
-refreshSuperUserGroupsConfiguration
-refreshUserToGroupsMappings
-refreshAdminAcls
-refreshServiceAcl
-getGroups [username]
-help [cmd]
-transitionToActive serviceId
-transitionToStandby serviceId
-failover [--forcefence] [--forceactive] serviceId serviceId
-getServiceState serviceId
-checkHealth serviceId
{code}
But some of the new options such as -getGroups, -transitionToActive, and
-transitionToStandby are not documented.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1692) ConcurrentModificationException in fair scheduler AppSchedulable
[
https://issues.apache.org/jira/browse/YARN-1692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899786#comment-13899786
]
Hudson commented on YARN-1692:
--
SUCCESS: Integrated in Hadoop-trunk-Commit #5159 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/5159/])
Move YARN-1692 in CHANGES.txt (sandy:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1567793)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
YARN-1692. ConcurrentModificationException in fair scheduler AppSchedulable
(Sangjin Lee via Sandy Ryza) (sandy:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1567788)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AppSchedulable.java
ConcurrentModificationException in fair scheduler AppSchedulable
Key: YARN-1692
URL: https://issues.apache.org/jira/browse/YARN-1692
Project: Hadoop YARN
Issue Type: Bug
Components: scheduler
Affects Versions: 2.0.5-alpha
Reporter: Sangjin Lee
Assignee: Sangjin Lee
Fix For: 2.4.0
Attachments: yarn-1692.patch
We saw a ConcurrentModificationException thrown in the fair scheduler:
{noformat}
2014-02-07 01:40:01,978 ERROR
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
Exception in fair scheduler UpdateThread
java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextEntry(HashMap.java:926)
at java.util.HashMap$ValueIterator.next(HashMap.java:954)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AppSchedulable.updateDemand(AppSchedulable.java:85)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.updateDemand(FSLeafQueue.java:125)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSParentQueue.updateDemand(FSParentQueue.java:82)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.update(FairScheduler.java:217)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler$UpdateThread.run(FairScheduler.java:195)
at java.lang.Thread.run(Thread.java:724)
{noformat}
The map that gets returned by FSSchedulerApp.getResourceRequests() are
iterated on without proper synchronization.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (YARN-1721) When moving app between queues in Fair Scheduler, grab lock on FSSchedulerApp
[ https://issues.apache.org/jira/browse/YARN-1721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated YARN-1721: - Attachment: YARN-1721.patch When moving app between queues in Fair Scheduler, grab lock on FSSchedulerApp - Key: YARN-1721 URL: https://issues.apache.org/jira/browse/YARN-1721 Project: Hadoop YARN Issue Type: Bug Components: scheduler Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-1721.patch FairScheduler.moveApplication should grab lock on FSSchedulerApp, so that allocate() can't be modifying it at the same time. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1721) When moving app between queues in Fair Scheduler, grab lock on FSSchedulerApp
[ https://issues.apache.org/jira/browse/YARN-1721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899809#comment-13899809 ] Sandy Ryza commented on YARN-1721: -- My above comment does not appear to be issue, as we never take locks out on the queues. (This is OK because of the coarser-grained synchronization) When moving app between queues in Fair Scheduler, grab lock on FSSchedulerApp - Key: YARN-1721 URL: https://issues.apache.org/jira/browse/YARN-1721 Project: Hadoop YARN Issue Type: Bug Components: scheduler Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-1721.patch FairScheduler.moveApplication should grab lock on FSSchedulerApp, so that allocate() can't be modifying it at the same time. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1578) Fix how to read history file in FileSystemApplicationHistoryStore
[
https://issues.apache.org/jira/browse/YARN-1578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899819#comment-13899819
]
Zhijie Shen commented on YARN-1578:
---
Committed to trunk and branch-2. Thanks, [~sinchii]!
Fix how to read history file in FileSystemApplicationHistoryStore
-
Key: YARN-1578
URL: https://issues.apache.org/jira/browse/YARN-1578
Project: Hadoop YARN
Issue Type: Sub-task
Affects Versions: YARN-321
Reporter: Shinichi Yamashita
Assignee: Shinichi Yamashita
Attachments: YARN-1578-2.patch, YARN-1578-3.patch, YARN-1578-4.patch,
YARN-1578.patch, application_1390978867235_0001, resoucemanager.log,
screenshot.png, screenshot2.pdf
I carried out PiEstimator job at Hadoop cluster which applied YARN-321.
After the job end and when I accessed Web UI of HistoryServer, it displayed
500. And HistoryServer daemon log was output as follows.
{code}
2014-01-09 13:31:12,227 ERROR org.apache.hadoop.yarn.webapp.Dispatcher: error
handling URI:
/applicationhistory/appattempt/appattempt_1389146249925_0008_01
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:153)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
(snip...)
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore.mergeContainerHistoryData(FileSystemApplicationHistoryStore.java:696)
at
org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore.getContainers(FileSystemApplicationHistoryStore.java:429)
at
org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl.getContainers(ApplicationHistoryManagerImpl.java:201)
at
org.apache.hadoop.yarn.server.webapp.AppAttemptBlock.render(AppAttemptBlock.java:110)
(snip...)
{code}
I confirmed that there was container which was not finished from
ApplicationHistory file.
In ResourceManager daemon log, ResourceManager reserved this container, but
did not allocate it.
When FileSystemApplicationHistoryStore reads container information without
finish data in history file, this problem occurs.
In consideration of the case which there is not finish data, we should fix
how to read history file in FileSystemApplicationHistoryStore.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (YARN-1570) Formatting the lines within 80 chars in YarnCommands.apt.vm
[
https://issues.apache.org/jira/browse/YARN-1570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated YARN-1570:
Attachment: YARN-1570.patch
Attaching a patch.
Formatting the lines within 80 chars in YarnCommands.apt.vm
---
Key: YARN-1570
URL: https://issues.apache.org/jira/browse/YARN-1570
Project: Hadoop YARN
Issue Type: Improvement
Components: documentation
Affects Versions: 2.2.0
Reporter: Akira AJISAKA
Priority: Minor
Labels: newbie
Attachments: YARN-1570.patch
In YarnCommands.apt.vm, there are some lines longer than 80 characters.
For example:
{code}
Yarn commands are invoked by the bin/yarn script. Running the yarn script
without any arguments prints the description for all commands.
{code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (YARN-1551) Allow user-specified reason for killApplication
[ https://issues.apache.org/jira/browse/YARN-1551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gera Shegalov updated YARN-1551: Attachment: YARN-1551.v06.patch Thanks [~kkambatl] for the review. Getting rid of the redundant branching in v07.patch Allow user-specified reason for killApplication --- Key: YARN-1551 URL: https://issues.apache.org/jira/browse/YARN-1551 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 2.4.0 Reporter: Gera Shegalov Assignee: Gera Shegalov Attachments: YARN-1551.v01.patch, YARN-1551.v02.patch, YARN-1551.v03.patch, YARN-1551.v04.patch, YARN-1551.v05.patch, YARN-1551.v06.patch This completes MAPREDUCE-5648 -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1551) Allow user-specified reason for killApplication
[ https://issues.apache.org/jira/browse/YARN-1551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899833#comment-13899833 ] Gera Shegalov commented on YARN-1551: - s/v07/v06 in previous comment. Allow user-specified reason for killApplication --- Key: YARN-1551 URL: https://issues.apache.org/jira/browse/YARN-1551 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 2.4.0 Reporter: Gera Shegalov Assignee: Gera Shegalov Attachments: YARN-1551.v01.patch, YARN-1551.v02.patch, YARN-1551.v03.patch, YARN-1551.v04.patch, YARN-1551.v05.patch, YARN-1551.v06.patch This completes MAPREDUCE-5648 -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1685) [YARN-321] Logs link can be null so avoid NPE
[
https://issues.apache.org/jira/browse/YARN-1685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899837#comment-13899837
]
Zhijie Shen commented on YARN-1685:
---
Hi Mayank, some comments on the patch:
1. Please do not return # in the util method, which may not make sense to all
callers. Instead do null check in the related web page classes.
{code}
+ if(url == null){
+ return #;
+ }
{code}
2. Please fix the test failures. And in TestRMContainerImpl, please assert both
the urls of container at running and completed states.
[YARN-321] Logs link can be null so avoid NPE
-
Key: YARN-1685
URL: https://issues.apache.org/jira/browse/YARN-1685
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Mayank Bansal
Assignee: Mayank Bansal
Fix For: YARN-321
Attachments: YARN-1685-1.patch
https://issues.apache.org/jira/browse/YARN-1413?focusedCommentId=13866416page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13866416
https://issues.apache.org/jira/browse/YARN-1413?focusedCommentId=13866844page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13866844
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (YARN-1666) Make admin refreshNodes work across RM failover
[ https://issues.apache.org/jira/browse/YARN-1666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-1666: Attachment: YARN-1666.4.patch Make admin refreshNodes work across RM failover --- Key: YARN-1666 URL: https://issues.apache.org/jira/browse/YARN-1666 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong Attachments: YARN-1666.1.patch, YARN-1666.2.patch, YARN-1666.2.patch, YARN-1666.3.patch, YARN-1666.4.patch, YARN-1666.4.patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1666) Make admin refreshNodes work across RM failover
[ https://issues.apache.org/jira/browse/YARN-1666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899858#comment-13899858 ] Xuan Gong commented on YARN-1666: - add Private annotation to new APIs Make admin refreshNodes work across RM failover --- Key: YARN-1666 URL: https://issues.apache.org/jira/browse/YARN-1666 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong Attachments: YARN-1666.1.patch, YARN-1666.2.patch, YARN-1666.2.patch, YARN-1666.3.patch, YARN-1666.4.patch, YARN-1666.4.patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1345) Removing FINAL_SAVING from YarnApplicationAttemptState
[ https://issues.apache.org/jira/browse/YARN-1345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899865#comment-13899865 ] Jian He commented on YARN-1345: --- Committed to trunk and branch-2 , thanks Zhijie ! Removing FINAL_SAVING from YarnApplicationAttemptState -- Key: YARN-1345 URL: https://issues.apache.org/jira/browse/YARN-1345 Project: Hadoop YARN Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Attachments: YARN-1345.1.patch, YARN-1345.2.patch Whenever YARN-891 is done, we need to add the mapping of RMAppAttemptState.FINAL_SAVING - YarnApplicationAttemptState.FINAL_SAVING in RMServerUtils#createApplicationAttemptState -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1578) Fix how to read history file in FileSystemApplicationHistoryStore
[
https://issues.apache.org/jira/browse/YARN-1578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899873#comment-13899873
]
Hudson commented on YARN-1578:
--
SUCCESS: Integrated in Hadoop-trunk-Commit #5160 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/5160/])
YARN-1578. Fixed reading incomplete application attempt and container data in
FileSystemApplicationHistoryStore. Contributed by Shinichi Yamashita. (zjshen:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1567816)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/FileSystemApplicationHistoryStore.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestFileSystemApplicationHistoryStore.java
Fix how to read history file in FileSystemApplicationHistoryStore
-
Key: YARN-1578
URL: https://issues.apache.org/jira/browse/YARN-1578
Project: Hadoop YARN
Issue Type: Sub-task
Affects Versions: YARN-321
Reporter: Shinichi Yamashita
Assignee: Shinichi Yamashita
Fix For: 2.4.0
Attachments: YARN-1578-2.patch, YARN-1578-3.patch, YARN-1578-4.patch,
YARN-1578.patch, application_1390978867235_0001, resoucemanager.log,
screenshot.png, screenshot2.pdf
I carried out PiEstimator job at Hadoop cluster which applied YARN-321.
After the job end and when I accessed Web UI of HistoryServer, it displayed
500. And HistoryServer daemon log was output as follows.
{code}
2014-01-09 13:31:12,227 ERROR org.apache.hadoop.yarn.webapp.Dispatcher: error
handling URI:
/applicationhistory/appattempt/appattempt_1389146249925_0008_01
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:153)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
(snip...)
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore.mergeContainerHistoryData(FileSystemApplicationHistoryStore.java:696)
at
org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore.getContainers(FileSystemApplicationHistoryStore.java:429)
at
org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl.getContainers(ApplicationHistoryManagerImpl.java:201)
at
org.apache.hadoop.yarn.server.webapp.AppAttemptBlock.render(AppAttemptBlock.java:110)
(snip...)
{code}
I confirmed that there was container which was not finished from
ApplicationHistory file.
In ResourceManager daemon log, ResourceManager reserved this container, but
did not allocate it.
When FileSystemApplicationHistoryStore reads container information without
finish data in history file, this problem occurs.
In consideration of the case which there is not finish data, we should fix
how to read history file in FileSystemApplicationHistoryStore.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1345) Removing FINAL_SAVING from YarnApplicationAttemptState
[ https://issues.apache.org/jira/browse/YARN-1345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899872#comment-13899872 ] Hudson commented on YARN-1345: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5160 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5160/]) YARN-1345. Remove FINAL_SAVING state from YarnApplicationAttemptState. Contributed by Zhijie Shen (jianhe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1567820) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/YarnApplicationAttemptState.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMServerUtils.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java Removing FINAL_SAVING from YarnApplicationAttemptState -- Key: YARN-1345 URL: https://issues.apache.org/jira/browse/YARN-1345 Project: Hadoop YARN Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Attachments: YARN-1345.1.patch, YARN-1345.2.patch Whenever YARN-891 is done, we need to add the mapping of RMAppAttemptState.FINAL_SAVING - YarnApplicationAttemptState.FINAL_SAVING in RMServerUtils#createApplicationAttemptState -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1570) Formatting the lines within 80 chars in YarnCommands.apt.vm
[
https://issues.apache.org/jira/browse/YARN-1570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899890#comment-13899890
]
Hadoop QA commented on YARN-1570:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12628620/YARN-1570.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+0 tests included{color}. The patch appears to be a
documentation patch that doesn't require tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3088//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3088//console
This message is automatically generated.
Formatting the lines within 80 chars in YarnCommands.apt.vm
---
Key: YARN-1570
URL: https://issues.apache.org/jira/browse/YARN-1570
Project: Hadoop YARN
Issue Type: Improvement
Components: documentation
Affects Versions: 2.2.0
Reporter: Akira AJISAKA
Assignee: Akira AJISAKA
Priority: Minor
Labels: newbie
Attachments: YARN-1570.patch
In YarnCommands.apt.vm, there are some lines longer than 80 characters.
For example:
{code}
Yarn commands are invoked by the bin/yarn script. Running the yarn script
without any arguments prints the description for all commands.
{code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1721) When moving app between queues in Fair Scheduler, grab lock on FSSchedulerApp
[
https://issues.apache.org/jira/browse/YARN-1721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899905#comment-13899905
]
Hadoop QA commented on YARN-1721:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12628614/YARN-1721.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3089//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3089//console
This message is automatically generated.
When moving app between queues in Fair Scheduler, grab lock on FSSchedulerApp
-
Key: YARN-1721
URL: https://issues.apache.org/jira/browse/YARN-1721
Project: Hadoop YARN
Issue Type: Bug
Components: scheduler
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Attachments: YARN-1721.patch
FairScheduler.moveApplication should grab lock on FSSchedulerApp, so that
allocate() can't be modifying it at the same time.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1296) schedulerAllocateTimer is accessed without holding samplerLock in ResourceSchedulerWrapper
[
https://issues.apache.org/jira/browse/YARN-1296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899909#comment-13899909
]
Wei Yan commented on YARN-1296:
---
+1, the patch looks good to me. Looks also need to hold lock in the
handle(SchedulerEvent schedulerEvent) function.
schedulerAllocateTimer is accessed without holding samplerLock in
ResourceSchedulerWrapper
--
Key: YARN-1296
URL: https://issues.apache.org/jira/browse/YARN-1296
Project: Hadoop YARN
Issue Type: Bug
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Minor
Attachments: yarn-1296-v1.patch
Here is related code:
{code}
public Allocation allocate(ApplicationAttemptId attemptId,
ListResourceRequest resourceRequests,
ListContainerId containerIds,
ListString strings, ListString strings2) {
if (metricsON) {
final Timer.Context context = schedulerAllocateTimer.time();
{code}
samplerLock should be used to guard the access.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Created] (YARN-1722) AMRMProtocol should have a way of getting all the nodes in the cluster
Bikas Saha created YARN-1722: Summary: AMRMProtocol should have a way of getting all the nodes in the cluster Key: YARN-1722 URL: https://issues.apache.org/jira/browse/YARN-1722 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.2.0 Reporter: Bikas Saha Fix For: 2.4.0 There is no way for an AM to find out the names of all the nodes in the cluster via the AMRMProtocol. An AM can only at best ask for containers at * location. The only way to get that information is via the ClientRMProtocol but that is secured by Kerberos or RMDelegationToken while the AM has an AMRMToken. This is a pretty important piece of missing functionality. There are other jiras opened about getting cluster topology etc. but they havent been addressed due to a clear definition of cluster topology perhaps. Adding a means to at least get the node information would be a good first step. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1723) AMRMClientAsync missing blacklist addition and removal functionality
[ https://issues.apache.org/jira/browse/YARN-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated YARN-1723: - Affects Version/s: 2.2.0 AMRMClientAsync missing blacklist addition and removal functionality Key: YARN-1723 URL: https://issues.apache.org/jira/browse/YARN-1723 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.2.0 Reporter: Bikas Saha Fix For: 2.4.0 -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1692) ConcurrentModificationException in fair scheduler AppSchedulable
[
https://issues.apache.org/jira/browse/YARN-1692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899933#comment-13899933
]
Sangjin Lee commented on YARN-1692:
---
Hmm, I thought that refactoring had already happened in branch-2.3, but I guess
I was mistaken. I'll post a separate patch for branch-2.3.
ConcurrentModificationException in fair scheduler AppSchedulable
Key: YARN-1692
URL: https://issues.apache.org/jira/browse/YARN-1692
Project: Hadoop YARN
Issue Type: Bug
Components: scheduler
Affects Versions: 2.0.5-alpha
Reporter: Sangjin Lee
Assignee: Sangjin Lee
Fix For: 2.4.0
Attachments: yarn-1692.patch
We saw a ConcurrentModificationException thrown in the fair scheduler:
{noformat}
2014-02-07 01:40:01,978 ERROR
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
Exception in fair scheduler UpdateThread
java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextEntry(HashMap.java:926)
at java.util.HashMap$ValueIterator.next(HashMap.java:954)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AppSchedulable.updateDemand(AppSchedulable.java:85)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.updateDemand(FSLeafQueue.java:125)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSParentQueue.updateDemand(FSParentQueue.java:82)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.update(FairScheduler.java:217)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler$UpdateThread.run(FairScheduler.java:195)
at java.lang.Thread.run(Thread.java:724)
{noformat}
The map that gets returned by FSSchedulerApp.getResourceRequests() are
iterated on without proper synchronization.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Created] (YARN-1723) AMRMClientAsync missing blacklist addition and removal functionality
Bikas Saha created YARN-1723: Summary: AMRMClientAsync missing blacklist addition and removal functionality Key: YARN-1723 URL: https://issues.apache.org/jira/browse/YARN-1723 Project: Hadoop YARN Issue Type: Bug Reporter: Bikas Saha -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1723) AMRMClientAsync missing blacklist addition and removal functionality
[ https://issues.apache.org/jira/browse/YARN-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated YARN-1723: - Fix Version/s: 2.4.0 AMRMClientAsync missing blacklist addition and removal functionality Key: YARN-1723 URL: https://issues.apache.org/jira/browse/YARN-1723 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.2.0 Reporter: Bikas Saha Fix For: 2.4.0 -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (YARN-1724) Race condition in Fair Scheduler when continuous scheduling is turned on
Sandy Ryza created YARN-1724: Summary: Race condition in Fair Scheduler when continuous scheduling is turned on Key: YARN-1724 URL: https://issues.apache.org/jira/browse/YARN-1724 Project: Hadoop YARN Issue Type: Bug Components: scheduler Reporter: Sandy Ryza Assignee: Sandy Ryza If nodes resource allocations change during Collections.sort(nodeIdList, nodeAvailableResourceComparator); we'll hit: java.lang.IllegalArgumentException: Comparison method violates its general contract! -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1692) ConcurrentModificationException in fair scheduler AppSchedulable
[
https://issues.apache.org/jira/browse/YARN-1692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sangjin Lee updated YARN-1692:
--
Attachment: yarn-1692-branch-2.3.patch
This should apply for branch-2.3. It seems that the class in 2.3 is named
SchedulerApplication. Otherwise, the fix is identical.
ConcurrentModificationException in fair scheduler AppSchedulable
Key: YARN-1692
URL: https://issues.apache.org/jira/browse/YARN-1692
Project: Hadoop YARN
Issue Type: Bug
Components: scheduler
Affects Versions: 2.0.5-alpha
Reporter: Sangjin Lee
Assignee: Sangjin Lee
Fix For: 2.4.0
Attachments: yarn-1692-branch-2.3.patch, yarn-1692.patch
We saw a ConcurrentModificationException thrown in the fair scheduler:
{noformat}
2014-02-07 01:40:01,978 ERROR
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
Exception in fair scheduler UpdateThread
java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextEntry(HashMap.java:926)
at java.util.HashMap$ValueIterator.next(HashMap.java:954)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AppSchedulable.updateDemand(AppSchedulable.java:85)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.updateDemand(FSLeafQueue.java:125)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSParentQueue.updateDemand(FSParentQueue.java:82)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.update(FairScheduler.java:217)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler$UpdateThread.run(FairScheduler.java:195)
at java.lang.Thread.run(Thread.java:724)
{noformat}
The map that gets returned by FSSchedulerApp.getResourceRequests() are
iterated on without proper synchronization.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (YARN-1692) ConcurrentModificationException in fair scheduler AppSchedulable
[
https://issues.apache.org/jira/browse/YARN-1692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sangjin Lee updated YARN-1692:
--
Attachment: (was: yarn-1692-branch-2.3.patch)
ConcurrentModificationException in fair scheduler AppSchedulable
Key: YARN-1692
URL: https://issues.apache.org/jira/browse/YARN-1692
Project: Hadoop YARN
Issue Type: Bug
Components: scheduler
Affects Versions: 2.0.5-alpha
Reporter: Sangjin Lee
Assignee: Sangjin Lee
Fix For: 2.4.0
Attachments: yarn-1692.patch
We saw a ConcurrentModificationException thrown in the fair scheduler:
{noformat}
2014-02-07 01:40:01,978 ERROR
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
Exception in fair scheduler UpdateThread
java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextEntry(HashMap.java:926)
at java.util.HashMap$ValueIterator.next(HashMap.java:954)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AppSchedulable.updateDemand(AppSchedulable.java:85)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.updateDemand(FSLeafQueue.java:125)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSParentQueue.updateDemand(FSParentQueue.java:82)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.update(FairScheduler.java:217)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler$UpdateThread.run(FairScheduler.java:195)
at java.lang.Thread.run(Thread.java:724)
{noformat}
The map that gets returned by FSSchedulerApp.getResourceRequests() are
iterated on without proper synchronization.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (YARN-1692) ConcurrentModificationException in fair scheduler AppSchedulable
[
https://issues.apache.org/jira/browse/YARN-1692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sangjin Lee updated YARN-1692:
--
Attachment: yarn-1692-branch-2.3.patch
ConcurrentModificationException in fair scheduler AppSchedulable
Key: YARN-1692
URL: https://issues.apache.org/jira/browse/YARN-1692
Project: Hadoop YARN
Issue Type: Bug
Components: scheduler
Affects Versions: 2.0.5-alpha
Reporter: Sangjin Lee
Assignee: Sangjin Lee
Fix For: 2.4.0
Attachments: yarn-1692-branch-2.3.patch, yarn-1692.patch
We saw a ConcurrentModificationException thrown in the fair scheduler:
{noformat}
2014-02-07 01:40:01,978 ERROR
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
Exception in fair scheduler UpdateThread
java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextEntry(HashMap.java:926)
at java.util.HashMap$ValueIterator.next(HashMap.java:954)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AppSchedulable.updateDemand(AppSchedulable.java:85)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.updateDemand(FSLeafQueue.java:125)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSParentQueue.updateDemand(FSParentQueue.java:82)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.update(FairScheduler.java:217)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler$UpdateThread.run(FairScheduler.java:195)
at java.lang.Thread.run(Thread.java:724)
{noformat}
The map that gets returned by FSSchedulerApp.getResourceRequests() are
iterated on without proper synchronization.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (YARN-1525) Web UI should redirect to active RM when HA is enabled.
[ https://issues.apache.org/jira/browse/YARN-1525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cindy Li updated YARN-1525: --- Attachment: YARN1525.patch Made changes w.r.t Karthik's comment. @Karthik, The test couldn't be moved to testRMHA because it needs to use MiniYarnCluster which has been setup well in testRMFailover, but not used at all in testRMHA. It would be nice if you can test in a secure cluster. Thanks. Web UI should redirect to active RM when HA is enabled. --- Key: YARN-1525 URL: https://issues.apache.org/jira/browse/YARN-1525 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Cindy Li Attachments: YARN1525.patch, YARN1525.patch, YARN1525.patch.v1, YARN1525.patch.v2, YARN1525.patch.v3, YARN1525.v7.patch, YARN1525.v7.patch, YARN1525.v8.patch, YARN1525.v9.patch When failover happens, web UI should redirect to the current active rm. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1551) Allow user-specified reason for killApplication
[
https://issues.apache.org/jira/browse/YARN-1551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899971#comment-13899971
]
Hadoop QA commented on YARN-1551:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12628624/YARN-1551.v06.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.mapreduce.v2.TestUberAM
The following test timeouts occurred in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.client.api.impl.TestNMClient
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3087//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3087//console
This message is automatically generated.
Allow user-specified reason for killApplication
---
Key: YARN-1551
URL: https://issues.apache.org/jira/browse/YARN-1551
Project: Hadoop YARN
Issue Type: Improvement
Affects Versions: 2.4.0
Reporter: Gera Shegalov
Assignee: Gera Shegalov
Attachments: YARN-1551.v01.patch, YARN-1551.v02.patch,
YARN-1551.v03.patch, YARN-1551.v04.patch, YARN-1551.v05.patch,
YARN-1551.v06.patch
This completes MAPREDUCE-5648
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (YARN-1724) Race condition in Fair Scheduler when continuous scheduling is turned on
[ https://issues.apache.org/jira/browse/YARN-1724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated YARN-1724: - Attachment: YARN-1724.patch Race condition in Fair Scheduler when continuous scheduling is turned on - Key: YARN-1724 URL: https://issues.apache.org/jira/browse/YARN-1724 Project: Hadoop YARN Issue Type: Bug Components: scheduler Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-1724.patch If nodes resource allocations change during Collections.sort(nodeIdList, nodeAvailableResourceComparator); we'll hit: java.lang.IllegalArgumentException: Comparison method violates its general contract! -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1297) Miscellaneous Fair Scheduler speedups
[ https://issues.apache.org/jira/browse/YARN-1297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated YARN-1297: - Attachment: YARN-1297-2.patch Miscellaneous Fair Scheduler speedups - Key: YARN-1297 URL: https://issues.apache.org/jira/browse/YARN-1297 Project: Hadoop YARN Issue Type: Improvement Components: scheduler Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-1297-1.patch, YARN-1297-2.patch, YARN-1297.patch, YARN-1297.patch I ran the Fair Scheduler's core scheduling loop through a profiler to and identified a bunch of minimally invasive changes that can shave off a few milliseconds. The main one is demoting a couple INFO log messages to DEBUG, which brought my benchmark down from 16000 ms to 6000. A few others (which had way less of an impact) were * Most of the time in comparisons was being spent in Math.signum. I switched this to direct ifs and elses and it halved the percent of time spent in comparisons. * I removed some unnecessary instantiations of Resource objects * I made it so that queues' usage wasn't calculated from the applications up each time getResourceUsage was called. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1297) Miscellaneous Fair Scheduler speedups
[ https://issues.apache.org/jira/browse/YARN-1297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899976#comment-13899976 ] Sandy Ryza commented on YARN-1297: -- Ran more rigorous benchmarks and the FairSharePolicy changes don't seem to have much impact. Uploading a patch with just the log message changes and the usage calculation changes. Miscellaneous Fair Scheduler speedups - Key: YARN-1297 URL: https://issues.apache.org/jira/browse/YARN-1297 Project: Hadoop YARN Issue Type: Improvement Components: scheduler Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-1297-1.patch, YARN-1297-2.patch, YARN-1297.patch, YARN-1297.patch I ran the Fair Scheduler's core scheduling loop through a profiler to and identified a bunch of minimally invasive changes that can shave off a few milliseconds. The main one is demoting a couple INFO log messages to DEBUG, which brought my benchmark down from 16000 ms to 6000. A few others (which had way less of an impact) were * Most of the time in comparisons was being spent in Math.signum. I switched this to direct ifs and elses and it halved the percent of time spent in comparisons. * I removed some unnecessary instantiations of Resource objects * I made it so that queues' usage wasn't calculated from the applications up each time getResourceUsage was called. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (YARN-1725) RM should provide an easier way for the app to reject a bad allocation
Bikas Saha created YARN-1725: Summary: RM should provide an easier way for the app to reject a bad allocation Key: YARN-1725 URL: https://issues.apache.org/jira/browse/YARN-1725 Project: Hadoop YARN Issue Type: Bug Reporter: Bikas Saha Currently, if the app gets a bad allocation then it can release the container. However, the app now needs to request those resources again or else the RM will not give it a new container in lieu of the one just rejected. This makes the app writers life hard. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1725) RM should provide an easier way for the app to reject a bad allocation
[ https://issues.apache.org/jira/browse/YARN-1725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1380#comment-1380 ] Sandy Ryza commented on YARN-1725: -- +1 to this. Can it be accomplished on the AMRMClient side? RM should provide an easier way for the app to reject a bad allocation -- Key: YARN-1725 URL: https://issues.apache.org/jira/browse/YARN-1725 Project: Hadoop YARN Issue Type: Bug Reporter: Bikas Saha Currently, if the app gets a bad allocation then it can release the container. However, the app now needs to request those resources again or else the RM will not give it a new container in lieu of the one just rejected. This makes the app writers life hard. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1525) Web UI should redirect to active RM when HA is enabled.
[
https://issues.apache.org/jira/browse/YARN-1525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1383#comment-1383
]
Hadoop QA commented on YARN-1525:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12628660/YARN1525.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The following test timeouts occurred in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.client.api.impl.TestNMClient
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3091//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3091//console
This message is automatically generated.
Web UI should redirect to active RM when HA is enabled.
---
Key: YARN-1525
URL: https://issues.apache.org/jira/browse/YARN-1525
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Xuan Gong
Assignee: Cindy Li
Attachments: YARN1525.patch, YARN1525.patch, YARN1525.patch.v1,
YARN1525.patch.v2, YARN1525.patch.v3, YARN1525.v7.patch, YARN1525.v7.patch,
YARN1525.v8.patch, YARN1525.v9.patch
When failover happens, web UI should redirect to the current active rm.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1666) Make admin refreshNodes work across RM failover
[
https://issues.apache.org/jira/browse/YARN-1666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1386#comment-1386
]
Hadoop QA commented on YARN-1666:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12628631/YARN-1666.4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3090//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3090//console
This message is automatically generated.
Make admin refreshNodes work across RM failover
---
Key: YARN-1666
URL: https://issues.apache.org/jira/browse/YARN-1666
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Xuan Gong
Assignee: Xuan Gong
Attachments: YARN-1666.1.patch, YARN-1666.2.patch, YARN-1666.2.patch,
YARN-1666.3.patch, YARN-1666.4.patch, YARN-1666.4.patch
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (YARN-1551) Allow user-specified reason for killApplication
[ https://issues.apache.org/jira/browse/YARN-1551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gera Shegalov updated YARN-1551: Attachment: YARN-1551.v06.patch the failed tests pass locally and there was no semantic change in v06. Re-uploading the patch assuming it's an intermittent issue. Allow user-specified reason for killApplication --- Key: YARN-1551 URL: https://issues.apache.org/jira/browse/YARN-1551 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 2.4.0 Reporter: Gera Shegalov Assignee: Gera Shegalov Attachments: YARN-1551.v01.patch, YARN-1551.v02.patch, YARN-1551.v03.patch, YARN-1551.v04.patch, YARN-1551.v05.patch, YARN-1551.v06.patch, YARN-1551.v06.patch This completes MAPREDUCE-5648 -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1515) Ability to dump the container threads and stop the containers in a single RPC
[ https://issues.apache.org/jira/browse/YARN-1515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13900012#comment-13900012 ] Gera Shegalov commented on YARN-1515: - [~kkambatl] thanks for the suggestion! I am addressing it. Ability to dump the container threads and stop the containers in a single RPC - Key: YARN-1515 URL: https://issues.apache.org/jira/browse/YARN-1515 Project: Hadoop YARN Issue Type: New Feature Components: api, nodemanager Reporter: Gera Shegalov Assignee: Gera Shegalov Attachments: YARN-1515.v01.patch, YARN-1515.v02.patch This is needed to implement MAPREDUCE-5044 to enable thread diagnostics for timed-out task attempts. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1725) RM should provide an easier way for the app to reject a bad allocation
[ https://issues.apache.org/jira/browse/YARN-1725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13900029#comment-13900029 ] Karthik Kambatla commented on YARN-1725: +1 to doing this. The last time there was discussion on something similar, there was a concern of too much state that in the RM. Doing it on the client side does get us past that. RM should provide an easier way for the app to reject a bad allocation -- Key: YARN-1725 URL: https://issues.apache.org/jira/browse/YARN-1725 Project: Hadoop YARN Issue Type: Bug Reporter: Bikas Saha Currently, if the app gets a bad allocation then it can release the container. However, the app now needs to request those resources again or else the RM will not give it a new container in lieu of the one just rejected. This makes the app writers life hard. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
