[jira] [Commented] (YARN-3435) AM container to be allocated Appattempt AM container shown as null
[ https://issues.apache.org/jira/browse/YARN-3435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14396139#comment-14396139 ] Xuan Gong commented on YARN-3435: - +1 lgtm. will commit > AM container to be allocated Appattempt AM container shown as null > -- > > Key: YARN-3435 > URL: https://issues.apache.org/jira/browse/YARN-3435 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager > Environment: 1RM,1DN >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Trivial > Attachments: Screenshot.png, YARN-3435.001.patch > > > Submit yarn application > Open http://:8088/cluster/appattempt/appattempt_1427984982805_0003_01 > Before the AM container is allocated -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3435) AM container to be allocated Appattempt AM container shown as null
[ https://issues.apache.org/jira/browse/YARN-3435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14396142#comment-14396142 ] Hudson commented on YARN-3435: -- FAILURE: Integrated in Hadoop-trunk-Commit #7513 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7513/]) YARN-3435. AM container to be allocated Appattempt AM container shown as (xgong: rev 96d72118f5c81aa4e0dca0dc0241fbf1a3fff4d2) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMAppAttemptBlock.java > AM container to be allocated Appattempt AM container shown as null > -- > > Key: YARN-3435 > URL: https://issues.apache.org/jira/browse/YARN-3435 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager > Environment: 1RM,1DN >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Trivial > Fix For: 2.8.0 > > Attachments: Screenshot.png, YARN-3435.001.patch > > > Submit yarn application > Open http://:8088/cluster/appattempt/appattempt_1427984982805_0003_01 > Before the AM container is allocated -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3394) WebApplication proxy documentation is incomplete
[ https://issues.apache.org/jira/browse/YARN-3394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14396148#comment-14396148 ] Naganarasimha G R commented on YARN-3394: - hi [~ozawa]/ [~aw] Can any one of you please take look at this issue ? > WebApplication proxy documentation is incomplete > - > > Key: YARN-3394 > URL: https://issues.apache.org/jira/browse/YARN-3394 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.6.0 >Reporter: Bibin A Chundatt >Assignee: Naganarasimha G R >Priority: Minor > Attachments: WebApplicationProxy.html, YARN-3394.20150324-1.patch > > > Webproxy documentation is incomplete > hadoop-yarn/hadoop-yarn-site/WebApplicationProxy.html > 1.Configuration of service start/stop as separate server > 2.Steps to start as daemon service > 3.Secure mode for Web proxy -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1376) NM need to notify the log aggregation status to RM through Node heartbeat
[
https://issues.apache.org/jira/browse/YARN-1376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14396149#comment-14396149
]
Hadoop QA commented on YARN-1376:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12709457/YARN-1376.2015-04-04.patch
against trunk revision 4b3948e.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 2 new
Findbugs (version 2.0.3) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/7221//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/7221//artifact/patchprocess/patchReleaseAuditProblems.txt
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/7221//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7221//console
This message is automatically generated.
> NM need to notify the log aggregation status to RM through Node heartbeat
> -
>
> Key: YARN-1376
> URL: https://issues.apache.org/jira/browse/YARN-1376
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Attachments: YARN-1376.1.patch, YARN-1376.2.patch, YARN-1376.2.patch,
> YARN-1376.2015-04-04.patch, YARN-1376.3.patch, YARN-1376.4.patch
>
>
> Expose a client API to allow clients to figure if log aggregation is
> complete. The ticket is used to track the changes on NM side
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3110) Few issues in ApplicationHistory web ui
[
https://issues.apache.org/jira/browse/YARN-3110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14396151#comment-14396151
]
Naganarasimha G R commented on YARN-3110:
-
Hi [~xgong],
Can you please take a look at this jira ?
> Few issues in ApplicationHistory web ui
> ---
>
> Key: YARN-3110
> URL: https://issues.apache.org/jira/browse/YARN-3110
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: applications, timelineserver
>Affects Versions: 2.6.0
>Reporter: Bibin A Chundatt
>Assignee: Naganarasimha G R
>Priority: Minor
> Attachments: YARN-3110.20150209-1.patch, YARN-3110.20150315-1.patch
>
>
> Application state and History link wrong when Application is in unassigned
> state
>
> 1.Configure capacity schedular with queue size as 1 also max Absolute Max
> Capacity: 10.0%
> (Current application state is Accepted and Unassigned from resource manager
> side)
> 2.Submit application to queue and check the state and link in Application
> history
> State= null and History link shown as N/A in applicationhistory page
> Kill the same application . In timeline server logs the below is show when
> selecting application link.
> {quote}
> 2015-01-29 15:39:50,956 ERROR org.apache.hadoop.yarn.webapp.View: Failed to
> read the AM container of the application attempt
> appattempt_1422467063659_0007_01.
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore.getContainer(ApplicationHistoryManagerOnTimelineStore.java:162)
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore.getAMContainer(ApplicationHistoryManagerOnTimelineStore.java:184)
> at
> org.apache.hadoop.yarn.server.webapp.AppBlock$3.run(AppBlock.java:160)
> at
> org.apache.hadoop.yarn.server.webapp.AppBlock$3.run(AppBlock.java:157)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at
> org.apache.hadoop.yarn.server.webapp.AppBlock.render(AppBlock.java:156)
> at
> org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:67)
> at
> org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:77)
> at org.apache.hadoop.yarn.webapp.View.render(View.java:235)
> at
> org.apache.hadoop.yarn.webapp.view.HtmlPage$Page.subView(HtmlPage.java:49)
> at
> org.apache.hadoop.yarn.webapp.hamlet.HamletImpl$EImp._v(HamletImpl.java:117)
> at org.apache.hadoop.yarn.webapp.hamlet.Hamlet$TD._(Hamlet.java:845)
> at
> org.apache.hadoop.yarn.webapp.view.TwoColumnLayout.render(TwoColumnLayout.java:56)
> at org.apache.hadoop.yarn.webapp.view.HtmlPage.render(HtmlPage.java:82)
> at org.apache.hadoop.yarn.webapp.Controller.render(Controller.java:212)
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.AHSController.app(AHSController.java:38)
> at sun.reflect.GeneratedMethodAccessor63.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:153)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at
> com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
> at
> com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
> at
> com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
> at
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:900)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795)
> at
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
> at
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> at
> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
> at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at
> org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.ja
[jira] [Commented] (YARN-3110) Few issues in ApplicationHistory web ui
[
https://issues.apache.org/jira/browse/YARN-3110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14396183#comment-14396183
]
Xuan Gong commented on YARN-3110:
-
I will take a look shortly. Thanks for working on this issue.
> Few issues in ApplicationHistory web ui
> ---
>
> Key: YARN-3110
> URL: https://issues.apache.org/jira/browse/YARN-3110
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: applications, timelineserver
>Affects Versions: 2.6.0
>Reporter: Bibin A Chundatt
>Assignee: Naganarasimha G R
>Priority: Minor
> Attachments: YARN-3110.20150209-1.patch, YARN-3110.20150315-1.patch
>
>
> Application state and History link wrong when Application is in unassigned
> state
>
> 1.Configure capacity schedular with queue size as 1 also max Absolute Max
> Capacity: 10.0%
> (Current application state is Accepted and Unassigned from resource manager
> side)
> 2.Submit application to queue and check the state and link in Application
> history
> State= null and History link shown as N/A in applicationhistory page
> Kill the same application . In timeline server logs the below is show when
> selecting application link.
> {quote}
> 2015-01-29 15:39:50,956 ERROR org.apache.hadoop.yarn.webapp.View: Failed to
> read the AM container of the application attempt
> appattempt_1422467063659_0007_01.
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore.getContainer(ApplicationHistoryManagerOnTimelineStore.java:162)
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore.getAMContainer(ApplicationHistoryManagerOnTimelineStore.java:184)
> at
> org.apache.hadoop.yarn.server.webapp.AppBlock$3.run(AppBlock.java:160)
> at
> org.apache.hadoop.yarn.server.webapp.AppBlock$3.run(AppBlock.java:157)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at
> org.apache.hadoop.yarn.server.webapp.AppBlock.render(AppBlock.java:156)
> at
> org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:67)
> at
> org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:77)
> at org.apache.hadoop.yarn.webapp.View.render(View.java:235)
> at
> org.apache.hadoop.yarn.webapp.view.HtmlPage$Page.subView(HtmlPage.java:49)
> at
> org.apache.hadoop.yarn.webapp.hamlet.HamletImpl$EImp._v(HamletImpl.java:117)
> at org.apache.hadoop.yarn.webapp.hamlet.Hamlet$TD._(Hamlet.java:845)
> at
> org.apache.hadoop.yarn.webapp.view.TwoColumnLayout.render(TwoColumnLayout.java:56)
> at org.apache.hadoop.yarn.webapp.view.HtmlPage.render(HtmlPage.java:82)
> at org.apache.hadoop.yarn.webapp.Controller.render(Controller.java:212)
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.AHSController.app(AHSController.java:38)
> at sun.reflect.GeneratedMethodAccessor63.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:153)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at
> com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
> at
> com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
> at
> com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
> at
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:900)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795)
> at
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
> at
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> at
> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
> at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at
> org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109)
>
[jira] [Commented] (YARN-3435) AM container to be allocated Appattempt AM container shown as null
[ https://issues.apache.org/jira/browse/YARN-3435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14396188#comment-14396188 ] Hudson commented on YARN-3435: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #154 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/154/]) YARN-3435. AM container to be allocated Appattempt AM container shown as (xgong: rev 96d72118f5c81aa4e0dca0dc0241fbf1a3fff4d2) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMAppAttemptBlock.java * hadoop-yarn-project/CHANGES.txt > AM container to be allocated Appattempt AM container shown as null > -- > > Key: YARN-3435 > URL: https://issues.apache.org/jira/browse/YARN-3435 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager > Environment: 1RM,1DN >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Trivial > Fix For: 2.8.0 > > Attachments: Screenshot.png, YARN-3435.001.patch > > > Submit yarn application > Open http://:8088/cluster/appattempt/appattempt_1427984982805_0003_01 > Before the AM container is allocated -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3435) AM container to be allocated Appattempt AM container shown as null
[ https://issues.apache.org/jira/browse/YARN-3435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14396198#comment-14396198 ] Hudson commented on YARN-3435: -- FAILURE: Integrated in Hadoop-Yarn-trunk #888 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/888/]) YARN-3435. AM container to be allocated Appattempt AM container shown as (xgong: rev 96d72118f5c81aa4e0dca0dc0241fbf1a3fff4d2) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMAppAttemptBlock.java * hadoop-yarn-project/CHANGES.txt > AM container to be allocated Appattempt AM container shown as null > -- > > Key: YARN-3435 > URL: https://issues.apache.org/jira/browse/YARN-3435 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager > Environment: 1RM,1DN >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Trivial > Fix For: 2.8.0 > > Attachments: Screenshot.png, YARN-3435.001.patch > > > Submit yarn application > Open http://:8088/cluster/appattempt/appattempt_1427984982805_0003_01 > Before the AM container is allocated -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3435) AM container to be allocated Appattempt AM container shown as null
[ https://issues.apache.org/jira/browse/YARN-3435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14396234#comment-14396234 ] Hudson commented on YARN-3435: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2104 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2104/]) YARN-3435. AM container to be allocated Appattempt AM container shown as (xgong: rev 96d72118f5c81aa4e0dca0dc0241fbf1a3fff4d2) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMAppAttemptBlock.java * hadoop-yarn-project/CHANGES.txt > AM container to be allocated Appattempt AM container shown as null > -- > > Key: YARN-3435 > URL: https://issues.apache.org/jira/browse/YARN-3435 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager > Environment: 1RM,1DN >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Trivial > Fix For: 2.8.0 > > Attachments: Screenshot.png, YARN-3435.001.patch > > > Submit yarn application > Open http://:8088/cluster/appattempt/appattempt_1427984982805_0003_01 > Before the AM container is allocated -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3435) AM container to be allocated Appattempt AM container shown as null
[ https://issues.apache.org/jira/browse/YARN-3435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14396249#comment-14396249 ] Hudson commented on YARN-3435: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #145 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/145/]) YARN-3435. AM container to be allocated Appattempt AM container shown as (xgong: rev 96d72118f5c81aa4e0dca0dc0241fbf1a3fff4d2) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMAppAttemptBlock.java > AM container to be allocated Appattempt AM container shown as null > -- > > Key: YARN-3435 > URL: https://issues.apache.org/jira/browse/YARN-3435 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager > Environment: 1RM,1DN >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Trivial > Fix For: 2.8.0 > > Attachments: Screenshot.png, YARN-3435.001.patch > > > Submit yarn application > Open http://:8088/cluster/appattempt/appattempt_1427984982805_0003_01 > Before the AM container is allocated -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3448) Add Rolling Time To Lives Level DB Plugin Capabilities
[ https://issues.apache.org/jira/browse/YARN-3448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated YARN-3448: -- Attachment: (was: YARN-3448.1.patch) > Add Rolling Time To Lives Level DB Plugin Capabilities > -- > > Key: YARN-3448 > URL: https://issues.apache.org/jira/browse/YARN-3448 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > > For large applications, the majority of the time in LeveldbTimelineStore is > spent deleting old entities record at a time. An exclusive write lock is held > during the entire deletion phase which in practice can be hours. If we are to > relax some of the consistency constraints, other performance enhancing > techniques can be employed to maximize the throughput and minimize locking > time. > Split the 5 sections of the leveldb database (domain, owner, start time, > entity, index) into 5 separate databases. This allows each database to > maximize the read cache effectiveness based on the unique usage patterns of > each database. With 5 separate databases each lookup is much faster. This can > also help with I/O to have the entity and index databases on separate disks. > Rolling DBs for entity and index DBs. 99.9% of the data are in these two > sections 4:1 ration (index to entity) at least for tez. We replace DB record > removal with file system removal if we create a rolling set of databases that > age out and can be efficiently removed. To do this we must place a constraint > to always place an entity's events into it's correct rolling db instance > based on start time. This allows us to stitching the data back together while > reading and artificial paging. > Relax the synchronous writes constraints. If we are willing to accept losing > some records that we not flushed in the operating system during a crash, we > can use async writes that can be much faster. > Prefer Sequential writes. sequential writes can be several times faster than > random writes. Spend some small effort arranging the writes in such a way > that will trend towards sequential write performance over random write > performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3448) Add Rolling Time To Lives Level DB Plugin Capabilities
[ https://issues.apache.org/jira/browse/YARN-3448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated YARN-3448: -- Attachment: YARN-3448.1.patch > Add Rolling Time To Lives Level DB Plugin Capabilities > -- > > Key: YARN-3448 > URL: https://issues.apache.org/jira/browse/YARN-3448 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: YARN-3448.1.patch > > > For large applications, the majority of the time in LeveldbTimelineStore is > spent deleting old entities record at a time. An exclusive write lock is held > during the entire deletion phase which in practice can be hours. If we are to > relax some of the consistency constraints, other performance enhancing > techniques can be employed to maximize the throughput and minimize locking > time. > Split the 5 sections of the leveldb database (domain, owner, start time, > entity, index) into 5 separate databases. This allows each database to > maximize the read cache effectiveness based on the unique usage patterns of > each database. With 5 separate databases each lookup is much faster. This can > also help with I/O to have the entity and index databases on separate disks. > Rolling DBs for entity and index DBs. 99.9% of the data are in these two > sections 4:1 ration (index to entity) at least for tez. We replace DB record > removal with file system removal if we create a rolling set of databases that > age out and can be efficiently removed. To do this we must place a constraint > to always place an entity's events into it's correct rolling db instance > based on start time. This allows us to stitching the data back together while > reading and artificial paging. > Relax the synchronous writes constraints. If we are willing to accept losing > some records that we not flushed in the operating system during a crash, we > can use async writes that can be much faster. > Prefer Sequential writes. sequential writes can be several times faster than > random writes. Spend some small effort arranging the writes in such a way > that will trend towards sequential write performance over random write > performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3435) AM container to be allocated Appattempt AM container shown as null
[ https://issues.apache.org/jira/browse/YARN-3435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14396257#comment-14396257 ] Hudson commented on YARN-3435: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #2086 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2086/]) YARN-3435. AM container to be allocated Appattempt AM container shown as (xgong: rev 96d72118f5c81aa4e0dca0dc0241fbf1a3fff4d2) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMAppAttemptBlock.java * hadoop-yarn-project/CHANGES.txt > AM container to be allocated Appattempt AM container shown as null > -- > > Key: YARN-3435 > URL: https://issues.apache.org/jira/browse/YARN-3435 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager > Environment: 1RM,1DN >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Trivial > Fix For: 2.8.0 > > Attachments: Screenshot.png, YARN-3435.001.patch > > > Submit yarn application > Open http://:8088/cluster/appattempt/appattempt_1427984982805_0003_01 > Before the AM container is allocated -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3435) AM container to be allocated Appattempt AM container shown as null
[ https://issues.apache.org/jira/browse/YARN-3435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14396312#comment-14396312 ] Bibin A Chundatt commented on YARN-3435: Thank you [~xgong] for committing the same > AM container to be allocated Appattempt AM container shown as null > -- > > Key: YARN-3435 > URL: https://issues.apache.org/jira/browse/YARN-3435 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager > Environment: 1RM,1DN >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Trivial > Fix For: 2.8.0 > > Attachments: Screenshot.png, YARN-3435.001.patch > > > Submit yarn application > Open http://:8088/cluster/appattempt/appattempt_1427984982805_0003_01 > Before the AM container is allocated -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3436) Doc WebServicesIntro.html Example Rest API url wrong
[
https://issues.apache.org/jira/browse/YARN-3436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14396314#comment-14396314
]
Bibin A Chundatt commented on YARN-3436:
Please review the same .
> Doc WebServicesIntro.html Example Rest API url wrong
>
>
> Key: YARN-3436
> URL: https://issues.apache.org/jira/browse/YARN-3436
> Project: Hadoop YARN
> Issue Type: Bug
> Components: documentation, resourcemanager
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Minor
> Attachments: YARN-3436.001.patch
>
>
> /docs/current/hadoop-yarn/hadoop-yarn-site/WebServicesIntro.html
> {quote}
> Response Examples
> JSON response with single resource
> HTTP Request: GET
> http://rmhost.domain:8088/ws/v1/cluster/{color:red}app{color}/application_1324057493980_0001
> Response Status Line: HTTP/1.1 200 OK
> {quote}
> Url should be ws/v1/cluster/{color:red}apps{color} .
> 2 examples on same page are wrong
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3435) AM container to be allocated Appattempt AM container shown as null
[ https://issues.apache.org/jira/browse/YARN-3435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14396348#comment-14396348 ] Hudson commented on YARN-3435: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #155 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/155/]) YARN-3435. AM container to be allocated Appattempt AM container shown as (xgong: rev 96d72118f5c81aa4e0dca0dc0241fbf1a3fff4d2) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMAppAttemptBlock.java * hadoop-yarn-project/CHANGES.txt > AM container to be allocated Appattempt AM container shown as null > -- > > Key: YARN-3435 > URL: https://issues.apache.org/jira/browse/YARN-3435 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager > Environment: 1RM,1DN >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Trivial > Fix For: 2.8.0 > > Attachments: Screenshot.png, YARN-3435.001.patch > > > Submit yarn application > Open http://:8088/cluster/appattempt/appattempt_1427984982805_0003_01 > Before the AM container is allocated -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3289) Docker images should be downloaded during localization
[ https://issues.apache.org/jira/browse/YARN-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14445885#comment-14445885 ] Chen He commented on YARN-3289: --- Looks like the bottleneck of registry can be resolved by chaining registry [https://docs.docker.com/reference/api/hub_registry_spec/#chaining-registries] > Docker images should be downloaded during localization > -- > > Key: YARN-3289 > URL: https://issues.apache.org/jira/browse/YARN-3289 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Ravi Prakash > > We currently call docker run on images while launching containers. If the > image size if sufficiently big, the task will timeout. We should download the > image we want to run during localization (if possible) to prevent this -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3386) Cgroups feature should work with hierarchy path containing comma
[ https://issues.apache.org/jira/browse/YARN-3386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated YARN-3386: --- Summary: Cgroups feature should work with hierarchy path containing comma (was: Cgroups feature should work with default hierarchy settings of CentOS 7) > Cgroups feature should work with hierarchy path containing comma > > > Key: YARN-3386 > URL: https://issues.apache.org/jira/browse/YARN-3386 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki > > The path found by CgroupsLCEResourcesHandler#parseMtab contains comma and > results in failure of container-executor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (YARN-3386) Cgroups feature should work with hierarchy path containing comma
[ https://issues.apache.org/jira/browse/YARN-3386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki resolved YARN-3386. Resolution: Duplicate Thanks for the comment, [~kasha]. I closed this as duplicate of YARN-2194. I will reopen this if there is other platform needs this solely. > Cgroups feature should work with hierarchy path containing comma > > > Key: YARN-3386 > URL: https://issues.apache.org/jira/browse/YARN-3386 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki > > The path found by CgroupsLCEResourcesHandler#parseMtab contains comma and > results in failure of container-executor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3443) Create a 'ResourceHandler' subsystem to ease addition of support for new resource types on the NM
[
https://issues.apache.org/jira/browse/YARN-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14480958#comment-14480958
]
Varun Vasudev commented on YARN-3443:
-
Thanks for uploading the patch [~sidharta-s]!
Some comments
# In PrivilegedOperationExecutor.java
{noformat}
catch (ExitCodeException e) {
if (LOG.isWarnEnabled()) {
LOG.warn("Shell execution returned exit code: " + exec.getExitCode());
LOG.warn("Privileged Execution Operation Output:");
LOG.warn(exec.getOutput());
}
}
{noformat}
Can you merge the warnings into one warn message and add the exception to the
log message? This is so that alerts based on the number of errors/warnings
reflect the actual number.
# In CGroupHandlerImpl.java
{noformat}
if (enableCGroupMount) {
//nothing to do here - we support 'deferred' mounting of specific
//controllers - we'll populate the path for a given controller when an
//explicit mountCGroupController request is issued.
} else {
//cluster admins are expected to have mounted controllers in specific
//locations - we'll attempt to figure out mount points
initializeControllerPathsFromMtab();
}
{noformat}
Can we remove the empty if block?
# In CGroupHandlerImpl.java
{noformat}
LOG.error("Mount point Based on mtab file : " + MTAB_FILE);
LOG.error("Controller mount point not writable for: " + name);
{noformat}
Merge the log messages
# In CGroupHandlerImpl.java
{noformat}
LOG.warn("Failed to initialize controller paths!");
LOG.warn(e);
{noformat}
Merge the log messages
# In CGroupHandlerImpl.java
In initializeControllerPathsFromMtab, when the error and warnings are logged,
should we throw an exception, since the functionality being requested will not
work?
# In CGroupHandlerImpl.java
In mountCGroupController
{noformat}
if (!enableCGroupMount) {
if (LOG.isWarnEnabled()) {
LOG.warn("CGroup mounting is disabled - ignoring mount request for: " +
controller.getName());
return;
}
}
{noformat}
Should this be an error with an exception being thrown since the functionality
being requested won't be possible?
# In CGroupHandlerImpl.java
{noformat}
@param file object referring to the cgroup to be deleted mislabelled
{noformat}
The param name is mislabeled. It should be cgf instead of file
# In ResourceHandler.java
{noformat}
* @param containerId if of the container being reacquired.
* @return
* @throws ResourceHandlerException
{noformat}
return is missing description
# In ResourceHandler.java
{noformat}
List postComplete(ContainerId containerId) throws
ResourceHandlerException;
;
{noformat}
There's an extra semi-colon.
# In TestPrivilegedOperationExecutor.java
{noformat}
expectedPath = customExecutorPath;
containerExePath = PrivilegedOperationExecutor
.getContainerExecutorExecutablePath(confWithExecutorPath);
Assert.assertEquals(expectedPath, customExecutorPath);
{noformat}
containerExePath is not used. The assert will always succeed since you set them
to be equal before calling the function. Did you mean to compare
containerExePath and expectedPath?
# In TestPrivilegedOperationExecutor.java
{noformat}
prefixLength + 0
{noformat}
Remove the 0?
# In TestPrivilegedOperationExecutor.java
{noformat}
try {
PrivilegedOperationExecutor.squashCGroupOperations(ops);
} catch (PrivilegedOperationException e) {
if (LOG.isInfoEnabled()) {
LOG.info("Caught expected exception : " + e);
}
caughtPrivilegedOperationException = true;
}
Assert.assertTrue("Expected squash operation to fail!",
caughtPrivilegedOperationException);
{noformat}
Instead of using the boolean to check if the exception was thrown, you can use
Assert.fail in the try block. This pattern occurs multiple times.
# I think the formatting is a bit off in a couple of files. Can you apply the
formatter and check?
> Create a 'ResourceHandler' subsystem to ease addition of support for new
> resource types on the NM
> -
>
> Key: YARN-3443
> URL: https://issues.apache.org/jira/browse/YARN-3443
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Sidharta Seethana
>Assignee: Sidharta Seethana
> Attachments: YARN-3443.001.patch, YARN-3443.002.patch
>
>
> The current cgroups implementation is closely tied to supporting CPU as a
> resource . We need to separate out CGroups support as well a provide a simple
> ResourceHandler subsystem that will enable us to add support for new resource
> types on the NM - e.g Network, Disk etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3410) YARN admin should be able to remove individual application records from RMStateStore
[
https://issues.apache.org/jira/browse/YARN-3410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14481002#comment-14481002
]
Hadoop QA commented on YARN-3410:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12709200/0001-YARN-3410-v1.patch
against trunk revision 96d7211.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/7222//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7222//console
This message is automatically generated.
> YARN admin should be able to remove individual application records from
> RMStateStore
>
>
> Key: YARN-3410
> URL: https://issues.apache.org/jira/browse/YARN-3410
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager, yarn
>Reporter: Wangda Tan
>Assignee: Rohith
>Priority: Critical
> Attachments: 0001-YARN-3410-v1.patch
>
>
> When RM state store entered an unexpected state, one example is YARN-2340,
> when an attempt is not in final state but app already completed, RM can never
> get up unless format RMStateStore.
> I think we should support remove individual application records from
> RMStateStore to unblock RM admin make choice of either waiting for a fix or
> format state store.
> In addition, RM should be able to report all fatal errors (which will
> shutdown RM) when doing app recovery, this can save admin some time to remove
> apps in bad state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
