[jira] [Created] (YARN-4201) AMBlacklist does not work for minicluster
Jun Gong created YARN-4201: -- Summary: AMBlacklist does not work for minicluster Key: YARN-4201 URL: https://issues.apache.org/jira/browse/YARN-4201 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Reporter: Jun Gong Assignee: Jun Gong For minicluster (scheduler.include-port-in-node-name is set to TRUE), AMBlacklist does not work. It is because RM just puts host to AMBlacklist whether scheduler.include-port-in-node-name is set or not. In fact RM should put "host + port" to AMBlacklist when it is set. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4201) AMBlacklist does not work for minicluster
[
https://issues.apache.org/jira/browse/YARN-4201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904433#comment-14904433
]
Hadoop QA commented on YARN-4201:
-
\\
\\
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 16m 52s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 8m 4s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 5s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 44s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 29s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 29s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 54m 28s | Tests passed in
hadoop-yarn-server-resourcemanager. |
| | | 94m 12s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12761865/YARN-4021.001.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / a2c76e5 |
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9240/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/9240/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/9240/console |
This message was automatically generated.
> AMBlacklist does not work for minicluster
> -
>
> Key: YARN-4201
> URL: https://issues.apache.org/jira/browse/YARN-4201
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Reporter: Jun Gong
>Assignee: Jun Gong
> Attachments: YARN-4021.001.patch
>
>
> For minicluster (scheduler.include-port-in-node-name is set to TRUE),
> AMBlacklist does not work. It is because RM just puts host to AMBlacklist
> whether scheduler.include-port-in-node-name is set or not. In fact RM should
> put "host + port" to AMBlacklist when it is set.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4176) Resync NM nodelabels with RM every x interval for distributed nodelabels
[
https://issues.apache.org/jira/browse/YARN-4176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904358#comment-14904358
]
Bibin A Chundatt commented on YARN-4176:
{{hadoop.yarn.logaggregation.TestAggregatedLogsBlock}} test failure is not
related to patch attached.Checkstyle due to lines more than 2 K
> Resync NM nodelabels with RM every x interval for distributed nodelabels
>
>
> Key: YARN-4176
> URL: https://issues.apache.org/jira/browse/YARN-4176
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: 0001-YARN-4176.patch, 0002-YARN-4176.patch,
> 0003-YARN-4176.patch, 0004-YARN-4176.patch, 0005-YARN-4176.patch
>
>
> This JIRA is for handling the below set of issue
> # Distributed nodelabels after NM registered with RM if cluster nodelabels
> are removed and added then NM doesnt resend labels in heartbeat again untils
> any change in labels
> # NM registration failed with Nodelabels should resend labels again to RM
> The above cases can be handled by resync nodeLabels with RM every x interval
> # Add property {{yarn.nodemanager.node-labels.provider.resync-interval-ms}}
> and will resend nodelabels to RM based on config no matter what the
> registration fails or success.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4176) Resync NM nodelabels with RM every x interval for distributed nodelabels
[
https://issues.apache.org/jira/browse/YARN-4176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904349#comment-14904349
]
Hadoop QA commented on YARN-4176:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 19m 32s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 53s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 6s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 55s | The applied patch generated 1
new checkstyle issues (total was 211, now 211). |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 29s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 4m 26s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 0m 26s | Tests passed in
hadoop-yarn-api. |
| {color:red}-1{color} | yarn tests | 1m 58s | Tests failed in
hadoop-yarn-common. |
| {color:green}+1{color} | yarn tests | 7m 58s | Tests passed in
hadoop-yarn-server-nodemanager. |
| | | 57m 23s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.yarn.logaggregation.TestAggregatedLogsBlock |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12761859/0005-YARN-4176.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / a2c76e5 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/9239/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
|
| hadoop-yarn-api test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9239/artifact/patchprocess/testrun_hadoop-yarn-api.txt
|
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9239/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| hadoop-yarn-server-nodemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9239/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/9239/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/9239/console |
This message was automatically generated.
> Resync NM nodelabels with RM every x interval for distributed nodelabels
>
>
> Key: YARN-4176
> URL: https://issues.apache.org/jira/browse/YARN-4176
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: 0001-YARN-4176.patch, 0002-YARN-4176.patch,
> 0003-YARN-4176.patch, 0004-YARN-4176.patch, 0005-YARN-4176.patch
>
>
> This JIRA is for handling the below set of issue
> # Distributed nodelabels after NM registered with RM if cluster nodelabels
> are removed and added then NM doesnt resend labels in heartbeat again untils
> any change in labels
> # NM registration failed with Nodelabels should resend labels again to RM
> The above cases can be handled by resync nodeLabels with RM every x interval
> # Add property {{yarn.nodemanager.node-labels.provider.resync-interval-ms}}
> and will resend nodelabels to RM based on config no matter what the
> registration fails or success.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3433) Jersey tests failing with Port in Use -again
[ https://issues.apache.org/jira/browse/YARN-3433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904354#comment-14904354 ] Bibin A Chundatt commented on YARN-3433: Hi [~ste...@apache.org] The same failed with port in use again during one of ci build https://builds.apache.org/job/PreCommit-YARN-Build/9232/testReport > Jersey tests failing with Port in Use -again > > > Key: YARN-3433 > URL: https://issues.apache.org/jira/browse/YARN-3433 > Project: Hadoop YARN > Issue Type: Bug > Components: build, test >Affects Versions: 3.0.0 > Environment: ASF Jenkins >Reporter: Steve Loughran >Assignee: Brahma Reddy Battula >Priority: Critical > Fix For: 2.8.0 > > Attachments: YARN-3433.patch > > > ASF Jenkins jersey tests failing with port in use exceptions. > The YARN-2912 patch tried to fix it, but it defaults to port 9998 and doesn't > scan for a spare port —so is too brittle on a busy server -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4176) Resync NM nodelabels with RM every x interval for distributed nodelabels
[
https://issues.apache.org/jira/browse/YARN-4176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-4176:
---
Attachment: 0005-YARN-4176.patch
Thanks [~Naganarasimha] for comments
{quote}
increase this sync duration to a greater value like 10 mins.
{quote}
The udpate interval is 10 mins so resync should be less than 10.
All other comments have handled
> Resync NM nodelabels with RM every x interval for distributed nodelabels
>
>
> Key: YARN-4176
> URL: https://issues.apache.org/jira/browse/YARN-4176
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: 0001-YARN-4176.patch, 0002-YARN-4176.patch,
> 0003-YARN-4176.patch, 0004-YARN-4176.patch, 0005-YARN-4176.patch
>
>
> This JIRA is for handling the below set of issue
> # Distributed nodelabels after NM registered with RM if cluster nodelabels
> are removed and added then NM doesnt resend labels in heartbeat again untils
> any change in labels
> # NM registration failed with Nodelabels should resend labels again to RM
> The above cases can be handled by resync nodeLabels with RM every x interval
> # Add property {{yarn.nodemanager.node-labels.provider.resync-interval-ms}}
> and will resend nodelabels to RM based on config no matter what the
> registration fails or success.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3964) Support NodeLabelsProvider at Resource Manager side
[
https://issues.apache.org/jira/browse/YARN-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904346#comment-14904346
]
Hadoop QA commented on YARN-3964:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 19m 48s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 4 new or modified test files. |
| {color:green}+1{color} | javac | 8m 4s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 54s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 58s | The applied patch generated 1
new checkstyle issues (total was 211, now 211). |
| {color:green}+1{color} | whitespace | 0m 3s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 32s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 35s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 4m 39s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 0m 24s | Tests passed in
hadoop-yarn-api. |
| {color:green}+1{color} | yarn tests | 2m 2s | Tests passed in
hadoop-yarn-common. |
| {color:red}-1{color} | yarn tests | 40m 6s | Tests failed in
hadoop-yarn-server-resourcemanager. |
| | | 90m 14s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueParsing |
| | hadoop.yarn.server.resourcemanager.resourcetracker.TestRMNMRPCResponseId |
| | hadoop.yarn.server.resourcemanager.rmcontainer.TestRMContainerImpl |
| |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerEventLog |
| | hadoop.yarn.server.resourcemanager.TestApplicationCleanup |
| | hadoop.yarn.server.resourcemanager.TestRMHA |
| | hadoop.yarn.server.resourcemanager.TestRMEmbeddedElector |
| |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerQueueACLs |
| | hadoop.yarn.server.resourcemanager.TestResourceManager |
| | hadoop.yarn.server.resourcemanager.TestApplicationMasterService |
| | hadoop.yarn.server.resourcemanager.nodelabels.TestRMNodeLabelsManager |
| | hadoop.yarn.server.resourcemanager.TestRMHAForNodeLabels |
| | hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA |
| | hadoop.yarn.server.resourcemanager.scheduler.TestSchedulerHealth |
| | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue |
| | hadoop.yarn.server.resourcemanager.scheduler.TestSchedulerUtils |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestWorkPreservingRMRestartForNodeLabel
|
| |
hadoop.yarn.server.resourcemanager.monitor.capacity.TestProportionalCapacityPreemptionPolicy
|
| | hadoop.yarn.server.resourcemanager.TestAMAuthorization |
| | hadoop.yarn.server.resourcemanager.resourcetracker.TestNMReconnect |
| | hadoop.yarn.server.resourcemanager.TestApplicationMasterLauncher |
| | hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppAttempt |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation |
| | hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerQueueACLs
|
| | hadoop.yarn.server.resourcemanager.monitor.TestSchedulingMonitor |
| | hadoop.yarn.server.resourcemanager.TestRMProxyUsersConf |
| | hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerDynamicBehavior
|
| | hadoop.yarn.server.resourcemanager.TestApplicationACLs |
| | hadoop.yarn.server.resourcemanager.scheduler.TestAbstractYarnScheduler |
| | hadoop.yarn.server.resourcemanager.TestContainerResourceUsage |
| | hadoop.yarn.server.resourcemanager.scheduler.fifo.TestFifoScheduler |
| | hadoop.yarn.server.resourcemanager.scheduler.TestQueueMetrics |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerNodeLabelUpdate
|
| | hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation
|
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler |
| |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerFairShare |
| |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption |
| |
[jira] [Commented] (YARN-3603) Application Attempts page confusing
[ https://issues.apache.org/jira/browse/YARN-3603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904342#comment-14904342 ] Jeff Zhang commented on YARN-3603: -- Still feel excluding failed container doesn't make sense. If some of my containers fail during job running, how can I check its logs through web ui if I don't want to kill the job ? > Application Attempts page confusing > --- > > Key: YARN-3603 > URL: https://issues.apache.org/jira/browse/YARN-3603 > Project: Hadoop YARN > Issue Type: Bug > Components: webapp >Affects Versions: 2.8.0 >Reporter: Thomas Graves >Assignee: Sunil G > Attachments: 0001-YARN-3603.patch, 0002-YARN-3603.patch, > 0003-YARN-3603.patch, ahs1.png > > > The application attempts page > (http://RM:8088/cluster/appattempt/appattempt_1431101480046_0003_01) > is a bit confusing on what is going on. I think the table of containers > there is for only Running containers and when the app is completed or killed > its empty. The table should have a label on it stating so. > Also the "AM Container" field is a link when running but not when its killed. > That might be confusing. > There is no link to the logs in this page but there is in the app attempt > table when looking at http:// > rm:8088/cluster/app/application_1431101480046_0003 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4201) AMBlacklist does not work for minicluster
[ https://issues.apache.org/jira/browse/YARN-4201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Gong updated YARN-4201: --- Attachment: YARN-4021.001.patch > AMBlacklist does not work for minicluster > - > > Key: YARN-4201 > URL: https://issues.apache.org/jira/browse/YARN-4201 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Reporter: Jun Gong >Assignee: Jun Gong > Attachments: YARN-4021.001.patch > > > For minicluster (scheduler.include-port-in-node-name is set to TRUE), > AMBlacklist does not work. It is because RM just puts host to AMBlacklist > whether scheduler.include-port-in-node-name is set or not. In fact RM should > put "host + port" to AMBlacklist when it is set. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4095) Avoid sharing AllocatorPerContext object in LocalDirAllocator between ShuffleHandler and LocalDirsHandlerService.
[
https://issues.apache.org/jira/browse/YARN-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904687#comment-14904687
]
Jason Lowe commented on YARN-4095:
--
+1 lgtm. Committing this.
> Avoid sharing AllocatorPerContext object in LocalDirAllocator between
> ShuffleHandler and LocalDirsHandlerService.
> -
>
> Key: YARN-4095
> URL: https://issues.apache.org/jira/browse/YARN-4095
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
>Reporter: zhihai xu
>Assignee: zhihai xu
> Attachments: YARN-4095.000.patch, YARN-4095.001.patch
>
>
> Currently {{ShuffleHandler}} and {{LocalDirsHandlerService}} share
> {{AllocatorPerContext}} object in {{LocalDirAllocator}} for configuration
> {{NM_LOCAL_DIRS}} because {{AllocatorPerContext}} are stored in a static
> TreeMap with configuration name as key
> {code}
> private static Map contexts =
> new TreeMap();
> {code}
> {{LocalDirsHandlerService}} and {{ShuffleHandler}} both create a
> {{LocalDirAllocator}} using {{NM_LOCAL_DIRS}}. Even they don't use the same
> {{Configuration}} object, but they will use the same {{AllocatorPerContext}}
> object. Also {{LocalDirsHandlerService}} may change {{NM_LOCAL_DIRS}} value
> in its {{Configuration}} object to exclude full and bad local dirs,
> {{ShuffleHandler}} always uses the original {{NM_LOCAL_DIRS}} value in its
> {{Configuration}} object. So every time {{AllocatorPerContext#confChanged}}
> is called by {{ShuffleHandler}} after {{LocalDirsHandlerService}},
> {{AllocatorPerContext}} need be reinitialized because {{NM_LOCAL_DIRS}} value
> is changed. This will cause some overhead.
> {code}
> String newLocalDirs = conf.get(contextCfgItemName);
> if (!newLocalDirs.equals(savedLocalDirs)) {
> {code}
> So it will be a good improvement to not share the same
> {{AllocatorPerContext}} instance between {{ShuffleHandler}} and
> {{LocalDirsHandlerService}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4075) [reader REST API] implement support for querying for flows and flow runs
[ https://issues.apache.org/jira/browse/YARN-4075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-4075: --- Attachment: YARN-4075-YARN-2928.03.patch > [reader REST API] implement support for querying for flows and flow runs > > > Key: YARN-4075 > URL: https://issues.apache.org/jira/browse/YARN-4075 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Varun Saxena > Attachments: YARN-4075-YARN-2928.03.patch, > YARN-4075-YARN-2928.POC.1.patch, YARN-4075-YARN-2928.POC.2.patch > > > We need to be able to query for flows and flow runs via REST. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3975) WebAppProxyServlet should not redirect to RM page if AHS is enabled
[ https://issues.apache.org/jira/browse/YARN-3975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mit Desai updated YARN-3975: Attachment: YARN-3975.9.patch Attached branch-2.7 patch. Also attached b2 patch agian. I realized, I left a debug statement in the previous one. > WebAppProxyServlet should not redirect to RM page if AHS is enabled > --- > > Key: YARN-3975 > URL: https://issues.apache.org/jira/browse/YARN-3975 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Mit Desai >Assignee: Mit Desai > Attachments: YARN-3975.2.b2.patch, YARN-3975.3.patch, > YARN-3975.4.patch, YARN-3975.5.patch, YARN-3975.6.patch, YARN-3975.7.patch, > YARN-3975.8.patch, YARN-3975.9.b2.7.patch, YARN-3975.9.patch, > YARN-3975.9.patch > > > WebAppProxyServlet should be updated to handle the case when the appreport > doesn't have a tracking URL and the Application History Server is eanbled. > As we would have already tried the RM and got the > ApplicationNotFoundException we should not direct the user to the RM app page. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4075) [reader REST API] implement support for querying for flows and flow runs
[
https://issues.apache.org/jira/browse/YARN-4075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904632#comment-14904632
]
Hadoop QA commented on YARN-4075:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 17m 19s | Findbugs (version ) appears to
be broken on YARN-2928. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 3 new or modified test files. |
| {color:green}+1{color} | javac | 8m 6s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 17s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 22s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 3s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 38s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 39s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 24s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 0m 24s | Tests passed in
hadoop-yarn-api. |
| {color:green}+1{color} | yarn tests | 3m 6s | Tests passed in
hadoop-yarn-server-timelineservice. |
| | | 45m 46s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12761898/YARN-4075-YARN-2928.03.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | YARN-2928 / 2e7e0f0 |
| hadoop-yarn-api test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9241/artifact/patchprocess/testrun_hadoop-yarn-api.txt
|
| hadoop-yarn-server-timelineservice test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9241/artifact/patchprocess/testrun_hadoop-yarn-server-timelineservice.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/9241/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/9241/console |
This message was automatically generated.
> [reader REST API] implement support for querying for flows and flow runs
>
>
> Key: YARN-4075
> URL: https://issues.apache.org/jira/browse/YARN-4075
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Varun Saxena
> Attachments: YARN-4075-YARN-2928.03.patch,
> YARN-4075-YARN-2928.POC.1.patch, YARN-4075-YARN-2928.POC.2.patch
>
>
> We need to be able to query for flows and flow runs via REST.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4075) [reader REST API] implement support for querying for flows and flow runs
[ https://issues.apache.org/jira/browse/YARN-4075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904578#comment-14904578 ] Varun Saxena commented on YARN-4075: In addition to rebasing on latest branch code, patch has following changes : * Removed userId as an optional query param from REST API for flows. * 2 different REST API Endpoints, one with cluster and one without it. * Take cluster id from config if not supplied by client. * Instead of using TimelineMetric#toString, for the sake of clarity, wrote 2 helper methods in test class to compare metrics. * Added VisibleForTesting annotation for TimelineSchemaCreator#createAllTables. The method had been changed to public in updated code. Also, the class itself is marked as private. I have not changed Set to TimelineEntities as there are issues around deserializing a JSON to a POJO if a polymorphic collection is wrapped inside a class as is the case with TimelineEntities. I tried a few things but they did not work. Either each derived class will have to be named separately for deserialization or object mapper configurations will have to be changed. As this is not directly related to this JIRA, this problem should not block this JIRA. I can take care of this in some other JIRA. > [reader REST API] implement support for querying for flows and flow runs > > > Key: YARN-4075 > URL: https://issues.apache.org/jira/browse/YARN-4075 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Varun Saxena > Attachments: YARN-4075-YARN-2928.03.patch, > YARN-4075-YARN-2928.POC.1.patch, YARN-4075-YARN-2928.POC.2.patch > > > We need to be able to query for flows and flow runs via REST. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3975) WebAppProxyServlet should not redirect to RM page if AHS is enabled
[
https://issues.apache.org/jira/browse/YARN-3975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904624#comment-14904624
]
Hadoop QA commented on YARN-3975:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12761903/YARN-3975.9.b2.7.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / a2c76e5 |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/9242/console |
This message was automatically generated.
> WebAppProxyServlet should not redirect to RM page if AHS is enabled
> ---
>
> Key: YARN-3975
> URL: https://issues.apache.org/jira/browse/YARN-3975
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.7.1
>Reporter: Mit Desai
>Assignee: Mit Desai
> Attachments: YARN-3975.2.b2.patch, YARN-3975.3.patch,
> YARN-3975.4.patch, YARN-3975.5.patch, YARN-3975.6.patch, YARN-3975.7.patch,
> YARN-3975.8.patch, YARN-3975.9.b2.7.patch, YARN-3975.9.patch
>
>
> WebAppProxyServlet should be updated to handle the case when the appreport
> doesn't have a tracking URL and the Application History Server is eanbled.
> As we would have already tried the RM and got the
> ApplicationNotFoundException we should not direct the user to the RM app page.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4165) An outstanding container request makes all nodes to be reserved causing all jobs pending
[ https://issues.apache.org/jira/browse/YARN-4165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904577#comment-14904577 ] MENG DING commented on YARN-4165: - I think this is expected behavior according to the current scheduling logic. Assuming resource usage of the app has not gone over user/queue limit, then because of the starvation of the application, the scheduler will keep reserving containers for unsatisfied requests. > An outstanding container request makes all nodes to be reserved causing all > jobs pending > > > Key: YARN-4165 > URL: https://issues.apache.org/jira/browse/YARN-4165 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler, resourcemanager, scheduler >Affects Versions: 2.7.1 >Reporter: Weiwei Yang >Assignee: Weiwei Yang > > We have a long running service in YARN, it has a outstanding container > request that YARN cannot satisfy (require more memory that nodemanager can > supply). Then YARN reserves all nodes for this application, when I submit > other jobs (require relative small memory that nodemanager can supply), all > jobs are pending because YARN skips scheduling containers on the nodes that > have been reserved. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3975) WebAppProxyServlet should not redirect to RM page if AHS is enabled
[ https://issues.apache.org/jira/browse/YARN-3975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mit Desai updated YARN-3975: Attachment: YARN-3975.9.b2.7.patch > WebAppProxyServlet should not redirect to RM page if AHS is enabled > --- > > Key: YARN-3975 > URL: https://issues.apache.org/jira/browse/YARN-3975 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Mit Desai >Assignee: Mit Desai > Attachments: YARN-3975.2.b2.patch, YARN-3975.3.patch, > YARN-3975.4.patch, YARN-3975.5.patch, YARN-3975.6.patch, YARN-3975.7.patch, > YARN-3975.8.patch, YARN-3975.9.b2.7.patch, YARN-3975.9.patch > > > WebAppProxyServlet should be updated to handle the case when the appreport > doesn't have a tracking URL and the Application History Server is eanbled. > As we would have already tried the RM and got the > ApplicationNotFoundException we should not direct the user to the RM app page. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4176) Resync NM nodelabels with RM every x interval for distributed nodelabels
[
https://issues.apache.org/jira/browse/YARN-4176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904655#comment-14904655
]
Naganarasimha G R commented on YARN-4176:
-
Thanks for changes [~bibinchundatt],
Patch lgtm, +1.
> Resync NM nodelabels with RM every x interval for distributed nodelabels
>
>
> Key: YARN-4176
> URL: https://issues.apache.org/jira/browse/YARN-4176
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: 0001-YARN-4176.patch, 0002-YARN-4176.patch,
> 0003-YARN-4176.patch, 0004-YARN-4176.patch, 0005-YARN-4176.patch
>
>
> This JIRA is for handling the below set of issue
> # Distributed nodelabels after NM registered with RM if cluster nodelabels
> are removed and added then NM doesnt resend labels in heartbeat again untils
> any change in labels
> # NM registration failed with Nodelabels should resend labels again to RM
> The above cases can be handled by resync nodeLabels with RM every x interval
> # Add property {{yarn.nodemanager.node-labels.provider.resync-interval-ms}}
> and will resend nodelabels to RM based on config no matter what the
> registration fails or success.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3975) WebAppProxyServlet should not redirect to RM page if AHS is enabled
[
https://issues.apache.org/jira/browse/YARN-3975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904762#comment-14904762
]
Mit Desai commented on YARN-3975:
-
Findbugs not broken due to the patch.
This is from the console output of this run.
{noformat}
Running findbugs in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
/home/jenkins/tools/maven/latest/bin/mvn clean test findbugs:findbugs
-DskipTests -DhadoopPatchProcess >
/home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/patchprocess/trunkFindBugsOutputhadoop-yarn-client.txt
2>&1
Exception in thread "main" java.io.FileNotFoundException:
/home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/patchprocess/trunkFindbugsWarningshadoop-yarn-client.xml
(No such file or directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.(FileInputStream.java:146)
at
edu.umd.cs.findbugs.SortedBugCollection.progessMonitoredInputStream(SortedBugCollection.java:1231)
at
edu.umd.cs.findbugs.SortedBugCollection.readXML(SortedBugCollection.java:308)
at
edu.umd.cs.findbugs.SortedBugCollection.readXML(SortedBugCollection.java:295)
at edu.umd.cs.findbugs.workflow.Filter.main(Filter.java:712)
{noformat}
Found a similar issue in YARN-4075.
Console Output: https://builds.apache.org/job/PreCommit-YARN-Build/9241/console
TestYarnClient passes when I run it on my local box.
Filed HADOOP-12435 and YARN-4202 for investigating the issues.
> WebAppProxyServlet should not redirect to RM page if AHS is enabled
> ---
>
> Key: YARN-3975
> URL: https://issues.apache.org/jira/browse/YARN-3975
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.7.1
>Reporter: Mit Desai
>Assignee: Mit Desai
> Attachments: YARN-3975.2.b2.patch, YARN-3975.3.patch,
> YARN-3975.4.patch, YARN-3975.5.patch, YARN-3975.6.patch, YARN-3975.7.patch,
> YARN-3975.8.patch, YARN-3975.9.b2.7.patch, YARN-3975.9.patch,
> YARN-3975.9.patch
>
>
> WebAppProxyServlet should be updated to handle the case when the appreport
> doesn't have a tracking URL and the Application History Server is eanbled.
> As we would have already tried the RM and got the
> ApplicationNotFoundException we should not direct the user to the RM app page.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (YARN-4202) TestYarnClient#testReservationAPIs fails intermittently
Mit Desai created YARN-4202:
---
Summary: TestYarnClient#testReservationAPIs fails intermittently
Key: YARN-4202
URL: https://issues.apache.org/jira/browse/YARN-4202
Project: Hadoop YARN
Issue Type: Bug
Reporter: Mit Desai
Fix For: 3.0.0
Found this failure while looking at the Pre-run on one of my Jiras.
{noformat}
org.apache.hadoop.yarn.server.resourcemanager.reservation.exceptions.PlanningException:
The planning algorithm could not find a valid allocation for your request
at org.apache.hadoop.yarn.ipc.RPCUtil.getRemoteException(RPCUtil.java:38)
at
org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitReservation(ClientRMService.java:1149)
at
org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.submitReservation(ApplicationClientProtocolPBServiceImpl.java:428)
at
org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:465)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:636)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:976)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2230)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2226)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1667)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2224)
Caused by:
org.apache.hadoop.yarn.server.resourcemanager.reservation.exceptions.PlanningException:
The planning algorithm could not find a valid allocation for your request
at
org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.PlanningAlgorithm.allocateUser(PlanningAlgorithm.java:69)
at
org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.PlanningAlgorithm.createReservation(PlanningAlgorithm.java:140)
at
org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.TryManyReservationAgents.createReservation(TryManyReservationAgents.java:55)
at
org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.AlignedPlannerWithGreedy.createReservation(AlignedPlannerWithGreedy.java:84)
at
org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitReservation(ClientRMService.java:1132)
... 10 more
{noformat}
TestReport Link:
https://builds.apache.org/job/PreCommit-YARN-Build/9243/testReport/
When I ran this on my local box branch-2, it succeeds.
{noformat}
---
T E S T S
---
Running org.apache.hadoop.yarn.client.api.impl.TestYarnClient
Tests run: 21, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.999 sec -
in org.apache.hadoop.yarn.client.api.impl.TestYarnClient
Results :
Tests run: 21, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 52.029 s
[INFO] Finished at: 2015-09-23T11:25:04-06:00
[INFO] Final Memory: 31M/391M
[INFO]
{noformat}
Haven't tried if it is a problem in branch-2.7 or not.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4095) Avoid sharing AllocatorPerContext object in LocalDirAllocator between ShuffleHandler and LocalDirsHandlerService.
[
https://issues.apache.org/jira/browse/YARN-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904766#comment-14904766
]
Hudson commented on YARN-4095:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #427 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/427/])
YARN-4095. Avoid sharing AllocatorPerContext object in LocalDirAllocator
between ShuffleHandler and LocalDirsHandlerService. Contributed by Zhihai Xu
(jlowe: rev c890c51a916894a985439497b8a44e8eee82d762)
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LocalDirsHandlerService.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLocalDirsHandlerService.java
> Avoid sharing AllocatorPerContext object in LocalDirAllocator between
> ShuffleHandler and LocalDirsHandlerService.
> -
>
> Key: YARN-4095
> URL: https://issues.apache.org/jira/browse/YARN-4095
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
>Reporter: zhihai xu
>Assignee: zhihai xu
> Fix For: 2.8.0
>
> Attachments: YARN-4095.000.patch, YARN-4095.001.patch
>
>
> Currently {{ShuffleHandler}} and {{LocalDirsHandlerService}} share
> {{AllocatorPerContext}} object in {{LocalDirAllocator}} for configuration
> {{NM_LOCAL_DIRS}} because {{AllocatorPerContext}} are stored in a static
> TreeMap with configuration name as key
> {code}
> private static Map contexts =
> new TreeMap();
> {code}
> {{LocalDirsHandlerService}} and {{ShuffleHandler}} both create a
> {{LocalDirAllocator}} using {{NM_LOCAL_DIRS}}. Even they don't use the same
> {{Configuration}} object, but they will use the same {{AllocatorPerContext}}
> object. Also {{LocalDirsHandlerService}} may change {{NM_LOCAL_DIRS}} value
> in its {{Configuration}} object to exclude full and bad local dirs,
> {{ShuffleHandler}} always uses the original {{NM_LOCAL_DIRS}} value in its
> {{Configuration}} object. So every time {{AllocatorPerContext#confChanged}}
> is called by {{ShuffleHandler}} after {{LocalDirsHandlerService}},
> {{AllocatorPerContext}} need be reinitialized because {{NM_LOCAL_DIRS}} value
> is changed. This will cause some overhead.
> {code}
> String newLocalDirs = conf.get(contextCfgItemName);
> if (!newLocalDirs.equals(savedLocalDirs)) {
> {code}
> So it will be a good improvement to not share the same
> {{AllocatorPerContext}} instance between {{ShuffleHandler}} and
> {{LocalDirsHandlerService}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4165) An outstanding container request makes all nodes to be reserved causing all jobs pending
[ https://issues.apache.org/jira/browse/YARN-4165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904878#comment-14904878 ] Wangda Tan commented on YARN-4165: -- I think this is not expected behavior, we shouldn't consider one application is starving if it reserved #containers as same as pending #containers. [~cheersyang], could you write a test about this? It will be easier to identify this issue if we have a unit test to prove it. Thanks, > An outstanding container request makes all nodes to be reserved causing all > jobs pending > > > Key: YARN-4165 > URL: https://issues.apache.org/jira/browse/YARN-4165 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler, resourcemanager, scheduler >Affects Versions: 2.7.1 >Reporter: Weiwei Yang >Assignee: Weiwei Yang > > We have a long running service in YARN, it has a outstanding container > request that YARN cannot satisfy (require more memory that nodemanager can > supply). Then YARN reserves all nodes for this application, when I submit > other jobs (require relative small memory that nodemanager can supply), all > jobs are pending because YARN skips scheduling containers on the nodes that > have been reserved. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3813) Support Application timeout feature in YARN.
[ https://issues.apache.org/jira/browse/YARN-3813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904688#comment-14904688 ] nijel commented on YARN-3813: - thanks [~rohithsharma] and [~sunilg] for the comments Updated patch with the the comment fix and test case for recovery. bq. we are starting the monitor thread always regardless whether application demands for applicationtimeout or not. I feel we can have a configuration to enable this feature in RM level. Thoughts? As i pinged you offline, this service will consider only apps which are configured with a timeout. So leaving as a default service. bq.RMAppTimeOutMonitor : When InterruptedException is thrown in the below code, thread should break or throw back exception. So, thread will die else thread wil be alive for ever The while loop is guarded for interrupted state > Support Application timeout feature in YARN. > - > > Key: YARN-3813 > URL: https://issues.apache.org/jira/browse/YARN-3813 > Project: Hadoop YARN > Issue Type: New Feature > Components: scheduler >Reporter: nijel >Assignee: nijel > Attachments: 0001-YARN-3813.patch, YARN Application Timeout .pdf > > > It will be useful to support Application Timeout in YARN. Some use cases are > not worried about the output of the applications if the application is not > completed in a specific time. > *Background:* > The requirement is to show the CDR statistics of last few minutes, say for > every 5 minutes. The same Job will run continuously with different dataset. > So one job will be started in every 5 minutes. The estimate time for this > task is 2 minutes or lesser time. > If the application is not completing in the given time the output is not > useful. > *Proposal* > So idea is to support application timeout, with which timeout parameter is > given while submitting the job. > Here, user is expecting to finish (complete or kill) the application in the > given time. > One option for us is to move this logic to Application client (who submit the > job). > But it will be nice if it can be generic logic and can make more robust. > Kindly provide your suggestions/opinion on this feature. If it sounds good, i > will update the design doc and prototype patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4095) Avoid sharing AllocatorPerContext object in LocalDirAllocator between ShuffleHandler and LocalDirsHandlerService.
[
https://issues.apache.org/jira/browse/YARN-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904704#comment-14904704
]
Hudson commented on YARN-4095:
--
FAILURE: Integrated in Hadoop-trunk-Commit #8503 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/8503/])
YARN-4095. Avoid sharing AllocatorPerContext object in LocalDirAllocator
between ShuffleHandler and LocalDirsHandlerService. Contributed by Zhihai Xu
(jlowe: rev c890c51a916894a985439497b8a44e8eee82d762)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LocalDirsHandlerService.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLocalDirsHandlerService.java
* hadoop-yarn-project/CHANGES.txt
> Avoid sharing AllocatorPerContext object in LocalDirAllocator between
> ShuffleHandler and LocalDirsHandlerService.
> -
>
> Key: YARN-4095
> URL: https://issues.apache.org/jira/browse/YARN-4095
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
>Reporter: zhihai xu
>Assignee: zhihai xu
> Fix For: 2.8.0
>
> Attachments: YARN-4095.000.patch, YARN-4095.001.patch
>
>
> Currently {{ShuffleHandler}} and {{LocalDirsHandlerService}} share
> {{AllocatorPerContext}} object in {{LocalDirAllocator}} for configuration
> {{NM_LOCAL_DIRS}} because {{AllocatorPerContext}} are stored in a static
> TreeMap with configuration name as key
> {code}
> private static Map contexts =
> new TreeMap();
> {code}
> {{LocalDirsHandlerService}} and {{ShuffleHandler}} both create a
> {{LocalDirAllocator}} using {{NM_LOCAL_DIRS}}. Even they don't use the same
> {{Configuration}} object, but they will use the same {{AllocatorPerContext}}
> object. Also {{LocalDirsHandlerService}} may change {{NM_LOCAL_DIRS}} value
> in its {{Configuration}} object to exclude full and bad local dirs,
> {{ShuffleHandler}} always uses the original {{NM_LOCAL_DIRS}} value in its
> {{Configuration}} object. So every time {{AllocatorPerContext#confChanged}}
> is called by {{ShuffleHandler}} after {{LocalDirsHandlerService}},
> {{AllocatorPerContext}} need be reinitialized because {{NM_LOCAL_DIRS}} value
> is changed. This will cause some overhead.
> {code}
> String newLocalDirs = conf.get(contextCfgItemName);
> if (!newLocalDirs.equals(savedLocalDirs)) {
> {code}
> So it will be a good improvement to not share the same
> {{AllocatorPerContext}} instance between {{ShuffleHandler}} and
> {{LocalDirsHandlerService}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4009) CORS support for ResourceManager REST API
[ https://issues.apache.org/jira/browse/YARN-4009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-4009: Attachment: YARN-4009.005.patch Uploaded a new version of the patch that adds additional configs to enable/disable CORS globally. This will allow individual services to enable CORS without it affecting all web services. > CORS support for ResourceManager REST API > - > > Key: YARN-4009 > URL: https://issues.apache.org/jira/browse/YARN-4009 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Prakash Ramachandran >Assignee: Varun Vasudev > Attachments: YARN-4009.001.patch, YARN-4009.002.patch, > YARN-4009.003.patch, YARN-4009.004.patch, YARN-4009.005.patch > > > Currently the REST API's do not have CORS support. This means any UI (running > in browser) cannot consume the REST API's. For ex Tez UI would like to use > the REST API for getting application, application attempt information exposed > by the API's. > It would be very useful if CORS is enabled for the REST API's. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3975) WebAppProxyServlet should not redirect to RM page if AHS is enabled
[
https://issues.apache.org/jira/browse/YARN-3975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904767#comment-14904767
]
Mit Desai commented on YARN-3975:
-
Moreover this is the exact same change form the one that was uploaded
previously and got a +1.
Only difference between the two trunk patches is:
{{@Test//(timeout=5000)}} and {{@Test(timeout=5000)}}
> WebAppProxyServlet should not redirect to RM page if AHS is enabled
> ---
>
> Key: YARN-3975
> URL: https://issues.apache.org/jira/browse/YARN-3975
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.7.1
>Reporter: Mit Desai
>Assignee: Mit Desai
> Attachments: YARN-3975.2.b2.patch, YARN-3975.3.patch,
> YARN-3975.4.patch, YARN-3975.5.patch, YARN-3975.6.patch, YARN-3975.7.patch,
> YARN-3975.8.patch, YARN-3975.9.b2.7.patch, YARN-3975.9.patch,
> YARN-3975.9.patch
>
>
> WebAppProxyServlet should be updated to handle the case when the appreport
> doesn't have a tracking URL and the Application History Server is eanbled.
> As we would have already tried the RM and got the
> ApplicationNotFoundException we should not direct the user to the RM app page.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3975) WebAppProxyServlet should not redirect to RM page if AHS is enabled
[ https://issues.apache.org/jira/browse/YARN-3975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904770#comment-14904770 ] Jason Lowe commented on YARN-3975: -- +1 lgtm. Committing this. > WebAppProxyServlet should not redirect to RM page if AHS is enabled > --- > > Key: YARN-3975 > URL: https://issues.apache.org/jira/browse/YARN-3975 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Mit Desai >Assignee: Mit Desai > Attachments: YARN-3975.2.b2.patch, YARN-3975.3.patch, > YARN-3975.4.patch, YARN-3975.5.patch, YARN-3975.6.patch, YARN-3975.7.patch, > YARN-3975.8.patch, YARN-3975.9.b2.7.patch, YARN-3975.9.patch, > YARN-3975.9.patch > > > WebAppProxyServlet should be updated to handle the case when the appreport > doesn't have a tracking URL and the Application History Server is eanbled. > As we would have already tried the RM and got the > ApplicationNotFoundException we should not direct the user to the RM app page. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4141) Runtime Application Priority change should not throw exception for applications at finishing states
[ https://issues.apache.org/jira/browse/YARN-4141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-4141: -- Attachment: 0006-YARN-4141.patch Thankyou [~jlowe] for sharing the comments. Updated patch addressing the same. > Runtime Application Priority change should not throw exception for > applications at finishing states > --- > > Key: YARN-4141 > URL: https://issues.apache.org/jira/browse/YARN-4141 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Sunil G >Assignee: Sunil G > Attachments: 0001-YARN-4141.patch, 0002-YARN-4141.patch, > 0003-YARN-4141.patch, 0004-YARN-4141.patch, 0005-YARN-4141.patch, > 0006-YARN-4141.patch > > > As suggested by [~jlowe] in > [MAPREDUCE-5870-comment|https://issues.apache.org/jira/browse/MAPREDUCE-5870?focusedCommentId=14737035=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14737035] > , its good that if YARN can suppress exceptions during change application > priority calls for applications at its finishing stages. > Currently it will be difficult for clients to handle this. This will be > similar to kill application behavior. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4075) [reader REST API] implement support for querying for flows and flow runs
[
https://issues.apache.org/jira/browse/YARN-4075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904820#comment-14904820
]
Li Lu commented on YARN-4075:
-
LGTM. Only one question: I'm not an expert in RESTful API implementation so
this may sounds silly. Seems like in some endpoint names we're having the final
slash ('/') but in some others we don't. Are there any special requirements on
this? Thanks!
> [reader REST API] implement support for querying for flows and flow runs
>
>
> Key: YARN-4075
> URL: https://issues.apache.org/jira/browse/YARN-4075
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Varun Saxena
> Attachments: YARN-4075-YARN-2928.03.patch,
> YARN-4075-YARN-2928.POC.1.patch, YARN-4075-YARN-2928.POC.2.patch
>
>
> We need to be able to query for flows and flow runs via REST.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4075) [reader REST API] implement support for querying for flows and flow runs
[ https://issues.apache.org/jira/browse/YARN-4075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904888#comment-14904888 ] Varun Saxena commented on YARN-4075: Ok, will update a patch. > [reader REST API] implement support for querying for flows and flow runs > > > Key: YARN-4075 > URL: https://issues.apache.org/jira/browse/YARN-4075 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Varun Saxena > Attachments: YARN-4075-YARN-2928.03.patch, > YARN-4075-YARN-2928.POC.1.patch, YARN-4075-YARN-2928.POC.2.patch > > > We need to be able to query for flows and flow runs via REST. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3813) Support Application timeout feature in YARN.
[ https://issues.apache.org/jira/browse/YARN-3813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nijel updated YARN-3813: Attachment: 0002_YARN-3813.patch > Support Application timeout feature in YARN. > - > > Key: YARN-3813 > URL: https://issues.apache.org/jira/browse/YARN-3813 > Project: Hadoop YARN > Issue Type: New Feature > Components: scheduler >Reporter: nijel >Assignee: nijel > Attachments: 0001-YARN-3813.patch, 0002_YARN-3813.patch, YARN > Application Timeout .pdf > > > It will be useful to support Application Timeout in YARN. Some use cases are > not worried about the output of the applications if the application is not > completed in a specific time. > *Background:* > The requirement is to show the CDR statistics of last few minutes, say for > every 5 minutes. The same Job will run continuously with different dataset. > So one job will be started in every 5 minutes. The estimate time for this > task is 2 minutes or lesser time. > If the application is not completing in the given time the output is not > useful. > *Proposal* > So idea is to support application timeout, with which timeout parameter is > given while submitting the job. > Here, user is expecting to finish (complete or kill) the application in the > given time. > One option for us is to move this logic to Application client (who submit the > job). > But it will be nice if it can be generic logic and can make more robust. > Kindly provide your suggestions/opinion on this feature. If it sounds good, i > will update the design doc and prototype patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3975) WebAppProxyServlet should not redirect to RM page if AHS is enabled
[ https://issues.apache.org/jira/browse/YARN-3975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904826#comment-14904826 ] Hudson commented on YARN-3975: -- FAILURE: Integrated in Hadoop-trunk-Commit #8504 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8504/]) YARN-3975. WebAppProxyServlet should not redirect to RM page if AHS is enabled. Contributed by Mit Desai (jlowe: rev 692d51c09d3668cde47cc297296d095ddfa933a3) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestWebAppProxyServlet.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/AppReportFetcher.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java * hadoop-yarn-project/CHANGES.txt > WebAppProxyServlet should not redirect to RM page if AHS is enabled > --- > > Key: YARN-3975 > URL: https://issues.apache.org/jira/browse/YARN-3975 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Mit Desai >Assignee: Mit Desai > Fix For: 2.7.2 > > Attachments: YARN-3975.2.b2.patch, YARN-3975.3.patch, > YARN-3975.4.patch, YARN-3975.5.patch, YARN-3975.6.patch, YARN-3975.7.patch, > YARN-3975.8.patch, YARN-3975.9.b2.7.patch, YARN-3975.9.patch, > YARN-3975.9.patch > > > WebAppProxyServlet should be updated to handle the case when the appreport > doesn't have a tracking URL and the Application History Server is eanbled. > As we would have already tried the RM and got the > ApplicationNotFoundException we should not direct the user to the RM app page. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4075) [reader REST API] implement support for querying for flows and flow runs
[ https://issues.apache.org/jira/browse/YARN-4075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904866#comment-14904866 ] Varun Saxena commented on YARN-4075: I am not an expert either. Infact this has not been done knowingly. Actually URLs' ending in a slash and not ending with them are treated the same. Refer to Section 2.1.1.of JAX-RS documentation below. https://jersey.java.net/documentation/1.19/jax-rs.html It states: "A @Path value may or may not begin with a '/', it makes no difference. Likewise, by default, a @Path value may or may not end in a '/', it makes no difference, and thus request URLs that end or do not end in a '/' will both be matched." > [reader REST API] implement support for querying for flows and flow runs > > > Key: YARN-4075 > URL: https://issues.apache.org/jira/browse/YARN-4075 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Varun Saxena > Attachments: YARN-4075-YARN-2928.03.patch, > YARN-4075-YARN-2928.POC.1.patch, YARN-4075-YARN-2928.POC.2.patch > > > We need to be able to query for flows and flow runs via REST. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4075) [reader REST API] implement support for querying for flows and flow runs
[ https://issues.apache.org/jira/browse/YARN-4075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904877#comment-14904877 ] Li Lu commented on YARN-4075: - Thanks for the reference [~varun_saxena]. Maybe we'd like to keep them consistent in this sense? +1 after this clean up. > [reader REST API] implement support for querying for flows and flow runs > > > Key: YARN-4075 > URL: https://issues.apache.org/jira/browse/YARN-4075 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Varun Saxena > Attachments: YARN-4075-YARN-2928.03.patch, > YARN-4075-YARN-2928.POC.1.patch, YARN-4075-YARN-2928.POC.2.patch > > > We need to be able to query for flows and flow runs via REST. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4095) Avoid sharing AllocatorPerContext object in LocalDirAllocator between ShuffleHandler and LocalDirsHandlerService.
[
https://issues.apache.org/jira/browse/YARN-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904872#comment-14904872
]
Hudson commented on YARN-4095:
--
SUCCESS: Integrated in Hadoop-Yarn-trunk #1167 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/1167/])
YARN-4095. Avoid sharing AllocatorPerContext object in LocalDirAllocator
between ShuffleHandler and LocalDirsHandlerService. Contributed by Zhihai Xu
(jlowe: rev c890c51a916894a985439497b8a44e8eee82d762)
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LocalDirsHandlerService.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLocalDirsHandlerService.java
> Avoid sharing AllocatorPerContext object in LocalDirAllocator between
> ShuffleHandler and LocalDirsHandlerService.
> -
>
> Key: YARN-4095
> URL: https://issues.apache.org/jira/browse/YARN-4095
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
>Reporter: zhihai xu
>Assignee: zhihai xu
> Fix For: 2.8.0
>
> Attachments: YARN-4095.000.patch, YARN-4095.001.patch
>
>
> Currently {{ShuffleHandler}} and {{LocalDirsHandlerService}} share
> {{AllocatorPerContext}} object in {{LocalDirAllocator}} for configuration
> {{NM_LOCAL_DIRS}} because {{AllocatorPerContext}} are stored in a static
> TreeMap with configuration name as key
> {code}
> private static Map contexts =
> new TreeMap();
> {code}
> {{LocalDirsHandlerService}} and {{ShuffleHandler}} both create a
> {{LocalDirAllocator}} using {{NM_LOCAL_DIRS}}. Even they don't use the same
> {{Configuration}} object, but they will use the same {{AllocatorPerContext}}
> object. Also {{LocalDirsHandlerService}} may change {{NM_LOCAL_DIRS}} value
> in its {{Configuration}} object to exclude full and bad local dirs,
> {{ShuffleHandler}} always uses the original {{NM_LOCAL_DIRS}} value in its
> {{Configuration}} object. So every time {{AllocatorPerContext#confChanged}}
> is called by {{ShuffleHandler}} after {{LocalDirsHandlerService}},
> {{AllocatorPerContext}} need be reinitialized because {{NM_LOCAL_DIRS}} value
> is changed. This will cause some overhead.
> {code}
> String newLocalDirs = conf.get(contextCfgItemName);
> if (!newLocalDirs.equals(savedLocalDirs)) {
> {code}
> So it will be a good improvement to not share the same
> {{AllocatorPerContext}} instance between {{ShuffleHandler}} and
> {{LocalDirsHandlerService}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3975) WebAppProxyServlet should not redirect to RM page if AHS is enabled
[
https://issues.apache.org/jira/browse/YARN-3975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904726#comment-14904726
]
Hadoop QA commented on YARN-3975:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 16m 1s | Findbugs (version 3.0.0)
appears to be broken on trunk. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 48s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 24s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 39s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 41s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 38s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:red}-1{color} | yarn tests | 6m 59s | Tests failed in
hadoop-yarn-client. |
| {color:green}+1{color} | yarn tests | 0m 22s | Tests passed in
hadoop-yarn-server-web-proxy. |
| | | 46m 33s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.yarn.client.api.impl.TestYarnClient |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12761904/YARN-3975.9.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / a2c76e5 |
| hadoop-yarn-client test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9243/artifact/patchprocess/testrun_hadoop-yarn-client.txt
|
| hadoop-yarn-server-web-proxy test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9243/artifact/patchprocess/testrun_hadoop-yarn-server-web-proxy.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/9243/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/9243/console |
This message was automatically generated.
> WebAppProxyServlet should not redirect to RM page if AHS is enabled
> ---
>
> Key: YARN-3975
> URL: https://issues.apache.org/jira/browse/YARN-3975
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.7.1
>Reporter: Mit Desai
>Assignee: Mit Desai
> Attachments: YARN-3975.2.b2.patch, YARN-3975.3.patch,
> YARN-3975.4.patch, YARN-3975.5.patch, YARN-3975.6.patch, YARN-3975.7.patch,
> YARN-3975.8.patch, YARN-3975.9.b2.7.patch, YARN-3975.9.patch,
> YARN-3975.9.patch
>
>
> WebAppProxyServlet should be updated to handle the case when the appreport
> doesn't have a tracking URL and the Application History Server is eanbled.
> As we would have already tried the RM and got the
> ApplicationNotFoundException we should not direct the user to the RM app page.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4011) Jobs fail since nm-local-dir not cleaned up when rogue job fills up disk
[ https://issues.apache.org/jira/browse/YARN-4011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904697#comment-14904697 ] Maysam Yabandeh commented on YARN-4011: --- Thanks [~jlowe]. I created MAPREDUCE-6489 for this. > Jobs fail since nm-local-dir not cleaned up when rogue job fills up disk > > > Key: YARN-4011 > URL: https://issues.apache.org/jira/browse/YARN-4011 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Affects Versions: 2.4.0 >Reporter: Ashwin Shankar > > We observed jobs failed since tasks couldn't launch on nodes due to > "java.io.IOException No space left on device". > On digging in further, we found a rogue job which filled up disk. > Specifically it was wrote a lot of map spills(like > attempt_1432082376223_461647_m_000421_0_spill_1.out) to nm-local-dir > causing disk to fill up, and it failed/got killed, but didn't clean up these > files in nm-local-dir. > So the disk remained full, causing subsequent jobs to fail. > This jira is created to address why files under nm-local-dir doesn't get > cleaned up when job fails after filling up disk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3975) WebAppProxyServlet should not redirect to RM page if AHS is enabled
[ https://issues.apache.org/jira/browse/YARN-3975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904899#comment-14904899 ] Hudson commented on YARN-3975: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #428 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/428/]) YARN-3975. WebAppProxyServlet should not redirect to RM page if AHS is enabled. Contributed by Mit Desai (jlowe: rev 692d51c09d3668cde47cc297296d095ddfa933a3) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/AppReportFetcher.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestWebAppProxyServlet.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java > WebAppProxyServlet should not redirect to RM page if AHS is enabled > --- > > Key: YARN-3975 > URL: https://issues.apache.org/jira/browse/YARN-3975 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Mit Desai >Assignee: Mit Desai > Fix For: 2.7.2 > > Attachments: YARN-3975.2.b2.patch, YARN-3975.3.patch, > YARN-3975.4.patch, YARN-3975.5.patch, YARN-3975.6.patch, YARN-3975.7.patch, > YARN-3975.8.patch, YARN-3975.9.b2.7.patch, YARN-3975.9.patch, > YARN-3975.9.patch > > > WebAppProxyServlet should be updated to handle the case when the appreport > doesn't have a tracking URL and the Application History Server is eanbled. > As we would have already tried the RM and got the > ApplicationNotFoundException we should not direct the user to the RM app page. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3603) Application Attempts page confusing
[
https://issues.apache.org/jira/browse/YARN-3603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904945#comment-14904945
]
Sunil G commented on YARN-3603:
---
Hi [~jeffzhang]
Thank you for sharing the thoughts. I could remember an old JIRA YARN-1809,
where UI is made similar for RM and HistoryServer.
Here we use {{getContainers)), where History/Timeline fills containers info
from ApplicationContext (has all container info). But RM was taking from
RMContext where only live containers are taken. I understood the need of killed
containers here. it can help in debugging. However we have to segregate
killed/preempted containers from completed containers.
Hi [~leftnoteasy]
I also feel that we can have the information about killed/preempted container
along with running containers in app attempt UI. In YARN-1809, we added for
only running containers. Could you pls share your thoughts here.
> Application Attempts page confusing
> ---
>
> Key: YARN-3603
> URL: https://issues.apache.org/jira/browse/YARN-3603
> Project: Hadoop YARN
> Issue Type: Bug
> Components: webapp
>Affects Versions: 2.8.0
>Reporter: Thomas Graves
>Assignee: Sunil G
> Attachments: 0001-YARN-3603.patch, 0002-YARN-3603.patch,
> 0003-YARN-3603.patch, ahs1.png
>
>
> The application attempts page
> (http://RM:8088/cluster/appattempt/appattempt_1431101480046_0003_01)
> is a bit confusing on what is going on. I think the table of containers
> there is for only Running containers and when the app is completed or killed
> its empty. The table should have a label on it stating so.
> Also the "AM Container" field is a link when running but not when its killed.
> That might be confusing.
> There is no link to the logs in this page but there is in the app attempt
> table when looking at http://
> rm:8088/cluster/app/application_1431101480046_0003
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Reopened] (YARN-3976) Catch ApplicationNotFoundException instead of parent YarnException in YarnClient and AppReportFetcher
[
https://issues.apache.org/jira/browse/YARN-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe reopened YARN-3976:
--
> Catch ApplicationNotFoundException instead of parent YarnException in
> YarnClient and AppReportFetcher
> -
>
> Key: YARN-3976
> URL: https://issues.apache.org/jira/browse/YARN-3976
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.7.1
>Reporter: Mit Desai
>Assignee: Mit Desai
>Priority: Trivial
> Fix For: 2.7.2
>
>
> It's is better to catch the ApplicationNotFoundException rather than the
> parent YarnException and rethrow it when it's not ApplicationNotFoundExcepton
> {noformat}
> catch (YarnException e) {
> if (!historyServiceEnabled) {
> // Just throw it as usual if historyService is not enabled.
> throw e;
> }
> // Even if history-service is enabled, treat all exceptions still the
> same
> // except the following
> if (!(e.getClass() == ApplicationNotFoundException.class)) {
> throw e;
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4095) Avoid sharing AllocatorPerContext object in LocalDirAllocator between ShuffleHandler and LocalDirsHandlerService.
[
https://issues.apache.org/jira/browse/YARN-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905003#comment-14905003
]
Hudson commented on YARN-4095:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #2346 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2346/])
YARN-4095. Avoid sharing AllocatorPerContext object in LocalDirAllocator
between ShuffleHandler and LocalDirsHandlerService. Contributed by Zhihai Xu
(jlowe: rev c890c51a916894a985439497b8a44e8eee82d762)
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLocalDirsHandlerService.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LocalDirsHandlerService.java
> Avoid sharing AllocatorPerContext object in LocalDirAllocator between
> ShuffleHandler and LocalDirsHandlerService.
> -
>
> Key: YARN-4095
> URL: https://issues.apache.org/jira/browse/YARN-4095
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
>Reporter: zhihai xu
>Assignee: zhihai xu
> Fix For: 2.8.0
>
> Attachments: YARN-4095.000.patch, YARN-4095.001.patch
>
>
> Currently {{ShuffleHandler}} and {{LocalDirsHandlerService}} share
> {{AllocatorPerContext}} object in {{LocalDirAllocator}} for configuration
> {{NM_LOCAL_DIRS}} because {{AllocatorPerContext}} are stored in a static
> TreeMap with configuration name as key
> {code}
> private static Map contexts =
> new TreeMap();
> {code}
> {{LocalDirsHandlerService}} and {{ShuffleHandler}} both create a
> {{LocalDirAllocator}} using {{NM_LOCAL_DIRS}}. Even they don't use the same
> {{Configuration}} object, but they will use the same {{AllocatorPerContext}}
> object. Also {{LocalDirsHandlerService}} may change {{NM_LOCAL_DIRS}} value
> in its {{Configuration}} object to exclude full and bad local dirs,
> {{ShuffleHandler}} always uses the original {{NM_LOCAL_DIRS}} value in its
> {{Configuration}} object. So every time {{AllocatorPerContext#confChanged}}
> is called by {{ShuffleHandler}} after {{LocalDirsHandlerService}},
> {{AllocatorPerContext}} need be reinitialized because {{NM_LOCAL_DIRS}} value
> is changed. This will cause some overhead.
> {code}
> String newLocalDirs = conf.get(contextCfgItemName);
> if (!newLocalDirs.equals(savedLocalDirs)) {
> {code}
> So it will be a good improvement to not share the same
> {{AllocatorPerContext}} instance between {{ShuffleHandler}} and
> {{LocalDirsHandlerService}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4095) Avoid sharing AllocatorPerContext object in LocalDirAllocator between ShuffleHandler and LocalDirsHandlerService.
[
https://issues.apache.org/jira/browse/YARN-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905044#comment-14905044
]
Hudson commented on YARN-4095:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2373 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2373/])
YARN-4095. Avoid sharing AllocatorPerContext object in LocalDirAllocator
between ShuffleHandler and LocalDirsHandlerService. Contributed by Zhihai Xu
(jlowe: rev c890c51a916894a985439497b8a44e8eee82d762)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LocalDirsHandlerService.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLocalDirsHandlerService.java
* hadoop-yarn-project/CHANGES.txt
> Avoid sharing AllocatorPerContext object in LocalDirAllocator between
> ShuffleHandler and LocalDirsHandlerService.
> -
>
> Key: YARN-4095
> URL: https://issues.apache.org/jira/browse/YARN-4095
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
>Reporter: zhihai xu
>Assignee: zhihai xu
> Fix For: 2.8.0
>
> Attachments: YARN-4095.000.patch, YARN-4095.001.patch
>
>
> Currently {{ShuffleHandler}} and {{LocalDirsHandlerService}} share
> {{AllocatorPerContext}} object in {{LocalDirAllocator}} for configuration
> {{NM_LOCAL_DIRS}} because {{AllocatorPerContext}} are stored in a static
> TreeMap with configuration name as key
> {code}
> private static Map contexts =
> new TreeMap();
> {code}
> {{LocalDirsHandlerService}} and {{ShuffleHandler}} both create a
> {{LocalDirAllocator}} using {{NM_LOCAL_DIRS}}. Even they don't use the same
> {{Configuration}} object, but they will use the same {{AllocatorPerContext}}
> object. Also {{LocalDirsHandlerService}} may change {{NM_LOCAL_DIRS}} value
> in its {{Configuration}} object to exclude full and bad local dirs,
> {{ShuffleHandler}} always uses the original {{NM_LOCAL_DIRS}} value in its
> {{Configuration}} object. So every time {{AllocatorPerContext#confChanged}}
> is called by {{ShuffleHandler}} after {{LocalDirsHandlerService}},
> {{AllocatorPerContext}} need be reinitialized because {{NM_LOCAL_DIRS}} value
> is changed. This will cause some overhead.
> {code}
> String newLocalDirs = conf.get(contextCfgItemName);
> if (!newLocalDirs.equals(savedLocalDirs)) {
> {code}
> So it will be a good improvement to not share the same
> {{AllocatorPerContext}} instance between {{ShuffleHandler}} and
> {{LocalDirsHandlerService}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4192) Add YARN metric logging periodically to a seperate file
[ https://issues.apache.org/jira/browse/YARN-4192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904937#comment-14904937 ] Allen Wittenauer commented on YARN-4192: bq. Any thoughts ? The file sink for metrics2 almost certainly meets the needs here as well as what was added in HDFS-8880. > Add YARN metric logging periodically to a seperate file > --- > > Key: YARN-4192 > URL: https://issues.apache.org/jira/browse/YARN-4192 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: nijel >Assignee: nijel >Priority: Minor > > HDFS-8880 added a framework for logging metrics in a given interval. > This can be added to YARN as well > Any thoughts ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (YARN-3976) Catch ApplicationNotFoundException instead of parent YarnException in YarnClient and AppReportFetcher
[
https://issues.apache.org/jira/browse/YARN-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe resolved YARN-3976.
--
Resolution: Duplicate
Fixing resolution since this was implemented by YARN-3975.
> Catch ApplicationNotFoundException instead of parent YarnException in
> YarnClient and AppReportFetcher
> -
>
> Key: YARN-3976
> URL: https://issues.apache.org/jira/browse/YARN-3976
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.7.1
>Reporter: Mit Desai
>Assignee: Mit Desai
>Priority: Trivial
> Fix For: 2.7.2
>
>
> It's is better to catch the ApplicationNotFoundException rather than the
> parent YarnException and rethrow it when it's not ApplicationNotFoundExcepton
> {noformat}
> catch (YarnException e) {
> if (!historyServiceEnabled) {
> // Just throw it as usual if historyService is not enabled.
> throw e;
> }
> // Even if history-service is enabled, treat all exceptions still the
> same
> // except the following
> if (!(e.getClass() == ApplicationNotFoundException.class)) {
> throw e;
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4140) RM container allocation delayed incase of app submitted to Nodelabel partition
[
https://issues.apache.org/jira/browse/YARN-4140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905012#comment-14905012
]
Naganarasimha G R commented on YARN-4140:
-
thanks for the patch [~bibinchundatt], and for the approach by [~wangda],
But i had few queries :
# bq. Previous we don't have ANY request, this may not happen, but we can
protect this corner case.
Can this scenario be possible ? IIRC, MR its not possible but may be possible
in other apps ?
If so is it req to handle {{incPendingResourcesForLabel}} and
{{decPendingResourceForLabel}} for such requests currently we are doing for
only {{previousAnyRequest}} only .
# I think the modifications for updates would not be required when
{{recoverPreemptedRequest}} is true right ?
Small nit, not exactly related to the patch, method parameter {{requests}}
could be renamed as {{updatedRequests}} and the local variable in the for loop
as {{updatedRequest}}, so that its more readable as currently a member with the
same name {{requests}}
> RM container allocation delayed incase of app submitted to Nodelabel partition
> --
>
> Key: YARN-4140
> URL: https://issues.apache.org/jira/browse/YARN-4140
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api, client, resourcemanager
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: 0001-YARN-4140.patch, 0002-YARN-4140.patch,
> 0003-YARN-4140.patch, 0004-YARN-4140.patch, 0005-YARN-4140.patch,
> 0006-YARN-4140.patch, 0007-YARN-4140.patch, 0008-YARN-4140.patch,
> 0009-YARN-4140.patch
>
>
> Trying to run application on Nodelabel partition I found that the
> application execution time is delayed by 5 – 10 min for 500 containers .
> Total 3 machines 2 machines were in same partition and app submitted to same.
> After enabling debug was able to find the below
> # From AM the container ask is for OFF-SWITCH
> # RM allocating all containers to NODE_LOCAL as shown in logs below.
> # So since I was having about 500 containers time taken was about – 6 minutes
> to allocate 1st map after AM allocation.
> # Tested with about 1K maps using PI job took 17 minutes to allocate next
> container after AM allocation
> Once 500 container allocation on NODE_LOCAL is done the next container
> allocation is done on OFF_SWITCH
> {code}
> 2015-09-09 15:21:58,954 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt:
> showRequests: application=application_1441791998224_0001 request={Priority:
> 20, Capability: , # Containers: 500, Location:
> /default-rack, Relax Locality: true, Node Label Expression: }
> 2015-09-09 15:21:58,954 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt:
> showRequests: application=application_1441791998224_0001 request={Priority:
> 20, Capability: , # Containers: 500, Location: *, Relax
> Locality: true, Node Label Expression: 3}
> 2015-09-09 15:21:58,954 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt:
> showRequests: application=application_1441791998224_0001 request={Priority:
> 20, Capability: , # Containers: 500, Location:
> host-10-19-92-143, Relax Locality: true, Node Label Expression: }
> 2015-09-09 15:21:58,954 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt:
> showRequests: application=application_1441791998224_0001 request={Priority:
> 20, Capability: , # Containers: 500, Location:
> host-10-19-92-117, Relax Locality: true, Node Label Expression: }
> 2015-09-09 15:21:58,954 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
> Assigned to queue: root.b.b1 stats: b1: capacity=1.0, absoluteCapacity=0.5,
> usedResources=, usedCapacity=0.0,
> absoluteUsedCapacity=0.0, numApps=1, numContainers=1 --> vCores:0>, NODE_LOCAL
> {code}
>
> {code}
> 2015-09-09 14:35:45,467 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
> Assigned to queue: root.b.b1 stats: b1: capacity=1.0, absoluteCapacity=0.5,
> usedResources=, usedCapacity=0.0,
> absoluteUsedCapacity=0.0, numApps=1, numContainers=1 --> vCores:0>, NODE_LOCAL
> 2015-09-09 14:35:45,831 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
> Assigned to queue: root.b.b1 stats: b1: capacity=1.0, absoluteCapacity=0.5,
> usedResources=, usedCapacity=0.0,
> absoluteUsedCapacity=0.0, numApps=1, numContainers=1 --> vCores:0>, NODE_LOCAL
> 2015-09-09 14:35:46,469 DEBUG
>
[jira] [Commented] (YARN-4141) Runtime Application Priority change should not throw exception for applications at finishing states
[
https://issues.apache.org/jira/browse/YARN-4141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905089#comment-14905089
]
Hadoop QA commented on YARN-4141:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 6s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 8m 4s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 20s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 50s | The applied patch generated 2
new checkstyle issues (total was 33, now 35). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 28s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 29s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 55m 14s | Tests passed in
hadoop-yarn-server-resourcemanager. |
| | | 95m 34s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12761940/0006-YARN-4141.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 692d51c |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/9246/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9246/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/9246/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/9246/console |
This message was automatically generated.
> Runtime Application Priority change should not throw exception for
> applications at finishing states
> ---
>
> Key: YARN-4141
> URL: https://issues.apache.org/jira/browse/YARN-4141
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: 0001-YARN-4141.patch, 0002-YARN-4141.patch,
> 0003-YARN-4141.patch, 0004-YARN-4141.patch, 0005-YARN-4141.patch,
> 0006-YARN-4141.patch
>
>
> As suggested by [~jlowe] in
> [MAPREDUCE-5870-comment|https://issues.apache.org/jira/browse/MAPREDUCE-5870?focusedCommentId=14737035=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14737035]
> , its good that if YARN can suppress exceptions during change application
> priority calls for applications at its finishing stages.
> Currently it will be difficult for clients to handle this. This will be
> similar to kill application behavior.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4075) [reader REST API] implement support for querying for flows and flow runs
[ https://issues.apache.org/jira/browse/YARN-4075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-4075: --- Attachment: YARN-4075-YARN-2928.04.patch > [reader REST API] implement support for querying for flows and flow runs > > > Key: YARN-4075 > URL: https://issues.apache.org/jira/browse/YARN-4075 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Varun Saxena > Attachments: YARN-4075-YARN-2928.03.patch, > YARN-4075-YARN-2928.04.patch, YARN-4075-YARN-2928.POC.1.patch, > YARN-4075-YARN-2928.POC.2.patch > > > We need to be able to query for flows and flow runs via REST. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3976) Catch ApplicationNotFoundException instead of parent YarnException in YarnClient and AppReportFetcher
[
https://issues.apache.org/jira/browse/YARN-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904940#comment-14904940
]
Mit Desai commented on YARN-3976:
-
[~jlowe], Sorry did not see your comment to move this jira out of 2.7.2 until
now.
> Catch ApplicationNotFoundException instead of parent YarnException in
> YarnClient and AppReportFetcher
> -
>
> Key: YARN-3976
> URL: https://issues.apache.org/jira/browse/YARN-3976
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.7.1
>Reporter: Mit Desai
>Assignee: Mit Desai
>Priority: Trivial
> Fix For: 2.7.2
>
>
> It's is better to catch the ApplicationNotFoundException rather than the
> parent YarnException and rethrow it when it's not ApplicationNotFoundExcepton
> {noformat}
> catch (YarnException e) {
> if (!historyServiceEnabled) {
> // Just throw it as usual if historyService is not enabled.
> throw e;
> }
> // Even if history-service is enabled, treat all exceptions still the
> same
> // except the following
> if (!(e.getClass() == ApplicationNotFoundException.class)) {
> throw e;
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4009) CORS support for ResourceManager REST API
[ https://issues.apache.org/jira/browse/YARN-4009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905075#comment-14905075 ] Varun Vasudev commented on YARN-4009: - The test failures are unrelated to the patch. The tests pass on my machine - it looks like some other process took the ports and the http servers couldn't start. > CORS support for ResourceManager REST API > - > > Key: YARN-4009 > URL: https://issues.apache.org/jira/browse/YARN-4009 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Prakash Ramachandran >Assignee: Varun Vasudev > Attachments: YARN-4009.001.patch, YARN-4009.002.patch, > YARN-4009.003.patch, YARN-4009.004.patch, YARN-4009.005.patch > > > Currently the REST API's do not have CORS support. This means any UI (running > in browser) cannot consume the REST API's. For ex Tez UI would like to use > the REST API for getting application, application attempt information exposed > by the API's. > It would be very useful if CORS is enabled for the REST API's. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4165) An outstanding container request makes all nodes to be reserved causing all jobs pending
[ https://issues.apache.org/jira/browse/YARN-4165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904028#comment-14904028 ] Weiwei Yang commented on YARN-4165: --- Hello Jason Thanks for looking this. I checked YARN 957 but I think this is a different problem. I have 3 nodes NM1 8G NM2 8G NM3 8G I submitted an application, requires 4 containers and each of them relative big memory like 5G, its app master requires 1G, so RM fills 3 containers and 1 app master but leaving 1 outstanding request, *unexpectedly* RM reserved 1 container on all 3 nodes like NM1 - 1 container, 1 app master - 6G used - 2G left - 5G reserved NM2 - 1 container - 5G used - 3G left - 5G reserved NM3 - 1 container - 5G used - 3G left - 5G reserved I am not sure yet why we run into such situation, but it might be related to YARN-1769, I am still investigating, if you have any pointers or comments, please let me know. Thanks. > An outstanding container request makes all nodes to be reserved causing all > jobs pending > > > Key: YARN-4165 > URL: https://issues.apache.org/jira/browse/YARN-4165 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler, resourcemanager, scheduler >Affects Versions: 2.7.1 >Reporter: Weiwei Yang >Assignee: Weiwei Yang > > We have a long running service in YARN, it has a outstanding container > request that YARN cannot satisfy (require more memory that nodemanager can > supply). Then YARN reserves all nodes for this application, when I submit > other jobs (require relative small memory that nodemanager can supply), all > jobs are pending because YARN skips scheduling containers on the nodes that > have been reserved. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4165) An outstanding container request makes all nodes to be reserved causing all jobs pending
[ https://issues.apache.org/jira/browse/YARN-4165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904033#comment-14904033 ] Weiwei Yang commented on YARN-4165: --- Thanks Jason, it doesn't look like the YARN-957, the reserved memory did less than node manager reports. > An outstanding container request makes all nodes to be reserved causing all > jobs pending > > > Key: YARN-4165 > URL: https://issues.apache.org/jira/browse/YARN-4165 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler, resourcemanager, scheduler >Affects Versions: 2.7.1 >Reporter: Weiwei Yang >Assignee: Weiwei Yang > > We have a long running service in YARN, it has a outstanding container > request that YARN cannot satisfy (require more memory that nodemanager can > supply). Then YARN reserves all nodes for this application, when I submit > other jobs (require relative small memory that nodemanager can supply), all > jobs are pending because YARN skips scheduling containers on the nodes that > have been reserved. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4176) Resync NM nodelabels with RM every x interval for distributed nodelabels
[
https://issues.apache.org/jira/browse/YARN-4176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904169#comment-14904169
]
Naganarasimha G R commented on YARN-4176:
-
Thanks [~bibinchundatt] for working on this issue, few small nits remaining :
# {{nodeLabelsProvider}} and {{resyncInterval}} can be final
# {{lastNodeLabelSendMills}} needs to be set even in case validation fails this
will avoid invalid labels to be validated for every heartbeat
# In TestNodeStatusUpdaterForLabels,
{{dummyLabelsProviderRef.setNodeLabels(toNodeLabelSet("P1"));}} can be outside
the for loop and {{assertNull("Null resourcelabels are expected",
resourceTracker.labels);}} is not required in side the if check.
# Modify {{updatedLabelsSentToRM}} -> areLabelsSentToRM
# instead of logging on every resync in
{{verifyRMHeartbeatResponseForNodeLabels}} we can either make it debug log or
based on labels update/RM reject we can log. I prefer for the former approach.
Apart from this every thing else is fine and have tested the patch in the
cluster
[~wangda] we have kept the sync interval as 2 mins so that when cluster
nodelabels are added, NM HB can update latest labels fast, please share weather
we need to increase this sync duration to a greater value like 10 mins.
> Resync NM nodelabels with RM every x interval for distributed nodelabels
>
>
> Key: YARN-4176
> URL: https://issues.apache.org/jira/browse/YARN-4176
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: 0001-YARN-4176.patch, 0002-YARN-4176.patch,
> 0003-YARN-4176.patch, 0004-YARN-4176.patch
>
>
> This JIRA is for handling the below set of issue
> # Distributed nodelabels after NM registered with RM if cluster nodelabels
> are removed and added then NM doesnt resend labels in heartbeat again untils
> any change in labels
> # NM registration failed with Nodelabels should resend labels again to RM
> The above cases can be handled by resync nodeLabels with RM every x interval
> # Add property {{yarn.nodemanager.node-labels.provider.resync-interval-ms}}
> and will resend nodelabels to RM based on config no matter what the
> registration fails or success.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4169) jenkins trunk+java build failed in TestNodeStatusUpdaterForLabels
[
https://issues.apache.org/jira/browse/YARN-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904178#comment-14904178
]
Steve Loughran commented on YARN-4169:
--
I'm not the person to review YARN patches, I'm afraid.
Looking at the tests briefly
# you can use Java 7 <> tags in constructors; the old code was for java 6 and
can be upgraded on the changed lines.
# no need to catch InterruptedExceptions and {{AssertFail()}}. Simply have the
test declare it throws {{InterruptedException}} and JUnit will do the rest.
> jenkins trunk+java build failed in TestNodeStatusUpdaterForLabels
> -
>
> Key: YARN-4169
> URL: https://issues.apache.org/jira/browse/YARN-4169
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0
> Environment: Jenkins
>Reporter: Steve Loughran
>Assignee: Naganarasimha G R
>Priority: Critical
> Attachments: YARN-4169.v1.001.patch
>
>
> Test failing in [[Jenkins build
> 402|https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Yarn-trunk-Java8/402/testReport/junit/org.apache.hadoop.yarn.server.nodemanager/TestNodeStatusUpdaterForLabels/testNodeStatusUpdaterForNodeLabels/]
> {code}
> java.lang.NullPointerException: null
> at java.util.HashSet.(HashSet.java:118)
> at
> org.apache.hadoop.yarn.nodelabels.NodeLabelTestBase.assertNLCollectionEquals(NodeLabelTestBase.java:103)
> at
> org.apache.hadoop.yarn.server.nodemanager.TestNodeStatusUpdaterForLabels.testNodeStatusUpdaterForNodeLabels(TestNodeStatusUpdaterForLabels.java:268)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4169) jenkins trunk+java build failed in TestNodeStatusUpdaterForLabels
[
https://issues.apache.org/jira/browse/YARN-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904222#comment-14904222
]
Naganarasimha G R commented on YARN-4169:
-
Thanks for the comments [~ste...@apache.org], Will try to get it reviewed from
either [~wangda] / [~rohithsharma] for YARN specific modifications.
But i dint get which part of modifications(class and line number) were you
referring to for the first comment !
> jenkins trunk+java build failed in TestNodeStatusUpdaterForLabels
> -
>
> Key: YARN-4169
> URL: https://issues.apache.org/jira/browse/YARN-4169
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0
> Environment: Jenkins
>Reporter: Steve Loughran
>Assignee: Naganarasimha G R
>Priority: Critical
> Attachments: YARN-4169.v1.001.patch
>
>
> Test failing in [[Jenkins build
> 402|https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Yarn-trunk-Java8/402/testReport/junit/org.apache.hadoop.yarn.server.nodemanager/TestNodeStatusUpdaterForLabels/testNodeStatusUpdaterForNodeLabels/]
> {code}
> java.lang.NullPointerException: null
> at java.util.HashSet.(HashSet.java:118)
> at
> org.apache.hadoop.yarn.nodelabels.NodeLabelTestBase.assertNLCollectionEquals(NodeLabelTestBase.java:103)
> at
> org.apache.hadoop.yarn.server.nodemanager.TestNodeStatusUpdaterForLabels.testNodeStatusUpdaterForNodeLabels(TestNodeStatusUpdaterForLabels.java:268)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3866) AM-RM protocol changes to support container resizing
[ https://issues.apache.org/jira/browse/YARN-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905316#comment-14905316 ] Hudson commented on YARN-3866: -- FAILURE: Integrated in Hadoop-trunk-Commit #8505 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8505/]) YARN-3866. AM-RM protocol changes to support container resizing. Contributed by Meng Ding (wangda: rev dfe2cb849fcb273d32e840d8562eaaee904af054) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestAllocateResponse.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestPBImplRecords.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_service_protos.proto * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ContainerResourceIncreasePBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/AllocateRequestPBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/AllocateResponsePBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ContainerStatusPBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ContainerResourceChangeRequestPBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ContainerResourceIncreaseRequestPBImpl.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/local/TestLocalContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ContainerResourceIncrease.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/AllocateResponse.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ContainerResourceIncreaseRequest.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ContainerResourceDecrease.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ContainerResourceDecreasePBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ContainerResourceChangeRequest.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestContainerResourceDecrease.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestContainerResourceIncrease.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ContainerStatus.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/AllocateRequest.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestContainerResourceIncreaseRequest.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestAllocateRequest.java > AM-RM protocol changes to support container resizing > > > Key: YARN-3866 > URL: https://issues.apache.org/jira/browse/YARN-3866 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api >Reporter: MENG DING >Assignee: MENG DING > Fix For: 3.0.0 > > Attachments: YARN-3866-YARN-1197.4.patch, YARN-3866.1.patch, > YARN-3866.2.patch, YARN-3866.3.patch > > > YARN-1447 and YARN-1448 are outdated. > This ticket deals with AM-RM Protocol changes to support container resize > according to the latest design in YARN-1197. > 1) Add increase/decrease requests in AllocateRequest > 2) Get approved increase/decrease requests from RM in AllocateResponse > 3) Add relevant test cases -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3942) Timeline store to read events from HDFS
[ https://issues.apache.org/jira/browse/YARN-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905398#comment-14905398 ] Xuan Gong commented on YARN-3942: - The patch while implements LevelDBCacheTimelineStore as cache used in EntityFileTimelineStore looks fine. [~jlowe] [~jeagles] Could you take a look, please ? > Timeline store to read events from HDFS > --- > > Key: YARN-3942 > URL: https://issues.apache.org/jira/browse/YARN-3942 > Project: Hadoop YARN > Issue Type: Improvement > Components: timelineserver >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: YARN-3942-leveldb.001.patch, YARN-3942.001.patch > > > This adds a new timeline store plugin that is intended as a stop-gap measure > to mitigate some of the issues we've seen with ATS v1 while waiting for ATS > v2. The intent of this plugin is to provide a workable solution for running > the Tez UI against the timeline server on a large-scale clusters running many > thousands of jobs per day. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1645) ContainerManager implementation to support container resizing
[ https://issues.apache.org/jira/browse/YARN-1645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905410#comment-14905410 ] Wangda Tan commented on YARN-1645: -- This feature is targeting to support both, cpu support is tracked by YARN-4166. > ContainerManager implementation to support container resizing > - > > Key: YARN-1645 > URL: https://issues.apache.org/jira/browse/YARN-1645 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Wangda Tan >Assignee: MENG DING > Fix For: 3.0.0 > > Attachments: YARN-1645-YARN-1197.3.patch, > YARN-1645-YARN-1197.4.patch, YARN-1645-YARN-1197.5.patch, YARN-1645.1.patch, > YARN-1645.2.patch, yarn-1645.1.patch > > > Implementation of ContainerManager for container resize, including: > 1) ContainerManager resize logic > 2) Relevant test cases -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (YARN-3976) Catch ApplicationNotFoundException instead of parent YarnException in YarnClient and AppReportFetcher
[
https://issues.apache.org/jira/browse/YARN-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mit Desai resolved YARN-3976.
-
Resolution: Fixed
Fix Version/s: 2.7.2
Made the change along with YARN-3975.
Resolving this jira as YARN-3975 is committed.
> Catch ApplicationNotFoundException instead of parent YarnException in
> YarnClient and AppReportFetcher
> -
>
> Key: YARN-3976
> URL: https://issues.apache.org/jira/browse/YARN-3976
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.7.1
>Reporter: Mit Desai
>Assignee: Mit Desai
>Priority: Trivial
> Fix For: 2.7.2
>
>
> It's is better to catch the ApplicationNotFoundException rather than the
> parent YarnException and rethrow it when it's not ApplicationNotFoundExcepton
> {noformat}
> catch (YarnException e) {
> if (!historyServiceEnabled) {
> // Just throw it as usual if historyService is not enabled.
> throw e;
> }
> // Even if history-service is enabled, treat all exceptions still the
> same
> // except the following
> if (!(e.getClass() == ApplicationNotFoundException.class)) {
> throw e;
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4075) [reader REST API] implement support for querying for flows and flow runs
[ https://issues.apache.org/jira/browse/YARN-4075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904911#comment-14904911 ] Li Lu commented on YARN-4075: - Latest patch LGTM. > [reader REST API] implement support for querying for flows and flow runs > > > Key: YARN-4075 > URL: https://issues.apache.org/jira/browse/YARN-4075 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Varun Saxena > Attachments: YARN-4075-YARN-2928.03.patch, > YARN-4075-YARN-2928.04.patch, YARN-4075-YARN-2928.POC.1.patch, > YARN-4075-YARN-2928.POC.2.patch > > > We need to be able to query for flows and flow runs via REST. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4165) An outstanding container request makes all nodes to be reserved causing all jobs pending
[
https://issues.apache.org/jira/browse/YARN-4165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904960#comment-14904960
]
MENG DING commented on YARN-4165:
-
Hi, [~leftnoteasy]
Please correct me if I am wrong. According to the current logic, in
{{shouldAllocOrReserveNewContainer}}, the starvation number will grow with
re-reservations. Based on the use case in this ticket, I think the
re-reservations will increase for the first reserved container, and to the
point that the scheduler will try to reserve on another host. Isn't it?
{code}
starvation =
(int) ((application.getReReservations(priority) /
(float) reservedContainers) * (1.0f - (Math.min(
nodeFactor, application.getCSLeafQueue()
.getMinimumAllocationFactor();
{code}
> An outstanding container request makes all nodes to be reserved causing all
> jobs pending
>
>
> Key: YARN-4165
> URL: https://issues.apache.org/jira/browse/YARN-4165
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacity scheduler, resourcemanager, scheduler
>Affects Versions: 2.7.1
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>
> We have a long running service in YARN, it has a outstanding container
> request that YARN cannot satisfy (require more memory that nodemanager can
> supply). Then YARN reserves all nodes for this application, when I submit
> other jobs (require relative small memory that nodemanager can supply), all
> jobs are pending because YARN skips scheduling containers on the nodes that
> have been reserved.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4095) Avoid sharing AllocatorPerContext object in LocalDirAllocator between ShuffleHandler and LocalDirsHandlerService.
[
https://issues.apache.org/jira/browse/YARN-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904969#comment-14904969
]
Hudson commented on YARN-4095:
--
SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #435 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/435/])
YARN-4095. Avoid sharing AllocatorPerContext object in LocalDirAllocator
between ShuffleHandler and LocalDirsHandlerService. Contributed by Zhihai Xu
(jlowe: rev c890c51a916894a985439497b8a44e8eee82d762)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LocalDirsHandlerService.java
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLocalDirsHandlerService.java
> Avoid sharing AllocatorPerContext object in LocalDirAllocator between
> ShuffleHandler and LocalDirsHandlerService.
> -
>
> Key: YARN-4095
> URL: https://issues.apache.org/jira/browse/YARN-4095
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
>Reporter: zhihai xu
>Assignee: zhihai xu
> Fix For: 2.8.0
>
> Attachments: YARN-4095.000.patch, YARN-4095.001.patch
>
>
> Currently {{ShuffleHandler}} and {{LocalDirsHandlerService}} share
> {{AllocatorPerContext}} object in {{LocalDirAllocator}} for configuration
> {{NM_LOCAL_DIRS}} because {{AllocatorPerContext}} are stored in a static
> TreeMap with configuration name as key
> {code}
> private static Map contexts =
> new TreeMap();
> {code}
> {{LocalDirsHandlerService}} and {{ShuffleHandler}} both create a
> {{LocalDirAllocator}} using {{NM_LOCAL_DIRS}}. Even they don't use the same
> {{Configuration}} object, but they will use the same {{AllocatorPerContext}}
> object. Also {{LocalDirsHandlerService}} may change {{NM_LOCAL_DIRS}} value
> in its {{Configuration}} object to exclude full and bad local dirs,
> {{ShuffleHandler}} always uses the original {{NM_LOCAL_DIRS}} value in its
> {{Configuration}} object. So every time {{AllocatorPerContext#confChanged}}
> is called by {{ShuffleHandler}} after {{LocalDirsHandlerService}},
> {{AllocatorPerContext}} need be reinitialized because {{NM_LOCAL_DIRS}} value
> is changed. This will cause some overhead.
> {code}
> String newLocalDirs = conf.get(contextCfgItemName);
> if (!newLocalDirs.equals(savedLocalDirs)) {
> {code}
> So it will be a good improvement to not share the same
> {{AllocatorPerContext}} instance between {{ShuffleHandler}} and
> {{LocalDirsHandlerService}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3975) WebAppProxyServlet should not redirect to RM page if AHS is enabled
[ https://issues.apache.org/jira/browse/YARN-3975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904970#comment-14904970 ] Hudson commented on YARN-3975: -- SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #435 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/435/]) YARN-3975. WebAppProxyServlet should not redirect to RM page if AHS is enabled. Contributed by Mit Desai (jlowe: rev 692d51c09d3668cde47cc297296d095ddfa933a3) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/AppReportFetcher.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestWebAppProxyServlet.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java * hadoop-yarn-project/CHANGES.txt > WebAppProxyServlet should not redirect to RM page if AHS is enabled > --- > > Key: YARN-3975 > URL: https://issues.apache.org/jira/browse/YARN-3975 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Mit Desai >Assignee: Mit Desai > Fix For: 2.7.2 > > Attachments: YARN-3975.2.b2.patch, YARN-3975.3.patch, > YARN-3975.4.patch, YARN-3975.5.patch, YARN-3975.6.patch, YARN-3975.7.patch, > YARN-3975.8.patch, YARN-3975.9.b2.7.patch, YARN-3975.9.patch, > YARN-3975.9.patch > > > WebAppProxyServlet should be updated to handle the case when the appreport > doesn't have a tracking URL and the Application History Server is eanbled. > As we would have already tried the RM and got the > ApplicationNotFoundException we should not direct the user to the RM app page. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1449) AM-NM protocol changes to support container resizing
[ https://issues.apache.org/jira/browse/YARN-1449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905347#comment-14905347 ] Hudson commented on YARN-1449: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #436 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/436/]) YARN-1449. AM-NM protocol changes to support container resizing. Contributed by Meng Ding & Wangda Tan) (wangda: rev 83a18add10ee937a04e833a66e0a4642e776e510) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/TestRPC.java * hadoop-yarn-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher/TestContainerLauncher.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher/TestContainerLauncherImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/IncreaseContainersResourceResponse.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/TestContainerLaunchRPC.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ContainerManagementProtocol.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAMAuthorization.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/TestContainerResourceIncreaseRPC.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/containermanagement_protocol.proto * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/client/ContainerManagementProtocolPBClientImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/service/ContainerManagementProtocolPBServiceImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/IncreaseContainersResourceRequest.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/IncreaseContainersResourceResponsePBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestPBImplRecords.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/IncreaseContainersResourceRequestPBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationMasterLauncher.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/NodeManager.java > AM-NM protocol changes to support container resizing > > > Key: YARN-1449 > URL: https://issues.apache.org/jira/browse/YARN-1449 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api >Reporter: Wangda Tan (No longer used) >Assignee: MENG DING > Fix For: 3.0.0 > > Attachments: YARN-1449-YARN-1197.5.patch, YARN-1449.1.patch, > YARN-1449.2.patch, YARN-1449.3.patch, YARN-1449.4.patch, yarn-1449.1.patch, > yarn-1449.3.patch, yarn-1449.4.patch, yarn-1449.5.patch > > > AM-NM protocol changes to support container resizing > 1) "IncreaseContainersResourceRequest" and > "IncreaseContainersResourceResponse" PB protocol and implementation > 2) "increaseContainersResources" method in ContainerManagementProtocol > 3) Update "ContainerStatus" protocol to include Resource > 4) Relevant test cases -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3248) Display count of nodes blacklisted by apps in the web UI
[ https://issues.apache.org/jira/browse/YARN-3248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905343#comment-14905343 ] Hudson commented on YARN-3248: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #436 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/436/]) CHANGES.txt: Moving YARN-1884, YARN-3171, YARN-3740, YARN-3248, YARN-3544 to 2.6.1 given the backport. (vinodkv: rev f3e5bc67661ebc7e14509f2a267131ae314699dc) * hadoop-yarn-project/CHANGES.txt > Display count of nodes blacklisted by apps in the web UI > > > Key: YARN-3248 > URL: https://issues.apache.org/jira/browse/YARN-3248 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler, resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Labels: 2.6.1-candidate, 2.7.2-candidate > Fix For: 2.6.1, 2.8.0 > > Attachments: All applications.png, App page.png, Screenshot.jpg, > YARN-3248-branch-2.6.1.txt, apache-yarn-3248.0.patch, > apache-yarn-3248.1.patch, apache-yarn-3248.2.patch, apache-yarn-3248.3.patch, > apache-yarn-3248.4.patch > > > It would be really useful when debugging app performance and failure issues > to get a count of the nodes blacklisted by individual apps displayed in the > web UI. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1643) Make ContainersMonitor can support change monitoring size of an allocated container in NM side
[ https://issues.apache.org/jira/browse/YARN-1643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905342#comment-14905342 ] Hudson commented on YARN-1643: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #436 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/436/]) YARN-1643. Make ContainersMonitor support changing monitoring size of an allocated container. Contributed by Meng Ding and Wangda Tan (wangda: rev c59ae4eeb17e52e8fc659b9962d20628719fc621) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerManagerWithLCE.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitorResourceChange.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/MockResourceCalculatorProcessTree.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/MockResourceCalculatorPlugin.java > Make ContainersMonitor can support change monitoring size of an allocated > container in NM side > -- > > Key: YARN-1643 > URL: https://issues.apache.org/jira/browse/YARN-1643 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Wangda Tan >Assignee: MENG DING > Fix For: 3.0.0 > > Attachments: YARN-1643-YARN-1197.4.patch, > YARN-1643-YARN-1197.5.patch, YARN-1643-YARN-1197.6.patch, > YARN-1643-YARN-1197.7.patch, YARN-1643.1.patch, YARN-1643.2.patch, > YARN-1643.3.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1884) ContainerReport should have nodeHttpAddress
[ https://issues.apache.org/jira/browse/YARN-1884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905354#comment-14905354 ] Hudson commented on YARN-1884: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #436 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/436/]) CHANGES.txt: Moving YARN-1884, YARN-3171, YARN-3740, YARN-3248, YARN-3544 to 2.6.1 given the backport. (vinodkv: rev f3e5bc67661ebc7e14509f2a267131ae314699dc) * hadoop-yarn-project/CHANGES.txt > ContainerReport should have nodeHttpAddress > --- > > Key: YARN-1884 > URL: https://issues.apache.org/jira/browse/YARN-1884 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Xuan Gong > Labels: 2.6.1-candidate > Fix For: 2.7.0, 2.6.1 > > Attachments: YARN-1884-branch-2.6.1.txt, YARN-1884.1.patch, > YARN-1884.2.patch, YARN-1884.3.patch, YARN-1884.4.patch > > > In web UI, we're going to show the node, which used to be to link to the NM > web page. However, on AHS web UI, and RM web UI after YARN-1809, the node > field has to be set to nodeID where the container is allocated. We need to > add nodeHttpAddress to the containerReport to link users to NM web page -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3866) AM-RM protocol changes to support container resizing
[ https://issues.apache.org/jira/browse/YARN-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905350#comment-14905350 ] Hudson commented on YARN-3866: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #436 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/436/]) YARN-3866. AM-RM protocol changes to support container resizing. Contributed by Meng Ding (wangda: rev dfe2cb849fcb273d32e840d8562eaaee904af054) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestAllocateRequest.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/local/TestLocalContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestPBImplRecords.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestAllocateResponse.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ContainerResourceIncreaseRequestPBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ContainerResourceIncreaseRequest.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestContainerResourceDecrease.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ContainerResourceDecrease.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestContainerResourceIncrease.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ContainerStatus.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/AllocateRequestPBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestContainerResourceIncreaseRequest.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ContainerResourceChangeRequestPBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ContainerResourceDecreasePBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ContainerResourceIncreasePBImpl.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ContainerResourceIncrease.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ContainerStatusPBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/AllocateRequest.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/AllocateResponse.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ContainerResourceChangeRequest.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_service_protos.proto * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/AllocateResponsePBImpl.java > AM-RM protocol changes to support container resizing > > > Key: YARN-3866 > URL: https://issues.apache.org/jira/browse/YARN-3866 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api >Reporter: MENG DING >Assignee: MENG DING > Fix For: 3.0.0 > > Attachments: YARN-3866-YARN-1197.4.patch, YARN-3866.1.patch, > YARN-3866.2.patch, YARN-3866.3.patch > > > YARN-1447 and YARN-1448 are outdated. > This ticket deals with AM-RM Protocol changes to support container resize > according to the latest design in YARN-1197. > 1) Add increase/decrease requests in AllocateRequest > 2) Get approved increase/decrease requests from RM in AllocateResponse > 3) Add relevant test cases -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3867) ContainerImpl changes to support container resizing
[ https://issues.apache.org/jira/browse/YARN-3867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905352#comment-14905352 ] Hudson commented on YARN-3867: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #436 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/436/]) YARN-3867. ContainerImpl changes to support container resizing. Contributed by Meng Ding (wangda: rev 5f5a968d65c44a831176764439e00db9203999ed) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorEventType.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/metrics/NodeManagerMetrics.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/NodeManager.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdater.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationCleanup.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerEventType.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ChangeContainerResourceEvent.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/BuilderUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ChangeMonitoringContainerResourceEvent.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/metrics/TestNodeManagerMetrics.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/MockContainer.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestAMRMTokens.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/Container.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNM.java > ContainerImpl changes to support container resizing > --- > > Key: YARN-3867 > URL: https://issues.apache.org/jira/browse/YARN-3867 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: MENG DING >Assignee: MENG DING > Fix For: 3.0.0 > > Attachments: YARN-3867-YARN-1197.3.patch, > YARN-3867-YARN-1197.4.patch, YARN-3867-YARN-1197.5.patch, YARN-3867.1.patch, > YARN-3867.2.patch > > > 1) ContainerImpl logic changes in NM to handle events related to container > resizing > 2) Relevant test cases -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1651) CapacityScheduler side changes to support increase/decrease container resource.
[ https://issues.apache.org/jira/browse/YARN-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905348#comment-14905348 ] Hudson commented on YARN-1651: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #436 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/436/]) YARN-1651. CapacityScheduler side changes to support container resize. Contributed by Wangda Tan (wangda: rev 89cab1ba5f0671f8ef30dbe7432079c18362b434) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeEventType.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/TestRMContainerImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNode.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/AbstractContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/IncreaseContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/YarnScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAuditLogger.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockAM.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeDecreaseContainerEvent.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java * hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SLSCapacityScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestReservations.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerImpl.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/rm/TestRMContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/Allocation.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/RegularContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestContinuousScheduling.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/nodemanager/NodeInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/resource/ResourceCalculator.java *
[jira] [Commented] (YARN-3171) Sort by Application id, AppAttempt & ContainerID doesn't work in ATS / RM web ui
[ https://issues.apache.org/jira/browse/YARN-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905353#comment-14905353 ] Hudson commented on YARN-3171: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #436 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/436/]) CHANGES.txt: Moving YARN-1884, YARN-3171, YARN-3740, YARN-3248, YARN-3544 to 2.6.1 given the backport. (vinodkv: rev f3e5bc67661ebc7e14509f2a267131ae314699dc) * hadoop-yarn-project/CHANGES.txt > Sort by Application id, AppAttempt & ContainerID doesn't work in ATS / RM web > ui > > > Key: YARN-3171 > URL: https://issues.apache.org/jira/browse/YARN-3171 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: 2.6.0 >Reporter: Jeff Zhang >Assignee: Naganarasimha G R >Priority: Minor > Labels: 2.6.1-candidate > Fix For: 2.7.0, 2.6.1 > > Attachments: YARN-3171.20150213-1.patch, YARN-3171.20150215-1.patch, > ats_webui.png > > > The order doesn't change when I click the column header -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3868) ContainerManager recovery for container resizing
[ https://issues.apache.org/jira/browse/YARN-3868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905349#comment-14905349 ] Hudson commented on YARN-3868: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #436 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/436/]) YARN-3868. Recovery support for container resizing. Contributed by Meng Ding (wangda: rev c57eac5dfe277845ab4522a1188023a73ee41539) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManagerRecovery.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/recovery/NMStateStoreService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/recovery/TestNMLeveldbStateStoreService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/recovery/NMMemoryStateStoreService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/recovery/NMNullStateStoreService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/recovery/NMLeveldbStateStoreService.java > ContainerManager recovery for container resizing > > > Key: YARN-3868 > URL: https://issues.apache.org/jira/browse/YARN-3868 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: MENG DING >Assignee: MENG DING > Fix For: 3.0.0 > > Attachments: YARN-3868-YARN-1197.3.patch, > YARN-3868-YARN-1197.4.patch, YARN-3868-YARN-1197.5.patch, YARN-3868.1.patch, > YARN-3868.2.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1197) Support changing resources of an allocated container
[ https://issues.apache.org/jira/browse/YARN-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905355#comment-14905355 ] Hudson commented on YARN-1197: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #436 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/436/]) YARN-4171. Fix findbugs warnings in YARN-1197 branch. Contributed by Wangda Tan (wangda: rev b3f6b641dccb0d59df78855e2951d2cae7dff8ad) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeStatusEvent.java > Support changing resources of an allocated container > > > Key: YARN-1197 > URL: https://issues.apache.org/jira/browse/YARN-1197 > Project: Hadoop YARN > Issue Type: Task > Components: api, nodemanager, resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Wangda Tan > Attachments: YARN-1197 old-design-docs-patches-for-reference.zip, > YARN-1197_Design.2015.06.24.pdf, YARN-1197_Design.2015.07.07.pdf, > YARN-1197_Design.2015.08.21.pdf, YARN-1197_Design.pdf > > > The current YARN resource management logic assumes resource allocated to a > container is fixed during the lifetime of it. When users want to change a > resource > of an allocated container the only way is releasing it and allocating a new > container with expected size. > Allowing run-time changing resources of an allocated container will give us > better control of resource usage in application side -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3740) Fixed the typo with the configuration name: APPLICATION_HISTORY_PREFIX_MAX_APPS
[ https://issues.apache.org/jira/browse/YARN-3740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-3740: -- Fix Version/s: 2.7.2 Pulled this into branch-2.7 given its backport to 2.6.1. Applies cleanly. Ran compilation and TestApplicationHistoryClientService before the push. > Fixed the typo with the configuration name: > APPLICATION_HISTORY_PREFIX_MAX_APPS > --- > > Key: YARN-3740 > URL: https://issues.apache.org/jira/browse/YARN-3740 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager, webapp, yarn >Reporter: Xuan Gong >Assignee: Xuan Gong > Labels: 2.6.1-candidate, 2.7.2-candidate > Fix For: 2.6.1, 2.8.0, 2.7.2 > > Attachments: YARN-3740.1.patch > > > YARN-3700 introduces a new configuration named > APPLICATION_HISTORY_PREFIX_MAX_APPS, which need be changed to > APPLICATION_HISTORY_MAX_APPS. > This is not an incompatibility change since YARN-3700 is in 2.8 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3248) Display count of nodes blacklisted by apps in the web UI
[ https://issues.apache.org/jira/browse/YARN-3248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-3248: -- Attachment: YARN-3248-branch-2.7.2.txt Pulled this into branch-2.7 given its backport to 2.6.1. Merge conflicts like with 2.6.1 patch. Ran compilation and TestRMWebServicesApps before the push. Attaching the patch that I put in 2.7.2. > Display count of nodes blacklisted by apps in the web UI > > > Key: YARN-3248 > URL: https://issues.apache.org/jira/browse/YARN-3248 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler, resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Labels: 2.6.1-candidate, 2.7.2-candidate > Fix For: 2.6.1, 2.8.0, 2.7.2 > > Attachments: All applications.png, App page.png, Screenshot.jpg, > YARN-3248-branch-2.6.1.txt, YARN-3248-branch-2.7.2.txt, > apache-yarn-3248.0.patch, apache-yarn-3248.1.patch, apache-yarn-3248.2.patch, > apache-yarn-3248.3.patch, apache-yarn-3248.4.patch > > > It would be really useful when debugging app performance and failure issues > to get a count of the nodes blacklisted by individual apps displayed in the > web UI. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4129) Refactor the SystemMetricPublisher in RM to better support newer events
[ https://issues.apache.org/jira/browse/YARN-4129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905386#comment-14905386 ] Li Lu commented on YARN-4129: - Hi [~Naganarasimha], I'll take a look at this patch soon. > Refactor the SystemMetricPublisher in RM to better support newer events > --- > > Key: YARN-4129 > URL: https://issues.apache.org/jira/browse/YARN-4129 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > Attachments: YARN-4129.YARN-2928.001.patch > > > Currently to add new timeline event/ entity in RM side, one has to add a > method in publisher and a method in handler and create a new event class > which looks cumbersome and redundant. also further all the events might not > be required to be published in V1 & V2. So adopting the approach similar to > what was adopted in YARN-3045(NM side) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4075) [reader REST API] implement support for querying for flows and flow runs
[ https://issues.apache.org/jira/browse/YARN-4075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905407#comment-14905407 ] Vrushali C commented on YARN-4075: -- Alright, I have restarted the build on jenkins as well. > [reader REST API] implement support for querying for flows and flow runs > > > Key: YARN-4075 > URL: https://issues.apache.org/jira/browse/YARN-4075 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Varun Saxena > Attachments: YARN-4075-YARN-2928.03.patch, > YARN-4075-YARN-2928.04.patch, YARN-4075-YARN-2928.POC.1.patch, > YARN-4075-YARN-2928.POC.2.patch > > > We need to be able to query for flows and flow runs via REST. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1645) ContainerManager implementation to support container resizing
[ https://issues.apache.org/jira/browse/YARN-1645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905310#comment-14905310 ] Hudson commented on YARN-1645: -- FAILURE: Integrated in Hadoop-trunk-Commit #8505 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8505/]) YARN-1645. ContainerManager implementation to support container resizing. Contributed by Meng Ding & Wangda Tan (wangda: rev ffd820c27a4f8cf4676ad8758696ed89fde80218) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/BaseContainerManagerTest.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerEventType.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerManagerEventType.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerManagerWithLCE.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ChangeContainerResourceEvent.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/CMgrDecreaseContainersResourceEvent.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/DummyContainerManager.java > ContainerManager implementation to support container resizing > - > > Key: YARN-1645 > URL: https://issues.apache.org/jira/browse/YARN-1645 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Wangda Tan >Assignee: MENG DING > Fix For: 3.0.0 > > Attachments: YARN-1645-YARN-1197.3.patch, > YARN-1645-YARN-1197.4.patch, YARN-1645-YARN-1197.5.patch, YARN-1645.1.patch, > YARN-1645.2.patch, yarn-1645.1.patch > > > Implementation of ContainerManager for container resize, including: > 1) ContainerManager resize logic > 2) Relevant test cases -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3867) ContainerImpl changes to support container resizing
[ https://issues.apache.org/jira/browse/YARN-3867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905318#comment-14905318 ] Hudson commented on YARN-3867: -- FAILURE: Integrated in Hadoop-trunk-Commit #8505 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8505/]) YARN-3867. ContainerImpl changes to support container resizing. Contributed by Meng Ding (wangda: rev 5f5a968d65c44a831176764439e00db9203999ed) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ChangeContainerResourceEvent.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdater.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/MockContainer.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorEventType.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerEventType.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestAMRMTokens.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/metrics/NodeManagerMetrics.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/Container.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/BuilderUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNM.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationCleanup.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/metrics/TestNodeManagerMetrics.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/NodeManager.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ChangeMonitoringContainerResourceEvent.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java > ContainerImpl changes to support container resizing > --- > > Key: YARN-3867 > URL: https://issues.apache.org/jira/browse/YARN-3867 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: MENG DING >Assignee: MENG DING > Fix For: 3.0.0 > > Attachments: YARN-3867-YARN-1197.3.patch, > YARN-3867-YARN-1197.4.patch, YARN-3867-YARN-1197.5.patch, YARN-3867.1.patch, > YARN-3867.2.patch > > > 1) ContainerImpl logic changes in NM to handle events related to container > resizing > 2) Relevant test cases -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1651) CapacityScheduler side changes to support increase/decrease container resource.
[ https://issues.apache.org/jira/browse/YARN-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905314#comment-14905314 ] Hudson commented on YARN-1651: -- FAILURE: Integrated in Hadoop-trunk-Commit #8505 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8505/]) YARN-1651. CapacityScheduler side changes to support container resize. Contributed by Wangda Tan (wangda: rev 89cab1ba5f0671f8ef30dbe7432079c18362b434) * hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/RMNodeWrapper.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerNode.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/rm/TestRMContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerResizing.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/ContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRestart.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/Application.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeDecreaseContainerEvent.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/YarnScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNode.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/QueueMetrics.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/AbstractContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainer.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodeHeartbeatResponsePBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerUpdatesAcquiredEvent.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerChangeResourceEvent.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerEventType.java *
[jira] [Commented] (YARN-1644) RM-NM protocol changes and NodeStatusUpdater implementation to support container resizing
[ https://issues.apache.org/jira/browse/YARN-1644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905311#comment-14905311 ] Hudson commented on YARN-1644: -- FAILURE: Integrated in Hadoop-trunk-Commit #8505 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8505/]) YARN-1644. RM-NM protocol changes and NodeStatusUpdater implementation to support container resizing. Contributed by Meng Ding (wangda: rev c3dc1af072574f5890a8d43e4d60526951b4b8bc) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/MockResourceManagerFacade.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/Context.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/NodeStatus.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_service_protos.proto * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodeHeartbeatResponsePBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/impl/pb/NodeStatusPBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/BaseAMRMProxyTest.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestResourceTrackerOnHA.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_protos.proto * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/TestYarnServerApiClasses.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeManagerResync.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/NodeHeartbeatResponse.java > RM-NM protocol changes and NodeStatusUpdater implementation to support > container resizing > - > > Key: YARN-1644 > URL: https://issues.apache.org/jira/browse/YARN-1644 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Wangda Tan >Assignee: MENG DING > Fix For: 3.0.0 > > Attachments: YARN-1644-YARN-1197.4.patch, > YARN-1644-YARN-1197.5.patch, YARN-1644-YARN-1197.6.patch, YARN-1644.1.patch, > YARN-1644.2.patch, YARN-1644.3.patch, yarn-1644.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4171) Resolve findbugs/javac warnings in YARN-1197 branch
[ https://issues.apache.org/jira/browse/YARN-4171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905317#comment-14905317 ] Hudson commented on YARN-4171: -- FAILURE: Integrated in Hadoop-trunk-Commit #8505 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8505/]) YARN-4171. Fix findbugs warnings in YARN-1197 branch. Contributed by Wangda Tan (wangda: rev b3f6b641dccb0d59df78855e2951d2cae7dff8ad) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeStatusEvent.java > Resolve findbugs/javac warnings in YARN-1197 branch > --- > > Key: YARN-4171 > URL: https://issues.apache.org/jira/browse/YARN-4171 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: Wangda Tan > Fix For: 3.0.0 > > Attachments: YARN-4171-YARN-1197.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1884) ContainerReport should have nodeHttpAddress
[ https://issues.apache.org/jira/browse/YARN-1884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905320#comment-14905320 ] Hudson commented on YARN-1884: -- FAILURE: Integrated in Hadoop-trunk-Commit #8505 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8505/]) CHANGES.txt: Moving YARN-1884, YARN-3171, YARN-3740, YARN-3248, YARN-3544 to 2.6.1 given the backport. (vinodkv: rev f3e5bc67661ebc7e14509f2a267131ae314699dc) * hadoop-yarn-project/CHANGES.txt > ContainerReport should have nodeHttpAddress > --- > > Key: YARN-1884 > URL: https://issues.apache.org/jira/browse/YARN-1884 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Xuan Gong > Labels: 2.6.1-candidate > Fix For: 2.7.0, 2.6.1 > > Attachments: YARN-1884-branch-2.6.1.txt, YARN-1884.1.patch, > YARN-1884.2.patch, YARN-1884.3.patch, YARN-1884.4.patch > > > In web UI, we're going to show the node, which used to be to link to the NM > web page. However, on AHS web UI, and RM web UI after YARN-1809, the node > field has to be set to nodeID where the container is allocated. We need to > add nodeHttpAddress to the containerReport to link users to NM web page -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1449) AM-NM protocol changes to support container resizing
[ https://issues.apache.org/jira/browse/YARN-1449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905313#comment-14905313 ] Hudson commented on YARN-1449: -- FAILURE: Integrated in Hadoop-trunk-Commit #8505 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8505/]) YARN-1449. AM-NM protocol changes to support container resizing. Contributed by Meng Ding & Wangda Tan) (wangda: rev 83a18add10ee937a04e833a66e0a4642e776e510) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAMAuthorization.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestPBImplRecords.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationMasterLauncher.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/TestContainerResourceIncreaseRPC.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher/TestContainerLauncher.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/TestRPC.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/client/ContainerManagementProtocolPBClientImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/IncreaseContainersResourceResponsePBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/containermanagement_protocol.proto * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/service/ContainerManagementProtocolPBServiceImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ContainerManagementProtocol.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/IncreaseContainersResourceResponse.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/TestContainerLaunchRPC.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/IncreaseContainersResourceRequest.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher/TestContainerLauncherImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/NodeManager.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/IncreaseContainersResourceRequestPBImpl.java > AM-NM protocol changes to support container resizing > > > Key: YARN-1449 > URL: https://issues.apache.org/jira/browse/YARN-1449 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api >Reporter: Wangda Tan (No longer used) >Assignee: MENG DING > Fix For: 3.0.0 > > Attachments: YARN-1449-YARN-1197.5.patch, YARN-1449.1.patch, > YARN-1449.2.patch, YARN-1449.3.patch, YARN-1449.4.patch, yarn-1449.1.patch, > yarn-1449.3.patch, yarn-1449.4.patch, yarn-1449.5.patch > > > AM-NM protocol changes to support container resizing > 1) "IncreaseContainersResourceRequest" and > "IncreaseContainersResourceResponse" PB protocol and implementation > 2) "increaseContainersResources" method in ContainerManagementProtocol > 3) Update "ContainerStatus" protocol to include Resource > 4) Relevant test cases -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1197) Support changing resources of an allocated container
[ https://issues.apache.org/jira/browse/YARN-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905321#comment-14905321 ] Hudson commented on YARN-1197: -- FAILURE: Integrated in Hadoop-trunk-Commit #8505 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8505/]) YARN-4171. Fix findbugs warnings in YARN-1197 branch. Contributed by Wangda Tan (wangda: rev b3f6b641dccb0d59df78855e2951d2cae7dff8ad) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeStatusEvent.java > Support changing resources of an allocated container > > > Key: YARN-1197 > URL: https://issues.apache.org/jira/browse/YARN-1197 > Project: Hadoop YARN > Issue Type: Task > Components: api, nodemanager, resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Wangda Tan > Attachments: YARN-1197 old-design-docs-patches-for-reference.zip, > YARN-1197_Design.2015.06.24.pdf, YARN-1197_Design.2015.07.07.pdf, > YARN-1197_Design.2015.08.21.pdf, YARN-1197_Design.pdf > > > The current YARN resource management logic assumes resource allocated to a > container is fixed during the lifetime of it. When users want to change a > resource > of an allocated container the only way is releasing it and allocating a new > container with expected size. > Allowing run-time changing resources of an allocated container will give us > better control of resource usage in application side -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3740) Fixed the typo with the configuration name: APPLICATION_HISTORY_PREFIX_MAX_APPS
[ https://issues.apache.org/jira/browse/YARN-3740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905312#comment-14905312 ] Hudson commented on YARN-3740: -- FAILURE: Integrated in Hadoop-trunk-Commit #8505 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8505/]) CHANGES.txt: Moving YARN-1884, YARN-3171, YARN-3740, YARN-3248, YARN-3544 to 2.6.1 given the backport. (vinodkv: rev f3e5bc67661ebc7e14509f2a267131ae314699dc) * hadoop-yarn-project/CHANGES.txt > Fixed the typo with the configuration name: > APPLICATION_HISTORY_PREFIX_MAX_APPS > --- > > Key: YARN-3740 > URL: https://issues.apache.org/jira/browse/YARN-3740 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager, webapp, yarn >Reporter: Xuan Gong >Assignee: Xuan Gong > Labels: 2.6.1-candidate, 2.7.2-candidate > Fix For: 2.6.1, 2.8.0 > > Attachments: YARN-3740.1.patch > > > YARN-3700 introduces a new configuration named > APPLICATION_HISTORY_PREFIX_MAX_APPS, which need be changed to > APPLICATION_HISTORY_MAX_APPS. > This is not an incompatibility change since YARN-3700 is in 2.8 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3544) AM logs link missing in the RM UI for a completed app
[ https://issues.apache.org/jira/browse/YARN-3544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905307#comment-14905307 ] Hudson commented on YARN-3544: -- FAILURE: Integrated in Hadoop-trunk-Commit #8505 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8505/]) CHANGES.txt: Moving YARN-1884, YARN-3171, YARN-3740, YARN-3248, YARN-3544 to 2.6.1 given the backport. (vinodkv: rev f3e5bc67661ebc7e14509f2a267131ae314699dc) * hadoop-yarn-project/CHANGES.txt > AM logs link missing in the RM UI for a completed app > -- > > Key: YARN-3544 > URL: https://issues.apache.org/jira/browse/YARN-3544 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.7.0 >Reporter: Hitesh Shah >Assignee: Xuan Gong >Priority: Blocker > Labels: 2.6.1-candidate > Fix For: 2.6.1, 2.7.1 > > Attachments: Screen Shot 2015-04-27 at 6.24.05 PM.png, > YARN-3544-branch-2.6.1.txt, YARN-3544-branch-2.7-1.2.patch, > YARN-3544-branch-2.7-1.patch, YARN-3544.1.patch > > > AM log links should always be present ( for both running and completed apps). > Likewise node info is also empty. This is usually quite crucial when trying > to debug where an AM was launched and a pointer to which NM's logs to look at > if the AM failed to launch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3868) ContainerManager recovery for container resizing
[ https://issues.apache.org/jira/browse/YARN-3868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905315#comment-14905315 ] Hudson commented on YARN-3868: -- FAILURE: Integrated in Hadoop-trunk-Commit #8505 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8505/]) YARN-3868. Recovery support for container resizing. Contributed by Meng Ding (wangda: rev c57eac5dfe277845ab4522a1188023a73ee41539) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/recovery/NMNullStateStoreService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManagerRecovery.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/recovery/NMMemoryStateStoreService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/recovery/NMStateStoreService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/recovery/TestNMLeveldbStateStoreService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/recovery/NMLeveldbStateStoreService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java > ContainerManager recovery for container resizing > > > Key: YARN-3868 > URL: https://issues.apache.org/jira/browse/YARN-3868 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: MENG DING >Assignee: MENG DING > Fix For: 3.0.0 > > Attachments: YARN-3868-YARN-1197.3.patch, > YARN-3868-YARN-1197.4.patch, YARN-3868-YARN-1197.5.patch, YARN-3868.1.patch, > YARN-3868.2.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3248) Display count of nodes blacklisted by apps in the web UI
[ https://issues.apache.org/jira/browse/YARN-3248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905309#comment-14905309 ] Hudson commented on YARN-3248: -- FAILURE: Integrated in Hadoop-trunk-Commit #8505 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8505/]) CHANGES.txt: Moving YARN-1884, YARN-3171, YARN-3740, YARN-3248, YARN-3544 to 2.6.1 given the backport. (vinodkv: rev f3e5bc67661ebc7e14509f2a267131ae314699dc) * hadoop-yarn-project/CHANGES.txt > Display count of nodes blacklisted by apps in the web UI > > > Key: YARN-3248 > URL: https://issues.apache.org/jira/browse/YARN-3248 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler, resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Labels: 2.6.1-candidate, 2.7.2-candidate > Fix For: 2.6.1, 2.8.0 > > Attachments: All applications.png, App page.png, Screenshot.jpg, > YARN-3248-branch-2.6.1.txt, apache-yarn-3248.0.patch, > apache-yarn-3248.1.patch, apache-yarn-3248.2.patch, apache-yarn-3248.3.patch, > apache-yarn-3248.4.patch > > > It would be really useful when debugging app performance and failure issues > to get a count of the nodes blacklisted by individual apps displayed in the > web UI. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3171) Sort by Application id, AppAttempt & ContainerID doesn't work in ATS / RM web ui
[ https://issues.apache.org/jira/browse/YARN-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905319#comment-14905319 ] Hudson commented on YARN-3171: -- FAILURE: Integrated in Hadoop-trunk-Commit #8505 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8505/]) CHANGES.txt: Moving YARN-1884, YARN-3171, YARN-3740, YARN-3248, YARN-3544 to 2.6.1 given the backport. (vinodkv: rev f3e5bc67661ebc7e14509f2a267131ae314699dc) * hadoop-yarn-project/CHANGES.txt > Sort by Application id, AppAttempt & ContainerID doesn't work in ATS / RM web > ui > > > Key: YARN-3171 > URL: https://issues.apache.org/jira/browse/YARN-3171 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: 2.6.0 >Reporter: Jeff Zhang >Assignee: Naganarasimha G R >Priority: Minor > Labels: 2.6.1-candidate > Fix For: 2.7.0, 2.6.1 > > Attachments: YARN-3171.20150213-1.patch, YARN-3171.20150215-1.patch, > ats_webui.png > > > The order doesn't change when I click the column header -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1643) Make ContainersMonitor can support change monitoring size of an allocated container in NM side
[ https://issues.apache.org/jira/browse/YARN-1643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905308#comment-14905308 ] Hudson commented on YARN-1643: -- FAILURE: Integrated in Hadoop-trunk-Commit #8505 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8505/]) YARN-1643. Make ContainersMonitor support changing monitoring size of an allocated container. Contributed by Meng Ding and Wangda Tan (wangda: rev c59ae4eeb17e52e8fc659b9962d20628719fc621) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/MockResourceCalculatorPlugin.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitorResourceChange.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/MockResourceCalculatorProcessTree.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerManagerWithLCE.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java > Make ContainersMonitor can support change monitoring size of an allocated > container in NM side > -- > > Key: YARN-1643 > URL: https://issues.apache.org/jira/browse/YARN-1643 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Wangda Tan >Assignee: MENG DING > Fix For: 3.0.0 > > Attachments: YARN-1643-YARN-1197.4.patch, > YARN-1643-YARN-1197.5.patch, YARN-1643-YARN-1197.6.patch, > YARN-1643-YARN-1197.7.patch, YARN-1643.1.patch, YARN-1643.2.patch, > YARN-1643.3.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3544) AM logs link missing in the RM UI for a completed app
[ https://issues.apache.org/jira/browse/YARN-3544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905341#comment-14905341 ] Hudson commented on YARN-3544: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #436 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/436/]) CHANGES.txt: Moving YARN-1884, YARN-3171, YARN-3740, YARN-3248, YARN-3544 to 2.6.1 given the backport. (vinodkv: rev f3e5bc67661ebc7e14509f2a267131ae314699dc) * hadoop-yarn-project/CHANGES.txt > AM logs link missing in the RM UI for a completed app > -- > > Key: YARN-3544 > URL: https://issues.apache.org/jira/browse/YARN-3544 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.7.0 >Reporter: Hitesh Shah >Assignee: Xuan Gong >Priority: Blocker > Labels: 2.6.1-candidate > Fix For: 2.6.1, 2.7.1 > > Attachments: Screen Shot 2015-04-27 at 6.24.05 PM.png, > YARN-3544-branch-2.6.1.txt, YARN-3544-branch-2.7-1.2.patch, > YARN-3544-branch-2.7-1.patch, YARN-3544.1.patch > > > AM log links should always be present ( for both running and completed apps). > Likewise node info is also empty. This is usually quite crucial when trying > to debug where an AM was launched and a pointer to which NM's logs to look at > if the AM failed to launch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4171) Resolve findbugs/javac warnings in YARN-1197 branch
[ https://issues.apache.org/jira/browse/YARN-4171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905351#comment-14905351 ] Hudson commented on YARN-4171: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #436 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/436/]) YARN-4171. Fix findbugs warnings in YARN-1197 branch. Contributed by Wangda Tan (wangda: rev b3f6b641dccb0d59df78855e2951d2cae7dff8ad) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeStatusEvent.java > Resolve findbugs/javac warnings in YARN-1197 branch > --- > > Key: YARN-4171 > URL: https://issues.apache.org/jira/browse/YARN-4171 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: Wangda Tan > Fix For: 3.0.0 > > Attachments: YARN-4171-YARN-1197.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1644) RM-NM protocol changes and NodeStatusUpdater implementation to support container resizing
[ https://issues.apache.org/jira/browse/YARN-1644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905345#comment-14905345 ] Hudson commented on YARN-1644: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #436 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/436/]) YARN-1644. RM-NM protocol changes and NodeStatusUpdater implementation to support container resizing. Contributed by Meng Ding (wangda: rev c3dc1af072574f5890a8d43e4d60526951b4b8bc) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/Context.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_service_protos.proto * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/impl/pb/NodeStatusPBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeManagerResync.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodeHeartbeatResponsePBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/TestYarnServerApiClasses.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_protos.proto * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestResourceTrackerOnHA.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/BaseAMRMProxyTest.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/NodeStatus.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/NodeHeartbeatResponse.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/MockResourceManagerFacade.java > RM-NM protocol changes and NodeStatusUpdater implementation to support > container resizing > - > > Key: YARN-1644 > URL: https://issues.apache.org/jira/browse/YARN-1644 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Wangda Tan >Assignee: MENG DING > Fix For: 3.0.0 > > Attachments: YARN-1644-YARN-1197.4.patch, > YARN-1644-YARN-1197.5.patch, YARN-1644-YARN-1197.6.patch, YARN-1644.1.patch, > YARN-1644.2.patch, YARN-1644.3.patch, yarn-1644.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3248) Display count of nodes blacklisted by apps in the web UI
[ https://issues.apache.org/jira/browse/YARN-3248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-3248: -- Fix Version/s: 2.7.2 > Display count of nodes blacklisted by apps in the web UI > > > Key: YARN-3248 > URL: https://issues.apache.org/jira/browse/YARN-3248 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler, resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Labels: 2.6.1-candidate, 2.7.2-candidate > Fix For: 2.6.1, 2.8.0, 2.7.2 > > Attachments: All applications.png, App page.png, Screenshot.jpg, > YARN-3248-branch-2.6.1.txt, YARN-3248-branch-2.7.2.txt, > apache-yarn-3248.0.patch, apache-yarn-3248.1.patch, apache-yarn-3248.2.patch, > apache-yarn-3248.3.patch, apache-yarn-3248.4.patch > > > It would be really useful when debugging app performance and failure issues > to get a count of the nodes blacklisted by individual apps displayed in the > web UI. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4075) [reader REST API] implement support for querying for flows and flow runs
[ https://issues.apache.org/jira/browse/YARN-4075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905385#comment-14905385 ] Li Lu commented on YARN-4075: - Tried the failed UTs on my local machine and could not reproduce the failures. Those failures also appear to be irrelevant. > [reader REST API] implement support for querying for flows and flow runs > > > Key: YARN-4075 > URL: https://issues.apache.org/jira/browse/YARN-4075 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Varun Saxena > Attachments: YARN-4075-YARN-2928.03.patch, > YARN-4075-YARN-2928.04.patch, YARN-4075-YARN-2928.POC.1.patch, > YARN-4075-YARN-2928.POC.2.patch > > > We need to be able to query for flows and flow runs via REST. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3975) WebAppProxyServlet should not redirect to RM page if AHS is enabled
[ https://issues.apache.org/jira/browse/YARN-3975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905142#comment-14905142 ] Hudson commented on YARN-3975: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #1168 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/1168/]) YARN-3975. WebAppProxyServlet should not redirect to RM page if AHS is enabled. Contributed by Mit Desai (jlowe: rev 692d51c09d3668cde47cc297296d095ddfa933a3) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestWebAppProxyServlet.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/AppReportFetcher.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java > WebAppProxyServlet should not redirect to RM page if AHS is enabled > --- > > Key: YARN-3975 > URL: https://issues.apache.org/jira/browse/YARN-3975 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Mit Desai >Assignee: Mit Desai > Fix For: 2.7.2 > > Attachments: YARN-3975.2.b2.patch, YARN-3975.3.patch, > YARN-3975.4.patch, YARN-3975.5.patch, YARN-3975.6.patch, YARN-3975.7.patch, > YARN-3975.8.patch, YARN-3975.9.b2.7.patch, YARN-3975.9.patch, > YARN-3975.9.patch > > > WebAppProxyServlet should be updated to handle the case when the appreport > doesn't have a tracking URL and the Application History Server is eanbled. > As we would have already tried the RM and got the > ApplicationNotFoundException we should not direct the user to the RM app page. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3975) WebAppProxyServlet should not redirect to RM page if AHS is enabled
[ https://issues.apache.org/jira/browse/YARN-3975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905136#comment-14905136 ] Hudson commented on YARN-3975: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #408 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/408/]) YARN-3975. WebAppProxyServlet should not redirect to RM page if AHS is enabled. Contributed by Mit Desai (jlowe: rev 692d51c09d3668cde47cc297296d095ddfa933a3) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestWebAppProxyServlet.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/AppReportFetcher.java > WebAppProxyServlet should not redirect to RM page if AHS is enabled > --- > > Key: YARN-3975 > URL: https://issues.apache.org/jira/browse/YARN-3975 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Mit Desai >Assignee: Mit Desai > Fix For: 2.7.2 > > Attachments: YARN-3975.2.b2.patch, YARN-3975.3.patch, > YARN-3975.4.patch, YARN-3975.5.patch, YARN-3975.6.patch, YARN-3975.7.patch, > YARN-3975.8.patch, YARN-3975.9.b2.7.patch, YARN-3975.9.patch, > YARN-3975.9.patch > > > WebAppProxyServlet should be updated to handle the case when the appreport > doesn't have a tracking URL and the Application History Server is eanbled. > As we would have already tried the RM and got the > ApplicationNotFoundException we should not direct the user to the RM app page. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4095) Avoid sharing AllocatorPerContext object in LocalDirAllocator between ShuffleHandler and LocalDirsHandlerService.
[
https://issues.apache.org/jira/browse/YARN-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905135#comment-14905135
]
Hudson commented on YARN-4095:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #408 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/408/])
YARN-4095. Avoid sharing AllocatorPerContext object in LocalDirAllocator
between ShuffleHandler and LocalDirsHandlerService. Contributed by Zhihai Xu
(jlowe: rev c890c51a916894a985439497b8a44e8eee82d762)
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLocalDirsHandlerService.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LocalDirsHandlerService.java
> Avoid sharing AllocatorPerContext object in LocalDirAllocator between
> ShuffleHandler and LocalDirsHandlerService.
> -
>
> Key: YARN-4095
> URL: https://issues.apache.org/jira/browse/YARN-4095
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
>Reporter: zhihai xu
>Assignee: zhihai xu
> Fix For: 2.8.0
>
> Attachments: YARN-4095.000.patch, YARN-4095.001.patch
>
>
> Currently {{ShuffleHandler}} and {{LocalDirsHandlerService}} share
> {{AllocatorPerContext}} object in {{LocalDirAllocator}} for configuration
> {{NM_LOCAL_DIRS}} because {{AllocatorPerContext}} are stored in a static
> TreeMap with configuration name as key
> {code}
> private static Map contexts =
> new TreeMap();
> {code}
> {{LocalDirsHandlerService}} and {{ShuffleHandler}} both create a
> {{LocalDirAllocator}} using {{NM_LOCAL_DIRS}}. Even they don't use the same
> {{Configuration}} object, but they will use the same {{AllocatorPerContext}}
> object. Also {{LocalDirsHandlerService}} may change {{NM_LOCAL_DIRS}} value
> in its {{Configuration}} object to exclude full and bad local dirs,
> {{ShuffleHandler}} always uses the original {{NM_LOCAL_DIRS}} value in its
> {{Configuration}} object. So every time {{AllocatorPerContext#confChanged}}
> is called by {{ShuffleHandler}} after {{LocalDirsHandlerService}},
> {{AllocatorPerContext}} need be reinitialized because {{NM_LOCAL_DIRS}} value
> is changed. This will cause some overhead.
> {code}
> String newLocalDirs = conf.get(contextCfgItemName);
> if (!newLocalDirs.equals(savedLocalDirs)) {
> {code}
> So it will be a good improvement to not share the same
> {{AllocatorPerContext}} instance between {{ShuffleHandler}} and
> {{LocalDirsHandlerService}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4165) An outstanding container request makes all nodes to be reserved causing all jobs pending
[ https://issues.apache.org/jira/browse/YARN-4165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905153#comment-14905153 ] Wangda Tan commented on YARN-4165: -- [~mding], IIRC, re-reservation >= 1 only if it allowed to reserve again, if shouldAllocOrReserveNewContainer check failed, scheduler won't allow application to re-reserve the container. > An outstanding container request makes all nodes to be reserved causing all > jobs pending > > > Key: YARN-4165 > URL: https://issues.apache.org/jira/browse/YARN-4165 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler, resourcemanager, scheduler >Affects Versions: 2.7.1 >Reporter: Weiwei Yang >Assignee: Weiwei Yang > > We have a long running service in YARN, it has a outstanding container > request that YARN cannot satisfy (require more memory that nodemanager can > supply). Then YARN reserves all nodes for this application, when I submit > other jobs (require relative small memory that nodemanager can supply), all > jobs are pending because YARN skips scheduling containers on the nodes that > have been reserved. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4075) [reader REST API] implement support for querying for flows and flow runs
[
https://issues.apache.org/jira/browse/YARN-4075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905173#comment-14905173
]
Hadoop QA commented on YARN-4075:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 17m 13s | Findbugs (version ) appears to
be broken on YARN-2928. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 3 new or modified test files. |
| {color:green}+1{color} | javac | 8m 5s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 16s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 22s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 4s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 38s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 31s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 0m 25s | Tests passed in
hadoop-yarn-api. |
| {color:red}-1{color} | yarn tests | 19m 24s | Tests failed in
hadoop-yarn-server-timelineservice. |
| | | 62m 1s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests |
hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorage |
| |
hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesFlowRun |
| |
hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowActivity |
| |
hadoop.yarn.server.timelineservice.storage.TestPhoenixOfflineAggregationWriterImpl
|
| | hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRun |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12761945/YARN-4075-YARN-2928.04.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | YARN-2928 / 2e7e0f0 |
| hadoop-yarn-api test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9247/artifact/patchprocess/testrun_hadoop-yarn-api.txt
|
| hadoop-yarn-server-timelineservice test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9247/artifact/patchprocess/testrun_hadoop-yarn-server-timelineservice.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/9247/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/9247/console |
This message was automatically generated.
> [reader REST API] implement support for querying for flows and flow runs
>
>
> Key: YARN-4075
> URL: https://issues.apache.org/jira/browse/YARN-4075
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Varun Saxena
> Attachments: YARN-4075-YARN-2928.03.patch,
> YARN-4075-YARN-2928.04.patch, YARN-4075-YARN-2928.POC.1.patch,
> YARN-4075-YARN-2928.POC.2.patch
>
>
> We need to be able to query for flows and flow runs via REST.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
