[jira] [Created] (YARN-1183) MiniYARNCluster shutdown takes several minutes intermittently

Andrey Klochkov created YARN-1183: - Summary: MiniYARNCluster shutdown takes several minutes intermittently Key: YARN-1183 URL: https://issues.apache.org/jira/browse/YARN-1183 Project: Hadoop YARN

[jira] [Updated] (YARN-1183) MiniYARNCluster shutdown takes several minutes intermittently

[ https://issues.apache.org/jira/browse/YARN-1183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Klochkov updated YARN-1183: -- Attachment: YARN-1183.patch Attaching a patch which modifies MiniYARNCluter so it waits until

[jira] [Commented] (YARN-1183) MiniYARNCluster shutdown takes several minutes intermittently

[ https://issues.apache.org/jira/browse/YARN-1183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13765735#comment-13765735 ] Andrey Klochkov commented on YARN-1183: --- Yep, you're right about trackingUrl being

[jira] [Updated] (YARN-1183) MiniYARNCluster shutdown takes several minutes intermittently

[ https://issues.apache.org/jira/browse/YARN-1183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Klochkov updated YARN-1183: -- Attachment: YARN-1183--n2.patch Attaching an updated patch. Updated the name of the wait

[jira] [Commented] (YARN-1183) MiniYARNCluster shutdown takes several minutes intermittently

[ https://issues.apache.org/jira/browse/YARN-1183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13766237#comment-13766237 ] Andrey Klochkov commented on YARN-1183: --- bq. MiniYARNCluster is used by several

[jira] [Updated] (YARN-1183) MiniYARNCluster shutdown takes several minutes intermittently

[ https://issues.apache.org/jira/browse/YARN-1183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Klochkov updated YARN-1183: -- Attachment: YARN-1183--n4.patch MiniYARNCluster shutdown takes several minutes

[jira] [Updated] (YARN-1183) MiniYARNCluster shutdown takes several minutes intermittently

[ https://issues.apache.org/jira/browse/YARN-1183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Klochkov updated YARN-1183: -- Attachment: YARN-1183--n3.patch Attaching an updated patch MiniYARNCluster

[jira] [Updated] (YARN-261) Ability to kill AM attempts

[ https://issues.apache.org/jira/browse/YARN-261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Klochkov updated YARN-261: - Attachment: YARN-261.patch Attaching a patch implementing application restart feature.

[jira] [Updated] (YARN-261) Ability to kill AM attempts

[ https://issues.apache.org/jira/browse/YARN-261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Klochkov updated YARN-261: - Attachment: YARN-261--n2.patch Updated the patch to prevent the exception in the log when

[jira] [Updated] (YARN-261) Ability to kill AM attempts

[ https://issues.apache.org/jira/browse/YARN-261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Klochkov updated YARN-261: - Attachment: YARN-261--n3.patch Fixing Javadoc warning. Ability to kill AM

[jira] [Commented] (YARN-261) Ability to kill AM attempts

[ https://issues.apache.org/jira/browse/YARN-261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13769772#comment-13769772 ] Andrey Klochkov commented on YARN-261: -- On a closer look it is indeed possible to reuse

[jira] [Assigned] (YARN-415) Capture memory utilization at the app-level for chargeback

[ https://issues.apache.org/jira/browse/YARN-415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Klochkov reassigned YARN-415: Assignee: Andrey Klochkov Capture memory utilization at the app-level for chargeback

[jira] [Updated] (YARN-261) Ability to kill AM attempts

[ https://issues.apache.org/jira/browse/YARN-261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Klochkov updated YARN-261: - Attachment: YARN-261--n4.patch This patch is implementing this with less changes in state machines

[jira] [Commented] (YARN-261) Ability to kill AM attempts

[ https://issues.apache.org/jira/browse/YARN-261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770160#comment-13770160 ] Andrey Klochkov commented on YARN-261: -- Seems that the reported test failures are all

[jira] [Updated] (YARN-415) Capture memory utilization at the app-level for chargeback

[ https://issues.apache.org/jira/browse/YARN-415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Klochkov updated YARN-415: - Attachment: YARN-415.patch The patch exposes MB-seconds and CPU-seconds through CLI, REST API and

[jira] [Updated] (YARN-415) Capture memory utilization at the app-level for chargeback

[ https://issues.apache.org/jira/browse/YARN-415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Klochkov updated YARN-415: - Attachment: YARN-415--n3.patch Updating the patch with fixes in tests. Capture

[jira] [Updated] (YARN-415) Capture memory utilization at the app-level for chargeback

[ https://issues.apache.org/jira/browse/YARN-415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Klochkov updated YARN-415: - Attachment: YARN-415--n2.patch Updating the patch with fixes of findbugs warnings on multithreaded

[jira] [Commented] (YARN-415) Capture memory utilization at the app-level for chargeback

[ https://issues.apache.org/jira/browse/YARN-415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13773419#comment-13773419 ] Andrey Klochkov commented on YARN-415: -- The proposed implementation uses events fired

[jira] [Commented] (YARN-415) Capture aggregate memory allocation at the app-level for chargeback

[ https://issues.apache.org/jira/browse/YARN-415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14130368#comment-14130368 ] Andrey Klochkov commented on YARN-415: -- [~eepayne], congratulations and thanks for this

[jira] [Updated] (YARN-415) Capture aggregate memory allocation at the app-level for chargeback

[ https://issues.apache.org/jira/browse/YARN-415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Klochkov updated YARN-415: - Assignee: Eric Payne (was: Andrey Klochkov) Capture aggregate memory allocation at the app-level

[jira] [Commented] (YARN-415) Capture memory utilization at the app-level for chargeback

[ https://issues.apache.org/jira/browse/YARN-415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14017225#comment-14017225 ] Andrey Klochkov commented on YARN-415: -- [~eepayne], thanks a lot for finishing this.

[jira] [Updated] (YARN-445) Ability to signal containers

[ https://issues.apache.org/jira/browse/YARN-445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Klochkov updated YARN-445: - Attachment: YARN-445.patch Attaching a patch that provides the simplest implementation: -

[jira] [Commented] (YARN-445) Ability to signal containers

[ https://issues.apache.org/jira/browse/YARN-445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13784199#comment-13784199 ] Andrey Klochkov commented on YARN-445: -- Bikas, on Windows JVM prints full thread dump

[jira] [Updated] (YARN-445) Ability to signal containers

[ https://issues.apache.org/jira/browse/YARN-445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Klochkov updated YARN-445: - Attachment: YARN-445--n2.patch fixing javadoc warnings and the failed test Ability to signal

[jira] [Commented] (YARN-445) Ability to signal containers

[ https://issues.apache.org/jira/browse/YARN-445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13784292#comment-13784292 ] Andrey Klochkov commented on YARN-445: -- As I understand this Findbugs warning should be

[jira] [Updated] (YARN-465) fix coverage org.apache.hadoop.yarn.server.webproxy

[ https://issues.apache.org/jira/browse/YARN-465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Klochkov updated YARN-465: - Attachment: YARN-465-branch-2--n3.patch YARN-465-trunk--n3.patch Attaching updated

[jira] [Commented] (YARN-465) fix coverage org.apache.hadoop.yarn.server.webproxy

[ https://issues.apache.org/jira/browse/YARN-465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13785381#comment-13785381 ] Andrey Klochkov commented on YARN-465: -- The robot failed when testing the branch-2

[jira] [Assigned] (YARN-677) Increase coverage to FairScheduler

[ https://issues.apache.org/jira/browse/YARN-677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Klochkov reassigned YARN-677: Assignee: Andrey Klochkov Increase coverage to FairScheduler

[jira] [Commented] (YARN-677) Increase coverage to FairScheduler

[ https://issues.apache.org/jira/browse/YARN-677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13785407#comment-13785407 ] Andrey Klochkov commented on YARN-677: -- I looked at the difference in coverage before

[jira] [Assigned] (YARN-465) fix coverage org.apache.hadoop.yarn.server.webproxy

[ https://issues.apache.org/jira/browse/YARN-465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Klochkov reassigned YARN-465: Assignee: Andrey Klochkov (was: Aleksey Gorshkov) fix coverage

[jira] [Commented] (YARN-445) Ability to signal containers

[ https://issues.apache.org/jira/browse/YARN-445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13786506#comment-13786506 ] Andrey Klochkov commented on YARN-445: -- The large diffs in the tests are not due to

[jira] [Commented] (YARN-445) Ability to signal containers

[ https://issues.apache.org/jira/browse/YARN-445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13786572#comment-13786572 ] Andrey Klochkov commented on YARN-445: -- Steve, the current implementation will send the

[jira] [Assigned] (YARN-1183) MiniYARNCluster shutdown takes several minutes intermittently

[ https://issues.apache.org/jira/browse/YARN-1183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Klochkov reassigned YARN-1183: - Assignee: Andrey Klochkov MiniYARNCluster shutdown takes several minutes intermittently

[jira] [Updated] (YARN-465) fix coverage org.apache.hadoop.yarn.server.webproxy

[ https://issues.apache.org/jira/browse/YARN-465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Klochkov updated YARN-465: - Attachment: YARN-465-trunk--n4.patch Ravi, this is not my patch so please keep in mind I'm digging

[jira] [Updated] (YARN-465) fix coverage org.apache.hadoop.yarn.server.webproxy

[ https://issues.apache.org/jira/browse/YARN-465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Klochkov updated YARN-465: - Attachment: YARN-465-branch-2--n4.patch Attaching the updated patch for branch-2 fix coverage

[jira] [Updated] (YARN-415) Capture memory utilization at the app-level for chargeback

[ https://issues.apache.org/jira/browse/YARN-415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Klochkov updated YARN-415: - Attachment: YARN-415--n4.patch Jason, thanks for the thorough review. Attaching the patch with

[jira] [Updated] (YARN-445) Ability to signal containers

[ https://issues.apache.org/jira/browse/YARN-445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Klochkov updated YARN-445: - Attachment: YARN-445--n3.patch Attaching the patch that marks all new interfaces/methods as

[jira] [Updated] (YARN-465) fix coverage org.apache.hadoop.yarn.server.webproxy

[ https://issues.apache.org/jira/browse/YARN-465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Klochkov updated YARN-465: - Attachment: YARN-465-trunk--n5.patch YARN-465-branch-2--n5.patch Ravi, The main

[jira] [Commented] (YARN-465) fix coverage org.apache.hadoop.yarn.server.webproxy

[ https://issues.apache.org/jira/browse/YARN-465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13788623#comment-13788623 ] Andrey Klochkov commented on YARN-465: -- Actually the only difference is in how

[jira] [Updated] (YARN-415) Capture memory utilization at the app-level for chargeback

[ https://issues.apache.org/jira/browse/YARN-415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Klochkov updated YARN-415: - Attachment: YARN-415--n5.patch Fixing the failed test. Capture memory utilization at the

[jira] [Updated] (YARN-415) Capture memory utilization at the app-level for chargeback

[ https://issues.apache.org/jira/browse/YARN-415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Klochkov updated YARN-415: - Attachment: YARN-415--n6.patch With the 1st option it's not clear how to implement a protection

[jira] [Updated] (YARN-445) Ability to signal containers

[ https://issues.apache.org/jira/browse/YARN-445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Klochkov updated YARN-445: - Attachment: YARN-445--n4.patch Attaching a patch which implements Ctrl+C and uses it instead of

[jira] [Updated] (YARN-261) Ability to kill AM attempts

[ https://issues.apache.org/jira/browse/YARN-261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Klochkov updated YARN-261: - Attachment: YARN-261--n5.patch Jason, thanks for review. All your points make sense for me.

[jira] [Commented] (YARN-445) Ability to signal containers

[ https://issues.apache.org/jira/browse/YARN-445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13795753#comment-13795753 ] Andrey Klochkov commented on YARN-445: -- Vinod, Accepting a mapping of arbitrary

[jira] [Resolved] (YARN-677) Increase coverage to FairScheduler

[ https://issues.apache.org/jira/browse/YARN-677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Klochkov resolved YARN-677. -- Resolution: Won't Fix Increase coverage to FairScheduler --

[jira] [Updated] (YARN-415) Capture memory utilization at the app-level for chargeback

[ https://issues.apache.org/jira/browse/YARN-415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Klochkov updated YARN-415: - Attachment: YARN-415--n7.patch Thanks Jason. Attaching a fixed patch. Capture memory utilization

[jira] [Updated] (YARN-415) Capture memory utilization at the app-level for chargeback

[ https://issues.apache.org/jira/browse/YARN-415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Klochkov updated YARN-415: - Attachment: YARN-415--n8.patch Adding changes in REST API docs to the patch. Capture memory

[jira] [Commented] (YARN-415) Capture memory utilization at the app-level for chargeback

[ https://issues.apache.org/jira/browse/YARN-415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13799363#comment-13799363 ] Andrey Klochkov commented on YARN-415: -- Arun, the idea is to have the stats being

[jira] [Updated] (YARN-261) Ability to kill AM attempts

[ https://issues.apache.org/jira/browse/YARN-261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Klochkov updated YARN-261: - Attachment: YARN-261--n6.patch Jason, makes sense. See the updated patch. Ability to kill AM

[jira] [Commented] (YARN-415) Capture memory utilization at the app-level for chargeback

[ https://issues.apache.org/jira/browse/YARN-415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13801286#comment-13801286 ] Andrey Klochkov commented on YARN-415: -- IMO it makes sense to move this tracking into

[jira] [Commented] (YARN-415) Capture memory utilization at the app-level for chargeback

[ https://issues.apache.org/jira/browse/YARN-415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13801308#comment-13801308 ] Andrey Klochkov commented on YARN-415: -- Adding YARN-1335 as a dependency. Capture

[jira] [Commented] (YARN-415) Capture memory utilization at the app-level for chargeback

[ https://issues.apache.org/jira/browse/YARN-415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13801332#comment-13801332 ] Andrey Klochkov commented on YARN-415: -- On a second thought, scheduler does not seem

[jira] [Commented] (YARN-1183) MiniYARNCluster shutdown takes several minutes intermittently

[ https://issues.apache.org/jira/browse/YARN-1183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13802033#comment-13802033 ] Andrey Klochkov commented on YARN-1183: --- I don't think concurrency level can make any

[jira] [Updated] (YARN-1183) MiniYARNCluster shutdown takes several minutes intermittently

[ https://issues.apache.org/jira/browse/YARN-1183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Klochkov updated YARN-1183: -- Attachment: YARN-1183--n5.patch Attaching an updated patch. MiniYARNCluster shutdown takes

[jira] [Commented] (YARN-1183) MiniYARNCluster shutdown takes several minutes intermittently

[ https://issues.apache.org/jira/browse/YARN-1183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13802258#comment-13802258 ] Andrey Klochkov commented on YARN-1183: --- Jonathan, the issue occurred when I just run

[jira] [Updated] (YARN-415) Capture memory utilization at the app-level for chargeback

[ https://issues.apache.org/jira/browse/YARN-415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Klochkov updated YARN-415: - Attachment: YARN-415--n9.patch Updated the patch moving tracking logic into Scheduler: -

[jira] [Commented] (YARN-415) Capture memory utilization at the app-level for chargeback

[ https://issues.apache.org/jira/browse/YARN-415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13805804#comment-13805804 ] Andrey Klochkov commented on YARN-415: -- This scheme has a downside that the stats would

[jira] [Updated] (YARN-415) Capture memory utilization at the app-level for chargeback

[ https://issues.apache.org/jira/browse/YARN-415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Klochkov updated YARN-415: - Attachment: YARN-415--n10.patch Updating the patch: - got rid of the runningContainers map -

[jira] [Updated] (YARN-261) Ability to kill AM attempts

[ https://issues.apache.org/jira/browse/YARN-261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Klochkov updated YARN-261: - Attachment: YARN-261--n7.patch Uploading a patch rebased after YARN-891 and with fixes according

[jira] [Commented] (YARN-415) Capture memory utilization at the app-level for chargeback

[ https://issues.apache.org/jira/browse/YARN-415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13878308#comment-13878308 ] Andrey Klochkov commented on YARN-415: -- I'm updating the patch to make it applicable to

[jira] [Assigned] (YARN-445) Ability to signal containers

[ https://issues.apache.org/jira/browse/YARN-445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Klochkov reassigned YARN-445: Assignee: (was: Andrey Klochkov) After I submitted the patch for this there were

[jira] [Commented] (YARN-261) Ability to kill AM attempts

[ https://issues.apache.org/jira/browse/YARN-261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14731080#comment-14731080 ] Andrey Klochkov commented on YARN-261: -- [~rohithsharma], please feel free to reassign to yourself. I

[jira] [Assigned] (YARN-261) Ability to kill AM attempts

[ https://issues.apache.org/jira/browse/YARN-261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Klochkov reassigned YARN-261: Assignee: (was: Andrey Klochkov) > Ability to kill AM attempts >