[jira] [Updated] (YARN-2280) Resource manager web service fields are not accessible
[ https://issues.apache.org/jira/browse/YARN-2280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Horvath updated YARN-2280: Attachment: (was: YARN-2280.patch) Resource manager web service fields are not accessible -- Key: YARN-2280 URL: https://issues.apache.org/jira/browse/YARN-2280 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.4.0, 2.4.1 Reporter: Krisztian Horvath Assignee: Krisztian Horvath Priority: Trivial Fix For: 2.6.0 Attachments: YARN-2280.patch Using the resource manager's rest api (org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices) some rest call returns a class where the fields after the unmarshal cannot be accessible. For example SchedulerTypeInfo - schedulerInfo. Using the same classes on client side these fields only accessible via reflection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2280) Resource manager web service fields are not accessible
[ https://issues.apache.org/jira/browse/YARN-2280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Horvath updated YARN-2280: Attachment: YARN-2280.patch Resource manager web service fields are not accessible -- Key: YARN-2280 URL: https://issues.apache.org/jira/browse/YARN-2280 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.4.0, 2.4.1 Reporter: Krisztian Horvath Assignee: Krisztian Horvath Priority: Trivial Fix For: 2.6.0 Attachments: YARN-2280.patch Using the resource manager's rest api (org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices) some rest call returns a class where the fields after the unmarshal cannot be accessible. For example SchedulerTypeInfo - schedulerInfo. Using the same classes on client side these fields only accessible via reflection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2280) Resource manager web service fields are not accessible
[ https://issues.apache.org/jira/browse/YARN-2280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Horvath updated YARN-2280: Priority: Trivial (was: Minor) Resource manager web service fields are not accessible -- Key: YARN-2280 URL: https://issues.apache.org/jira/browse/YARN-2280 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.4.0, 2.4.1 Reporter: Krisztian Horvath Assignee: Krisztian Horvath Priority: Trivial Fix For: 2.6.0 Attachments: YARN-2280.patch Using the resource manager's rest api (org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices) some rest call returns a class where the fields after the unmarshal cannot be accessible. For example SchedulerTypeInfo - schedulerInfo. Using the same classes on client side these fields only accessible via reflection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1964) Create Docker analog of the LinuxContainerExecutor in YARN
[ https://issues.apache.org/jira/browse/YARN-1964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14131581#comment-14131581 ] Krisztian Horvath commented on YARN-1964: - Will containers be able to communicate with each other, e.g with slider I can run HBase inside containers. Create Docker analog of the LinuxContainerExecutor in YARN -- Key: YARN-1964 URL: https://issues.apache.org/jira/browse/YARN-1964 Project: Hadoop YARN Issue Type: New Feature Affects Versions: 2.2.0 Reporter: Arun C Murthy Assignee: Abin Shahab Attachments: yarn-1964-branch-2.2.0-docker.patch, yarn-1964-branch-2.2.0-docker.patch, yarn-1964-docker.patch, yarn-1964-docker.patch, yarn-1964-docker.patch, yarn-1964-docker.patch, yarn-1964-docker.patch Docker (https://www.docker.io/) is, increasingly, a very popular container technology. In context of YARN, the support for Docker will provide a very elegant solution to allow applications to *package* their software into a Docker container (entire Linux file system incl. custom versions of perl, python etc.) and use it as a blueprint to launch all their YARN containers with requisite software environment. This provides both consistency (all YARN containers will have the same software environment) and isolation (no interference with whatever is installed on the physical machine). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2280) Resource manager web service fields are not accessible
[ https://issues.apache.org/jira/browse/YARN-2280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Horvath updated YARN-2280: Attachment: (was: YARN-2280.patch) Resource manager web service fields are not accessible -- Key: YARN-2280 URL: https://issues.apache.org/jira/browse/YARN-2280 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.4.0, 2.4.1 Reporter: Krisztian Horvath Assignee: Krisztian Horvath Priority: Minor Fix For: 2.6.0 Attachments: YARN-2280.patch Using the resource manager's rest api (org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices) some rest call returns a class where the fields after the unmarshal cannot be accessible. For example SchedulerTypeInfo - schedulerInfo. Using the same classes on client side these fields only accessible via reflection. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-2280) Resource manager web service fields are not accessible
[ https://issues.apache.org/jira/browse/YARN-2280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Horvath updated YARN-2280: Attachment: YARN-2280.patch Resource manager web service fields are not accessible -- Key: YARN-2280 URL: https://issues.apache.org/jira/browse/YARN-2280 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.4.0, 2.4.1 Reporter: Krisztian Horvath Assignee: Krisztian Horvath Priority: Minor Fix For: 2.6.0 Attachments: YARN-2280.patch Using the resource manager's rest api (org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices) some rest call returns a class where the fields after the unmarshal cannot be accessible. For example SchedulerTypeInfo - schedulerInfo. Using the same classes on client side these fields only accessible via reflection. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2248) Capacity Scheduler changes for moving apps between queues
[ https://issues.apache.org/jira/browse/YARN-2248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14099695#comment-14099695 ] Krisztian Horvath commented on YARN-2248: - We will and let you know if anything seems abnormal. Capacity Scheduler changes for moving apps between queues - Key: YARN-2248 URL: https://issues.apache.org/jira/browse/YARN-2248 Project: Hadoop YARN Issue Type: Sub-task Components: capacityscheduler Reporter: Janos Matyas Assignee: Janos Matyas Fix For: 2.6.0 Attachments: YARN-2248-1.patch, YARN-2248-2.patch, YARN-2248-3.patch We would like to have the capability (same as the Fair Scheduler has) to move applications between queues. We have made a baseline implementation and tests to start with - and we would like the community to review, come up with suggestions and finally have this contributed. The current implementation is available for 2.4.1 - so the first thing is that we'd need to identify the target version as there are differences between 2.4.* and 3.* interfaces. The story behind is available at http://blog.sequenceiq.com/blog/2014/07/02/move-applications-between-queues/ and the baseline implementation and test at: https://github.com/sequenceiq/hadoop-common/blob/branch-2.4.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/a/ExtendedCapacityScheduler.java#L924 https://github.com/sequenceiq/hadoop-common/blob/branch-2.4.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/a/TestExtendedCapacitySchedulerAppMove.java -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2248) Capacity Scheduler changes for moving apps between queues
[ https://issues.apache.org/jira/browse/YARN-2248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14088185#comment-14088185 ] Krisztian Horvath commented on YARN-2248: - Hi, As long as we don't break the functionality we can merge them and try to take the best out of them, so yes. Have you tried your patch with the queue metrics test, yet? Capacity Scheduler changes for moving apps between queues - Key: YARN-2248 URL: https://issues.apache.org/jira/browse/YARN-2248 Project: Hadoop YARN Issue Type: Sub-task Components: capacityscheduler Reporter: Janos Matyas Assignee: Janos Matyas Fix For: 2.6.0 Attachments: YARN-2248-1.patch, YARN-2248-2.patch, YARN-2248-3.patch We would like to have the capability (same as the Fair Scheduler has) to move applications between queues. We have made a baseline implementation and tests to start with - and we would like the community to review, come up with suggestions and finally have this contributed. The current implementation is available for 2.4.1 - so the first thing is that we'd need to identify the target version as there are differences between 2.4.* and 3.* interfaces. The story behind is available at http://blog.sequenceiq.com/blog/2014/07/02/move-applications-between-queues/ and the baseline implementation and test at: https://github.com/sequenceiq/hadoop-common/blob/branch-2.4.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/a/ExtendedCapacityScheduler.java#L924 https://github.com/sequenceiq/hadoop-common/blob/branch-2.4.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/a/TestExtendedCapacitySchedulerAppMove.java -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2248) Capacity Scheduler changes for moving apps between queues
[ https://issues.apache.org/jira/browse/YARN-2248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14088832#comment-14088832 ] Krisztian Horvath commented on YARN-2248: - Is there a change we can get this committed in 2.6.0? Capacity Scheduler changes for moving apps between queues - Key: YARN-2248 URL: https://issues.apache.org/jira/browse/YARN-2248 Project: Hadoop YARN Issue Type: Sub-task Components: capacityscheduler Reporter: Janos Matyas Assignee: Janos Matyas Fix For: 2.6.0 Attachments: YARN-2248-1.patch, YARN-2248-2.patch, YARN-2248-3.patch We would like to have the capability (same as the Fair Scheduler has) to move applications between queues. We have made a baseline implementation and tests to start with - and we would like the community to review, come up with suggestions and finally have this contributed. The current implementation is available for 2.4.1 - so the first thing is that we'd need to identify the target version as there are differences between 2.4.* and 3.* interfaces. The story behind is available at http://blog.sequenceiq.com/blog/2014/07/02/move-applications-between-queues/ and the baseline implementation and test at: https://github.com/sequenceiq/hadoop-common/blob/branch-2.4.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/a/ExtendedCapacityScheduler.java#L924 https://github.com/sequenceiq/hadoop-common/blob/branch-2.4.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/a/TestExtendedCapacitySchedulerAppMove.java -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-2280) Resource manager web service fields are not accessible
[ https://issues.apache.org/jira/browse/YARN-2280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Horvath updated YARN-2280: Attachment: YARN-2280.patch Resource manager web service fields are not accessible -- Key: YARN-2280 URL: https://issues.apache.org/jira/browse/YARN-2280 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.4.0, 2.4.1 Reporter: Krisztian Horvath Assignee: Krisztian Horvath Priority: Minor Fix For: 2.5.0 Attachments: YARN-2280.patch Using the resource manager's rest api (org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices) some rest call returns a class where the fields after the unmarshal cannot be accessible. For example SchedulerTypeInfo - schedulerInfo. Using the same classes on client side these fields only accessible via reflection. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-2280) Resource manager web service fields are not accessible
[ https://issues.apache.org/jira/browse/YARN-2280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Horvath updated YARN-2280: Attachment: (was: YARN-2280.patch) Resource manager web service fields are not accessible -- Key: YARN-2280 URL: https://issues.apache.org/jira/browse/YARN-2280 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.4.0, 2.4.1 Reporter: Krisztian Horvath Assignee: Krisztian Horvath Priority: Minor Fix For: 2.5.0 Attachments: YARN-2280.patch Using the resource manager's rest api (org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices) some rest call returns a class where the fields after the unmarshal cannot be accessible. For example SchedulerTypeInfo - schedulerInfo. Using the same classes on client side these fields only accessible via reflection. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-2248) Capacity Scheduler changes for moving apps between queues
[ https://issues.apache.org/jira/browse/YARN-2248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Horvath updated YARN-2248: Fix Version/s: 2.6.0 Capacity Scheduler changes for moving apps between queues - Key: YARN-2248 URL: https://issues.apache.org/jira/browse/YARN-2248 Project: Hadoop YARN Issue Type: Sub-task Components: capacityscheduler Reporter: Janos Matyas Assignee: Janos Matyas Fix For: 2.6.0 Attachments: YARN-2248-1.patch, YARN-2248-2.patch, YARN-2248-3.patch We would like to have the capability (same as the Fair Scheduler has) to move applications between queues. We have made a baseline implementation and tests to start with - and we would like the community to review, come up with suggestions and finally have this contributed. The current implementation is available for 2.4.1 - so the first thing is that we'd need to identify the target version as there are differences between 2.4.* and 3.* interfaces. The story behind is available at http://blog.sequenceiq.com/blog/2014/07/02/move-applications-between-queues/ and the baseline implementation and test at: https://github.com/sequenceiq/hadoop-common/blob/branch-2.4.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/a/ExtendedCapacityScheduler.java#L924 https://github.com/sequenceiq/hadoop-common/blob/branch-2.4.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/a/TestExtendedCapacitySchedulerAppMove.java -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (YARN-2280) Resource manager web service fields are not accessible
Krisztian Horvath created YARN-2280: --- Summary: Resource manager web service fields are not accessible Key: YARN-2280 URL: https://issues.apache.org/jira/browse/YARN-2280 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.4.1, 2.4.0 Reporter: Krisztian Horvath Assignee: Krisztian Horvath Priority: Minor Fix For: 2.5.0 Using the resource manager's rest api (org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices) some rest call returns a class where the fields after the unmarshal cannot be accessible. For example SchedulerTypeInfo - schedulerInfo. Using the same classes on client side these fields only accessible via reflection. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-2280) Resource manager web service fields are not accessible
[ https://issues.apache.org/jira/browse/YARN-2280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Horvath updated YARN-2280: Attachment: YARN-2280.patch Resource manager web service fields are not accessible -- Key: YARN-2280 URL: https://issues.apache.org/jira/browse/YARN-2280 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.4.0, 2.4.1 Reporter: Krisztian Horvath Assignee: Krisztian Horvath Priority: Minor Fix For: 2.5.0 Attachments: YARN-2280.patch Using the resource manager's rest api (org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices) some rest call returns a class where the fields after the unmarshal cannot be accessible. For example SchedulerTypeInfo - schedulerInfo. Using the same classes on client side these fields only accessible via reflection. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2280) Resource manager web service fields are not accessible
[ https://issues.apache.org/jira/browse/YARN-2280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14058992#comment-14058992 ] Krisztian Horvath commented on YARN-2280: - Test failures are not relevant to this patch, and no new tests were added because there is no point to test those 2 getters. Resource manager web service fields are not accessible -- Key: YARN-2280 URL: https://issues.apache.org/jira/browse/YARN-2280 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.4.0, 2.4.1 Reporter: Krisztian Horvath Assignee: Krisztian Horvath Priority: Minor Fix For: 2.5.0 Attachments: YARN-2280.patch Using the resource manager's rest api (org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices) some rest call returns a class where the fields after the unmarshal cannot be accessible. For example SchedulerTypeInfo - schedulerInfo. Using the same classes on client side these fields only accessible via reflection. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-2248) Capacity Scheduler changes for moving apps between queues
[ https://issues.apache.org/jira/browse/YARN-2248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Horvath updated YARN-2248: Attachment: YARN-2248-3.patch The order of the live container resource release is important as the user can go under Resource usage 0, 0. To avoid it changed the order. More metrics test added. Capacity Scheduler changes for moving apps between queues - Key: YARN-2248 URL: https://issues.apache.org/jira/browse/YARN-2248 Project: Hadoop YARN Issue Type: Sub-task Components: capacityscheduler Reporter: Janos Matyas Assignee: Janos Matyas Attachments: YARN-2248-1.patch, YARN-2248-2.patch, YARN-2248-3.patch We would like to have the capability (same as the Fair Scheduler has) to move applications between queues. We have made a baseline implementation and tests to start with - and we would like the community to review, come up with suggestions and finally have this contributed. The current implementation is available for 2.4.1 - so the first thing is that we'd need to identify the target version as there are differences between 2.4.* and 3.* interfaces. The story behind is available at http://blog.sequenceiq.com/blog/2014/07/02/move-applications-between-queues/ and the baseline implementation and test at: https://github.com/sequenceiq/hadoop-common/blob/branch-2.4.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/a/ExtendedCapacityScheduler.java#L924 https://github.com/sequenceiq/hadoop-common/blob/branch-2.4.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/a/TestExtendedCapacitySchedulerAppMove.java -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-2248) Capacity Scheduler changes for moving apps between queues
[ https://issues.apache.org/jira/browse/YARN-2248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Horvath updated YARN-2248: Attachment: YARN-2248-2.patch Capacity Scheduler changes for moving apps between queues - Key: YARN-2248 URL: https://issues.apache.org/jira/browse/YARN-2248 Project: Hadoop YARN Issue Type: Sub-task Components: capacityscheduler Reporter: Janos Matyas Assignee: Janos Matyas Attachments: YARN-2248-1.patch, YARN-2248-2.patch We would like to have the capability (same as the Fair Scheduler has) to move applications between queues. We have made a baseline implementation and tests to start with - and we would like the community to review, come up with suggestions and finally have this contributed. The current implementation is available for 2.4.1 - so the first thing is that we'd need to identify the target version as there are differences between 2.4.* and 3.* interfaces. The story behind is available at http://blog.sequenceiq.com/blog/2014/07/02/move-applications-between-queues/ and the baseline implementation and test at: https://github.com/sequenceiq/hadoop-common/blob/branch-2.4.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/a/ExtendedCapacityScheduler.java#L924 https://github.com/sequenceiq/hadoop-common/blob/branch-2.4.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/a/TestExtendedCapacitySchedulerAppMove.java -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2248) Capacity Scheduler changes for moving apps between queues
[ https://issues.apache.org/jira/browse/YARN-2248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14053750#comment-14053750 ] Krisztian Horvath commented on YARN-2248: - Can anyone take a look at the patch? I've some concerns regarding the live containers. Movement steps: 1, Check if the target queue has enough capacity and some more validation, exception otherwise (same as with FairScheduler) 2, Remove the app attempt from the current queue 3, Release resources used by live containers on this queue 4, Remove application upwards root (--numApplications) 5, QueueMetrics update 6, Set new queue in application 7, Allocate resources consumed by the live containers (basically the resource usage moved here from the original queue) 8, Submit new app attempt 9, Add application (++numApplications) Capacity Scheduler changes for moving apps between queues - Key: YARN-2248 URL: https://issues.apache.org/jira/browse/YARN-2248 Project: Hadoop YARN Issue Type: Sub-task Components: capacityscheduler Reporter: Janos Matyas Assignee: Janos Matyas Attachments: YARN-2248-1.patch, YARN-2248-2.patch We would like to have the capability (same as the Fair Scheduler has) to move applications between queues. We have made a baseline implementation and tests to start with - and we would like the community to review, come up with suggestions and finally have this contributed. The current implementation is available for 2.4.1 - so the first thing is that we'd need to identify the target version as there are differences between 2.4.* and 3.* interfaces. The story behind is available at http://blog.sequenceiq.com/blog/2014/07/02/move-applications-between-queues/ and the baseline implementation and test at: https://github.com/sequenceiq/hadoop-common/blob/branch-2.4.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/a/ExtendedCapacityScheduler.java#L924 https://github.com/sequenceiq/hadoop-common/blob/branch-2.4.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/a/TestExtendedCapacitySchedulerAppMove.java -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2250) Moving apps between queues - FairScheduler
[ https://issues.apache.org/jira/browse/YARN-2250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14051192#comment-14051192 ] Krisztian Horvath commented on YARN-2250: - Hi Sandy, The problem is not that the metrics are not updated, but the findLowestCommonAncestorQueue method always returns null causing the update always go upwards till the root. root.queue1.a root.queue1.b common should be queue1, but returning null, and goes upward till root Moving apps between queues - FairScheduler -- Key: YARN-2250 URL: https://issues.apache.org/jira/browse/YARN-2250 Project: Hadoop YARN Issue Type: Bug Components: scheduler Affects Versions: 2.4.0, 2.4.1 Reporter: Krisztian Horvath We need to update the queue metrics until to lowest common ancestor of the target and source queue. This method fails to retrieve the right queue. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2250) Moving apps between queues - FairScheduler
[ https://issues.apache.org/jira/browse/YARN-2250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14051217#comment-14051217 ] Krisztian Horvath commented on YARN-2250: - I'm preparing the patch for this and you can check that. Moving apps between queues - FairScheduler -- Key: YARN-2250 URL: https://issues.apache.org/jira/browse/YARN-2250 Project: Hadoop YARN Issue Type: Bug Components: scheduler Affects Versions: 2.4.0, 2.4.1 Reporter: Krisztian Horvath We need to update the queue metrics until to lowest common ancestor of the target and source queue. This method fails to retrieve the right queue. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-2250) Moving apps between queues - FairScheduler
[ https://issues.apache.org/jira/browse/YARN-2250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Horvath updated YARN-2250: Attachment: YARN-2250-1.patch Moving apps between queues - FairScheduler -- Key: YARN-2250 URL: https://issues.apache.org/jira/browse/YARN-2250 Project: Hadoop YARN Issue Type: Bug Components: scheduler Affects Versions: 2.4.0, 2.4.1 Reporter: Krisztian Horvath Attachments: YARN-2250-1.patch We need to update the queue metrics until to lowest common ancestor of the target and source queue. This method fails to retrieve the right queue. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2250) Moving apps between queues - FairScheduler
[ https://issues.apache.org/jira/browse/YARN-2250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14051241#comment-14051241 ] Krisztian Horvath commented on YARN-2250: - Do I need to create a review request for this? Moving apps between queues - FairScheduler -- Key: YARN-2250 URL: https://issues.apache.org/jira/browse/YARN-2250 Project: Hadoop YARN Issue Type: Bug Components: scheduler Affects Versions: 2.4.0, 2.4.1 Reporter: Krisztian Horvath Attachments: YARN-2250-1.patch We need to update the queue metrics until to lowest common ancestor of the target and source queue. This method fails to retrieve the right queue. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-2250) FairScheduler.findLowestCommonAncestorQueue returns null when queues not identical
[ https://issues.apache.org/jira/browse/YARN-2250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Horvath updated YARN-2250: Attachment: YARN-2250-2.patch FairScheduler.findLowestCommonAncestorQueue returns null when queues not identical -- Key: YARN-2250 URL: https://issues.apache.org/jira/browse/YARN-2250 Project: Hadoop YARN Issue Type: Bug Components: scheduler Affects Versions: 2.4.0, 2.4.1 Reporter: Krisztian Horvath Attachments: YARN-2250-1.patch, YARN-2250-2.patch We need to update the queue metrics until to lowest common ancestor of the target and source queue. This method fails to retrieve the right queue. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2250) FairScheduler.findLowestCommonAncestorQueue returns null when queues not identical
[ https://issues.apache.org/jira/browse/YARN-2250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14051918#comment-14051918 ] Krisztian Horvath commented on YARN-2250: - Indeed, made changes accordingly. Thanks. FairScheduler.findLowestCommonAncestorQueue returns null when queues not identical -- Key: YARN-2250 URL: https://issues.apache.org/jira/browse/YARN-2250 Project: Hadoop YARN Issue Type: Bug Components: scheduler Affects Versions: 2.4.0, 2.4.1 Reporter: Krisztian Horvath Attachments: YARN-2250-1.patch, YARN-2250-2.patch We need to update the queue metrics until to lowest common ancestor of the target and source queue. This method fails to retrieve the right queue. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-1842) InvalidApplicationMasterRequestException raised during AM-requested shutdown
[ https://issues.apache.org/jira/browse/YARN-1842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Horvath updated YARN-1842: Attachment: hoyalogs.tar.gz InvalidApplicationMasterRequestException raised during AM-requested shutdown Key: YARN-1842 URL: https://issues.apache.org/jira/browse/YARN-1842 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.3.0 Reporter: Steve Loughran Priority: Minor Attachments: hoyalogs.tar.gz Report of the RM raising a stack trace [https://gist.github.com/matyix/9596735] during AM-initiated shutdown. The AM could just swallow this and exit, but it could be a sign of a race condition YARN-side, or maybe just in the RM client code/AM dual signalling the shutdown. I haven't replicated this myself; maybe the stack will help track down the problem. Otherwise: what is the policy YARN apps should adopt for AM's handling errors on shutdown? go straight to an exit(-1)? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1842) InvalidApplicationMasterRequestException raised during AM-requested shutdown
[ https://issues.apache.org/jira/browse/YARN-1842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13939658#comment-13939658 ] Krisztian Horvath commented on YARN-1842: - The original exception occurred on osx with hadoop 2.3.0, but after trying on debian with hadoop 2.2.0 the result is the same. I attached some logs. The Flume agent starts with a provider creating the shell command to exec. Other than that, the application should run in the requested container, right? Because it seems that it is running outside of it (setting the yarn.memory to 8mb and the jvm heap size of the application a much larger and works fine). InvalidApplicationMasterRequestException raised during AM-requested shutdown Key: YARN-1842 URL: https://issues.apache.org/jira/browse/YARN-1842 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.3.0 Reporter: Steve Loughran Priority: Minor Attachments: hoyalogs.tar.gz Report of the RM raising a stack trace [https://gist.github.com/matyix/9596735] during AM-initiated shutdown. The AM could just swallow this and exit, but it could be a sign of a race condition YARN-side, or maybe just in the RM client code/AM dual signalling the shutdown. I haven't replicated this myself; maybe the stack will help track down the problem. Otherwise: what is the policy YARN apps should adopt for AM's handling errors on shutdown? go straight to an exit(-1)? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1842) InvalidApplicationMasterRequestException raised during AM-requested shutdown
[ https://issues.apache.org/jira/browse/YARN-1842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13939798#comment-13939798 ] Krisztian Horvath commented on YARN-1842: - I'm curious whether this bug is causing the application to hang up after container shutdown. InvalidApplicationMasterRequestException raised during AM-requested shutdown Key: YARN-1842 URL: https://issues.apache.org/jira/browse/YARN-1842 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.3.0 Reporter: Steve Loughran Priority: Minor Attachments: hoyalogs.tar.gz Report of the RM raising a stack trace [https://gist.github.com/matyix/9596735] during AM-initiated shutdown. The AM could just swallow this and exit, but it could be a sign of a race condition YARN-side, or maybe just in the RM client code/AM dual signalling the shutdown. I haven't replicated this myself; maybe the stack will help track down the problem. Otherwise: what is the policy YARN apps should adopt for AM's handling errors on shutdown? go straight to an exit(-1)? -- This message was sent by Atlassian JIRA (v6.2#6252)
