[jira] [Created] (YARN-4093) Encapsulate additional information through AM to RM heartbeat
Robert Grandl created YARN-4093: --- Summary: Encapsulate additional information through AM to RM heartbeat Key: YARN-4093 URL: https://issues.apache.org/jira/browse/YARN-4093 Project: Hadoop YARN Issue Type: Improvement Components: api, yarn Reporter: Robert Grandl Assignee: Robert Grandl In this JIRA we propose to enhance the AM RM protocol with a new message which encapsulates additional information about group of tasks. The RM scheduler will benefit of the additional information to take better decisions at the scheduling time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4093) Encapsulate additional information through AM to RM heartbeat
[ https://issues.apache.org/jira/browse/YARN-4093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Grandl updated YARN-4093: Attachment: AllocateRequest_extension.docx Added a proposed design doc Encapsulate additional information through AM to RM heartbeat - Key: YARN-4093 URL: https://issues.apache.org/jira/browse/YARN-4093 Project: Hadoop YARN Issue Type: Sub-task Components: api, yarn Affects Versions: 2.7.1 Reporter: Robert Grandl Assignee: Robert Grandl Attachments: AllocateRequest_extension.docx In this JIRA we propose to enhance the AM RM protocol with a new message which encapsulates additional information about group of tasks. The RM scheduler will benefit of the additional information to take better decisions at the scheduling time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4093) Encapsulate additional information through AM to RM heartbeat
[ https://issues.apache.org/jira/browse/YARN-4093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Grandl updated YARN-4093: Issue Type: Sub-task (was: Improvement) Parent: YARN-2745 Encapsulate additional information through AM to RM heartbeat - Key: YARN-4093 URL: https://issues.apache.org/jira/browse/YARN-4093 Project: Hadoop YARN Issue Type: Sub-task Components: api, yarn Affects Versions: 2.7.1 Reporter: Robert Grandl Assignee: Robert Grandl In this JIRA we propose to enhance the AM RM protocol with a new message which encapsulates additional information about group of tasks. The RM scheduler will benefit of the additional information to take better decisions at the scheduling time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4093) Encapsulate additional information through AM to RM heartbeat
[ https://issues.apache.org/jira/browse/YARN-4093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Grandl updated YARN-4093: Attachment: YARN-4093.patch Encapsulate additional information through AM to RM heartbeat - Key: YARN-4093 URL: https://issues.apache.org/jira/browse/YARN-4093 Project: Hadoop YARN Issue Type: Sub-task Components: api, yarn Affects Versions: 2.7.1 Reporter: Robert Grandl Assignee: Robert Grandl Attachments: AllocateRequest_extension.docx, YARN-4093.patch In this JIRA we propose to enhance the AM RM protocol with a new message which encapsulates additional information about group of tasks. The RM scheduler will benefit of the additional information to take better decisions at the scheduling time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4093) Encapsulate additional group information in the AM to RM heartbeat
[ https://issues.apache.org/jira/browse/YARN-4093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Grandl updated YARN-4093: Summary: Encapsulate additional group information in the AM to RM heartbeat (was: Encapsulate additional information through AM to RM heartbeat) Encapsulate additional group information in the AM to RM heartbeat -- Key: YARN-4093 URL: https://issues.apache.org/jira/browse/YARN-4093 Project: Hadoop YARN Issue Type: Sub-task Components: api, yarn Affects Versions: 2.7.1 Reporter: Robert Grandl Assignee: Robert Grandl Labels: patch Attachments: AllocateRequest_extension.docx, YARN-4093.patch In this JIRA we propose to enhance the AM RM protocol with a new message which encapsulates additional information about group of tasks. The RM scheduler will benefit of the additional information to take better decisions at the scheduling time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (YARN-4056) Bundling: Searching for multiple containers in a single pass over {queues, applications, priorities}
[
https://issues.apache.org/jira/browse/YARN-4056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Grandl reassigned YARN-4056:
---
Assignee: Robert Grandl
Bundling: Searching for multiple containers in a single pass over {queues,
applications, priorities}
Key: YARN-4056
URL: https://issues.apache.org/jira/browse/YARN-4056
Project: Hadoop YARN
Issue Type: New Feature
Components: capacityscheduler, resourcemanager, scheduler
Reporter: Srikanth Kandula
Assignee: Robert Grandl
Attachments: bundling.docx
More than one container is allocated on many NM heartbeats. Yet, the current
scheduler allocates exactly one container per iteration over {{queues,
applications, priorities}}. When there are many queues, applications, or
priorities allocating only one container per iteration can needlessly
increase the duration of the NM heartbeat.
In this JIRA, we propose bundling. That is, allow arbitrarily many containers
to be allocated in a single iteration over {{queues, applications and
priorities}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3820) Collect disks usages on the node
[ https://issues.apache.org/jira/browse/YARN-3820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596463#comment-14596463 ] Robert Grandl commented on YARN-3820: - [~elgoiri], I fixed the warning because HadoopQA javadoc was -1. I will revert the change if HadoopQA will return +1. Collect disks usages on the node Key: YARN-3820 URL: https://issues.apache.org/jira/browse/YARN-3820 Project: Hadoop YARN Issue Type: New Feature Affects Versions: 3.0.0 Reporter: Robert Grandl Assignee: Robert Grandl Labels: yarn-common, yarn-util Attachments: YARN-3820-1.patch, YARN-3820-2.patch, YARN-3820-3.patch, YARN-3820-4.patch In this JIRA we propose to collect disks usages on a node. This JIRA is part of a larger effort of monitoring resource usages on the nodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3820) Collect disks usages on the node
[ https://issues.apache.org/jira/browse/YARN-3820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594231#comment-14594231 ] Robert Grandl commented on YARN-3820: - [~hadoopqa] - could you please rerun the checks ? I cannot fix checkstyle as it does not show anything to fix here: https://builds.apache.org/job/PreCommit-YARN-Build/8288/artifact/patchprocess/diffcheckstylehadoop-yarn-common.txt. Collect disks usages on the node Key: YARN-3820 URL: https://issues.apache.org/jira/browse/YARN-3820 Project: Hadoop YARN Issue Type: New Feature Affects Versions: 3.0.0 Reporter: Robert Grandl Assignee: Robert Grandl Labels: yarn-common, yarn-util Attachments: YARN-3820-1.patch, YARN-3820-2.patch, YARN-3820-3.patch, YARN-3820-4.patch In this JIRA we propose to collect disks usages on a node. This JIRA is part of a larger effort of monitoring resource usages on the nodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3820) Collect disks usages on the node
[ https://issues.apache.org/jira/browse/YARN-3820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Grandl updated YARN-3820: Attachment: YARN-3820-4.patch Fixed checkstyles and javadoc Collect disks usages on the node Key: YARN-3820 URL: https://issues.apache.org/jira/browse/YARN-3820 Project: Hadoop YARN Issue Type: New Feature Affects Versions: 3.0.0 Reporter: Robert Grandl Assignee: Robert Grandl Labels: yarn-common, yarn-util Attachments: YARN-3820-1.patch, YARN-3820-2.patch, YARN-3820-3.patch, YARN-3820-4.patch In this JIRA we propose to collect disks usages on a node. This JIRA is part of a larger effort of monitoring resource usages on the nodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3819) Collect network usage on the node
[ https://issues.apache.org/jira/browse/YARN-3819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Grandl updated YARN-3819: Attachment: YARN-3819-5.patch Fixed checkstyle issues. Collect network usage on the node - Key: YARN-3819 URL: https://issues.apache.org/jira/browse/YARN-3819 Project: Hadoop YARN Issue Type: New Feature Affects Versions: 3.0.0 Reporter: Robert Grandl Assignee: Robert Grandl Labels: yarn-common, yarn-util Attachments: YARN-3819-1.patch, YARN-3819-2.patch, YARN-3819-3.patch, YARN-3819-4.patch, YARN-3819-5.patch In this JIRA we propose to collect the network usage on a node. This JIRA is part of a larger effort of monitoring resource usages on the nodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3819) Collect network usage on the node
[ https://issues.apache.org/jira/browse/YARN-3819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Grandl updated YARN-3819: Attachment: YARN-3819-2.patch Updates to DummyResourceCalculatorPlugin.java Collect network usage on the node - Key: YARN-3819 URL: https://issues.apache.org/jira/browse/YARN-3819 Project: Hadoop YARN Issue Type: New Feature Affects Versions: 3.0.0 Reporter: Robert Grandl Assignee: Robert Grandl Labels: yarn-common, yarn-util Attachments: YARN-3819-1.patch, YARN-3819-2.patch In this JIRA we propose to collect the network usage on a node. This JIRA is part of a larger effort of monitoring resource usages on the nodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3820) Collect disks usages on the node
[ https://issues.apache.org/jira/browse/YARN-3820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Grandl updated YARN-3820: Attachment: YARN-3820-1.patch Added first cut patch Collect disks usages on the node Key: YARN-3820 URL: https://issues.apache.org/jira/browse/YARN-3820 Project: Hadoop YARN Issue Type: New Feature Affects Versions: 3.0.0 Reporter: Robert Grandl Assignee: Robert Grandl Labels: yarn-common, yarn-util Attachments: YARN-3820-1.patch In this JIRA we propose to collect disks usages on a node. This JIRA is part of a larger effort of monitoring resource usages on the nodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3820) Collect disks usages on the node
[ https://issues.apache.org/jira/browse/YARN-3820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14590297#comment-14590297 ] Robert Grandl commented on YARN-3820: - [~srikanthkandula] and I were proposing to collect the disks usages on a node. This is part of a larger effort of multi-resource scheduling. Currently, yarn does not have any mechanism to monitor the amount of bytes read/written from/to disks. Collect disks usages on the node Key: YARN-3820 URL: https://issues.apache.org/jira/browse/YARN-3820 Project: Hadoop YARN Issue Type: New Feature Affects Versions: 3.0.0 Reporter: Robert Grandl Assignee: Robert Grandl Labels: yarn-common, yarn-util In this JIRA we propose to collect disks usages on a node. This JIRA is part of a larger effort of monitoring resource usages on the nodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3819) Collect network usage on the node
[ https://issues.apache.org/jira/browse/YARN-3819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Grandl updated YARN-3819: Flags: Patch Collect network usage on the node - Key: YARN-3819 URL: https://issues.apache.org/jira/browse/YARN-3819 Project: Hadoop YARN Issue Type: New Feature Affects Versions: 3.0.0 Reporter: Robert Grandl Assignee: Robert Grandl Labels: yarn-common, yarn-util In this JIRA we propose to collect the network usage on a node. This JIRA is part of a larger effort of monitoring resource usages on the nodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3819) Collect network usage on the node
[ https://issues.apache.org/jira/browse/YARN-3819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14590217#comment-14590217 ] Robert Grandl commented on YARN-3819: - [~srikanthkandula] and I were proposing to collect the network usage on a node. This is part of a larger effort of multi-resource scheduling. Previous efforts in collecting network usage per containers is not enough for the purpose of multi-resource scheduling, as it is not able to capture other traffic activities on the node such as ingestion or evacuation. Collect network usage on the node - Key: YARN-3819 URL: https://issues.apache.org/jira/browse/YARN-3819 Project: Hadoop YARN Issue Type: New Feature Affects Versions: 3.0.0 Reporter: Robert Grandl Assignee: Robert Grandl Labels: yarn-common, yarn-util Attachments: YARN-3819-1.patch In this JIRA we propose to collect the network usage on a node. This JIRA is part of a larger effort of monitoring resource usages on the nodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3819) Collect network usage on the node
[ https://issues.apache.org/jira/browse/YARN-3819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Grandl updated YARN-3819: Attachment: YARN-3819-1.patch Collect network usage on the node - Key: YARN-3819 URL: https://issues.apache.org/jira/browse/YARN-3819 Project: Hadoop YARN Issue Type: New Feature Affects Versions: 3.0.0 Reporter: Robert Grandl Assignee: Robert Grandl Labels: yarn-common, yarn-util Attachments: YARN-3819-1.patch In this JIRA we propose to collect the network usage on a node. This JIRA is part of a larger effort of monitoring resource usages on the nodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3819) Collect network usage on the node
[ https://issues.apache.org/jira/browse/YARN-3819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Grandl updated YARN-3819: Attachment: YARN-3819-3.patch Collect network usage on the node - Key: YARN-3819 URL: https://issues.apache.org/jira/browse/YARN-3819 Project: Hadoop YARN Issue Type: New Feature Affects Versions: 3.0.0 Reporter: Robert Grandl Assignee: Robert Grandl Labels: yarn-common, yarn-util Attachments: YARN-3819-1.patch, YARN-3819-2.patch, YARN-3819-3.patch In this JIRA we propose to collect the network usage on a node. This JIRA is part of a larger effort of monitoring resource usages on the nodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-3820) Collect disks usages on the node
Robert Grandl created YARN-3820: --- Summary: Collect disks usages on the node Key: YARN-3820 URL: https://issues.apache.org/jira/browse/YARN-3820 Project: Hadoop YARN Issue Type: New Feature Affects Versions: 3.0.0 Reporter: Robert Grandl Assignee: Robert Grandl In this JIRA we propose to collect disks usages on a node. This JIRA is part of a larger effort of monitoring resource usages on the nodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3819) Collect network usage on the node
[ https://issues.apache.org/jira/browse/YARN-3819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14590555#comment-14590555 ] Robert Grandl commented on YARN-3819: - Short description of this JIRA: We process /proc/net/dev file which reports for every network interface present on the node, the cumulative amount of bytes read/written. We aggregate these numbers across all the interfaces except loopback. We tested the existence of these files in the following Linux kernel versions: Linux 3.2.0 Linux 2.6.32 Linux 3.13.0 Also, doing further search on the web, it seems people are using/recommending these files for extracting read/written network bytes counters. Collect network usage on the node - Key: YARN-3819 URL: https://issues.apache.org/jira/browse/YARN-3819 Project: Hadoop YARN Issue Type: New Feature Affects Versions: 3.0.0 Reporter: Robert Grandl Assignee: Robert Grandl Labels: yarn-common, yarn-util Attachments: YARN-3819-1.patch, YARN-3819-2.patch, YARN-3819-3.patch In this JIRA we propose to collect the network usage on a node. This JIRA is part of a larger effort of monitoring resource usages on the nodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3820) Collect disks usages on the node
[ https://issues.apache.org/jira/browse/YARN-3820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Grandl updated YARN-3820: Attachment: YARN-3820-2.patch Collect disks usages on the node Key: YARN-3820 URL: https://issues.apache.org/jira/browse/YARN-3820 Project: Hadoop YARN Issue Type: New Feature Affects Versions: 3.0.0 Reporter: Robert Grandl Assignee: Robert Grandl Labels: yarn-common, yarn-util Attachments: YARN-3820-1.patch, YARN-3820-2.patch In this JIRA we propose to collect disks usages on a node. This JIRA is part of a larger effort of monitoring resource usages on the nodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3819) Collect network usage on the node
[ https://issues.apache.org/jira/browse/YARN-3819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Grandl updated YARN-3819: Attachment: YARN-3819-4.patch Updated patch to address the failure and the whitespaces Collect network usage on the node - Key: YARN-3819 URL: https://issues.apache.org/jira/browse/YARN-3819 Project: Hadoop YARN Issue Type: New Feature Affects Versions: 3.0.0 Reporter: Robert Grandl Assignee: Robert Grandl Labels: yarn-common, yarn-util Attachments: YARN-3819-1.patch, YARN-3819-2.patch, YARN-3819-3.patch, YARN-3819-4.patch In this JIRA we propose to collect the network usage on a node. This JIRA is part of a larger effort of monitoring resource usages on the nodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-3819) Collect network usage on the node
Robert Grandl created YARN-3819: --- Summary: Collect network usage on the node Key: YARN-3819 URL: https://issues.apache.org/jira/browse/YARN-3819 Project: Hadoop YARN Issue Type: New Feature Reporter: Robert Grandl Assignee: Robert Grandl In this JIRA we propose to collect the network usage on a node. This JIRA is part of a larger effort of monitoring resource usages on the nodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3820) Collect disks usages on the node
[ https://issues.apache.org/jira/browse/YARN-3820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14590533#comment-14590533 ] Robert Grandl commented on YARN-3820: - Short description: This JIRA collects bytes read/written from/to disks in Linux. Step 1: We exploit the /proc/diskstats counters, extract the number of sectors read/written for every disk, and return the aggregation of these counters among all the disks. Step 2: To convert sectors into bytes, for every disk, we extract the sector size from /sys/block/diskName/queue/hw_sector_size. Step 3: Finally by multiplying the number of sectors from Step 1 with sector size from Step 2 we compute the number of bytes. We tested the existence of these files in the following Linux kernel versions: Linux 3.2.0 Linux 2.6.32 Linux 3.13.0 Also, doing further search on the web, it seems people are using/recommending these files for extracting read/written disks counters Collect disks usages on the node Key: YARN-3820 URL: https://issues.apache.org/jira/browse/YARN-3820 Project: Hadoop YARN Issue Type: New Feature Affects Versions: 3.0.0 Reporter: Robert Grandl Assignee: Robert Grandl Labels: yarn-common, yarn-util Attachments: YARN-3820-1.patch In this JIRA we propose to collect disks usages on a node. This JIRA is part of a larger effort of monitoring resource usages on the nodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3819) Collect network usage on the node
[ https://issues.apache.org/jira/browse/YARN-3819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14590553#comment-14590553 ] Robert Grandl commented on YARN-3819: - [~grey], YARN-2745 is an effort to schedule multiple resources. The resources taken in account are CPU/Memory/Disk/Network. For fungible resources such as disk and network, the counters required are the total number of bytes read/written from/to disk/network. This JIRA extends the ResourceCalculatorPlugin which is able to extract the amount of available CPU and Memory on a node. YARN-1012 is already using this information and YARN-1012 is aggregating this information in a heartbeat from NM to RM. Collect network usage on the node - Key: YARN-3819 URL: https://issues.apache.org/jira/browse/YARN-3819 Project: Hadoop YARN Issue Type: New Feature Affects Versions: 3.0.0 Reporter: Robert Grandl Assignee: Robert Grandl Labels: yarn-common, yarn-util Attachments: YARN-3819-1.patch, YARN-3819-2.patch, YARN-3819-3.patch In this JIRA we propose to collect the network usage on a node. This JIRA is part of a larger effort of monitoring resource usages on the nodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3820) Collect disks usages on the node
[ https://issues.apache.org/jira/browse/YARN-3820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Grandl updated YARN-3820: Attachment: YARN-3820-3.patch Remove whitespaces and same build crash as YARN-3819-3.patch Collect disks usages on the node Key: YARN-3820 URL: https://issues.apache.org/jira/browse/YARN-3820 Project: Hadoop YARN Issue Type: New Feature Affects Versions: 3.0.0 Reporter: Robert Grandl Assignee: Robert Grandl Labels: yarn-common, yarn-util Attachments: YARN-3820-1.patch, YARN-3820-2.patch, YARN-3820-3.patch In this JIRA we propose to collect disks usages on a node. This JIRA is part of a larger effort of monitoring resource usages on the nodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2745) Extend YARN to support multi-resource packing of tasks
[ https://issues.apache.org/jira/browse/YARN-2745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Grandl updated YARN-2745: Component/s: nodemanager Description: In this umbrella JIRA we propose an extension to existing scheduling techniques, which accounts for all resources used by a task (CPU, memory, disk, network) and it is able to achieve three competing objectives: fairness, improve cluster utilization and reduces average job completion time. (was: In this umbrella JIRA we propose a new pluggable scheduler, which accounts for all resources used by a task (CPU, memory, disk, network) and it is able to achieve three competing objectives: fairness, improve cluster utilization and reduces average job completion time.) Summary: Extend YARN to support multi-resource packing of tasks (was: YARN new pluggable scheduler which does multi-resource packing) Summary of main changes * Update the container allocation logic in the RM scheduler. This change is the core. It enables “packing” tasks, preferring jobs with less remaining work and trades off fairness for efficiency. * Expand the AM-RM resource ask to expose tasks’ disk and network resource demands to the scheduler. *Support for cluster-wide resource tracking: want per-machine resource usage information available at the RM. Extend YARN to support multi-resource packing of tasks -- Key: YARN-2745 URL: https://issues.apache.org/jira/browse/YARN-2745 Project: Hadoop YARN Issue Type: Improvement Components: nodemanager, resourcemanager, scheduler Reporter: Robert Grandl Attachments: sigcomm_14_tetris_talk.pptx, tetris_paper.pdf In this umbrella JIRA we propose an extension to existing scheduling techniques, which accounts for all resources used by a task (CPU, memory, disk, network) and it is able to achieve three competing objectives: fairness, improve cluster utilization and reduces average job completion time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-2965) Enhance Node Managers to monitor and report the resource usage on the machines
Robert Grandl created YARN-2965: --- Summary: Enhance Node Managers to monitor and report the resource usage on the machines Key: YARN-2965 URL: https://issues.apache.org/jira/browse/YARN-2965 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager, resourcemanager Reporter: Robert Grandl This JIRA is about augmenting Node Managers to monitor the resource usage on the machine, aggregates these reports and exposes them to the RM. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2965) Enhance Node Managers to monitor and report the resource usage on the machines
[ https://issues.apache.org/jira/browse/YARN-2965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Grandl updated YARN-2965: Attachment: ddoc_RT.docx Attached proposed design document. Enhance Node Managers to monitor and report the resource usage on the machines -- Key: YARN-2965 URL: https://issues.apache.org/jira/browse/YARN-2965 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager, resourcemanager Reporter: Robert Grandl Attachments: ddoc_RT.docx This JIRA is about augmenting Node Managers to monitor the resource usage on the machine, aggregates these reports and exposes them to the RM. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2965) Enhance Node Managers to monitor and report the resource usage on the machines
[ https://issues.apache.org/jira/browse/YARN-2965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Grandl updated YARN-2965: Attachment: (was: ddoc_RT.docx) Enhance Node Managers to monitor and report the resource usage on the machines -- Key: YARN-2965 URL: https://issues.apache.org/jira/browse/YARN-2965 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager, resourcemanager Reporter: Robert Grandl This JIRA is about augmenting Node Managers to monitor the resource usage on the machine, aggregates these reports and exposes them to the RM. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2965) Enhance Node Managers to monitor and report the resource usage on the machines
[ https://issues.apache.org/jira/browse/YARN-2965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Grandl updated YARN-2965: Attachment: ddoc_RT.pdf Proposed design document attached. Enhance Node Managers to monitor and report the resource usage on the machines -- Key: YARN-2965 URL: https://issues.apache.org/jira/browse/YARN-2965 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager, resourcemanager Reporter: Robert Grandl Attachments: ddoc_RT.pdf This JIRA is about augmenting Node Managers to monitor the resource usage on the machine, aggregates these reports and exposes them to the RM. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-2966) Extend ask request to include additional fields
Robert Grandl created YARN-2966: --- Summary: Extend ask request to include additional fields Key: YARN-2966 URL: https://issues.apache.org/jira/browse/YARN-2966 Project: Hadoop YARN Issue Type: Sub-task Reporter: Robert Grandl This JIRA is about extending the ask request from AM to RM to include additional information that describe tasks' resource requirements other than cpu and memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2966) Extend ask request to include additional fields
[ https://issues.apache.org/jira/browse/YARN-2966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Grandl updated YARN-2966: Attachment: ddoc_expanded_ask.docx Extend ask request to include additional fields --- Key: YARN-2966 URL: https://issues.apache.org/jira/browse/YARN-2966 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager, resourcemanager, scheduler Reporter: Robert Grandl Attachments: ddoc_expanded_ask.docx This JIRA is about extending the ask request from AM to RM to include additional information that describe tasks' resource requirements other than cpu and memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2965) Enhance Node Managers to monitor and report the resource usage on the machines
[ https://issues.apache.org/jira/browse/YARN-2965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Grandl updated YARN-2965: Attachment: ddoc_RT.docx Enhance Node Managers to monitor and report the resource usage on the machines -- Key: YARN-2965 URL: https://issues.apache.org/jira/browse/YARN-2965 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager, resourcemanager Reporter: Robert Grandl Attachments: ddoc_RT.docx This JIRA is about augmenting Node Managers to monitor the resource usage on the machine, aggregates these reports and exposes them to the RM. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2965) Enhance Node Managers to monitor and report the resource usage on the machines
[ https://issues.apache.org/jira/browse/YARN-2965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Grandl updated YARN-2965: Attachment: (was: ddoc_RT.pdf) Enhance Node Managers to monitor and report the resource usage on the machines -- Key: YARN-2965 URL: https://issues.apache.org/jira/browse/YARN-2965 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager, resourcemanager Reporter: Robert Grandl Attachments: ddoc_RT.docx This JIRA is about augmenting Node Managers to monitor the resource usage on the machine, aggregates these reports and exposes them to the RM. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-2967) New task matching logic at the RM
Robert Grandl created YARN-2967: --- Summary: New task matching logic at the RM Key: YARN-2967 URL: https://issues.apache.org/jira/browse/YARN-2967 Project: Hadoop YARN Issue Type: Sub-task Reporter: Robert Grandl This sub-JIRA changes the matching logic at the RM. We expect different extensions to both the CS and the FS schedulers. These changes should work independent of the other changes. That is, with just CPU and memory in the asks, as is the case today, the matching logic should still work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2967) New task matching logic at the RM
[ https://issues.apache.org/jira/browse/YARN-2967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Grandl updated YARN-2967: Attachment: ddoc_matching_logic.docx new matching logic design document attached New task matching logic at the RM -- Key: YARN-2967 URL: https://issues.apache.org/jira/browse/YARN-2967 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager, resourcemanager, scheduler Reporter: Robert Grandl Attachments: ddoc_matching_logic.docx This sub-JIRA changes the matching logic at the RM. We expect different extensions to both the CS and the FS schedulers. These changes should work independent of the other changes. That is, with just CPU and memory in the asks, as is the case today, the matching logic should still work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2745) Extend YARN to support multi-resource packing of tasks
[ https://issues.apache.org/jira/browse/YARN-2745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Grandl updated YARN-2745: Attachment: tetris_design_doc.docx added design document Extend YARN to support multi-resource packing of tasks -- Key: YARN-2745 URL: https://issues.apache.org/jira/browse/YARN-2745 Project: Hadoop YARN Issue Type: Improvement Components: nodemanager, resourcemanager, scheduler Reporter: Robert Grandl Attachments: sigcomm_14_tetris_talk.pptx, tetris_design_doc.docx, tetris_paper.pdf In this umbrella JIRA we propose an extension to existing scheduling techniques, which accounts for all resources used by a task (CPU, memory, disk, network) and it is able to achieve three competing objectives: fairness, improve cluster utilization and reduces average job completion time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2965) Enhance Node Managers to monitor and report the resource usage on the machines
[ https://issues.apache.org/jira/browse/YARN-2965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Grandl updated YARN-2965: Attachment: (was: ddoc_RT.docx) Enhance Node Managers to monitor and report the resource usage on the machines -- Key: YARN-2965 URL: https://issues.apache.org/jira/browse/YARN-2965 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager, resourcemanager Reporter: Robert Grandl This JIRA is about augmenting Node Managers to monitor the resource usage on the machine, aggregates these reports and exposes them to the RM. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2967) New task matching logic at the RM
[ https://issues.apache.org/jira/browse/YARN-2967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Grandl updated YARN-2967: Attachment: (was: ddoc_matching_logic.docx) New task matching logic at the RM -- Key: YARN-2967 URL: https://issues.apache.org/jira/browse/YARN-2967 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager, resourcemanager, scheduler Reporter: Robert Grandl This sub-JIRA changes the matching logic at the RM. We expect different extensions to both the CS and the FS schedulers. These changes should work independent of the other changes. That is, with just CPU and memory in the asks, as is the case today, the matching logic should still work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2966) Extend ask request to include additional fields
[ https://issues.apache.org/jira/browse/YARN-2966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Grandl updated YARN-2966: Attachment: (was: ddoc_expanded_ask.docx) Extend ask request to include additional fields --- Key: YARN-2966 URL: https://issues.apache.org/jira/browse/YARN-2966 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager, resourcemanager, scheduler Reporter: Robert Grandl This JIRA is about extending the ask request from AM to RM to include additional information that describe tasks' resource requirements other than cpu and memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2745) Extend YARN to support multi-resource packing of tasks
[ https://issues.apache.org/jira/browse/YARN-2745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Grandl updated YARN-2745: Attachment: (was: tetris_design_doc.docx) Extend YARN to support multi-resource packing of tasks -- Key: YARN-2745 URL: https://issues.apache.org/jira/browse/YARN-2745 Project: Hadoop YARN Issue Type: Improvement Components: nodemanager, resourcemanager, scheduler Reporter: Robert Grandl Attachments: sigcomm_14_tetris_talk.pptx, tetris_paper.pdf In this umbrella JIRA we propose an extension to existing scheduling techniques, which accounts for all resources used by a task (CPU, memory, disk, network) and it is able to achieve three competing objectives: fairness, improve cluster utilization and reduces average job completion time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2965) Enhance Node Managers to monitor and report the resource usage on the machines
[ https://issues.apache.org/jira/browse/YARN-2965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Grandl updated YARN-2965: Attachment: ddoc_RT.docx Enhance Node Managers to monitor and report the resource usage on the machines -- Key: YARN-2965 URL: https://issues.apache.org/jira/browse/YARN-2965 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager, resourcemanager Reporter: Robert Grandl Attachments: ddoc_RT.docx This JIRA is about augmenting Node Managers to monitor the resource usage on the machine, aggregates these reports and exposes them to the RM. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2745) Extend YARN to support multi-resource packing of tasks
[ https://issues.apache.org/jira/browse/YARN-2745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Grandl updated YARN-2745: Attachment: tetris_design_doc.docx Extend YARN to support multi-resource packing of tasks -- Key: YARN-2745 URL: https://issues.apache.org/jira/browse/YARN-2745 Project: Hadoop YARN Issue Type: Improvement Components: nodemanager, resourcemanager, scheduler Reporter: Robert Grandl Attachments: sigcomm_14_tetris_talk.pptx, tetris_design_doc.docx, tetris_paper.pdf In this umbrella JIRA we propose an extension to existing scheduling techniques, which accounts for all resources used by a task (CPU, memory, disk, network) and it is able to achieve three competing objectives: fairness, improve cluster utilization and reduces average job completion time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2966) Extend ask request to include additional fields
[ https://issues.apache.org/jira/browse/YARN-2966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Grandl updated YARN-2966: Attachment: ddoc_expanded_ask.docx Extend ask request to include additional fields --- Key: YARN-2966 URL: https://issues.apache.org/jira/browse/YARN-2966 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager, resourcemanager, scheduler Reporter: Robert Grandl Attachments: ddoc_expanded_ask.docx This JIRA is about extending the ask request from AM to RM to include additional information that describe tasks' resource requirements other than cpu and memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2967) New task matching logic at the RM
[ https://issues.apache.org/jira/browse/YARN-2967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Grandl updated YARN-2967: Attachment: ddoc_matching_logic.docx New task matching logic at the RM -- Key: YARN-2967 URL: https://issues.apache.org/jira/browse/YARN-2967 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager, resourcemanager, scheduler Reporter: Robert Grandl Attachments: ddoc_matching_logic.docx This sub-JIRA changes the matching logic at the RM. We expect different extensions to both the CS and the FS schedulers. These changes should work independent of the other changes. That is, with just CPU and memory in the asks, as is the case today, the matching logic should still work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2745) YARN new pluggable scheduler which does multi-resource packing
[ https://issues.apache.org/jira/browse/YARN-2745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Grandl updated YARN-2745: Attachment: (was: tetris_design_doc.docx) YARN new pluggable scheduler which does multi-resource packing -- Key: YARN-2745 URL: https://issues.apache.org/jira/browse/YARN-2745 Project: Hadoop YARN Issue Type: Improvement Components: resourcemanager, scheduler Reporter: Robert Grandl Attachments: sigcomm_14_tetris_talk.pptx, tetris_paper.pdf In this umbrella JIRA we propose a new pluggable scheduler, which accounts for all resources used by a task (CPU, memory, disk, network) and it is able to achieve three competing objectives: fairness, improve cluster utilization and reduces average job completion time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-2745) YARN new pluggable scheduler which does multi-resource packing
Robert Grandl created YARN-2745: --- Summary: YARN new pluggable scheduler which does multi-resource packing Key: YARN-2745 URL: https://issues.apache.org/jira/browse/YARN-2745 Project: Hadoop YARN Issue Type: Improvement Components: resourcemanager, scheduler Reporter: Robert Grandl In this umbrella JIRA we propose a new pluggable scheduler, which accounts for all resources used by a task (CPU, memory, disk, network) and it is able to achieve three competing objectives: fairness, improve cluster utilization and reduces average job completion time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2745) YARN new pluggable scheduler which does multi-resource packing
[ https://issues.apache.org/jira/browse/YARN-2745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Grandl updated YARN-2745: Attachment: tetris_design_doc.docx YARN new pluggable scheduler which does multi-resource packing -- Key: YARN-2745 URL: https://issues.apache.org/jira/browse/YARN-2745 Project: Hadoop YARN Issue Type: Improvement Components: resourcemanager, scheduler Reporter: Robert Grandl Attachments: tetris_design_doc.docx In this umbrella JIRA we propose a new pluggable scheduler, which accounts for all resources used by a task (CPU, memory, disk, network) and it is able to achieve three competing objectives: fairness, improve cluster utilization and reduces average job completion time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2745) YARN new pluggable scheduler which does multi-resource packing
[ https://issues.apache.org/jira/browse/YARN-2745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Grandl updated YARN-2745: Attachment: sigcomm_14_tetris_talk.pptx YARN new pluggable scheduler which does multi-resource packing -- Key: YARN-2745 URL: https://issues.apache.org/jira/browse/YARN-2745 Project: Hadoop YARN Issue Type: Improvement Components: resourcemanager, scheduler Reporter: Robert Grandl Attachments: sigcomm_14_tetris_talk.pptx, tetris_design_doc.docx In this umbrella JIRA we propose a new pluggable scheduler, which accounts for all resources used by a task (CPU, memory, disk, network) and it is able to achieve three competing objectives: fairness, improve cluster utilization and reduces average job completion time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2745) YARN new pluggable scheduler which does multi-resource packing
[ https://issues.apache.org/jira/browse/YARN-2745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Grandl updated YARN-2745: Attachment: tetris_paper.pdf YARN new pluggable scheduler which does multi-resource packing -- Key: YARN-2745 URL: https://issues.apache.org/jira/browse/YARN-2745 Project: Hadoop YARN Issue Type: Improvement Components: resourcemanager, scheduler Reporter: Robert Grandl Attachments: sigcomm_14_tetris_talk.pptx, tetris_design_doc.docx, tetris_paper.pdf In this umbrella JIRA we propose a new pluggable scheduler, which accounts for all resources used by a task (CPU, memory, disk, network) and it is able to achieve three competing objectives: fairness, improve cluster utilization and reduces average job completion time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-1443) blacklist is not propagated from AM to RM
Robert Grandl created YARN-1443: --- Summary: blacklist is not propagated from AM to RM Key: YARN-1443 URL: https://issues.apache.org/jira/browse/YARN-1443 Project: Hadoop YARN Issue Type: Bug Components: api, client Reporter: Robert Grandl Priority: Minor I was trying to blacklist some nodes. I added a set of hosts as strings into blacklistAdditions list and propagated into RMContainerRequestor#makeRemoteRequest to the RM. However the blacklist is received empty at RM. I logged the path for blacklist in AM and I found that in ApplicationMasterProtocolPBClientImpl#allocate, this list is lost. I print request.getResourceBlacklistRequest().getBlacklistAdditions().toString() at the beginning of ApplicationMasterProtocolPBClientImpl#allocate and the blacklisted additions are there. After AllocateRequestProto requestProto is created based on this request, and I print again requestProto.getBlacklistRequest().getBlacklistAdditionsList().toString(), the blacklist additions is empty now. I looked even further and log what happened. At some point in yarn-api, I was lost with my logging as that code was regenerated every time I recompiled yarn-api. Thanks, robert -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (YARN-1444) RM crashes when force reducer to be node local
Robert Grandl created YARN-1444: --- Summary: RM crashes when force reducer to be node local Key: YARN-1444 URL: https://issues.apache.org/jira/browse/YARN-1444 Project: Hadoop YARN Issue Type: Bug Components: client, resourcemanager Reporter: Robert Grandl Priority: Minor I have tried to force reducers to execute on certain nodes. What I did is I changed for reduce tasks, the RMContainerRequestor#addResourceRequest(req.priority, ResourceRequest.ANY, req.capability) to RMContainerRequestor#addResourceRequest(req.priority, HOST_NAME, req.capability). However, this change lead to RM crashes when reducers needs to be assigned with the following exception: FATAL org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in handling event type NODE_UPDATE to the scheduler java.lang.NullPointerException at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:841) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:640) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:554) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:695) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:739) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:86) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:549) at java.lang.Thread.run(Thread.java:722) -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1444) RM crashes when force reducer to be node local
[ https://issues.apache.org/jira/browse/YARN-1444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13831919#comment-13831919 ] Robert Grandl commented on YARN-1444: - I have 2.3.0 RM crashes when force reducer to be node local -- Key: YARN-1444 URL: https://issues.apache.org/jira/browse/YARN-1444 Project: Hadoop YARN Issue Type: Bug Components: client, resourcemanager Reporter: Robert Grandl Priority: Minor I have tried to force reducers to execute on certain nodes. What I did is I changed for reduce tasks, the RMContainerRequestor#addResourceRequest(req.priority, ResourceRequest.ANY, req.capability) to RMContainerRequestor#addResourceRequest(req.priority, HOST_NAME, req.capability). However, this change lead to RM crashes when reducers needs to be assigned with the following exception: FATAL org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in handling event type NODE_UPDATE to the scheduler java.lang.NullPointerException at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:841) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:640) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:554) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:695) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:739) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:86) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:549) at java.lang.Thread.run(Thread.java:722) -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1443) blacklist is not propagated from AM to RM
[ https://issues.apache.org/jira/browse/YARN-1443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13832068#comment-13832068 ] Robert Grandl commented on YARN-1443: - I am not sure I fully understand what are you saying. What I did was to force to blacklist some node in RMContainerRequestor#computeIgnoreBlacklisting. So I saw in RMContainerRequest#makeRemoteRequest that I have for the ResourceRequest from the ask list, a blackAddition element with the name of the host I blacklisted. Unfortunately this list is not propagated to RM. I am not doing any writing against the protocol API I believe. I simply logged the blacklist in different files on the path from AM to RM. But this list is populated in computeIgnoreBlacklisting as default, not doing any fancy things. blacklist is not propagated from AM to RM - Key: YARN-1443 URL: https://issues.apache.org/jira/browse/YARN-1443 Project: Hadoop YARN Issue Type: Bug Components: api, client Reporter: Robert Grandl Priority: Minor I was trying to blacklist some nodes. I added a set of hosts as strings into blacklistAdditions list and propagated into RMContainerRequestor#makeRemoteRequest to the RM. However the blacklist is received empty at RM. I logged the path for blacklist in AM and I found that in ApplicationMasterProtocolPBClientImpl#allocate, this list is lost. I print request.getResourceBlacklistRequest().getBlacklistAdditions().toString() at the beginning of ApplicationMasterProtocolPBClientImpl#allocate and the blacklisted additions are there. After AllocateRequestProto requestProto is created based on this request, and I print again requestProto.getBlacklistRequest().getBlacklistAdditionsList().toString(), the blacklist additions is empty now. I looked even further and log what happened. At some point in yarn-api, I was lost with my logging as that code was regenerated every time I recompiled yarn-api. Thanks, robert -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1443) blacklist is not propagated from AM to RM
[ https://issues.apache.org/jira/browse/YARN-1443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13832075#comment-13832075 ] Robert Grandl commented on YARN-1443: - This happened in MR App master. However, it is not clear if the bug is due to MR implementation or Yarn API. From my side, the blacklist is NOT propagated from MR AM to RM. But as far as I did the tracing, it goes correctly to YARN-API. If this may need to be moved to MAPREDUCE project, can you take care of that ? Thanks. blacklist is not propagated from AM to RM - Key: YARN-1443 URL: https://issues.apache.org/jira/browse/YARN-1443 Project: Hadoop YARN Issue Type: Bug Components: api, client Reporter: Robert Grandl Priority: Minor I was trying to blacklist some nodes. I added a set of hosts as strings into blacklistAdditions list and propagated into RMContainerRequestor#makeRemoteRequest to the RM. However the blacklist is received empty at RM. I logged the path for blacklist in AM and I found that in ApplicationMasterProtocolPBClientImpl#allocate, this list is lost. I print request.getResourceBlacklistRequest().getBlacklistAdditions().toString() at the beginning of ApplicationMasterProtocolPBClientImpl#allocate and the blacklisted additions are there. After AllocateRequestProto requestProto is created based on this request, and I print again requestProto.getBlacklistRequest().getBlacklistAdditionsList().toString(), the blacklist additions is empty now. I looked even further and log what happened. At some point in yarn-api, I was lost with my logging as that code was regenerated every time I recompiled yarn-api. Thanks, robert -- This message was sent by Atlassian JIRA (v6.1#6144)
