[ https://issues.apache.org/jira/browse/AIRFLOW-398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15829167#comment-15829167 ]
ASF subversion and git services commented on AIRFLOW-398: --------------------------------------------------------- Commit b56cb5cc97de074bb0e520f66b79e7eb2d913fb1 in incubator-airflow's branch refs/heads/master from [~aoen] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=b56cb5c ] [AIRFLOW-219][AIRFLOW-398] Cgroups + impersonation Submitting on behalf of plypaul Please accept this PR that addresses the following issues: - https://issues.apache.org/jira/browse/AIRFLOW-219 - https://issues.apache.org/jira/browse/AIRFLOW-398 Testing Done: - Running on Airbnb prod (though on a different mergebase) for many months Credits: Impersonation Work: georgeke did most of the work but plypaul did quite a bit of work too. Cgroups: plypaul did most of the work, I just did some touch up/bug fixes (see commit history, cgroups + impersonation commit is actually plypaul 's not mine) Closes #1934 from aoen/ddavydov/cgroups_and_impers onation_after_rebase > Option to use cgroups to limit task resource utilization > -------------------------------------------------------- > > Key: AIRFLOW-398 > URL: https://issues.apache.org/jira/browse/AIRFLOW-398 > Project: Apache Airflow > Issue Type: Improvement > Components: core > Reporter: Paul Yang > Assignee: Paul Yang > Priority: Minor > > Now that tasks have resource requirements specified, Airflow can use the > defined resource requirements as limits when running the Airflow tasks. A > common use case is to limit the memory used by a task so that it doesn't > cause the worker machine to go OOM. > The proposed idea is to implement a TaskRunner class that, by default, > executes the Airflow run command in a bash subprocess as it is currently done > now. > Alternatively, the user can specify a TaskRunner class that uses cgroups to > ensure that resource requirements are respected. This would be a lightweight > approach to limiting resource usage by tasks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)