[jira] [Commented] (AIRFLOW-1467) allow tasks to use more than one pool slot
[ https://issues.apache.org/jira/browse/AIRFLOW-1467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17064139#comment-17064139 ] ASF subversion and git services commented on AIRFLOW-1467: -- Commit 3fc89bd67ff469940d97d182ce09dabef5634516 in airflow's branch refs/heads/v1-10-test from Lokesh Lal [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=3fc89bd ] [AIRFLOW-1467] Allow tasks to use more than one pool slot (#7160) > allow tasks to use more than one pool slot > -- > > Key: AIRFLOW-1467 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1467 > Project: Apache Airflow > Issue Type: Improvement > Components: scheduler >Reporter: Adrian Bridgett >Assignee: Lokesh Lal >Priority: Trivial > Labels: pool > Fix For: 1.10.10 > > > It would be useful to have tasks use more than a single pool slot. > Our use case is actually to limit how many tasks run on a head node (due to > memory constraints), currently we have to set a pool limit limiting how many > tasks. > Ideally we could set the pool size to e.g amount of memory and then set those > tasks pool_usage (or whatever the option would be called) to the amount of > memory we think they'll use. This way the pool would let lots of small tasks > run or just a few large tasks. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-1467) allow tasks to use more than one pool slot
[ https://issues.apache.org/jira/browse/AIRFLOW-1467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17062516#comment-17062516 ] ASF subversion and git services commented on AIRFLOW-1467: -- Commit 00921425b1c9583cdf2ca6beb39b22b8f93b34fa in airflow's branch refs/heads/v1-10-test from Lokesh Lal [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=0092142 ] [AIRFLOW-1467] Allow tasks to use more than one pool slot (#7160) > allow tasks to use more than one pool slot > -- > > Key: AIRFLOW-1467 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1467 > Project: Apache Airflow > Issue Type: Improvement > Components: scheduler >Reporter: Adrian Bridgett >Assignee: Lokesh Lal >Priority: Trivial > Labels: pool > Fix For: 1.10.10 > > > It would be useful to have tasks use more than a single pool slot. > Our use case is actually to limit how many tasks run on a head node (due to > memory constraints), currently we have to set a pool limit limiting how many > tasks. > Ideally we could set the pool size to e.g amount of memory and then set those > tasks pool_usage (or whatever the option would be called) to the amount of > memory we think they'll use. This way the pool would let lots of small tasks > run or just a few large tasks. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-1467) allow tasks to use more than one pool slot
[ https://issues.apache.org/jira/browse/AIRFLOW-1467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17062196#comment-17062196 ] ASF subversion and git services commented on AIRFLOW-1467: -- Commit 4bfa2b46929825efa11aac23ccd81cd7ad7e867d in airflow's branch refs/heads/v1-10-test from Lokesh Lal [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=4bfa2b4 ] [AIRFLOW-1467] Allow tasks to use more than one pool slot (#7160) > allow tasks to use more than one pool slot > -- > > Key: AIRFLOW-1467 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1467 > Project: Apache Airflow > Issue Type: Improvement > Components: scheduler >Reporter: Adrian Bridgett >Assignee: Lokesh Lal >Priority: Trivial > Labels: pool > Fix For: 1.10.10 > > > It would be useful to have tasks use more than a single pool slot. > Our use case is actually to limit how many tasks run on a head node (due to > memory constraints), currently we have to set a pool limit limiting how many > tasks. > Ideally we could set the pool size to e.g amount of memory and then set those > tasks pool_usage (or whatever the option would be called) to the amount of > memory we think they'll use. This way the pool would let lots of small tasks > run or just a few large tasks. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-1467) allow tasks to use more than one pool slot
[ https://issues.apache.org/jira/browse/AIRFLOW-1467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17018874#comment-17018874 ] ASF subversion and git services commented on AIRFLOW-1467: -- Commit 5878e48b787797c54b4138bc0166859232ea08d6 in airflow's branch refs/heads/master from Lokesh Lal [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=5878e48 ] [AIRFLOW-1467] Allow tasks to use more than one pool slot (#7160) > allow tasks to use more than one pool slot > -- > > Key: AIRFLOW-1467 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1467 > Project: Apache Airflow > Issue Type: Improvement > Components: scheduler >Reporter: Adrian Bridgett >Assignee: Lokesh Lal >Priority: Trivial > Labels: pool > Fix For: 1.10.8 > > > It would be useful to have tasks use more than a single pool slot. > Our use case is actually to limit how many tasks run on a head node (due to > memory constraints), currently we have to set a pool limit limiting how many > tasks. > Ideally we could set the pool size to e.g amount of memory and then set those > tasks pool_usage (or whatever the option would be called) to the amount of > memory we think they'll use. This way the pool would let lots of small tasks > run or just a few large tasks. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-1467) allow tasks to use more than one pool slot
[ https://issues.apache.org/jira/browse/AIRFLOW-1467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17018873#comment-17018873 ] ASF GitHub Bot commented on AIRFLOW-1467: - kaxil commented on pull request #7160: [AIRFLOW-1467] Dynamic pooling via allowing tasks to use more than one pool slot (depending upon the need) URL: https://github.com/apache/airflow/pull/7160 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > allow tasks to use more than one pool slot > -- > > Key: AIRFLOW-1467 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1467 > Project: Apache Airflow > Issue Type: Improvement > Components: scheduler >Reporter: Adrian Bridgett >Assignee: Lokesh Lal >Priority: Trivial > Labels: pool > Fix For: 1.10.8 > > > It would be useful to have tasks use more than a single pool slot. > Our use case is actually to limit how many tasks run on a head node (due to > memory constraints), currently we have to set a pool limit limiting how many > tasks. > Ideally we could set the pool size to e.g amount of memory and then set those > tasks pool_usage (or whatever the option would be called) to the amount of > memory we think they'll use. This way the pool would let lots of small tasks > run or just a few large tasks. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-1467) allow tasks to use more than one pool slot
[ https://issues.apache.org/jira/browse/AIRFLOW-1467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17014844#comment-17014844 ] ASF GitHub Bot commented on AIRFLOW-1467: - lokeshlal commented on pull request #7160: [AIRFLOW-1467] Dynamic pooling via allowing tasks to use more than one pool slot (depending upon the need) URL: https://github.com/apache/airflow/pull/7160 PR contains changes in pool and task instance to provide functionality to tasks to use more than one pool slot. - Added pool_capacity field in TaskInstance (pool_capacity is defaulted to 1, to maintain the current behavior) - Added pool_capacity in baseoperator - Modified pools functionality to calculate the used/queued/occupied/available slots - Modified pool_slots_available_dep.py to check against task _pool_capacity_ field instead of 1 - Modifed test case in file test_pool_slots_available_dep.py to include pool_capacity in the Mock object --- - [x] Description above provides context of the change - [X] Commit message contains [\[AIRFLOW-1467\]](https://issues.apache.org/jira/browse/AIRFLOW-1467) or `[AIRFLOW-]` for document-only changes - [X] Unit tests coverage for changes (not needed for documentation changes) - [X] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [X] Relevant documentation is updated including usage instructions. - [X] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > allow tasks to use more than one pool slot > -- > > Key: AIRFLOW-1467 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1467 > Project: Apache Airflow > Issue Type: Improvement > Components: scheduler >Reporter: Adrian Bridgett >Assignee: Lokesh Lal >Priority: Trivial > Labels: pool > Fix For: 1.10.8 > > > It would be useful to have tasks use more than a single pool slot. > Our use case is actually to limit how many tasks run on a head node (due to > memory constraints), currently we have to set a pool limit limiting how many > tasks. > Ideally we could set the pool size to e.g amount of memory and then set those > tasks pool_usage (or whatever the option would be called) to the amount of > memory we think they'll use. This way the pool would let lots of small tasks > run or just a few large tasks. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-1467) allow tasks to use more than one pool slot
[ https://issues.apache.org/jira/browse/AIRFLOW-1467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17014654#comment-17014654 ] ASF subversion and git services commented on AIRFLOW-1467: -- Commit 6df30a3ee78605e00376b3d2a0a537e27a6f20a9 in airflow's branch refs/heads/master from Jarek Potiuk [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=6df30a3 ] Revert "[AIRFLOW-1467] Dynamic pooling via allowing tasks to use more than one pool slot (depending upon the need) (#6975)" This reverts commit 277d01d4c57fe2470a8518c58bf0df9bf21e8f5e. > allow tasks to use more than one pool slot > -- > > Key: AIRFLOW-1467 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1467 > Project: Apache Airflow > Issue Type: Improvement > Components: scheduler >Reporter: Adrian Bridgett >Priority: Trivial > Labels: pool > Fix For: 1.10.8 > > > It would be useful to have tasks use more than a single pool slot. > Our use case is actually to limit how many tasks run on a head node (due to > memory constraints), currently we have to set a pool limit limiting how many > tasks. > Ideally we could set the pool size to e.g amount of memory and then set those > tasks pool_usage (or whatever the option would be called) to the amount of > memory we think they'll use. This way the pool would let lots of small tasks > run or just a few large tasks. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-1467) allow tasks to use more than one pool slot
[ https://issues.apache.org/jira/browse/AIRFLOW-1467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17014524#comment-17014524 ] ASF subversion and git services commented on AIRFLOW-1467: -- Commit 277d01d4c57fe2470a8518c58bf0df9bf21e8f5e in airflow's branch refs/heads/master from Lokesh Lal [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=277d01d ] [AIRFLOW-1467] Dynamic pooling via allowing tasks to use more than one pool slot (depending upon the need) (#6975) * adding pool capacity required for each task for dynamic pooling * Added pool_capacity column migration script * removed test checkedin file * removed extra space * correct test_database_schema_and_sqlalchemy_model_are_in_sync test case * Added description for pool_capacity property for task instance * Modified test cases to include pool_capacity along with pool in task instances * Modified test cases to include pool_capacity along with pool in task instances * Removed Column.name property, since property value is same as actual variable * check for pool_capacity property to be always >= 1 * removed unused variable ti * modified test cases for pool_capacity * modified test cases for pool_capacity > allow tasks to use more than one pool slot > -- > > Key: AIRFLOW-1467 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1467 > Project: Apache Airflow > Issue Type: Improvement > Components: scheduler >Reporter: Adrian Bridgett >Priority: Trivial > Labels: pool > > It would be useful to have tasks use more than a single pool slot. > Our use case is actually to limit how many tasks run on a head node (due to > memory constraints), currently we have to set a pool limit limiting how many > tasks. > Ideally we could set the pool size to e.g amount of memory and then set those > tasks pool_usage (or whatever the option would be called) to the amount of > memory we think they'll use. This way the pool would let lots of small tasks > run or just a few large tasks. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-1467) allow tasks to use more than one pool slot
[ https://issues.apache.org/jira/browse/AIRFLOW-1467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17014522#comment-17014522 ] ASF GitHub Bot commented on AIRFLOW-1467: - dimberman commented on pull request #6975: [AIRFLOW-1467] Dynamic pooling via allowing tasks to use more than one pool slot (depending upon the need) URL: https://github.com/apache/airflow/pull/6975 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > allow tasks to use more than one pool slot > -- > > Key: AIRFLOW-1467 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1467 > Project: Apache Airflow > Issue Type: Improvement > Components: scheduler >Reporter: Adrian Bridgett >Priority: Trivial > Labels: pool > > It would be useful to have tasks use more than a single pool slot. > Our use case is actually to limit how many tasks run on a head node (due to > memory constraints), currently we have to set a pool limit limiting how many > tasks. > Ideally we could set the pool size to e.g amount of memory and then set those > tasks pool_usage (or whatever the option would be called) to the amount of > memory we think they'll use. This way the pool would let lots of small tasks > run or just a few large tasks. -- This message was sent by Atlassian Jira (v8.3.4#803005)
