[jira] [Commented] (AIRFLOW-2371) SubdagOperator ignores pooling

2018-04-24 Thread Tao Feng (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16451482#comment-16451482 ] Tao Feng commented on AIRFLOW-2371: --- Thanks Xiao Zhu. I wonder if some airflow experts could help to

[jira] [Updated] (AIRFLOW-2372) SubDAGs should share dag_concurrency of parent DAG

2018-04-24 Thread Xiao Zhu (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-2372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Zhu updated AIRFLOW-2372: -- Description: It seems like right now subDAGs are scheduled just like normal DAGs, so if a DAG has a

[jira] [Updated] (AIRFLOW-2372) SubDAGs should share dag_concurrency of parent DAG

2018-04-24 Thread Xiao Zhu (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-2372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Zhu updated AIRFLOW-2372: -- Description: It seems like right now subDAGs are scheduled just like normal DAGs, so if a DAG has a

[jira] [Updated] (AIRFLOW-2372) SubDAGs should share dag_concurrency of parent DAG

2018-04-24 Thread Xiao Zhu (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-2372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Zhu updated AIRFLOW-2372: -- Description: It seems like right now subDAGs are scheduled just like normal DAGs, so if a DAG has a

[jira] [Updated] (AIRFLOW-2372) SubDAGs should share dag_concurrency of parent DAG

2018-04-24 Thread Xiao Zhu (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-2372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Zhu updated AIRFLOW-2372: -- Summary: SubDAGs should share dag_concurrency of parent DAG (was: SubDAGs should share parallelism

[jira] [Issue Comment Deleted] (AIRFLOW-2340) SQLalchemy pessimistic connection handling not working

2018-04-24 Thread Xiao Zhu (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-2340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Zhu updated AIRFLOW-2340: -- Comment: was deleted (was: We see that with our mysql db too) > SQLalchemy pessimistic connection

[jira] [Updated] (AIRFLOW-2372) SubDAGs should share parallelism of parent DAG

2018-04-24 Thread Xiao Zhu (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-2372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Zhu updated AIRFLOW-2372: -- Description: It seems like right now subDAGs are scheduled just like normal DAGs, so if a DAG has a

[jira] [Updated] (AIRFLOW-2372) SubDAGs should share parallelism of parent DAG

2018-04-24 Thread Xiao Zhu (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-2372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Zhu updated AIRFLOW-2372: -- Description: It seems like right now subDAGs are scheduled just like normal DAGs, so if a DAG has a

[jira] [Created] (AIRFLOW-2372) SubDAGs should share parallelism of parent DAG

2018-04-24 Thread Xiao Zhu (JIRA)
Xiao Zhu created AIRFLOW-2372: - Summary: SubDAGs should share parallelism of parent DAG Key: AIRFLOW-2372 URL: https://issues.apache.org/jira/browse/AIRFLOW-2372 Project: Apache Airflow Issue

[jira] [Updated] (AIRFLOW-2371) SubdagOperator ignores pooling

2018-04-24 Thread Xiao Zhu (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Zhu updated AIRFLOW-2371: -- Summary: SubdagOperator ignores pooling (was: SubdagOperator ignores pooling when not using

[jira] [Updated] (AIRFLOW-2371) SubdagOperator ignores pooling when not using SequentialExecutor

2018-04-24 Thread Xiao Zhu (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Zhu updated AIRFLOW-2371: -- Description: Basically we need a long term solution for

[jira] [Created] (AIRFLOW-2371) SubdagOperator ignores pooling when not using SequentialExecutor

2018-04-24 Thread Xiao Zhu (JIRA)
Xiao Zhu created AIRFLOW-2371: - Summary: SubdagOperator ignores pooling when not using SequentialExecutor Key: AIRFLOW-2371 URL: https://issues.apache.org/jira/browse/AIRFLOW-2371 Project: Apache Airflow

[jira] [Updated] (AIRFLOW-2361) Allow GoogleCloudStorageToGoogleCloudStorageOperator to store list of copied files to XCom

2018-04-24 Thread Berislav Lopac (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-2361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Berislav Lopac updated AIRFLOW-2361: Description: When {{GoogleCloudStorageToGoogleCloudStorageOperator}} is used with a

[jira] [Updated] (AIRFLOW-2361) Allow GoogleCloudStorageToGoogleCloudStorageOperator to store list of copied files to XCom

2018-04-24 Thread Berislav Lopac (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-2361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Berislav Lopac updated AIRFLOW-2361: Description: When {{GoogleCloudStorageToGoogleCloudStorageOperator}} is used with a

[jira] [Updated] (AIRFLOW-2361) Allow GoogleCloudStorageToGoogleCloudStorageOperator to store list of copied files to XCom

2018-04-24 Thread Berislav Lopac (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-2361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Berislav Lopac updated AIRFLOW-2361: Description: When {{GoogleCloudStorageToGoogleCloudStorageOperator}} is used with a

[jira] [Updated] (AIRFLOW-2361) Allow GoogleCloudStorageToGoogleCloudStorageOperator to store list of copied files to XCom

2018-04-24 Thread Berislav Lopac (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-2361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Berislav Lopac updated AIRFLOW-2361: Description: When {{GoogleCloudStorageToGoogleCloudStorageOperator}} is used with a

[jira] [Updated] (AIRFLOW-2361) Allow GoogleCloudStorageToGoogleCloudStorageOperator to store list of copied files to XCom

2018-04-24 Thread Berislav Lopac (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-2361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Berislav Lopac updated AIRFLOW-2361: Description: When {{GoogleCloudStorageToGoogleCloudStorageOperator}} is used with a

[jira] [Commented] (AIRFLOW-74) SubdagOperators can consume all celeryd worker processes

2018-04-24 Thread Xiao Zhu (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-74?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450416#comment-16450416 ] Xiao Zhu commented on AIRFLOW-74: - [~TaoFeng] fair. Can we have a JIRA ticket for the long term solution?

[jira] [Commented] (AIRFLOW-74) SubdagOperators can consume all celeryd worker processes

2018-04-24 Thread Tao Feng (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-74?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450411#comment-16450411 ] Tao Feng commented on AIRFLOW-74: - [~xzhu] , just to be clear, you could still use celeryExectuor etc for

[jira] [Commented] (AIRFLOW-74) SubdagOperators can consume all celeryd worker processes

2018-04-24 Thread Tao Feng (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-74?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450406#comment-16450406 ] Tao Feng commented on AIRFLOW-74: - [~xzhu] , There are some issues for subdag to use with other

[jira] [Commented] (AIRFLOW-74) SubdagOperators can consume all celeryd worker processes

2018-04-24 Thread Xiao Zhu (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-74?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450401#comment-16450401 ] Xiao Zhu commented on AIRFLOW-74: - So we have to use SequentialExecutor for subdags? isn't it possibly

[jira] [Resolved] (AIRFLOW-74) SubdagOperators can consume all celeryd worker processes

2018-04-24 Thread Arthur Wiedmer (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-74?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arthur Wiedmer resolved AIRFLOW-74. --- Resolution: Fixed Fix Version/s: 1.10.0 Issue resolved by pull request #3251

[jira] [Commented] (AIRFLOW-74) SubdagOperators can consume all celeryd worker processes

2018-04-24 Thread ASF subversion and git services (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-74?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450234#comment-16450234 ] ASF subversion and git services commented on AIRFLOW-74: Commit

incubator-airflow git commit: [AIRFLOW-74] SubdagOperators can consume all celeryd worker processes

2018-04-24 Thread arthur
Repository: incubator-airflow Updated Branches: refs/heads/master be886b986 -> 64d950166 [AIRFLOW-74] SubdagOperators can consume all celeryd worker processes Closes #3251 from feng-tao/airflow-74 Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit:

[jira] [Created] (AIRFLOW-2370) Unable to script user account creation

2018-04-24 Thread William Pursell (JIRA)
William Pursell created AIRFLOW-2370: Summary: Unable to script user account creation Key: AIRFLOW-2370 URL: https://issues.apache.org/jira/browse/AIRFLOW-2370 Project: Apache Airflow

[jira] [Commented] (AIRFLOW-2250) execution_date logic seems broken in 1.9

2018-04-24 Thread sulphide (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-2250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450009#comment-16450009 ] sulphide commented on AIRFLOW-2250: --- a workaround we've found is to manually set the date on the first

[jira] [Commented] (AIRFLOW-2363) S3 remote logging appending tuple instead of str

2018-04-24 Thread Kyle Hamlin (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-2363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449691#comment-16449691 ] Kyle Hamlin commented on AIRFLOW-2363: -- Yes, I'll test this today and let you know the outcome.

[jira] [Resolved] (AIRFLOW-2369) Fix failing GCS test

2018-04-24 Thread Bolke de Bruin (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-2369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin resolved AIRFLOW-2369. - Resolution: Fixed Fix Version/s: 1.10.0 Issue resolved by pull request #3260

[jira] [Commented] (AIRFLOW-2369) Fix failing GCS test

2018-04-24 Thread ASF subversion and git services (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-2369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449422#comment-16449422 ] ASF subversion and git services commented on AIRFLOW-2369: -- Commit

[jira] [Commented] (AIRFLOW-2369) Fix failing GCS test

2018-04-24 Thread ASF subversion and git services (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-2369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449421#comment-16449421 ] ASF subversion and git services commented on AIRFLOW-2369: -- Commit

incubator-airflow git commit: [AIRFLOW-2369] Fix gcs tests

2018-04-24 Thread bolke
Repository: incubator-airflow Updated Branches: refs/heads/v1-10-test a935e11ac -> c1264e714 [AIRFLOW-2369] Fix gcs tests The version was hardcoded and would break if you update the version of Apache Airflow Closes #3260 from Fokko/airflow-2369-fix-tests (cherry picked from commit

incubator-airflow git commit: [AIRFLOW-2369] Fix gcs tests

2018-04-24 Thread bolke
Repository: incubator-airflow Updated Branches: refs/heads/master 813136f45 -> be886b986 [AIRFLOW-2369] Fix gcs tests The version was hardcoded and would break if you update the version of Apache Airflow Closes #3260 from Fokko/airflow-2369-fix-tests Project:

[jira] [Commented] (AIRFLOW-1729) Ignore whole directories in .airflowignore

2018-04-24 Thread JIRA
[ https://issues.apache.org/jira/browse/AIRFLOW-1729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449376#comment-16449376 ] Christoph Hösler commented on AIRFLOW-1729: --- I don't think this was fixed with #3171, which

[jira] [Resolved] (AIRFLOW-2365) Fix autocommit test issue with SQLite

2018-04-24 Thread Bolke de Bruin (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-2365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin resolved AIRFLOW-2365. - Resolution: Fixed Fix Version/s: 1.10.0 > Fix autocommit test issue with SQLite

[jira] [Commented] (AIRFLOW-2365) Fix autocommit test issue with SQLite

2018-04-24 Thread ASF subversion and git services (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-2365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449363#comment-16449363 ] ASF subversion and git services commented on AIRFLOW-2365: -- Commit

incubator-airflow git commit: [AIRFLOW-2365] Fix autocommit attribute check

2018-04-24 Thread bolke
Repository: incubator-airflow Updated Branches: refs/heads/v1-10-test d96534c26 -> a935e11ac [AIRFLOW-2365] Fix autocommit attribute check Closes #3258 from artwr/awiedmer-fix-dbapi-test- issue (cherry picked from commit 813136f457b554561402b4a2510a2a4234f76b6a) Signed-off-by: Bolke de Bruin

[jira] [Commented] (AIRFLOW-2365) Fix autocommit test issue with SQLite

2018-04-24 Thread ASF subversion and git services (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-2365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449361#comment-16449361 ] ASF subversion and git services commented on AIRFLOW-2365: -- Commit

incubator-airflow git commit: [AIRFLOW-2365] Fix autocommit attribute check

2018-04-24 Thread bolke
Repository: incubator-airflow Updated Branches: refs/heads/master 1f86299cf -> 813136f45 [AIRFLOW-2365] Fix autocommit attribute check Closes #3258 from artwr/awiedmer-fix-dbapi-test- issue Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit:

[jira] [Assigned] (AIRFLOW-2369) Fix failing GCS test

2018-04-24 Thread Fokko Driesprong (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-2369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong reassigned AIRFLOW-2369: - Assignee: Fokko Driesprong > Fix failing GCS test > > >

[jira] [Created] (AIRFLOW-2369) Fix failing GCS test

2018-04-24 Thread Fokko Driesprong (JIRA)
Fokko Driesprong created AIRFLOW-2369: - Summary: Fix failing GCS test Key: AIRFLOW-2369 URL: https://issues.apache.org/jira/browse/AIRFLOW-2369 Project: Apache Airflow Issue Type: