[jira] [Created] (AIRFLOW-5887) User can be removed from CI images

2019-11-10 Thread Jarek Potiuk (Jira)
Jarek Potiuk created AIRFLOW-5887: - Summary: User can be removed from CI images Key: AIRFLOW-5887 URL: https://issues.apache.org/jira/browse/AIRFLOW-5887 Project: Apache Airflow Issue Type:

[jira] [Commented] (AIRFLOW-5888) Use psycopg2-binary package

2019-11-10 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971249#comment-16971249 ] ASF GitHub Bot commented on AIRFLOW-5888: - potiuk commented on pull request #6533:

[GitHub] [airflow] potiuk merged pull request #6533: [AIRFLOW-5888] Use psycopg2-binary for postgres operations

2019-11-10 Thread GitBox
potiuk merged pull request #6533: [AIRFLOW-5888] Use psycopg2-binary for postgres operations URL: https://github.com/apache/airflow/pull/6533 This is an automated message from the Apache Git Service. To respond to the

[jira] [Updated] (AIRFLOW-5889) AWS Batch Operator - API request limits should not fail a task

2019-11-10 Thread Darren Weber (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Darren Weber updated AIRFLOW-5889: -- Description: The AWS Batch Operator attempts to use a boto3 feature that is not available

[jira] [Updated] (AIRFLOW-5889) AWS Batch Operator - API request limits should not fail a task

2019-11-10 Thread Darren Weber (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Darren Weber updated AIRFLOW-5889: -- Description: The AWS Batch Operator attempts to use a boto3 feature that is not available

[GitHub] [airflow] codecov-io edited a comment on issue #6529: [AIRFLOW-5877][depends on AIRFLOW-5876][part of AIRFLOW-5716] Improve job_id detection in DataflowRunner

2019-11-10 Thread GitBox
codecov-io edited a comment on issue #6529: [AIRFLOW-5877][depends on AIRFLOW-5876][part of AIRFLOW-5716] Improve job_id detection in DataflowRunner URL: https://github.com/apache/airflow/pull/6529#issuecomment-551952578 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6529?src=pr=h1)

[GitHub] [airflow] codecov-io edited a comment on issue #6529: [AIRFLOW-5877][depends on AIRFLOW-5876][part of AIRFLOW-5716] Improve job_id detection in DataflowRunner

2019-11-10 Thread GitBox
codecov-io edited a comment on issue #6529: [AIRFLOW-5877][depends on AIRFLOW-5876][part of AIRFLOW-5716] Improve job_id detection in DataflowRunner URL: https://github.com/apache/airflow/pull/6529#issuecomment-551952578 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6529?src=pr=h1)

[GitHub] [airflow] codecov-io edited a comment on issue #6529: [AIRFLOW-5877][depends on AIRFLOW-5876][part of AIRFLOW-5716] Improve job_id detection in DataflowRunner

2019-11-10 Thread GitBox
codecov-io edited a comment on issue #6529: [AIRFLOW-5877][depends on AIRFLOW-5876][part of AIRFLOW-5716] Improve job_id detection in DataflowRunner URL: https://github.com/apache/airflow/pull/6529#issuecomment-551952578 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6529?src=pr=h1)

[jira] [Updated] (AIRFLOW-5889) AWS Batch Operator - API request limits should not fail a task

2019-11-10 Thread Darren Weber (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Darren Weber updated AIRFLOW-5889: -- Description: The AWS Batch Operator attempts to use a boto3 feature that is not available

[GitHub] [airflow] codecov-io edited a comment on issue #6540: [AIRFLOW-5887] User is removed from CI images. Depends on [AIRFLOW-5886]

2019-11-10 Thread GitBox
codecov-io edited a comment on issue #6540: [AIRFLOW-5887] User is removed from CI images. Depends on [AIRFLOW-5886] URL: https://github.com/apache/airflow/pull/6540#issuecomment-552254922 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6540?src=pr=h1) Report > Merging

[GitHub] [airflow] codecov-io commented on issue #6540: [AIRFLOW-5887] User is removed from CI images. Depends on [AIRFLOW-5886]

2019-11-10 Thread GitBox
codecov-io commented on issue #6540: [AIRFLOW-5887] User is removed from CI images. Depends on [AIRFLOW-5886] URL: https://github.com/apache/airflow/pull/6540#issuecomment-552254922 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6540?src=pr=h1) Report > Merging

[GitHub] [airflow] potiuk commented on issue #6533: [AIRFLOW-5888] Use psycopg2-binary for postgres operations

2019-11-10 Thread GitBox
potiuk commented on issue #6533: [AIRFLOW-5888] Use psycopg2-binary for postgres operations URL: https://github.com/apache/airflow/pull/6533#issuecomment-552243144 I also cherry-picked it to v1-10-test so that it will get to 1.10.7

[jira] [Resolved] (AIRFLOW-5888) Use psycopg2-binary package

2019-11-10 Thread Jarek Potiuk (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5888. --- Resolution: Fixed > Use psycopg2-binary package > --- > >

[jira] [Commented] (AIRFLOW-5888) Use psycopg2-binary package

2019-11-10 Thread ASF subversion and git services (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971252#comment-16971252 ] ASF subversion and git services commented on AIRFLOW-5888: -- Commit

[GitHub] [airflow] potiuk commented on issue #6533: [AIRFLOW-5888] Use psycopg2-binary for postgres operations

2019-11-10 Thread GitBox
potiuk commented on issue #6533: [AIRFLOW-5888] Use psycopg2-binary for postgres operations URL: https://github.com/apache/airflow/pull/6533#issuecomment-552242960 Thanks! This is an automated message from the Apache Git

[jira] [Commented] (AIRFLOW-5888) Use psycopg2-binary package

2019-11-10 Thread ASF subversion and git services (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971251#comment-16971251 ] ASF subversion and git services commented on AIRFLOW-5888: -- Commit

[GitHub] [airflow] mik-laj commented on a change in pull request #6530: [AIRFLOW-5878] Use JobID to monitor statuses when running a Dataflow …

2019-11-10 Thread GitBox
mik-laj commented on a change in pull request #6530: [AIRFLOW-5878] Use JobID to monitor statuses when running a Dataflow … URL: https://github.com/apache/airflow/pull/6530#discussion_r344520951 ## File path: tests/gcp/hooks/test_dataflow.py ## @@ -346,6 +348,15 @@ def

[jira] [Updated] (AIRFLOW-5886) Copying sources in Dockerfile can be selective

2019-11-10 Thread Jarek Potiuk (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk updated AIRFLOW-5886: -- Parent: AIRFLOW-5606 Issue Type: Sub-task (was: Bug) > Copying sources in

[jira] [Created] (AIRFLOW-5886) Copying sources in Dockerfile can be selective

2019-11-10 Thread Jarek Potiuk (Jira)
Jarek Potiuk created AIRFLOW-5886: - Summary: Copying sources in Dockerfile can be selective Key: AIRFLOW-5886 URL: https://issues.apache.org/jira/browse/AIRFLOW-5886 Project: Apache Airflow

[jira] [Commented] (AIRFLOW-5886) Copying sources in Dockerfile can be selective

2019-11-10 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971232#comment-16971232 ] ASF GitHub Bot commented on AIRFLOW-5886: - potiuk commented on pull request #6538:

[GitHub] [airflow] potiuk opened a new pull request #6538: [AIRFLOW-5886] Selective copying of sources in Docker image depends on [AIRFLOW-5885]

2019-11-10 Thread GitBox
potiuk opened a new pull request #6538: [AIRFLOW-5886] Selective copying of sources in Docker image depends on [AIRFLOW-5885] URL: https://github.com/apache/airflow/pull/6538 Note for reviewed - this change depends on #6536 so please review only last commit. This change further

[GitHub] [airflow] potiuk commented on issue #6538: [AIRFLOW-5886] Selective copying of sources in Docker image depends on [AIRFLOW-5885]

2019-11-10 Thread GitBox
potiuk commented on issue #6538: [AIRFLOW-5886] Selective copying of sources in Docker image depends on [AIRFLOW-5885] URL: https://github.com/apache/airflow/pull/6538#issuecomment-552235090 Another step in improving Docker image/building production one.

[jira] [Created] (AIRFLOW-5888) Use psycopg2-binary package

2019-11-10 Thread Matthew Bowden (Jira)
Matthew Bowden created AIRFLOW-5888: --- Summary: Use psycopg2-binary package Key: AIRFLOW-5888 URL: https://issues.apache.org/jira/browse/AIRFLOW-5888 Project: Apache Airflow Issue Type:

[GitHub] [airflow] mik-laj commented on issue #6527: [AIRFLOW-5876][part of AIRFLOW-5697][depends on AIRFLOW-5716] Fetch all Dataflow jobs during searching

2019-11-10 Thread GitBox
mik-laj commented on issue #6527: [AIRFLOW-5876][part of AIRFLOW-5697][depends on AIRFLOW-5716] Fetch all Dataflow jobs during searching URL: https://github.com/apache/airflow/pull/6527#issuecomment-552242315 Rebased. This

[GitHub] [airflow] mattbowden-ookla commented on issue #6533: [AIRFLOW-5888] Use psycopg2-binary for postgres operations

2019-11-10 Thread GitBox
mattbowden-ookla commented on issue #6533: [AIRFLOW-5888] Use psycopg2-binary for postgres operations URL: https://github.com/apache/airflow/pull/6533#issuecomment-552242349 Update to reference JIRA ticket! This is an

[GitHub] [airflow] mik-laj commented on a change in pull request #6530: [AIRFLOW-5878] Use JobID to monitor statuses when running a Dataflow …

2019-11-10 Thread GitBox
mik-laj commented on a change in pull request #6530: [AIRFLOW-5878] Use JobID to monitor statuses when running a Dataflow … URL: https://github.com/apache/airflow/pull/6530#discussion_r344520883 ## File path: tests/gcp/hooks/test_dataflow.py ## @@ -315,9 +315,10 @@ def

[GitHub] [airflow] codecov-io commented on issue #6530: [AIRFLOW-5878] Use JobID to monitor statuses when running a Dataflow …

2019-11-10 Thread GitBox
codecov-io commented on issue #6530: [AIRFLOW-5878] Use JobID to monitor statuses when running a Dataflow … URL: https://github.com/apache/airflow/pull/6530#issuecomment-552272723 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6530?src=pr=h1) Report > Merging

[GitHub] [airflow] codecov-io edited a comment on issue #6530: [AIRFLOW-5878] Use JobID to monitor statuses when running a Dataflow …

2019-11-10 Thread GitBox
codecov-io edited a comment on issue #6530: [AIRFLOW-5878] Use JobID to monitor statuses when running a Dataflow … URL: https://github.com/apache/airflow/pull/6530#issuecomment-552272723 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6530?src=pr=h1) Report > Merging

[GitHub] [airflow] codecov-io edited a comment on issue #6530: [AIRFLOW-5878] Use JobID to monitor statuses when running a Dataflow …

2019-11-10 Thread GitBox
codecov-io edited a comment on issue #6530: [AIRFLOW-5878] Use JobID to monitor statuses when running a Dataflow … URL: https://github.com/apache/airflow/pull/6530#issuecomment-552272723 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6530?src=pr=h1) Report > Merging

[GitHub] [airflow] ratb3rt opened a new pull request #6539: [AIRFLOW-5783] AIP-21 Move aws redshift into providers structure

2019-11-10 Thread GitBox
ratb3rt opened a new pull request #6539: [AIRFLOW-5783] AIP-21 Move aws redshift into providers structure URL: https://github.com/apache/airflow/pull/6539 Make sure you have checked _all_ steps below. ### Jira - [] My PR addresses the following [Airflow

[jira] [Commented] (AIRFLOW-5783) Migrate AWS Redshift to /providers/aws [AIP-21]

2019-11-10 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971235#comment-16971235 ] ASF GitHub Bot commented on AIRFLOW-5783: - ratb3rt commented on pull request #6539:

[GitHub] [airflow] potiuk commented on issue #6529: [AIRFLOW-5877][depends on AIRFLOW-5876, AIRFLOW-5716][part of AIRFLOW-5697] Improve job_id detection in DataflowRunner

2019-11-10 Thread GitBox
potiuk commented on issue #6529: [AIRFLOW-5877][depends on AIRFLOW-5876,AIRFLOW-5716][part of AIRFLOW-5697] Improve job_id detection in DataflowRunner URL: https://github.com/apache/airflow/pull/6529#issuecomment-552241885 Same here :(

[jira] [Commented] (AIRFLOW-5716) Simplify DataflowJobsController logic

2019-11-10 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971246#comment-16971246 ] ASF GitHub Bot commented on AIRFLOW-5716: - potiuk commented on pull request #6386:

[GitHub] [airflow] potiuk merged pull request #6386: [AIRFLOW-5716][part of AIRFLOW-5697] Simplify DataflowJobsController logic

2019-11-10 Thread GitBox
potiuk merged pull request #6386: [AIRFLOW-5716][part of AIRFLOW-5697] Simplify DataflowJobsController logic URL: https://github.com/apache/airflow/pull/6386 This is an automated message from the Apache Git Service. To

[jira] [Created] (AIRFLOW-5890) Many dag task status is skiped

2019-11-10 Thread yangming (Jira)
yangming created AIRFLOW-5890: - Summary: Many dag task status is skiped Key: AIRFLOW-5890 URL: https://issues.apache.org/jira/browse/AIRFLOW-5890 Project: Apache Airflow Issue Type: Task

[GitHub] [airflow] potiuk commented on issue #6536: [AIRFLOW-5885] List of tests is generated by pre-commit not Docker build

2019-11-10 Thread GitBox
potiuk commented on issue #6536: [AIRFLOW-5885] List of tests is generated by pre-commit not Docker build URL: https://github.com/apache/airflow/pull/6536#issuecomment-552232295 One of the next steps towards simpler / better image + production one. This make the CI images build faster +

[GitHub] [airflow] ratb3rt commented on issue #6539: [AIRFLOW-5783] AIP-21 Move aws redshift into providers structure

2019-11-10 Thread GitBox
ratb3rt commented on issue #6539: [AIRFLOW-5783] AIP-21 Move aws redshift into providers structure URL: https://github.com/apache/airflow/pull/6539#issuecomment-552235723 Should this PR also: a) also move the s3_to_redshift + redshift_to_s3 operators from /operators b) fix/noqa the

[GitHub] [airflow] potiuk commented on a change in pull request #6539: [AIRFLOW-5783] AIP-21 Move aws redshift into providers structure

2019-11-10 Thread GitBox
potiuk commented on a change in pull request #6539: [AIRFLOW-5783] AIP-21 Move aws redshift into providers structure URL: https://github.com/apache/airflow/pull/6539#discussion_r344516551 ## File path: scripts/ci/pylint_todo.txt ## @@ -20,7 +20,7 @@

[GitHub] [airflow] potiuk edited a comment on issue #6539: [AIRFLOW-5783] AIP-21 Move aws redshift into providers structure

2019-11-10 Thread GitBox
potiuk edited a comment on issue #6539: [AIRFLOW-5783] AIP-21 Move aws redshift into providers structure URL: https://github.com/apache/airflow/pull/6539#issuecomment-552237261 > Should this PR also: > a) also move the s3_to_redshift + redshift_to_s3 operators from /operators

[GitHub] [airflow] potiuk commented on issue #6539: [AIRFLOW-5783] AIP-21 Move aws redshift into providers structure

2019-11-10 Thread GitBox
potiuk commented on issue #6539: [AIRFLOW-5783] AIP-21 Move aws redshift into providers structure URL: https://github.com/apache/airflow/pull/6539#issuecomment-552237261 > Should this PR also: > a) also move the s3_to_redshift + redshift_to_s3 operators from /operators This can be

[GitHub] [airflow] potiuk commented on issue #6519: [AIRFLOW-5869] BugFix: Creating DagRuns fails for Deserialized tasks …

2019-11-10 Thread GitBox
potiuk commented on issue #6519: [AIRFLOW-5869] BugFix: Creating DagRuns fails for Deserialized tasks … URL: https://github.com/apache/airflow/pull/6519#issuecomment-552243817 Just make sure description fits in 80 chars when merging.

[GitHub] [airflow] potiuk merged pull request #6511: [AIRFLOW-XXX] Improve the PubSub documentation

2019-11-10 Thread GitBox
potiuk merged pull request #6511: [AIRFLOW-XXX] Improve the PubSub documentation URL: https://github.com/apache/airflow/pull/6511 This is an automated message from the Apache Git Service. To respond to the message, please

[jira] [Created] (AIRFLOW-5889) AWS Batch Operator - API request limits should not fail a task

2019-11-10 Thread Darren Weber (Jira)
Darren Weber created AIRFLOW-5889: - Summary: AWS Batch Operator - API request limits should not fail a task Key: AIRFLOW-5889 URL: https://issues.apache.org/jira/browse/AIRFLOW-5889 Project: Apache

[GitHub] [airflow] potiuk opened a new pull request #6540: [AIRFLOW-5887] User is removed from CI images. Depends on [AIRFLOW-5886]

2019-11-10 Thread GitBox
potiuk opened a new pull request #6540: [AIRFLOW-5887] User is removed from CI images. Depends on [AIRFLOW-5886] URL: https://github.com/apache/airflow/pull/6540 Note for reviewer: please review only last commit as it is based on #6538 The AIRFLOW_USER is not needed any more in CI

[jira] [Commented] (AIRFLOW-5887) User can be removed from CI images

2019-11-10 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971237#comment-16971237 ] ASF GitHub Bot commented on AIRFLOW-5887: - potiuk commented on pull request #6540:

[GitHub] [airflow] mik-laj commented on a change in pull request #6539: [AIRFLOW-5783] AIP-21 Move aws redshift into providers structure

2019-11-10 Thread GitBox
mik-laj commented on a change in pull request #6539: [AIRFLOW-5783] AIP-21 Move aws redshift into providers structure URL: https://github.com/apache/airflow/pull/6539#discussion_r344518572 ## File path: scripts/ci/pylint_todo.txt ## @@ -20,7 +20,7 @@

[GitHub] [airflow] codecov-io edited a comment on issue #6538: [AIRFLOW-5886] Selective copying of sources in Docker image. Depends on [AIRFLOW-5885] [AIRFLOW-5887]

2019-11-10 Thread GitBox
codecov-io edited a comment on issue #6538: [AIRFLOW-5886] Selective copying of sources in Docker image. Depends on [AIRFLOW-5885] [AIRFLOW-5887] URL: https://github.com/apache/airflow/pull/6538#issuecomment-552258036 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6538?src=pr=h1)

[GitHub] [airflow] codecov-io commented on issue #6538: [AIRFLOW-5886] Selective copying of sources in Docker image. Depends on [AIRFLOW-5885] [AIRFLOW-5887]

2019-11-10 Thread GitBox
codecov-io commented on issue #6538: [AIRFLOW-5886] Selective copying of sources in Docker image. Depends on [AIRFLOW-5885] [AIRFLOW-5887] URL: https://github.com/apache/airflow/pull/6538#issuecomment-552258036 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6538?src=pr=h1) Report

[GitHub] [airflow] codecov-io edited a comment on issue #6538: [AIRFLOW-5886] Selective copying of sources in Docker image. Depends on [AIRFLOW-5885] [AIRFLOW-5887]

2019-11-10 Thread GitBox
codecov-io edited a comment on issue #6538: [AIRFLOW-5886] Selective copying of sources in Docker image. Depends on [AIRFLOW-5885] [AIRFLOW-5887] URL: https://github.com/apache/airflow/pull/6538#issuecomment-552258036 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6538?src=pr=h1)

[GitHub] [airflow] potiuk commented on issue #6527: [AIRFLOW-5876][part of AIRFLOW-5697][depends on AIRFLOW-5716] Fetch all Dataflow jobs during searching

2019-11-10 Thread GitBox
potiuk commented on issue #6527: [AIRFLOW-5876][part of AIRFLOW-5697][depends on AIRFLOW-5716] Fetch all Dataflow jobs during searching URL: https://github.com/apache/airflow/pull/6527#issuecomment-552241864 You need to rebase @mik-laj :(

[GitHub] [airflow] potiuk edited a comment on issue #6529: [AIRFLOW-5877][depends on AIRFLOW-5876, AIRFLOW-5716][part of AIRFLOW-5697] Improve job_id detection in DataflowRunner

2019-11-10 Thread GitBox
potiuk edited a comment on issue #6529: [AIRFLOW-5877][depends on AIRFLOW-5876,AIRFLOW-5716][part of AIRFLOW-5697] Improve job_id detection in DataflowRunner URL: https://github.com/apache/airflow/pull/6529#issuecomment-552241885 Same here :( Rebase needeed.

[GitHub] [airflow] mik-laj commented on issue #6529: [AIRFLOW-5877][depends on AIRFLOW-5876, AIRFLOW-5716][part of AIRFLOW-5697] Improve job_id detection in DataflowRunner

2019-11-10 Thread GitBox
mik-laj commented on issue #6529: [AIRFLOW-5877][depends on AIRFLOW-5876,AIRFLOW-5716][part of AIRFLOW-5697] Improve job_id detection in DataflowRunner URL: https://github.com/apache/airflow/pull/6529#issuecomment-552241951 @potiuk Can you do a review per commit? Each changes contains

[jira] [Commented] (AIRFLOW-5716) Simplify DataflowJobsController logic

2019-11-10 Thread ASF subversion and git services (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971247#comment-16971247 ] ASF subversion and git services commented on AIRFLOW-5716: -- Commit

[GitHub] [airflow] mik-laj commented on issue #6529: [AIRFLOW-5877][depends on AIRFLOW-5876, AIRFLOW-5716][part of AIRFLOW-5697] Improve job_id detection in DataflowRunner

2019-11-10 Thread GitBox
mik-laj commented on issue #6529: [AIRFLOW-5877][depends on AIRFLOW-5876,AIRFLOW-5716][part of AIRFLOW-5697] Improve job_id detection in DataflowRunner URL: https://github.com/apache/airflow/pull/6529#issuecomment-552242068 In the meantime, it will delete the commit from the branch and

[GitHub] [airflow] mik-laj commented on a change in pull request #6539: [AIRFLOW-5783] AIP-21 Move aws redshift into providers structure

2019-11-10 Thread GitBox
mik-laj commented on a change in pull request #6539: [AIRFLOW-5783] AIP-21 Move aws redshift into providers structure URL: https://github.com/apache/airflow/pull/6539#discussion_r344518572 ## File path: scripts/ci/pylint_todo.txt ## @@ -20,7 +20,7 @@

[GitHub] [airflow] mik-laj commented on a change in pull request #6539: [AIRFLOW-5783] AIP-21 Move aws redshift into providers structure

2019-11-10 Thread GitBox
mik-laj commented on a change in pull request #6539: [AIRFLOW-5783] AIP-21 Move aws redshift into providers structure URL: https://github.com/apache/airflow/pull/6539#discussion_r344518572 ## File path: scripts/ci/pylint_todo.txt ## @@ -20,7 +20,7 @@

[GitHub] [airflow] mattbowden-ookla commented on issue #6533: [AIRFLOW-5888] Use psycopg2-binary for postgres operations

2019-11-10 Thread GitBox
mattbowden-ookla commented on issue #6533: [AIRFLOW-5888] Use psycopg2-binary for postgres operations URL: https://github.com/apache/airflow/pull/6533#issuecomment-552243439 > I also cherry-picked it to v1-10-test so that it will get to 1.10.7 Oh awesome, thanks!

[GitHub] [airflow] ddelange commented on issue #6523: [AIRFLOW-5873] KubernetesPodOperator fixes and test

2019-11-10 Thread GitBox
ddelange commented on issue #6523: [AIRFLOW-5873] KubernetesPodOperator fixes and test URL: https://github.com/apache/airflow/pull/6523#issuecomment-552190688 On my local machine I get the same error as Travis, so I added `origin/` to the command. ```sh $ git diff --name-only

[GitHub] [airflow] codecov-io edited a comment on issue #6489: [AIRFLOW-3959] [AIRFLOW-4026] Add filter by DAG tags

2019-11-10 Thread GitBox
codecov-io edited a comment on issue #6489: [AIRFLOW-3959] [AIRFLOW-4026] Add filter by DAG tags URL: https://github.com/apache/airflow/pull/6489#issuecomment-552128422 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6489?src=pr=h1) Report > Merging

[GitHub] [airflow] potiuk merged pull request #6502: [AIRFLOW-5786] Migrate AWS SNS to /providers/amazon/aws

2019-11-10 Thread GitBox
potiuk merged pull request #6502: [AIRFLOW-5786] Migrate AWS SNS to /providers/amazon/aws URL: https://github.com/apache/airflow/pull/6502 This is an automated message from the Apache Git Service. To respond to the message,

[jira] [Commented] (AIRFLOW-5786) Migrate AWS SNS to /providers/aws [AIP-21]

2019-11-10 Thread ASF subversion and git services (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971118#comment-16971118 ] ASF subversion and git services commented on AIRFLOW-5786: -- Commit

[jira] [Commented] (AIRFLOW-5786) Migrate AWS SNS to /providers/aws [AIP-21]

2019-11-10 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971117#comment-16971117 ] ASF GitHub Bot commented on AIRFLOW-5786: - potiuk commented on pull request #6502:

[jira] [Commented] (AIRFLOW-5786) Migrate AWS SNS to /providers/aws [AIP-21]

2019-11-10 Thread ASF subversion and git services (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971119#comment-16971119 ] ASF subversion and git services commented on AIRFLOW-5786: -- Commit

[jira] [Resolved] (AIRFLOW-5786) Migrate AWS SNS to /providers/aws [AIP-21]

2019-11-10 Thread Jarek Potiuk (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5786. --- Fix Version/s: 2.0.0 Resolution: Fixed > Migrate AWS SNS to /providers/aws [AIP-21]

[GitHub] [airflow] potiuk merged pull request #6535: [AIRFLOW-5803] Update S3Hook import paths [AIP-21]

2019-11-10 Thread GitBox
potiuk merged pull request #6535: [AIRFLOW-5803] Update S3Hook import paths [AIP-21] URL: https://github.com/apache/airflow/pull/6535 This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [airflow] potiuk commented on issue #6535: [AIRFLOW-5803] Update S3Hook import paths [AIP-21]

2019-11-10 Thread GitBox
potiuk commented on issue #6535: [AIRFLOW-5803] Update S3Hook import paths [AIP-21] URL: https://github.com/apache/airflow/pull/6535#issuecomment-552192592 Thanks @mingrammer ! This is an automated message from the Apache

[GitHub] [airflow] feluelle commented on a change in pull request #6489: [AIRFLOW-3959] [AIRFLOW-4026] Add filter by DAG tags

2019-11-10 Thread GitBox
feluelle commented on a change in pull request #6489: [AIRFLOW-3959] [AIRFLOW-4026] Add filter by DAG tags URL: https://github.com/apache/airflow/pull/6489#discussion_r344484478 ## File path: airflow/models/dag.py ## @@ -186,6 +187,8 @@ class DAG(BaseDag, LoggingMixin):

[jira] [Commented] (AIRFLOW-5883) Fix inefficient sqlalchemy .count() queries

2019-11-10 Thread ASF subversion and git services (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971104#comment-16971104 ] ASF subversion and git services commented on AIRFLOW-5883: -- Commit

[jira] [Resolved] (AIRFLOW-5883) Fix inefficient sqlalchemy .count() queries

2019-11-10 Thread Felix Uellendall (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall resolved AIRFLOW-5883. --- Resolution: Fixed > Fix inefficient sqlalchemy .count() queries >

[GitHub] [airflow] feluelle merged pull request #6532: [AIRFLOW-5883] Don't use .count() from sqlalchemy to count

2019-11-10 Thread GitBox
feluelle merged pull request #6532: [AIRFLOW-5883] Don't use .count() from sqlalchemy to count URL: https://github.com/apache/airflow/pull/6532 This is an automated message from the Apache Git Service. To respond to the

[jira] [Commented] (AIRFLOW-5883) Fix inefficient sqlalchemy .count() queries

2019-11-10 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971103#comment-16971103 ] ASF GitHub Bot commented on AIRFLOW-5883: - feluelle commented on pull request #6532:

[jira] [Updated] (AIRFLOW-5883) Fix inefficient sqlalchemy .count() queries

2019-11-10 Thread Felix Uellendall (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall updated AIRFLOW-5883: -- Fix Version/s: 1.10.7 > Fix inefficient sqlalchemy .count() queries >

[GitHub] [airflow] feluelle commented on a change in pull request #6530: [AIRFLOW-5878] Use JobID to monitor statuses when running a Dataflow …

2019-11-10 Thread GitBox
feluelle commented on a change in pull request #6530: [AIRFLOW-5878] Use JobID to monitor statuses when running a Dataflow … URL: https://github.com/apache/airflow/pull/6530#discussion_r344486802 ## File path: tests/gcp/hooks/test_dataflow.py ## @@ -315,9 +315,10 @@ def

[GitHub] [airflow] feluelle commented on a change in pull request #6530: [AIRFLOW-5878] Use JobID to monitor statuses when running a Dataflow …

2019-11-10 Thread GitBox
feluelle commented on a change in pull request #6530: [AIRFLOW-5878] Use JobID to monitor statuses when running a Dataflow … URL: https://github.com/apache/airflow/pull/6530#discussion_r344487085 ## File path: tests/gcp/hooks/test_dataflow.py ## @@ -346,6 +348,15 @@ def

[GitHub] [airflow] zacharya19 commented on issue #6489: [AIRFLOW-3959] [AIRFLOW-4026] Add filter by DAG tags

2019-11-10 Thread GitBox
zacharya19 commented on issue #6489: [AIRFLOW-3959] [AIRFLOW-4026] Add filter by DAG tags URL: https://github.com/apache/airflow/pull/6489#issuecomment-552188750 @feluelle Added. This is an automated message from the Apache

[GitHub] [airflow] potiuk commented on a change in pull request #6502: [AIRFLOW-5786] Migrate AWS SNS to /providers/amazon/aws

2019-11-10 Thread GitBox
potiuk commented on a change in pull request #6502: [AIRFLOW-5786] Migrate AWS SNS to /providers/amazon/aws URL: https://github.com/apache/airflow/pull/6502#discussion_r37847 ## File path: UPDATING.md ## @@ -53,8 +53,10 @@ Migrated are: | airflow.hooks.S3_hook.S3Hook

[GitHub] [airflow] codecov-io edited a comment on issue #6489: [AIRFLOW-3959] [AIRFLOW-4026] Add filter by DAG tags

2019-11-10 Thread GitBox
codecov-io edited a comment on issue #6489: [AIRFLOW-3959] [AIRFLOW-4026] Add filter by DAG tags URL: https://github.com/apache/airflow/pull/6489#issuecomment-552128422 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6489?src=pr=h1) Report > Merging

[GitHub] [airflow] codecov-io edited a comment on issue #6489: [AIRFLOW-3959] [AIRFLOW-4026] Add filter by DAG tags

2019-11-10 Thread GitBox
codecov-io edited a comment on issue #6489: [AIRFLOW-3959] [AIRFLOW-4026] Add filter by DAG tags URL: https://github.com/apache/airflow/pull/6489#issuecomment-552128422 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6489?src=pr=h1) Report > Merging

[GitHub] [airflow] codecov-io edited a comment on issue #6489: [AIRFLOW-3959] [AIRFLOW-4026] Add filter by DAG tags

2019-11-10 Thread GitBox
codecov-io edited a comment on issue #6489: [AIRFLOW-3959] [AIRFLOW-4026] Add filter by DAG tags URL: https://github.com/apache/airflow/pull/6489#issuecomment-552128422 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6489?src=pr=h1) Report > Merging

[GitHub] [airflow] codecov-io edited a comment on issue #6489: [AIRFLOW-3959] [AIRFLOW-4026] Add filter by DAG tags

2019-11-10 Thread GitBox
codecov-io edited a comment on issue #6489: [AIRFLOW-3959] [AIRFLOW-4026] Add filter by DAG tags URL: https://github.com/apache/airflow/pull/6489#issuecomment-552128422 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6489?src=pr=h1) Report > Merging

[jira] [Resolved] (AIRFLOW-5803) Update S3Hook import paths

2019-11-10 Thread Jarek Potiuk (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5803. --- Resolution: Fixed > Update S3Hook import paths > -- > >

[jira] [Commented] (AIRFLOW-5803) Update S3Hook import paths

2019-11-10 Thread ASF subversion and git services (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971122#comment-16971122 ] ASF subversion and git services commented on AIRFLOW-5803: -- Commit

[jira] [Commented] (AIRFLOW-5803) Update S3Hook import paths

2019-11-10 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971120#comment-16971120 ] ASF GitHub Bot commented on AIRFLOW-5803: - potiuk commented on pull request #6535:

[jira] [Commented] (AIRFLOW-5803) Update S3Hook import paths

2019-11-10 Thread ASF subversion and git services (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971121#comment-16971121 ] ASF subversion and git services commented on AIRFLOW-5803: -- Commit

[GitHub] [airflow] ddelange edited a comment on issue #6523: [AIRFLOW-5873] KubernetesPodOperator fixes and test

2019-11-10 Thread GitBox
ddelange edited a comment on issue #6523: [AIRFLOW-5873] KubernetesPodOperator fixes and test URL: https://github.com/apache/airflow/pull/6523#issuecomment-552190688 On my local machine I get the same error as Travis. When I add `origin/` to the command it works on my machine, but I get

[GitHub] [airflow] codecov-io edited a comment on issue #6535: [AIRFLOW-5803] Update S3Hook import paths [AIP-21]

2019-11-10 Thread GitBox
codecov-io edited a comment on issue #6535: [AIRFLOW-5803] Update S3Hook import paths [AIP-21] URL: https://github.com/apache/airflow/pull/6535#issuecomment-552180197 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6535?src=pr=h1) Report > Merging

[GitHub] [airflow] codecov-io edited a comment on issue #6535: [AIRFLOW-5803] Update S3Hook import paths [AIP-21]

2019-11-10 Thread GitBox
codecov-io edited a comment on issue #6535: [AIRFLOW-5803] Update S3Hook import paths [AIP-21] URL: https://github.com/apache/airflow/pull/6535#issuecomment-552180197 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6535?src=pr=h1) Report > Merging

[GitHub] [airflow] codecov-io commented on issue #6535: [AIRFLOW-5803] Update S3Hook import paths [AIP-21]

2019-11-10 Thread GitBox
codecov-io commented on issue #6535: [AIRFLOW-5803] Update S3Hook import paths [AIP-21] URL: https://github.com/apache/airflow/pull/6535#issuecomment-552180197 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6535?src=pr=h1) Report > Merging

[GitHub] [airflow] eladkal removed a comment on issue #2715: [AIRFLOW-1641] Handle executor events in the scheduler

2019-11-10 Thread GitBox
eladkal removed a comment on issue #2715: [AIRFLOW-1641] Handle executor events in the scheduler URL: https://github.com/apache/airflow/pull/2715#issuecomment-552182642 @bolkedebruin @saguziel I'm running Airflow 1.10.3 and I experience from time to time problem with DAG is running but

[GitHub] [airflow] eladkal commented on issue #2715: [AIRFLOW-1641] Handle executor events in the scheduler

2019-11-10 Thread GitBox
eladkal commented on issue #2715: [AIRFLOW-1641] Handle executor events in the scheduler URL: https://github.com/apache/airflow/pull/2715#issuecomment-552182642 @bolkedebruin @saguziel I'm running Airflow 1.10.3 and I experience from time to time problem with DAG is running but task is

[GitHub] [airflow] potiuk opened a new pull request #6536: [AIRFLOW-5885] List of tests is generated by pre-commit not Docker build

2019-11-10 Thread GitBox
potiuk opened a new pull request #6536: [AIRFLOW-5885] List of tests is generated by pre-commit not Docker build URL: https://github.com/apache/airflow/pull/6536 The list of tests does not change that often, yet it could not be generated in Docker before all files were added (due to

[GitHub] [airflow] potiuk commented on issue #6533: [AIRFLOW-XXX] Use psycopg2-binary for postgres operations

2019-11-10 Thread GitBox
potiuk commented on issue #6533: [AIRFLOW-XXX] Use psycopg2-binary for postgres operations URL: https://github.com/apache/airflow/pull/6533#issuecomment-552218290 Yep. This is a very nice change but needs a ticket :). This

[GitHub] [airflow] potiuk merged pull request #6528: Update company name and user admin of the platform

2019-11-10 Thread GitBox
potiuk merged pull request #6528: Update company name and user admin of the platform URL: https://github.com/apache/airflow/pull/6528 This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [airflow] nanco2sensei opened a new pull request #6537: [AIRFLOW-XXX] Fix typo in 5 files docstring

2019-11-10 Thread GitBox
nanco2sensei opened a new pull request #6537: [AIRFLOW-XXX] Fix typo in 5 files docstring URL: https://github.com/apache/airflow/pull/6537 I fixed typo from "the the" to "the". ### Jira - [ ] My PR addresses the following [Airflow

[jira] [Commented] (AIRFLOW-5870) Allow infinite pool for tasks

2019-11-10 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971175#comment-16971175 ] ASF GitHub Bot commented on AIRFLOW-5870: - potiuk commented on pull request #6520:

[jira] [Commented] (AIRFLOW-5870) Allow infinite pool for tasks

2019-11-10 Thread ASF subversion and git services (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971176#comment-16971176 ] ASF subversion and git services commented on AIRFLOW-5870: -- Commit

[jira] [Commented] (AIRFLOW-5870) Allow infinite pool for tasks

2019-11-10 Thread ASF subversion and git services (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971177#comment-16971177 ] ASF subversion and git services commented on AIRFLOW-5870: -- Commit

[GitHub] [airflow] potiuk merged pull request #6520: [AIRFLOW-5870] Allow -1 for infinite pool size

2019-11-10 Thread GitBox
potiuk merged pull request #6520: [AIRFLOW-5870] Allow -1 for infinite pool size URL: https://github.com/apache/airflow/pull/6520 This is an automated message from the Apache Git Service. To respond to the message, please

[jira] [Updated] (AIRFLOW-5870) Optimise pool query and allow infinite pool for tasks

2019-11-10 Thread Jarek Potiuk (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk updated AIRFLOW-5870: -- Summary: Optimise pool query and allow infinite pool for tasks (was: Allow infinite pool

[jira] [Resolved] (AIRFLOW-5870) Optimise pool query and allow infinite pool for tasks

2019-11-10 Thread Jarek Potiuk (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5870. --- Fix Version/s: 1.10.7 Resolution: Fixed > Optimise pool query and allow infinite

  1   2   >