[GitHub] [airflow] varundhussa commented on pull request #10673: Dataproc job sensor

2020-09-04 Thread GitBox
varundhussa commented on pull request #10673: URL: https://github.com/apache/airflow/pull/10673#issuecomment-687554661 Thanks @jaketf and @turbaszek This is an automated message from the Apache Git Service. To respond to

[GitHub] [airflow] ryanyuan opened a new pull request #10744: Update node installation cmd

2020-09-04 Thread GitBox
ryanyuan opened a new pull request #10744: URL: https://github.com/apache/airflow/pull/10744 In CONTRIBUTING.rst, we have brew install node --without-npm for installing node in macOS. The --without-npm flag is outdated and running this command will throw Error: invalid option:

[GitHub] [airflow] ryanyuan opened a new issue #10743: [Docs] Update node installation command

2020-09-04 Thread GitBox
ryanyuan opened a new issue #10743: URL: https://github.com/apache/airflow/issues/10743 **Environment**: - **OS** (e.g. from /etc/os-release): macOS 10.15.6 - **Install tools**: brew **What happened**: In CONTRIBUTING.rst, we have `brew install node --without-npm`

[airflow] tag nightly-master updated (4e09cb5 -> 079d7b5)

2020-09-04 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to tag nightly-master in repository https://gitbox.apache.org/repos/asf/airflow.git. *** WARNING: tag nightly-master was modified! *** from 4e09cb5 (commit) to 079d7b5 (commit) from

[GitHub] [airflow] yuqian90 commented on issue #10725: BranchPythonOperator skipping following tasks

2020-09-04 Thread GitBox
yuqian90 commented on issue #10725: URL: https://github.com/apache/airflow/issues/10725#issuecomment-687530055 @kaxil will do This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [airflow] mik-laj commented on issue #9708: Increase typing coverage

2020-09-04 Thread GitBox
mik-laj commented on issue #9708: URL: https://github.com/apache/airflow/issues/9708#issuecomment-68751 @turbaszek We still have many suppliers with coverage below 75 percent. ``` 0.00 dingding 0.00 opsgenie 0.00 presto 0.00 qubole 0.00 samba

[airflow] branch master updated: Enable more checks for pydocstyle (#10741)

2020-09-04 Thread kaxilnaik
This is an automated email from the ASF dual-hosted git repository. kaxilnaik pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/master by this push: new 079d7b5 Enable more checks for pydocstyle

[GitHub] [airflow] kaxil merged pull request #10741: Enable more checks for pydocstyle

2020-09-04 Thread GitBox
kaxil merged pull request #10741: URL: https://github.com/apache/airflow/pull/10741 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [airflow] kaxil commented on pull request #10741: Enable more checks for pydocstyle

2020-09-04 Thread GitBox
kaxil commented on pull request #10741: URL: https://github.com/apache/airflow/pull/10741#issuecomment-687506821 > Have you thought about creating a ticket about changes in documentation along with the label goodfirstissue? we have a large community willing to help if they know what to

[GitHub] [airflow] mik-laj commented on pull request #10741: Enable more checks for pydocstyle

2020-09-04 Thread GitBox
mik-laj commented on pull request #10741: URL: https://github.com/apache/airflow/pull/10741#issuecomment-687505217 Have you thought about creating a ticket about changes in documentation along with the label goodfirstissue? we have a large community willing to help if they know what to

[GitHub] [airflow] kaxil opened a new issue #10742: Enable More PyDocStyle Checks

2020-09-04 Thread GitBox
kaxil opened a new issue #10742: URL: https://github.com/apache/airflow/issues/10742 We use [PyDocStyle](https://github.com/PyCQA/pydocstyle) in pre-commit to enforce docstring style:

[GitHub] [airflow] mik-laj commented on pull request #10594: WIP: Add permissions for stable API

2020-09-04 Thread GitBox
mik-laj commented on pull request #10594: URL: https://github.com/apache/airflow/pull/10594#issuecomment-687503868 Alternatively, you can add the tables to the documentation and that will be enough for me too. ![Screenshot 2020-09-05 at 03 07

[GitHub] [airflow] mik-laj edited a comment on pull request #10594: WIP: Add permissions for stable API

2020-09-04 Thread GitBox
mik-laj edited a comment on pull request #10594: URL: https://github.com/apache/airflow/pull/10594#issuecomment-687500056 I think it is worth adding a description of the required permissions for each endpoint to the OpenAPI specification. What do you think about it? Currently, this

[GitHub] [airflow] kaxil opened a new pull request #10741: Enable more checks for pydocstyle

2020-09-04 Thread GitBox
kaxil opened a new pull request #10741: URL: https://github.com/apache/airflow/pull/10741 Enable D106, D207 and D208 D106 Missing docstring in public nested class D207 Docstring is under-indented D208 Docstring is over-indented --- **^ Add meaningful

[GitHub] [airflow] mik-laj edited a comment on pull request #10594: WIP: Add permissions for stable API

2020-09-04 Thread GitBox
mik-laj edited a comment on pull request #10594: URL: https://github.com/apache/airflow/pull/10594#issuecomment-687500056 I think it is worth adding a description of the required permissions for each endpoint to the OpenAPI specification. What do you think about it? Currently, this

[GitHub] [airflow] mik-laj commented on pull request #10594: WIP: Add permissions for stable API

2020-09-04 Thread GitBox
mik-laj commented on pull request #10594: URL: https://github.com/apache/airflow/pull/10594#issuecomment-687500056 I think it is worth adding a description of the required permissions for each endpoint to the OpenAPI specification. What do you think about it? Currently, this information

[GitHub] [airflow] kaxil opened a new pull request #10740: Improve test coverage for test_common_schema.py

2020-09-04 Thread GitBox
kaxil opened a new pull request #10740: URL: https://github.com/apache/airflow/pull/10740 Adds test that an error is raised with specific message when unkown object type is passed --- **^ Add meaningful description above** Read the **[Pull Request

[GitHub] [airflow] jhtimmins commented on pull request #10594: WIP: Add permissions for stable API

2020-09-04 Thread GitBox
jhtimmins commented on pull request #10594: URL: https://github.com/apache/airflow/pull/10594#issuecomment-687496912 @ashb @mik-laj @houqp Ok I think I've addressed or included all of your edits. This is an automated

[GitHub] [airflow] kaxil opened a new pull request #10739: Fix typo in test_dag_run_schema.py

2020-09-04 Thread GitBox
kaxil opened a new pull request #10739: URL: https://github.com/apache/airflow/pull/10739 `test_serialze` -> `test_serialize` --- **^ Add meaningful description above** Read the **[Pull Request

[GitHub] [airflow] kaxil opened a new pull request #10738: Improve test coverage for ConfObject in dag_run_schema

2020-09-04 Thread GitBox
kaxil opened a new pull request #10738: URL: https://github.com/apache/airflow/pull/10738 Adds test to verify that string can be passed to conf and ConfObject._deserialize works. --- **^ Add meaningful description above** Read the **[Pull Request

[airflow] branch master updated (8cb4fc1 -> 7b810b8)

2020-09-04 Thread kaxilnaik
This is an automated email from the ASF dual-hosted git repository. kaxilnaik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git. from 8cb4fc1 Add securitySchemes in openapi spec (#10652) add 7b810b8 Fix docs for generate-dag-yaml cli

[airflow] branch master updated: Fix docs for generate-dag-yaml cli command (#10735)

2020-09-04 Thread kaxilnaik
This is an automated email from the ASF dual-hosted git repository. kaxilnaik pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/master by this push: new 7b810b8 Fix docs for generate-dag-yaml cli

[GitHub] [airflow] kaxil merged pull request #10735: Fix docs for generate-dag-yaml cli command

2020-09-04 Thread GitBox
kaxil merged pull request #10735: URL: https://github.com/apache/airflow/pull/10735 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [airflow] kaxil opened a new pull request #10737: Add black to STATIC_CODE_CHECKS.rst

2020-09-04 Thread GitBox
kaxil opened a new pull request #10737: URL: https://github.com/apache/airflow/pull/10737 Add black to the table for Static Code Checkers --- **^ Add meaningful description above** Read the **[Pull Request

[GitHub] [airflow] kaxil opened a new pull request #10736: Remove duplicate entries from .mailmap

2020-09-04 Thread GitBox
kaxil opened a new pull request #10736: URL: https://github.com/apache/airflow/pull/10736 Remove duplicate entries from .mailmap --- **^ Add meaningful description above** Read the **[Pull Request

[GitHub] [airflow] kaxil opened a new pull request #10735: Fix docs for generate-dag-yaml cli command

2020-09-04 Thread GitBox
kaxil opened a new pull request #10735: URL: https://github.com/apache/airflow/pull/10735 Fixes grammar --- **^ Add meaningful description above** Read the **[Pull Request

[GitHub] [airflow] pcandoalmeida commented on pull request #10162: Add Airflow UI site_title configuration option

2020-09-04 Thread GitBox
pcandoalmeida commented on pull request #10162: URL: https://github.com/apache/airflow/pull/10162#issuecomment-687479611 Sure @mik-laj I'll have a look tomorrow! This is an automated message from the Apache Git Service. To

[GitHub] [airflow] dimberman commented on pull request #10153: [AIP-34] TaskGroup: A UI task grouping concept as an alternative to SubDagOperator

2020-09-04 Thread GitBox
dimberman commented on pull request #10153: URL: https://github.com/apache/airflow/pull/10153#issuecomment-687473532 @yuqian90 This is AWESOME! Thank you for your work on this. I'll leave the review to @kaxil as he knows more about the front-end, but super excited for this!

[GitHub] [airflow] mik-laj commented on pull request #10162: Add Airflow UI site_title configuration option

2020-09-04 Thread GitBox
mik-laj commented on pull request #10162: URL: https://github.com/apache/airflow/pull/10162#issuecomment-687469613 @pcandoalmeida I prepared the patch and push it on your branch. Can you check it? This is an automated

[GitHub] [airflow] potiuk commented on pull request #9652: Move dev docker images to airflow registry

2020-09-04 Thread GitBox
potiuk commented on pull request #9652: URL: https://github.com/apache/airflow/pull/9652#issuecomment-687461312 Updated to the Google Shell Giuide. I would love to merge this one, this way we completely close image case - all under our control, with easy way to rebuild by the users.

[GitHub] [airflow] potiuk opened a new pull request #10734: Make dockerfiles Google Shell Guide Compliant

2020-09-04 Thread GitBox
potiuk opened a new pull request #10734: URL: https://github.com/apache/airflow/pull/10734 --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more

[GitHub] [airflow] potiuk commented on pull request #10733: Make ci/backport_packages Google Shell guide compliant

2020-09-04 Thread GitBox
potiuk commented on pull request #10733: URL: https://github.com/apache/airflow/pull/10733#issuecomment-687454263 cc: @OmairK This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [airflow] potiuk opened a new pull request #10733: Make ci/backport_packages Google Shell guide compliant

2020-09-04 Thread GitBox
potiuk opened a new pull request #10733: URL: https://github.com/apache/airflow/pull/10733 --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more

[GitHub] [airflow] potiuk commented on pull request #10708: Make breeeze-complete Google Shell Guide compatible

2020-09-04 Thread GitBox
potiuk commented on pull request #10708: URL: https://github.com/apache/airflow/pull/10708#issuecomment-687450981 Rebased to the latest asserts and added unit tests for breeze-complete. This is an automated message from the

[GitHub] [airflow] mik-laj commented on a change in pull request #10162: Add Airflow UI site_title configuration option

2020-09-04 Thread GitBox
mik-laj commented on a change in pull request #10162: URL: https://github.com/apache/airflow/pull/10162#discussion_r483872791 ## File path: airflow/config_templates/config.yml ## @@ -1080,6 +1080,13 @@ type: string example: ~ default: "30" +- name:

[GitHub] [airflow] dimon222 commented on issue #10722: KubernetesPodOperator with XCom enabled errors: 403 forbidden

2020-09-04 Thread GitBox
dimon222 commented on issue #10722: URL: https://github.com/apache/airflow/issues/10722#issuecomment-687447007 I've previously encountered this issue and can confirm that it takes place. My solution was to not use xcom with KubernetesPodOperator, but obviously that's not a cure, but

[GitHub] [airflow-client-go] mik-laj commented on pull request #5: add test & doc for authentication

2020-09-04 Thread GitBox
mik-laj commented on pull request #5: URL: https://github.com/apache/airflow-client-go/pull/5#issuecomment-687442219 Have you tested how the client is generated for Google OpenID or Kerberos? This is an automated message

[airflow] branch master updated (6e3d7b6 -> 8cb4fc1)

2020-09-04 Thread kamilbregula
This is an automated email from the ASF dual-hosted git repository. kamilbregula pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git. from 6e3d7b6 Add masterConfig parameter to MLEngineStartTrainingJobOperator (#10578) add 8cb4fc1 Add

[airflow] branch master updated (6e3d7b6 -> 8cb4fc1)

2020-09-04 Thread kamilbregula
This is an automated email from the ASF dual-hosted git repository. kamilbregula pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git. from 6e3d7b6 Add masterConfig parameter to MLEngineStartTrainingJobOperator (#10578) add 8cb4fc1 Add

[GitHub] [airflow] mik-laj merged pull request #10652: Add securitySchemes in openapi spec

2020-09-04 Thread GitBox
mik-laj merged pull request #10652: URL: https://github.com/apache/airflow/pull/10652 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [airflow] mik-laj commented on a change in pull request #10652: add securitySchemes in openapi spec

2020-09-04 Thread GitBox
mik-laj commented on a change in pull request #10652: URL: https://github.com/apache/airflow/pull/10652#discussion_r483865385 ## File path: docs/conf.py ## @@ -553,6 +553,7 @@ def _get_rst_filepath_from_path(filepath: str): 'spec': OPENAPI_FILE, 'opts': {

[GitHub] [airflow] jhtimmins commented on a change in pull request #10652: add securitySchemes in openapi spec

2020-09-04 Thread GitBox
jhtimmins commented on a change in pull request #10652: URL: https://github.com/apache/airflow/pull/10652#discussion_r483864689 ## File path: docs/conf.py ## @@ -553,6 +553,7 @@ def _get_rst_filepath_from_path(filepath: str): 'spec': OPENAPI_FILE, 'opts': {

[GitHub] [airflow] potiuk commented on pull request #10731: The verbose functions will not exit immediately if not asked to

2020-09-04 Thread GitBox
potiuk commented on pull request #10731: URL: https://github.com/apache/airflow/pull/10731#issuecomment-687415943 That one has just quarantined tests failed. This is an automated message from the Apache Git Service. To

[airflow] branch master updated (e4de728 -> 6e3d7b6)

2020-09-04 Thread kamilbregula
This is an automated email from the ASF dual-hosted git repository. kamilbregula pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git. from e4de728 Switches to better BATS asserts (#10718) add 6e3d7b6 Add masterConfig parameter to

[airflow] branch master updated: Add masterConfig parameter to MLEngineStartTrainingJobOperator (#10578)

2020-09-04 Thread kamilbregula
This is an automated email from the ASF dual-hosted git repository. kamilbregula pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/master by this push: new 6e3d7b6 Add masterConfig parameter to

[GitHub] [airflow] mik-laj merged pull request #10578: Add masterConfig parameter to MLEngineStartTrainingJobOperator

2020-09-04 Thread GitBox
mik-laj merged pull request #10578: URL: https://github.com/apache/airflow/pull/10578 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #10578: Add masterConfig parameter to MLEngineStartTrainingJobOperator

2020-09-04 Thread GitBox
boring-cyborg[bot] commented on pull request #10578: URL: https://github.com/apache/airflow/pull/10578#issuecomment-687413011 Awesome work, congrats on your first merged pull request! This is an automated message from

[GitHub] [airflow] kaxil commented on a change in pull request #9467: Add airflow upgrade-check command (MVP) (v1-10-test)

2020-09-04 Thread GitBox
kaxil commented on a change in pull request #9467: URL: https://github.com/apache/airflow/pull/9467#discussion_r483854376 ## File path: airflow/bin/cli.py ## @@ -2151,6 +2153,19 @@ def info(args): print(info) +def upgrade_check(args): +if args.save: +

[GitHub] [airflow] mik-laj commented on issue #10549: option to disable "lots of circles" in error page

2020-09-04 Thread GitBox
mik-laj commented on issue #10549: URL: https://github.com/apache/airflow/issues/10549#issuecomment-687402016 @jpmarques66 I assigned you to this ticket  This is an automated message from the Apache Git Service. To respond

[GitHub] [airflow] mik-laj commented on pull request #9467: Add airflow upgrade-check command (MVP) (v1-10-test)

2020-09-04 Thread GitBox
mik-laj commented on pull request #9467: URL: https://github.com/apache/airflow/pull/9467#issuecomment-687401568 We want to build this tool together with the community. When this change is merged, we will create several tickets for other contributors. It will also be a good time to add a

[GitHub] [airflow] mik-laj commented on a change in pull request #10728: Add note on GKE metadata server Workload Identity

2020-09-04 Thread GitBox
mik-laj commented on a change in pull request #10728: URL: https://github.com/apache/airflow/pull/10728#discussion_r483852840 ## File path: docs/howto/connection/gcp.rst ## @@ -32,7 +32,13 @@ There are three ways to connect to Google Cloud using Airflow. 1. Use

[GitHub] [airflow] kaxil commented on issue #10686: Common tasks in downstream of multiple branches always set to skipped due to 'Not Previously Skipped' dependency

2020-09-04 Thread GitBox
kaxil commented on issue #10686: URL: https://github.com/apache/airflow/issues/10686#issuecomment-687396470 cc @yuqian90 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [airflow] mik-laj commented on issue #8061: Support DAGS folder being in different location on scheduler and worker.

2020-09-04 Thread GitBox
mik-laj commented on issue #8061: URL: https://github.com/apache/airflow/issues/8061#issuecomment-687396161 @aragaorenan I did not see any active contributions related to it. This is also not a big problem if you are using containers/Docker or other [mount

[GitHub] [airflow] ashb commented on pull request #10729: Detect orphaned task instances by SchedulerJob id and heartbeat

2020-09-04 Thread GitBox
ashb commented on pull request #10729: URL: https://github.com/apache/airflow/pull/10729#issuecomment-687393784 I need to extend the unit tests to check that it correctly tests the new functionality (not sure how I missed that) -- namely that TIs from a SchedulerJob that has

[GitHub] [airflow] kaxil commented on issue #10725: BranchPythonOperator skipping following tasks

2020-09-04 Thread GitBox
kaxil commented on issue #10725: URL: https://github.com/apache/airflow/issues/10725#issuecomment-687393402 @yuqian90 Looks like more people rely on this feature, would you be able to make a PR that restores this behavior to allow empty branch please

[GitHub] [airflow] dimberman commented on pull request #10732: Create a script to migrate KubernetesExecutor airflow.cfg configs to …

2020-09-04 Thread GitBox
dimberman commented on pull request #10732: URL: https://github.com/apache/airflow/pull/10732#issuecomment-687393242 It turns out that the script needed to migrate users to a pod_template_file was WAY easier than I had expected! We should add this to 1.10.13.

[GitHub] [airflow] dimberman opened a new pull request #10732: Create a script to migrate KubernetesExecutor airflow.cfg configs to …

2020-09-04 Thread GitBox
dimberman opened a new pull request #10732: URL: https://github.com/apache/airflow/pull/10732 Create a script to migrate KubernetesExecutor airflow.cfg configs to pod_template_file --- **^ Add meaningful description above** Read the **[Pull Request

[GitHub] [airflow] jhtimmins commented on a change in pull request #10594: WIP: Add permissions for stable API

2020-09-04 Thread GitBox
jhtimmins commented on a change in pull request #10594: URL: https://github.com/apache/airflow/pull/10594#discussion_r483844302 ## File path: airflow/www/security.py ## @@ -151,10 +152,12 @@ class AirflowSecurityManager(SecurityManager, LoggingMixin): WRITE_DAG_PERMS =

[GitHub] [airflow] XD-DENG commented on pull request #10729: Detect orphaned task instances by SchedulerJob id and heartbeat

2020-09-04 Thread GitBox
XD-DENG commented on pull request #10729: URL: https://github.com/apache/airflow/pull/10729#issuecomment-687370386 > Looks like this breaks on SQLite --- will need to take a look. Should be due to SQLite dialect doesn’t support multiple-table criteria within UPDATE (log line

[GitHub] [airflow] dimon222 commented on issue #10725: BranchPythonOperator skipping following tasks

2020-09-04 Thread GitBox
dimon222 commented on issue #10725: URL: https://github.com/apache/airflow/issues/10725#issuecomment-687370279 Okay, I thought I'm alone, but then saw #10686 and now this, I start to think that its a regression. I'm also affected, what is the proposed way around this? Adding dummy

[airflow] branch master updated (90c1505 -> e4de728)

2020-09-04 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git. from 90c1505 Make grace_period_seconds option on K8sPodOperator (#10727) add e4de728 Switches to better BATS

[GitHub] [airflow] ashb commented on pull request #10729: Detect orphaned task instances by SchedulerJob id and heartbeat

2020-09-04 Thread GitBox
ashb commented on pull request #10729: URL: https://github.com/apache/airflow/pull/10729#issuecomment-687365654 Looks like this breaks on SQLite --- will need to take a look. This is an automated message from the Apache Git

[GitHub] [airflow] potiuk commented on pull request #10718: Switches to better BATS asserts

2020-09-04 Thread GitBox
potiuk commented on pull request #10718: URL: https://github.com/apache/airflow/pull/10718#issuecomment-687365311 Thanks @kaxil! This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [airflow] potiuk merged pull request #10718: Switches to better BATS asserts

2020-09-04 Thread GitBox
potiuk merged pull request #10718: URL: https://github.com/apache/airflow/pull/10718 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [airflow] ashb commented on a change in pull request #10729: Detect orphaned task instances by SchedulerJob id and heartbeat

2020-09-04 Thread GitBox
ashb commented on a change in pull request #10729: URL: https://github.com/apache/airflow/pull/10729#discussion_r483829702 ## File path: airflow/jobs/scheduler_job.py ## @@ -1759,3 +1759,46 @@ def _emit_pool_metrics(self, session: Session = None) -> None:

[GitHub] [airflow] XD-DENG commented on a change in pull request #10729: Detect orphaned task instances by SchedulerJob id and heartbeat

2020-09-04 Thread GitBox
XD-DENG commented on a change in pull request #10729: URL: https://github.com/apache/airflow/pull/10729#discussion_r483827767 ## File path: airflow/jobs/scheduler_job.py ## @@ -1759,3 +1759,46 @@ def _emit_pool_metrics(self, session: Session = None) -> None:

[GitHub] [airflow] potiuk opened a new pull request #10731: The verbose functions will not exit immediately if not asked to

2020-09-04 Thread GitBox
potiuk opened a new pull request #10731: URL: https://github.com/apache/airflow/pull/10731 The docker(), helm(), kubectl() functions replace the real tools to get verbose behaviour (we can print the exact command being executed for those. But when 'set +e' was set before the command

[GitHub] [airflow] jaketf commented on a change in pull request #10673: Dataproc job sensor

2020-09-04 Thread GitBox
jaketf commented on a change in pull request #10673: URL: https://github.com/apache/airflow/pull/10673#discussion_r483811344 ## File path: airflow/providers/google/cloud/sensors/dataproc.py ## @@ -0,0 +1,83 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +#

[GitHub] [airflow] XD-DENG commented on a change in pull request #10729: Detect orphaned task instances by SchedulerJob id and heartbeat

2020-09-04 Thread GitBox
XD-DENG commented on a change in pull request #10729: URL: https://github.com/apache/airflow/pull/10729#discussion_r483806158 ## File path: tests/jobs/test_backfill_job.py ## @@ -1441,3 +1441,97 @@ def test_backfill_run_backwards(self): dag.clear()

[GitHub] [airflow] jaketf commented on a change in pull request #10673: Dataproc job sensor

2020-09-04 Thread GitBox
jaketf commented on a change in pull request #10673: URL: https://github.com/apache/airflow/pull/10673#discussion_r483810223 ## File path: airflow/providers/google/cloud/sensors/dataproc.py ## @@ -0,0 +1,83 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +#

[GitHub] [airflow] jaketf commented on a change in pull request #10673: Dataproc job sensor

2020-09-04 Thread GitBox
jaketf commented on a change in pull request #10673: URL: https://github.com/apache/airflow/pull/10673#discussion_r483809033 ## File path: airflow/providers/google/cloud/sensors/dataproc.py ## @@ -0,0 +1,83 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +#

[GitHub] [airflow] HenryLinTw commented on issue #10725: BranchPythonOperator skipping following tasks

2020-09-04 Thread GitBox
HenryLinTw commented on issue #10725: URL: https://github.com/apache/airflow/issues/10725#issuecomment-687337266 Just want to bring this up because we also encountered this. If what mentioned above is eventually finalized, the example in the doc here:

[GitHub] [airflow] ashb commented on a change in pull request #10729: Detect orphaned task instances by SchedulerJob id and heartbeat

2020-09-04 Thread GitBox
ashb commented on a change in pull request #10729: URL: https://github.com/apache/airflow/pull/10729#discussion_r483799618 ## File path: airflow/jobs/base_job.py ## @@ -73,6 +72,17 @@ class BaseJob(Base, LoggingMixin): Index('idx_job_state_heartbeat', state,

[GitHub] [airflow] millertracy closed pull request #10591: Added Plexus as an Airflow provider

2020-09-04 Thread GitBox
millertracy closed pull request #10591: URL: https://github.com/apache/airflow/pull/10591 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[airflow] branch master updated (3f7831c -> 90c1505)

2020-09-04 Thread dimberman
This is an automated email from the ASF dual-hosted git repository. dimberman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git. from 3f7831c Relax requirement to allow latest version of flask-login (#9990) add 90c1505 Make

[airflow] branch master updated (3f7831c -> 90c1505)

2020-09-04 Thread dimberman
This is an automated email from the ASF dual-hosted git repository. dimberman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git. from 3f7831c Relax requirement to allow latest version of flask-login (#9990) add 90c1505 Make

[GitHub] [airflow] dimberman merged pull request #10727: Make grace_period_seconds option on K8sPodOperator

2020-09-04 Thread GitBox
dimberman merged pull request #10727: URL: https://github.com/apache/airflow/pull/10727 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [airflow] kaxil commented on a change in pull request #10729: Detect orphaned task instances by SchedulerJob id and heartbeat

2020-09-04 Thread GitBox
kaxil commented on a change in pull request #10729: URL: https://github.com/apache/airflow/pull/10729#discussion_r483783889 ## File path: airflow/jobs/scheduler_job.py ## @@ -1759,3 +1759,46 @@ def _emit_pool_metrics(self, session: Session = None) -> None:

[GitHub] [airflow] ashb commented on a change in pull request #10729: Detect orphaned task instances by SchedulerJob id and heartbeat

2020-09-04 Thread GitBox
ashb commented on a change in pull request #10729: URL: https://github.com/apache/airflow/pull/10729#discussion_r483781796 ## File path: airflow/models/taskinstance.py ## @@ -1871,3 +1872,14 @@ def construct_task_instance(self, session=None, lock_for_update=False) -> TaskIn

[GitHub] [airflow] ashb commented on a change in pull request #10729: Detect orphaned task instances by SchedulerJob id and heartbeat

2020-09-04 Thread GitBox
ashb commented on a change in pull request #10729: URL: https://github.com/apache/airflow/pull/10729#discussion_r483781427 ## File path: airflow/jobs/scheduler_job.py ## @@ -1759,3 +1759,46 @@ def _emit_pool_metrics(self, session: Session = None) -> None:

[GitHub] [airflow] kaxil commented on a change in pull request #10729: Detect orphaned task instances by SchedulerJob id and heartbeat

2020-09-04 Thread GitBox
kaxil commented on a change in pull request #10729: URL: https://github.com/apache/airflow/pull/10729#discussion_r483775657 ## File path: airflow/jobs/scheduler_job.py ## @@ -1759,3 +1759,46 @@ def _emit_pool_metrics(self, session: Session = None) -> None:

[GitHub] [airflow] kaxil commented on a change in pull request #10729: Detect orphaned task instances by SchedulerJob id and heartbeat

2020-09-04 Thread GitBox
kaxil commented on a change in pull request #10729: URL: https://github.com/apache/airflow/pull/10729#discussion_r483775410 ## File path: airflow/jobs/scheduler_job.py ## @@ -1759,3 +1759,46 @@ def _emit_pool_metrics(self, session: Session = None) -> None:

[GitHub] [airflow] kaxil commented on a change in pull request #10729: Detect orphaned task instances by SchedulerJob id and heartbeat

2020-09-04 Thread GitBox
kaxil commented on a change in pull request #10729: URL: https://github.com/apache/airflow/pull/10729#discussion_r483774892 ## File path: airflow/jobs/base_job.py ## @@ -73,6 +72,17 @@ class BaseJob(Base, LoggingMixin): Index('idx_job_state_heartbeat', state,

[GitHub] [airflow] potiuk commented on issue #10730: Dockerfile - Enable gpg to receive keys behind a proxy

2020-09-04 Thread GitBox
potiuk commented on issue #10730: URL: https://github.com/apache/airflow/issues/10730#issuecomment-687299266 Or maybe ypu can simply add a PR with that @enriqueayala -> happy to review it and merge :) This is an automated

[GitHub] [airflow] potiuk commented on issue #10730: Dockerfile - Enable gpg to receive keys behind a proxy

2020-09-04 Thread GitBox
potiuk commented on issue #10730: URL: https://github.com/apache/airflow/issues/10730#issuecomment-687298766 Nice thing to add indeed. There are a couple of similar requests so I will address them in one go soon. This is an

[GitHub] [airflow] kaxil commented on a change in pull request #10729: Detect orphaned task instances by SchedulerJob id and heartbeat

2020-09-04 Thread GitBox
kaxil commented on a change in pull request #10729: URL: https://github.com/apache/airflow/pull/10729#discussion_r483774432 ## File path: airflow/models/taskinstance.py ## @@ -1871,3 +1872,14 @@ def construct_task_instance(self, session=None, lock_for_update=False) -> TaskIn

[GitHub] [airflow] kaxil commented on a change in pull request #10729: Detect orphaned task instances by SchedulerJob id and heartbeat

2020-09-04 Thread GitBox
kaxil commented on a change in pull request #10729: URL: https://github.com/apache/airflow/pull/10729#discussion_r483774046 ## File path: airflow/migrations/versions/b247b1e3d1ed_add_queued_by_job_id_to_ti.py ## @@ -0,0 +1,47 @@ +# +# Licensed to the Apache Software

[GitHub] [airflow] kaxil commented on a change in pull request #10727: Make grace_period_seconds option on K8sPodOperator

2020-09-04 Thread GitBox
kaxil commented on a change in pull request #10727: URL: https://github.com/apache/airflow/pull/10727#discussion_r483770896 ## File path: kubernetes_tests/test_kubernetes_pod_operator.py ## @@ -903,7 +904,6 @@ def test_on_kill(self, self.assertEqual(pod.status.phase,

[GitHub] [airflow] dimberman commented on a change in pull request #10727: Make grace_period_seconds option on K8sPodOperator

2020-09-04 Thread GitBox
dimberman commented on a change in pull request #10727: URL: https://github.com/apache/airflow/pull/10727#discussion_r483770449 ## File path: kubernetes_tests/test_kubernetes_pod_operator.py ## @@ -903,7 +904,6 @@ def test_on_kill(self,

[GitHub] [airflow] boring-cyborg[bot] commented on issue #10730: Dockerfile - Enable gpg to receive keys behind a proxy

2020-09-04 Thread GitBox
boring-cyborg[bot] commented on issue #10730: URL: https://github.com/apache/airflow/issues/10730#issuecomment-687293892 Thanks for opening your first issue here! Be sure to follow the issue template! This is an

[GitHub] [airflow] enriqueayala opened a new issue #10730: Dockerfile - Enable gpg to receive keys behind a proxy

2020-09-04 Thread GitBox
enriqueayala opened a new issue #10730: URL: https://github.com/apache/airflow/issues/10730 **Description** Allow build of airflow docker image behind a corporate firewall. **Use case / motivation** I would like to manually build an image by specifying the

[GitHub] [airflow] kaxil commented on a change in pull request #10727: Make grace_period_seconds option on K8sPodOperator

2020-09-04 Thread GitBox
kaxil commented on a change in pull request #10727: URL: https://github.com/apache/airflow/pull/10727#discussion_r483769070 ## File path: kubernetes_tests/test_kubernetes_pod_operator.py ## @@ -903,7 +904,6 @@ def test_on_kill(self, self.assertEqual(pod.status.phase,

[GitHub] [airflow] potiuk commented on pull request #10718: Switches to better BATS asserts

2020-09-04 Thread GitBox
potiuk commented on pull request #10718: URL: https://github.com/apache/airflow/pull/10718#issuecomment-687293173 Would love to get that in, so I can add more unit tests while adding the "Google Shell guidelines" compatibility :)

[GitHub] [airflow] ashb opened a new pull request #10729: Detect orphaned task instances by SchedulerJob id and heartbeat

2020-09-04 Thread GitBox
ashb opened a new pull request #10729: URL: https://github.com/apache/airflow/pull/10729 Once HA mode for scheduler lands, we can no longer reset orphaned task by looking at the tasks in (the memory of) the current executor. This changes it to keep track of which (Scheduler)Job

[GitHub] [airflow] medb commented on pull request #10714: Add support for non-preemptible secondary workers in Dataproc operator

2020-09-04 Thread GitBox
medb commented on pull request #10714: URL: https://github.com/apache/airflow/pull/10714#issuecomment-687290335 Thank you for clarification, in this case I will close this PR because it's already possible to use `DataprocCreateCluster` operator to specify secondary workers preemptibility.

[GitHub] [airflow] aragaorenan commented on issue #8061: Support DAGS folder being in different location on scheduler and worker.

2020-09-04 Thread GitBox
aragaorenan commented on issue #8061: URL: https://github.com/apache/airflow/issues/8061#issuecomment-687290298 Hello everyone, any update about this issue? This is an automated message from the Apache Git Service. To

[GitHub] [airflow] medb closed pull request #10714: Add support for non-preemptible secondary workers in Dataproc operator

2020-09-04 Thread GitBox
medb closed pull request #10714: URL: https://github.com/apache/airflow/pull/10714 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [airflow] kaxil commented on a change in pull request #10727: Make grace_period_seconds option on K8sPodOperator

2020-09-04 Thread GitBox
kaxil commented on a change in pull request #10727: URL: https://github.com/apache/airflow/pull/10727#discussion_r483763836 ## File path: airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py ## @@ -146,6 +146,9 @@ class KubernetesPodOperator(BaseOperator): # pylint:

[GitHub] [airflow] ashb commented on a change in pull request #10594: WIP: Add permissions for stable API

2020-09-04 Thread GitBox
ashb commented on a change in pull request #10594: URL: https://github.com/apache/airflow/pull/10594#discussion_r483762556 ## File path: airflow/api_connexion/security.py ## @@ -16,25 +16,63 @@ # under the License. from functools import wraps -from typing import Callable,

[GitHub] [airflow] jaketf opened a new pull request #10728: Add note on GKE metadata server Workload Identity

2020-09-04 Thread GitBox
jaketf opened a new pull request #10728: URL: https://github.com/apache/airflow/pull/10728 Workload Identity is the recommended way of managing authorization to GCP services from GKE. While it is notably not used by Cloud Composer, I think a note like this could be useful for those who

  1   2   >