[GitHub] [airflow] casassg commented on pull request #10839: Improvement: Populate 'Configuration JSON' form with DAG default params json in the Trigger-DAG UI

2020-09-18 Thread GitBox
casassg commented on pull request #10839: URL: https://github.com/apache/airflow/pull/10839#issuecomment-695124610 Can we add a followup task to make this into a form? Lgtm otherwise This is an automated message from the

[GitHub] [airflow] fjmacagno commented on issue #11013: I have a question about schedule interval

2020-09-18 Thread GitBox
fjmacagno commented on issue #11013: URL: https://github.com/apache/airflow/issues/11013#issuecomment-695143827 See the note under https://airflow.apache.org/docs/stable/dag-run.html#cron-presets. This is intended, the execution date for a dag run is at the beginning of the cron

[GitHub] [airflow] ryanahamilton commented on issue #11025: Airflow Logo hides half of the page

2020-09-18 Thread GitBox
ryanahamilton commented on issue #11025: URL: https://github.com/apache/airflow/issues/11025#issuecomment-695150147 This would surely be a side-effect of #11018, but I'm unable to reproduce in Chrome, Safari, or Firefox on Mac. I can certainly provide a simple solution for this if it is

[GitHub] [airflow] kaxil opened a new pull request #11026: Remove Edit Button from DagModel View

2020-09-18 Thread GitBox
kaxil opened a new pull request #11026: URL: https://github.com/apache/airflow/pull/11026 We don't allow Editing DagModel so it will fail with Permissions anyway so the button is useless **Before**

[GitHub] [airflow] kaxil edited a comment on issue #11014: RBAC UI: Editing dag runs gives forbidden page when admin

2020-09-18 Thread GitBox
kaxil edited a comment on issue #11014: URL: https://github.com/apache/airflow/issues/11014#issuecomment-695150017 Yes that button will be removed as part of https://github.com/apache/airflow/pull/11026 -- We no longer allow editing DagModel

[GitHub] [airflow] kaxil commented on issue #11014: RBAC UI: Editing dag runs gives forbidden page when admin

2020-09-18 Thread GitBox
kaxil commented on issue #11014: URL: https://github.com/apache/airflow/issues/11014#issuecomment-695150017 Yes that button will be removed as part of https://github.com/apache/airflow/pull/11026 This is an automated

[GitHub] [airflow] ryanahamilton opened a new pull request #11018: Replace logo with SVG that includes name in proper typography

2020-09-18 Thread GitBox
ryanahamilton opened a new pull request #11018: URL: https://github.com/apache/airflow/pull/11018 related: apache/airflow#10953 Replaces the PNG + HTML text version of the Airflow logo with an SVG that yields both the pinwheel and "Airflow" text. The prior implementation simply

[GitHub] [airflow] abhilash1in commented on pull request #10873: Add description field to connection (#10840)

2020-09-18 Thread GitBox
abhilash1in commented on pull request #10873: URL: https://github.com/apache/airflow/pull/10873#issuecomment-695032646 @potiuk that makes sense, thank you. I'm okay with this being added to 2.0. This is an automated message

[GitHub] [airflow] kaxil commented on a change in pull request #11020: Fix AWS DataSync tests failing (#10985)

2020-09-18 Thread GitBox
kaxil commented on a change in pull request #11020: URL: https://github.com/apache/airflow/pull/11020#discussion_r491165720 ## File path: airflow/providers/amazon/aws/operators/datasync.py ## @@ -335,9 +335,10 @@ def _execute_datasync_task(self): # Log some meaningful

[airflow] branch master updated (f16f474 -> 49c193f)

2020-09-18 Thread kaxilnaik
This is an automated email from the ASF dual-hosted git repository. kaxilnaik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git. from f16f474 Replace PNG/text with SVG that includes name in proper typography (#11018) add 49c193f [AIP-34]

[GitHub] [airflow] kaxil commented on a change in pull request #11020: Fix AWS DataSync tests failing (#10985)

2020-09-18 Thread GitBox
kaxil commented on a change in pull request #11020: URL: https://github.com/apache/airflow/pull/11020#discussion_r491247543 ## File path: setup.py ## @@ -455,8 +455,7 @@ def write_version(filename: str = os.path.join(*[my_dir, "airflow", "git_version 'ipdb', 'jira',

[airflow] branch master updated: Support extra_args in S3Hook and GCSToS3Operator (#11001)

2020-09-18 Thread kaxilnaik
This is an automated email from the ASF dual-hosted git repository. kaxilnaik pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/master by this push: new 9edfcb7 Support extra_args in S3Hook and

[GitHub] [airflow] kaxil closed issue #10298: GCSToS3Operator (and S3Hook) does not allow passing in of custom metadata

2020-09-18 Thread GitBox
kaxil closed issue #10298: URL: https://github.com/apache/airflow/issues/10298 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [airflow] abhilash1in commented on a change in pull request #10873: Add description field to connection (#10840)

2020-09-18 Thread GitBox
abhilash1in commented on a change in pull request #10873: URL: https://github.com/apache/airflow/pull/10873#discussion_r491254344 ## File path: airflow/migrations/versions/c2e6d7e1c6d2_merge_61ec_and_e1a.py ## @@ -0,0 +1,44 @@ +# +# Licensed to the Apache Software Foundation

[GitHub] [airflow] kaxil opened a new issue #11025: Airflow Logo hides half of the page

2020-09-18 Thread GitBox
kaxil opened a new issue #11025: URL: https://github.com/apache/airflow/issues/11025 Looks like a side-effect of https://github.com/apache/airflow/pull/11018 ![image](https://user-images.githubusercontent.com/8811558/93656441-65aead80-fa22-11ea-9cf5-a3ec5eed1c11.png) cc

[GitHub] [airflow] kaxil merged pull request #11001: Support extra_args in S3Hook and through GCSToS3Operator

2020-09-18 Thread GitBox
kaxil merged pull request #11001: URL: https://github.com/apache/airflow/pull/11001 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [airflow] kaxil opened a new pull request #11022: Fix typo in DagFileProcessorAgent._last_parsing_stat_received_at

2020-09-18 Thread GitBox
kaxil opened a new pull request #11022: URL: https://github.com/apache/airflow/pull/11022 `_last_parsing_stat_recieved_at` -> `_last_parsing_stat_received_at` --- **^ Add meaningful description above** Read the **[Pull Request

[GitHub] [airflow] chipmyersjr commented on pull request #10760: Create salesforce to gcs transfer

2020-09-18 Thread GitBox
chipmyersjr commented on pull request #10760: URL: https://github.com/apache/airflow/pull/10760#issuecomment-695136988 @eladkal thanks for checking. I added templating support for parameters where I felt is was appropriate.

[airflow] branch master updated: [AIP-34] TaskGroup: A UI task grouping concept as an alternative to SubDagOperator (#10153)

2020-09-18 Thread kaxilnaik
This is an automated email from the ASF dual-hosted git repository. kaxilnaik pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/master by this push: new 49c193f [AIP-34] TaskGroup: A UI task

[airflow] branch master updated (f16f474 -> 49c193f)

2020-09-18 Thread kaxilnaik
This is an automated email from the ASF dual-hosted git repository. kaxilnaik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git. from f16f474 Replace PNG/text with SVG that includes name in proper typography (#11018) add 49c193f [AIP-34]

[GitHub] [airflow] kaxil merged pull request #10153: [AIP-34] TaskGroup: A UI task grouping concept as an alternative to SubDagOperator

2020-09-18 Thread GitBox
kaxil merged pull request #10153: URL: https://github.com/apache/airflow/pull/10153 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [airflow] abhilash1in commented on pull request #10873: Add description field to connection (#10840)

2020-09-18 Thread GitBox
abhilash1in commented on pull request #10873: URL: https://github.com/apache/airflow/pull/10873#issuecomment-695146183 > Tests are failing @kaxil There were two alembic heads since a new migration was pushed to master after I created my migration (both had the same

[GitHub] [airflow] kaxil opened a new pull request #11023: Get Airflow configs with sensitive data from AWS Systems Manager

2020-09-18 Thread GitBox
kaxil opened a new pull request #11023: URL: https://github.com/apache/airflow/pull/11023 Adds support to AWS SSM for feature added in https://github.com/apache/airflow/pull/9645 --- **^ Add meaningful description above** Read the **[Pull Request

[GitHub] [airflow] kaxil commented on a change in pull request #10873: Add description field to connection (#10840)

2020-09-18 Thread GitBox
kaxil commented on a change in pull request #10873: URL: https://github.com/apache/airflow/pull/10873#discussion_r491252557 ## File path: airflow/migrations/versions/c2e6d7e1c6d2_merge_61ec_and_e1a.py ## @@ -0,0 +1,44 @@ +# +# Licensed to the Apache Software Foundation (ASF)

[GitHub] [airflow] kaxil opened a new pull request #11024: Get Airflow configs with sensitive data from CloudSecretManagerBackend

2020-09-18 Thread GitBox
kaxil opened a new pull request #11024: URL: https://github.com/apache/airflow/pull/11024 Adds support to Google Cloud CloudSecretManagerBackend for feature added in https://github.com/apache/airflow/pull/9645 --- **^ Add meaningful description above** Read the

[GitHub] [airflow] baolsen commented on issue #10985: AWS Datasync tests are failing with Moto 1.3.16

2020-09-18 Thread GitBox
baolsen commented on issue #10985: URL: https://github.com/apache/airflow/issues/10985#issuecomment-695151657 @kaxil Thanks, I had a look and there was also some skip logic within the test_datasync.py file Removed now and pushed, lets see

[GitHub] [airflow] abhilash1in commented on a change in pull request #10873: Add description field to connection (#10840)

2020-09-18 Thread GitBox
abhilash1in commented on a change in pull request #10873: URL: https://github.com/apache/airflow/pull/10873#discussion_r491260254 ## File path: airflow/migrations/versions/c2e6d7e1c6d2_merge_61ec_and_e1a.py ## @@ -0,0 +1,44 @@ +# +# Licensed to the Apache Software Foundation

[GitHub] [airflow] jedcunningham commented on pull request #10663: Show a few lines of tracebacks for DAG import errors in web UI

2020-09-18 Thread GitBox
jedcunningham commented on pull request #10663: URL: https://github.com/apache/airflow/pull/10663#issuecomment-695021112 Added `traceback` and `tracebacks` to the spelling wordlist, not sure why "Wait for PROD images" got stuck...

[GitHub] [airflow] baolsen commented on a change in pull request #11020: Fix AWS DataSync tests failing (#10985)

2020-09-18 Thread GitBox
baolsen commented on a change in pull request #11020: URL: https://github.com/apache/airflow/pull/11020#discussion_r491169884 ## File path: airflow/providers/amazon/aws/operators/datasync.py ## @@ -335,9 +335,10 @@ def _execute_datasync_task(self): # Log some

[GitHub] [airflow] daemon-demon commented on issue #10505: PostgressHook connection - pass additional extra paramters

2020-09-18 Thread GitBox
daemon-demon commented on issue #10505: URL: https://github.com/apache/airflow/issues/10505#issuecomment-694995529 @RosterIn @agivola I will be creating a PR today for the fix I put in to allow all parameters to pass without restrictions.

[airflow] branch master updated (1a46e9b -> 1539bd0)

2020-09-18 Thread dimberman
This is an automated email from the ASF dual-hosted git repository. dimberman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git. from 1a46e9b Remove redundant curly brace from breeze echo message (#11012) add 1539bd0 KubernetesJobWatcher

[GitHub] [airflow] ryanahamilton commented on pull request #11018: Replace logo with SVG that includes name in proper typography

2020-09-18 Thread GitBox
ryanahamilton commented on pull request #11018: URL: https://github.com/apache/airflow/pull/11018#issuecomment-695033144 @ashb "I don't see where the don't is used?" typo? For smaller icon/logo SVGs, I typically opt for inline rendering and forgo the extra request.

[GitHub] [airflow] dimberman commented on a change in pull request #10956: Officially support running more than one scheduler concurrently.

2020-09-18 Thread GitBox
dimberman commented on a change in pull request #10956: URL: https://github.com/apache/airflow/pull/10956#discussion_r491140287 ## File path: airflow/jobs/scheduler_job.py ## @@ -1464,23 +1070,21 @@ def _execute_task_instances( 2. Change the state for the TIs above

[GitHub] [airflow] baolsen opened a new pull request #11020: Fix AWS DataSync tests failing (#10985)

2020-09-18 Thread GitBox
baolsen opened a new pull request #11020: URL: https://github.com/apache/airflow/pull/11020 Simple fix for DataSync Operator tests failing. Moto library was not returning a response containing 'Result' key when calling describe_task_execution. This is normally returned by

[GitHub] [airflow] baolsen commented on pull request #11020: Fix AWS DataSync tests failing (#10985)

2020-09-18 Thread GitBox
baolsen commented on pull request #11020: URL: https://github.com/apache/airflow/pull/11020#issuecomment-695050318 My GitHub actions were timing out on my repo, so I decided to try creating a PR upstream and see if the CI checks work correctly here.

[GitHub] [airflow] SalAlba opened a new pull request #11021: Increase typing coverage JDBC provider

2020-09-18 Thread GitBox
SalAlba opened a new pull request #11021: URL: https://github.com/apache/airflow/pull/11021 This PR is about increasing typing coverage for the JDBC provider. Part of: #9708 CC: @turbaszek @mik-laj Can I ask for review? --- **^ Add meaningful description above**

[GitHub] [airflow] dimberman opened a new pull request #11017: KubernetesJobWatcher no longer inherits from Process

2020-09-18 Thread GitBox
dimberman opened a new pull request #11017: URL: https://github.com/apache/airflow/pull/11017 multiprocessing.Process is set up in a very unfortunate manner that pretty much makes it impossible to test a class that inherits from Process or use any of its internal functions. For this

[GitHub] [airflow] dimberman merged pull request #11017: KubernetesJobWatcher no longer inherits from Process

2020-09-18 Thread GitBox
dimberman merged pull request #11017: URL: https://github.com/apache/airflow/pull/11017 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [airflow] ashb merged pull request #11018: Replace logo with SVG that includes name in proper typography

2020-09-18 Thread GitBox
ashb merged pull request #11018: URL: https://github.com/apache/airflow/pull/11018 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[airflow] branch master updated (1539bd0 -> f16f474)

2020-09-18 Thread ash
This is an automated email from the ASF dual-hosted git repository. ash pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git. from 1539bd0 KubernetesJobWatcher no longer inherits from Process (#11017) add f16f474 Replace PNG/text with SVG

[airflow] branch master updated (1539bd0 -> f16f474)

2020-09-18 Thread ash
This is an automated email from the ASF dual-hosted git repository. ash pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git. from 1539bd0 KubernetesJobWatcher no longer inherits from Process (#11017) add f16f474 Replace PNG/text with SVG

[airflow] branch master updated (1a46e9b -> 1539bd0)

2020-09-18 Thread dimberman
This is an automated email from the ASF dual-hosted git repository. dimberman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git. from 1a46e9b Remove redundant curly brace from breeze echo message (#11012) add 1539bd0 KubernetesJobWatcher

[GitHub] [airflow] daemon-demon opened a new pull request #11019: [Airflow-10505] Change to pass all extra connection paramaters to psycopg2

2020-09-18 Thread GitBox
daemon-demon opened a new pull request #11019: URL: https://github.com/apache/airflow/pull/11019 related: #6991 Allows the use of extra parameters while connecting to Postgres. The prior implementation only allowed passing ['sslmode', 'sslcert', 'sslkey','sslrootcert',

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #11019: [Airflow-10505] Change to pass all extra connection paramaters to psycopg2

2020-09-18 Thread GitBox
boring-cyborg[bot] commented on pull request #11019: URL: https://github.com/apache/airflow/pull/11019#issuecomment-695023316 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our

[GitHub] [airflow] joshi95 commented on pull request #10762: Fetching databricks host from connection if not supplied in extras.

2020-09-18 Thread GitBox
joshi95 commented on pull request #10762: URL: https://github.com/apache/airflow/pull/10762#issuecomment-694739302 > I see. Thanks for the clarification. > > May you rebase? Once ci is green I would be happy to merge. > > Thanks. Yes thanks, Have rebased .

[GitHub] [airflow] turbaszek commented on a change in pull request #10947: Add amazon glacier to GCS transfer operator

2020-09-18 Thread GitBox
turbaszek commented on a change in pull request #10947: URL: https://github.com/apache/airflow/pull/10947#discussion_r490818822 ## File path: airflow/providers/amazon/aws/transfers/glacier_to_gcs.py ## @@ -0,0 +1,162 @@ +# +# Licensed to the Apache Software Foundation (ASF)

[GitHub] [airflow] ashb commented on a change in pull request #10956: Officially support running more than one scheduler concurrently.

2020-09-18 Thread GitBox
ashb commented on a change in pull request #10956: URL: https://github.com/apache/airflow/pull/10956#discussion_r490827114 ## File path: airflow/models/dag.py ## @@ -458,6 +458,101 @@ def previous_schedule(self, dttm): elif self.normalized_schedule_interval is not

[GitHub] [airflow] XD-DENG commented on pull request #10762: Fetching databricks host from connection if not supplied in extras.

2020-09-18 Thread GitBox
XD-DENG commented on pull request #10762: URL: https://github.com/apache/airflow/pull/10762#issuecomment-694766855 Possibly a transient error. I have restarted the CI, let’s see if it will be ok This is an automated message

[GitHub] [airflow] turbaszek commented on a change in pull request #10947: Add amazon glacier to GCS transfer operator

2020-09-18 Thread GitBox
turbaszek commented on a change in pull request #10947: URL: https://github.com/apache/airflow/pull/10947#discussion_r490836571 ## File path: tests/providers/amazon/aws/hooks/test_glacier.py ## @@ -0,0 +1,173 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one

[jira] [Commented] (AIRFLOW-3607) Decreasing scheduler delay between tasks

2020-09-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-3607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198270#comment-17198270 ] ASF GitHub Bot commented on AIRFLOW-3607: - yuqian90 opened a new pull request #11010: URL:

[GitHub] [airflow] yuqian90 opened a new pull request #11010: [AIRFLOW-3607] Only query DB once per DAG run for TriggerRuleDep

2020-09-18 Thread GitBox
yuqian90 opened a new pull request #11010: URL: https://github.com/apache/airflow/pull/11010 This is cherry-picked from #4751 for `v1-10-test`. Some investigation into the issue reported in #10790 led to the discovery that this loop in `scheduler_job.py` takes almost 90% of the

[GitHub] [airflow] ashb commented on pull request #10956: Officially support running more than one scheduler concurrently.

2020-09-18 Thread GitBox
ashb commented on pull request #10956: URL: https://github.com/apache/airflow/pull/10956#issuecomment-694789979 As disucssed with @potiuk , I'll extend this PR to add config option to disable all locking (a.k.a. escape hatch) in case of unforeseen MySQL performance issues. Locking

[airflow] 01/01: Add Secrets backend for Microsoft Azure Key Vault (#10898)

2020-09-18 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit 8644aa953e541672e61b1e1ba0dca0b2691457d9 Author: Kaxil Naik AuthorDate: Sun Sep 13 15:45:21 2020 +0100 Add

[airflow] branch v1-10-test updated (c9856f1 -> 8644aa9)

2020-09-18 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git. omit c9856f1 Test: Set UPGRADE_TO_LATEST_CONSTRAINTS to 'true' omit 74faf5e Add Secrets backend for Microsoft

[airflow] branch v1-10-test updated (c9856f1 -> 8644aa9)

2020-09-18 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git. omit c9856f1 Test: Set UPGRADE_TO_LATEST_CONSTRAINTS to 'true' omit 74faf5e Add Secrets backend for Microsoft

[jira] [Commented] (AIRFLOW-6786) Adding KafkaConsumerHook, KafkaProducerHook, and KafkaSensor

2020-09-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198299#comment-17198299 ] ASF GitHub Bot commented on AIRFLOW-6786: - dferguson992 commented on a change in pull request

[GitHub] [airflow] dimberman commented on pull request #10153: [AIP-34] TaskGroup: A UI task grouping concept as an alternative to SubDagOperator

2020-09-18 Thread GitBox
dimberman commented on pull request #10153: URL: https://github.com/apache/airflow/pull/10153#issuecomment-695161125 @yuqian90 Heck yes!!! I'm so excited for this  This is an automated message from the Apache Git Service.

[GitHub] [airflow] KevinYang21 commented on a change in pull request #10956: Officially support running more than one scheduler concurrently.

2020-09-18 Thread GitBox
KevinYang21 commented on a change in pull request #10956: URL: https://github.com/apache/airflow/pull/10956#discussion_r491234879 ## File path: airflow/models/dag.py ## @@ -1896,6 +2050,36 @@ def deactivate_deleted_dags(cls, alive_dag_filelocs: List[str], session=None):

[GitHub] [airflow] rootcss commented on pull request #10770: Add Plugins View in web UI

2020-09-18 Thread GitBox
rootcss commented on pull request #10770: URL: https://github.com/apache/airflow/pull/10770#issuecomment-695167454 @kaxil @mik-laj This PR can be reviewed now. This is an automated message from the Apache Git Service. To

[GitHub] [airflow] jhtimmins opened a new pull request #11027: Replace get accessible dag ids

2020-09-18 Thread GitBox
jhtimmins opened a new pull request #11027: URL: https://github.com/apache/airflow/pull/11027 This PR builds on #10594 and replaces the calls to the old `get_accessible_dag_ids` with the newer `get_readable_dag_ids`. The new method uses resource-based permissions.

[GitHub] [airflow] nabinkhadka commented on pull request #9464: Fix DockerOperator xcom

2020-09-18 Thread GitBox
nabinkhadka commented on pull request #9464: URL: https://github.com/apache/airflow/pull/9464#issuecomment-694672021 do_xcom_push does not work in case of DockerOperator. Waiting for this. This is an automated message from

[GitHub] [airflow] abhilash1in edited a comment on pull request #10873: Add description field to connection (#10840)

2020-09-18 Thread GitBox
abhilash1in edited a comment on pull request #10873: URL: https://github.com/apache/airflow/pull/10873#issuecomment-694610175 Can this be added to Airflow 1.10.13 [milestone](https://github.com/apache/airflow/milestone/7)?

[GitHub] [airflow] nullhack commented on pull request #9464: Fix DockerOperator xcom

2020-09-18 Thread GitBox
nullhack commented on pull request #9464: URL: https://github.com/apache/airflow/pull/9464#issuecomment-694689569 > Hey @nullhack can you rebase, please? In the meantime we introduced black formatter in providers packages :) Yes, I'll do during the weekend

[GitHub] [airflow] ephraimbuddy commented on pull request #10990: Add example dag and system test for MySQLToGCSOperator

2020-09-18 Thread GitBox
ephraimbuddy commented on pull request #10990: URL: https://github.com/apache/airflow/pull/10990#issuecomment-694709553 @turbaszek, @mik-laj, can I ask for a review? This is an automated message from the Apache Git Service.

[GitHub] [airflow] turbaszek commented on pull request #10663: Show a few lines of tracebacks for DAG import errors in web UI

2020-09-18 Thread GitBox
turbaszek commented on pull request #10663: URL: https://github.com/apache/airflow/pull/10663#issuecomment-694709824 > @ashb, would you like me to rebase to see if the tests go green? @jedcunningham yes, please rebase

[GitHub] [airflow] zbigniev commented on issue #10790: Copy of [AIRFLOW-5071] JIRA: Thousands of Executor reports task instance X finished (success) although the task says its queued. Was the task kil

2020-09-18 Thread GitBox
zbigniev commented on issue #10790: URL: https://github.com/apache/airflow/issues/10790#issuecomment-694712736 We are on Airflow 1.10.10 Besides the DAGs which have sensor tasks in them, we are even encountering this in tasks which have no sensors in them, for example a DAG which

[jira] [Commented] (AIRFLOW-5071) Thousand os Executor reports task instance X finished (success) although the task says its queued. Was the task killed externally?

2020-09-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198186#comment-17198186 ] ASF GitHub Bot commented on AIRFLOW-5071: - zbigniev commented on issue #10790: URL:

[GitHub] [airflow] zbigniev edited a comment on issue #10790: Copy of [AIRFLOW-5071] JIRA: Thousands of Executor reports task instance X finished (success) although the task says its queued. Was the t

2020-09-18 Thread GitBox
zbigniev edited a comment on issue #10790: URL: https://github.com/apache/airflow/issues/10790#issuecomment-694712736 We are on Airflow 1.10.10 Besides the DAGs which have sensor tasks in them, we are even encountering this in tasks which have no sensors in them, for example a DAG

[GitHub] [airflow] XD-DENG commented on pull request #10762: Fetching databricks host from connection if not supplied in extras.

2020-09-18 Thread GitBox
XD-DENG commented on pull request #10762: URL: https://github.com/apache/airflow/pull/10762#issuecomment-694729962 Hi @joshi95 , looks good to me. May you rebase? This is an automated message from the Apache Git Service. To

[GitHub] [airflow] turbaszek commented on a change in pull request #10947: Add amazon glacier to GCS transfer operator

2020-09-18 Thread GitBox
turbaszek commented on a change in pull request #10947: URL: https://github.com/apache/airflow/pull/10947#discussion_r490839696 ## File path: airflow/providers/amazon/aws/sensors/glacier.py ## @@ -0,0 +1,75 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +#

[GitHub] [airflow] ashb edited a comment on pull request #10956: Officially support running more than one scheduler concurrently.

2020-09-18 Thread GitBox
ashb edited a comment on pull request #10956: URL: https://github.com/apache/airflow/pull/10956#issuecomment-694789979 As discussed with @potiuk , I'll extend this PR to add config option to disable all locking (a.k.a. escape hatch) in case of unforeseen MySQL performance issues. Locking

[airflow] branch v1-10-test updated (8644aa9 -> ea36166)

2020-09-18 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git. discard 8644aa9 Add Secrets backend for Microsoft Azure Key Vault (#10898) add ea36166 Add Secrets backend for

[airflow] branch v1-10-test updated (8644aa9 -> ea36166)

2020-09-18 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git. discard 8644aa9 Add Secrets backend for Microsoft Azure Key Vault (#10898) add ea36166 Add Secrets backend for

[GitHub] [airflow] kaxil commented on issue #11011: AWS DataSync Operator does not cancel task on Exception

2020-09-18 Thread GitBox
kaxil commented on issue #11011: URL: https://github.com/apache/airflow/issues/11011#issuecomment-694813177 Assigned :) Btw let me know you want to work on https://github.com/apache/airflow/issues/10985 too (The new version of moto added support for Datasync) and tests are failing

[airflow-site] branch asf-site updated: Update asf-site to output generated at 83d8103

2020-09-18 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/airflow-site.git The following commit(s) were added to refs/heads/asf-site by this push: new 31c654d Update asf-site to

[GitHub] [airflow-site] XD-DENG commented on pull request #292: add geniumcloud in ecosystem

2020-09-18 Thread GitBox
XD-DENG commented on pull request #292: URL: https://github.com/apache/airflow-site/pull/292#issuecomment-694812808 This product didn't mention Airflow (implicitly or explicitly) at all in the site. I'm not sure if it's valid to be added here.

[GitHub] [airflow] kaxil edited a comment on issue #11011: AWS DataSync Operator does not cancel task on Exception

2020-09-18 Thread GitBox
kaxil edited a comment on issue #11011: URL: https://github.com/apache/airflow/issues/11011#issuecomment-694813177 Assigned :) Btw let me if know you want to work on https://github.com/apache/airflow/issues/10985 too (The new version of moto added support for Datasync) and tests

[GitHub] [airflow] baolsen closed issue #11011: AWS DataSync Operator does not cancel task on Exception

2020-09-18 Thread GitBox
baolsen closed issue #11011: URL: https://github.com/apache/airflow/issues/11011 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [airflow] turbaszek commented on a change in pull request #10947: Add amazon glacier to GCS transfer operator

2020-09-18 Thread GitBox
turbaszek commented on a change in pull request #10947: URL: https://github.com/apache/airflow/pull/10947#discussion_r490837624 ## File path: tests/providers/amazon/aws/operators/test_glacier_system.py ## @@ -0,0 +1,39 @@ +# Licensed to the Apache Software Foundation (ASF)

[GitHub] [airflow] turbaszek commented on a change in pull request #10947: Add amazon glacier to GCS transfer operator

2020-09-18 Thread GitBox
turbaszek commented on a change in pull request #10947: URL: https://github.com/apache/airflow/pull/10947#discussion_r490838580 ## File path: airflow/providers/amazon/aws/sensors/glacier.py ## @@ -0,0 +1,75 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +#

[GitHub] [airflow] turbaszek commented on a change in pull request #10947: Add amazon glacier to GCS transfer operator

2020-09-18 Thread GitBox
turbaszek commented on a change in pull request #10947: URL: https://github.com/apache/airflow/pull/10947#discussion_r490838264 ## File path: tests/providers/amazon/aws/sensors/test_glacier.py ## @@ -0,0 +1,67 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one

[GitHub] [airflow] turbaszek commented on a change in pull request #10947: Add amazon glacier to GCS transfer operator

2020-09-18 Thread GitBox
turbaszek commented on a change in pull request #10947: URL: https://github.com/apache/airflow/pull/10947#discussion_r490837922 ## File path: tests/providers/amazon/aws/sensors/test_glacier.py ## @@ -0,0 +1,67 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one

[GitHub] [airflow] kaxil commented on issue #10985: AWS Datasync tests are failing with Moto 1.3.16

2020-09-18 Thread GitBox
kaxil commented on issue #10985: URL: https://github.com/apache/airflow/issues/10985#issuecomment-694814050 awesome, thanks :) This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [airflow] baolsen commented on issue #10985: AWS Datasync tests are failing with Moto 1.3.16

2020-09-18 Thread GitBox
baolsen commented on issue #10985: URL: https://github.com/apache/airflow/issues/10985#issuecomment-694813656 @kaxil you can assign to me, thanks :) This is an automated message from the Apache Git Service. To respond to the

[jira] [Commented] (AIRFLOW-5071) Thousand os Executor reports task instance X finished (success) although the task says its queued. Was the task killed externally?

2020-09-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198187#comment-17198187 ] ASF GitHub Bot commented on AIRFLOW-5071: - zbigniev edited a comment on issue #10790: URL:

[GitHub] [airflow] XD-DENG commented on pull request #10762: Fetching databricks host from connection if not supplied in extras.

2020-09-18 Thread GitBox
XD-DENG commented on pull request #10762: URL: https://github.com/apache/airflow/pull/10762#issuecomment-694731894 On the other hand, before I can merge, the test you added doesn't make much sense to me. May you clarify a bit or enhance it further?

[GitHub] [airflow] turbaszek commented on a change in pull request #10947: Add amazon glacier to GCS transfer operator

2020-09-18 Thread GitBox
turbaszek commented on a change in pull request #10947: URL: https://github.com/apache/airflow/pull/10947#discussion_r490818258 ## File path: airflow/providers/amazon/aws/transfers/glacier_to_gcs.py ## @@ -0,0 +1,162 @@ +# +# Licensed to the Apache Software Foundation (ASF)

[GitHub] [airflow] yuqian90 commented on pull request #11010: [AIRFLOW-3607] Only query DB once per DAG run for TriggerRuleDep

2020-09-18 Thread GitBox
yuqian90 commented on pull request #11010: URL: https://github.com/apache/airflow/pull/11010#issuecomment-694792832 @amichai07 @kaxil @ashb anyone interested to take a look? I'm porting #4751 to 1.10.* to fix a bad `DagFileProcessor` slowness.

[airflow] 01/01: Add Secrets backend for Microsoft Azure Key Vault (#10898)

2020-09-18 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit 8644aa953e541672e61b1e1ba0dca0b2691457d9 Author: Kaxil Naik AuthorDate: Sun Sep 13 15:45:21 2020 +0100 Add

[airflow] 01/01: Add Secrets backend for Microsoft Azure Key Vault (#10898)

2020-09-18 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit 8644aa953e541672e61b1e1ba0dca0b2691457d9 Author: Kaxil Naik AuthorDate: Sun Sep 13 15:45:21 2020 +0100 Add

[airflow] 01/01: Add Secrets backend for Microsoft Azure Key Vault (#10898)

2020-09-18 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit 8644aa953e541672e61b1e1ba0dca0b2691457d9 Author: Kaxil Naik AuthorDate: Sun Sep 13 15:45:21 2020 +0100 Add

[airflow] 01/01: Add Secrets backend for Microsoft Azure Key Vault (#10898)

2020-09-18 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit ea36166961ca35fc51ddc262ec245590c3e236fb Author: Kaxil Naik AuthorDate: Sun Sep 13 15:45:21 2020 +0100 Add

[airflow] branch v1-10-test updated (8644aa9 -> ea36166)

2020-09-18 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git. discard 8644aa9 Add Secrets backend for Microsoft Azure Key Vault (#10898) new ea36166 Add Secrets backend for

[airflow] branch v1-10-test updated (c9856f1 -> 8644aa9)

2020-09-18 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git. omit c9856f1 Test: Set UPGRADE_TO_LATEST_CONSTRAINTS to 'true' omit 74faf5e Add Secrets backend for Microsoft

[airflow-site] branch master updated: Remove duplicate text under Principles section (#293)

2020-09-18 Thread xddeng
This is an automated email from the ASF dual-hosted git repository. xddeng pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/airflow-site.git The following commit(s) were added to refs/heads/master by this push: new 83d8103 Remove duplicate text under

[GitHub] [airflow-site] XD-DENG merged pull request #293: Remove duplicate text under Principles section

2020-09-18 Thread GitBox
XD-DENG merged pull request #293: URL: https://github.com/apache/airflow-site/pull/293 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [airflow] dferguson992 commented on a change in pull request #10660: [AIRFLOW-6786] Added Kafka components

2020-09-18 Thread GitBox
dferguson992 commented on a change in pull request #10660: URL: https://github.com/apache/airflow/pull/10660#discussion_r490895415 ## File path: airflow/providers/google/cloud/example_dags/example_dataproc.py ## @@ -65,10 +65,16 @@ # Update options # [START

[GitHub] [airflow] RosterIn commented on issue #10505: PostgressHook connection - pass additional extra paramters

2020-09-18 Thread GitBox
RosterIn commented on issue #10505: URL: https://github.com/apache/airflow/issues/10505#issuecomment-694836845 @daemon-demon There was a PR that tried to achieve that https://github.com/apache/airflow/pull/7632 maybe it will help you as a starting point

[GitHub] [airflow] turbaszek commented on a change in pull request #10947: Add amazon glacier to GCS transfer operator

2020-09-18 Thread GitBox
turbaszek commented on a change in pull request #10947: URL: https://github.com/apache/airflow/pull/10947#discussion_r490820641 ## File path: airflow/providers/amazon/aws/sensors/glacier.py ## @@ -0,0 +1,75 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +#

[GitHub] [airflow] turbaszek commented on a change in pull request #10947: Add amazon glacier to GCS transfer operator

2020-09-18 Thread GitBox
turbaszek commented on a change in pull request #10947: URL: https://github.com/apache/airflow/pull/10947#discussion_r490837373 ## File path: tests/providers/amazon/aws/operators/test_glacier_system.py ## @@ -0,0 +1,39 @@ +# Licensed to the Apache Software Foundation (ASF)

  1   2   >