[GitHub] [airflow] SamWheating commented on issue #18669: Failed to record duration of externally triggered DAGs

2021-10-05 Thread GitBox
SamWheating commented on issue #18669: URL: https://github.com/apache/airflow/issues/18669#issuecomment-935410862 Upon investigation it like this was already reported in https://github.com/apache/airflow/issues/18082 and fixed in https://github.com/apache/airflow/pull/18226. Feel

[GitHub] [airflow] github-actions[bot] commented on pull request #18447: Add RedshiftSQLHook, RedshiftSQLOperator

2021-10-05 Thread GitBox
github-actions[bot] commented on pull request #18447: URL: https://github.com/apache/airflow/pull/18447#issuecomment-935300096 The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and

[GitHub] [airflow] damon09273 closed issue #14267: GoogleCloudStorageToBigQueryOperator generate duplicate data when network issue happen

2021-10-05 Thread GitBox
damon09273 closed issue #14267: URL: https://github.com/apache/airflow/issues/14267 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] damon09273 commented on issue #14267: GoogleCloudStorageToBigQueryOperator generate duplicate data when network issue happen

2021-10-05 Thread GitBox
damon09273 commented on issue #14267: URL: https://github.com/apache/airflow/issues/14267#issuecomment-935281219 @eladkal The problem had been solved after upgrade Airflow 2.0+ and latest GCP provider packages. Thanks for response. -- This is an automated message from the Apache Git

[GitHub] [airflow] dstandish commented on a change in pull request #18692: Add support for storing conn as json in SSM secrets backend

2021-10-05 Thread GitBox
dstandish commented on a change in pull request #18692: URL: https://github.com/apache/airflow/pull/18692#discussion_r722833687 ## File path: airflow/providers/amazon/aws/secrets/systems_manager.py ## @@ -101,6 +105,37 @@ def get_conn_uri(self, conn_id: str) -> Optional[str]:

[GitHub] [airflow] denysivanov commented on issue #18598: SerializedDagNotFound - when create 1000 dags

2021-10-05 Thread GitBox
denysivanov commented on issue #18598: URL: https://github.com/apache/airflow/issues/18598#issuecomment-935235752 @potiuk I do realize this but in my example https://github.com/apache/airflow/files/7289250/DAG.load.process.docx I tried this in different way and it doesn't work as I

[GitHub] [airflow] guotongfei commented on a change in pull request #18755: Support mysql to s3 in parquet format

2021-10-05 Thread GitBox
guotongfei commented on a change in pull request #18755: URL: https://github.com/apache/airflow/pull/18755#discussion_r722813490 ## File path: airflow/providers/amazon/aws/transfers/mysql_to_s3.py ## @@ -82,6 +86,8 @@ def __init__( pd_csv_kwargs: Optional[dict] =

[GitHub] [airflow] mik-laj commented on a change in pull request #18760: Amazon SQS Example

2021-10-05 Thread GitBox
mik-laj commented on a change in pull request #18760: URL: https://github.com/apache/airflow/pull/18760#discussion_r722812772 ## File path: airflow/providers/amazon/aws/example_dags/example_sqs.py ## @@ -0,0 +1,83 @@ +# Licensed to the Apache Software Foundation (ASF) under

[GitHub] [airflow] uranusjr commented on issue #18714: Make it possible to delete a permission from a role using the stable REST API

2021-10-05 Thread GitBox
uranusjr commented on issue #18714: URL: https://github.com/apache/airflow/issues/18714#issuecomment-935177145 REST API is only a guide line without hard rules, but `PATCH` generally means the user can submit a subset of fields, and the missing fields would be filled by existing values on

[GitHub] [airflow] uranusjr commented on a change in pull request #18692: Add support for storing conn as json in SSM secrets backend

2021-10-05 Thread GitBox
uranusjr commented on a change in pull request #18692: URL: https://github.com/apache/airflow/pull/18692#discussion_r722806003 ## File path: airflow/providers/amazon/aws/secrets/systems_manager.py ## @@ -101,6 +105,37 @@ def get_conn_uri(self, conn_id: str) -> Optional[str]:

[GitHub] [airflow] john-jac commented on a change in pull request #18760: Amazon SQS Example

2021-10-05 Thread GitBox
john-jac commented on a change in pull request #18760: URL: https://github.com/apache/airflow/pull/18760#discussion_r722802150 ## File path: airflow/providers/amazon/aws/example_dags/example_sqs.py ## @@ -0,0 +1,83 @@ +# Licensed to the Apache Software Foundation (ASF) under

[GitHub] [airflow] uranusjr commented on a change in pull request #18760: Amazon SQS Example

2021-10-05 Thread GitBox
uranusjr commented on a change in pull request #18760: URL: https://github.com/apache/airflow/pull/18760#discussion_r722797835 ## File path: airflow/providers/amazon/aws/example_dags/example_sqs.py ## @@ -0,0 +1,83 @@ +# Licensed to the Apache Software Foundation (ASF) under

[GitHub] [airflow] john-jac commented on a change in pull request #18760: Amazon SQS Example

2021-10-05 Thread GitBox
john-jac commented on a change in pull request #18760: URL: https://github.com/apache/airflow/pull/18760#discussion_r722794720 ## File path: airflow/providers/amazon/aws/example_dags/example_sqs.py ## @@ -0,0 +1,83 @@ +# Licensed to the Apache Software Foundation (ASF) under

[GitHub] [airflow] mik-laj commented on a change in pull request #18760: Amazon SQS Example

2021-10-05 Thread GitBox
mik-laj commented on a change in pull request #18760: URL: https://github.com/apache/airflow/pull/18760#discussion_r722793467 ## File path: airflow/providers/amazon/aws/example_dags/example_sqs.py ## @@ -0,0 +1,83 @@ +# Licensed to the Apache Software Foundation (ASF) under

[GitHub] [airflow] uranusjr commented on pull request #18671: Remove duplicated code on S3ToRedshiftOperator

2021-10-05 Thread GitBox
uranusjr commented on pull request #18671: URL: https://github.com/apache/airflow/pull/18671#issuecomment-935121570 > even if we put (>2.3.0) there, this is not a "hard" requirement. Extras are very "soft" dependencies and they only work temporarily while installation is running - they

[GitHub] [airflow] mik-laj commented on a change in pull request #18760: Amazon SQS Example

2021-10-05 Thread GitBox
mik-laj commented on a change in pull request #18760: URL: https://github.com/apache/airflow/pull/18760#discussion_r722793467 ## File path: airflow/providers/amazon/aws/example_dags/example_sqs.py ## @@ -0,0 +1,83 @@ +# Licensed to the Apache Software Foundation (ASF) under

[GitHub] [airflow] john-jac opened a new pull request #18760: Amazon SQS Example

2021-10-05 Thread GitBox
john-jac opened a new pull request #18760: URL: https://github.com/apache/airflow/pull/18760 In this PR I've added a doc entry and example for using the SQS Publish Operator. The example included creating a new SQS queue, publishing to that queue, reading the queue, and deleting

[GitHub] [airflow] uranusjr commented on a change in pull request #18723: Remove duplicate line of code for configuration copy

2021-10-05 Thread GitBox
uranusjr commented on a change in pull request #18723: URL: https://github.com/apache/airflow/pull/18723#discussion_r722790213 ## File path: airflow/task/task_runner/base_task_runner.py ## @@ -61,13 +61,14 @@ def __init__(self, local_task_job): # Add sudo commands to

[GitHub] [airflow] uranusjr commented on a change in pull request #18715: Small improvements for Airflow UI

2021-10-05 Thread GitBox
uranusjr commented on a change in pull request #18715: URL: https://github.com/apache/airflow/pull/18715#discussion_r722784809 ## File path: airflow/www/templates/airflow/confirm.html ## @@ -21,7 +21,7 @@ {% block content %} {{ super() }} - Wait a minute. + Wait a

[GitHub] [airflow] uranusjr commented on a change in pull request #18715: Small improvements for Airflow UI

2021-10-05 Thread GitBox
uranusjr commented on a change in pull request #18715: URL: https://github.com/apache/airflow/pull/18715#discussion_r722784809 ## File path: airflow/www/templates/airflow/confirm.html ## @@ -21,7 +21,7 @@ {% block content %} {{ super() }} - Wait a minute. + Wait a

[GitHub] [airflow] uranusjr commented on a change in pull request #17378: Refactor SSHOperator so a subclass can run many commands (#10874)

2021-10-05 Thread GitBox
uranusjr commented on a change in pull request #17378: URL: https://github.com/apache/airflow/pull/17378#discussion_r686661494 ## File path: airflow/providers/ssh/operators/ssh.py ## @@ -80,103 +82,120 @@ def __init__( self.environment = environment

[GitHub] [airflow] uranusjr commented on pull request #18752: Open src and dst in binary for samba copy

2021-10-05 Thread GitBox
uranusjr commented on pull request #18752: URL: https://github.com/apache/airflow/pull/18752#issuecomment-935083037 Would it be possible to add a test for this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [airflow] zachliu commented on pull request #18733: ECSOperator: airflow exception on edge case when cloudwatch log stream is not found

2021-10-05 Thread GitBox
zachliu commented on pull request #18733: URL: https://github.com/apache/airflow/pull/18733#issuecomment-935080740 `prepare-provider-packages`! how do i forget this...

[GitHub] [airflow] uranusjr commented on pull request #18733: ECSOperator: airflow exception on edge case when cloudwatch log stream is not found

2021-10-05 Thread GitBox
uranusjr commented on pull request #18733: URL: https://github.com/apache/airflow/pull/18733#issuecomment-935059136 You can ``` # Check out the latest main branch... # This is basically how the actual release is built, sans the upload part. ./breeze

[GitHub] [airflow] uranusjr commented on pull request #18759: Movin trigger dag to operations folder

2021-10-05 Thread GitBox
uranusjr commented on pull request #18759: URL: https://github.com/apache/airflow/pull/18759#issuecomment-935025049 There are many more that should be moved, basically everything mentioned in `airflow.www.views` (and probably some more, I didn't check). They don't need to done in this PR,

[GitHub] [airflow] mik-laj edited a comment on pull request #18759: Movin trigger dag to operations folder

2021-10-05 Thread GitBox
mik-laj edited a comment on pull request #18759: URL: https://github.com/apache/airflow/pull/18759#issuecomment-934958612 Can you add backward-compatibility with deprecation warning? For example, see: airflow.contrib module. We should also add note in UPDATING.md. -- This is an

[GitHub] [airflow] mik-laj commented on pull request #18759: Movin trigger dag to operations folder

2021-10-05 Thread GitBox
mik-laj commented on pull request #18759: URL: https://github.com/apache/airflow/pull/18759#issuecomment-934958612 Can you add backward-compatibility with deprecation warning? For example, see: airflow.contrib module. -- This is an automated message from the Apache Git Service. To

[GitHub] [airflow] zachliu commented on pull request #18733: ECSOperator: airflow exception on edge case when cloudwatch log stream is not found

2021-10-05 Thread GitBox
zachliu commented on pull request #18733: URL: https://github.com/apache/airflow/pull/18733#issuecomment-934953163 > Or check-out main version nd run it there (with Breeze providers are by default installed from local sources) oh, so i cannot just test this provider with my 2.1.4?

[GitHub] [airflow] kaxil commented on pull request #18616: Retry deadlocked transactions on deleting old rendered task fields

2021-10-05 Thread GitBox
kaxil commented on pull request #18616: URL: https://github.com/apache/airflow/pull/18616#issuecomment-934951986 Yup, it will be part of 2.2.0 to be released next week - RCs within next 2 days -- This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [airflow] mariotaddeucci closed pull request #18660: Improve DbApi code readability

2021-10-05 Thread GitBox
mariotaddeucci closed pull request #18660: URL: https://github.com/apache/airflow/pull/18660 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] SamWheating commented on issue #18669: Failed to record duration of externally triggered DAGs

2021-10-05 Thread GitBox
SamWheating commented on issue #18669: URL: https://github.com/apache/airflow/issues/18669#issuecomment-934947045 I'm interested in looking into this - feel free to assign it to me. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [airflow] khalidmammadov opened a new pull request #18759: Movin trigger dag to operations folder

2021-10-05 Thread GitBox
khalidmammadov opened a new pull request #18759: URL: https://github.com/apache/airflow/pull/18759 This is to move/graduate trigger dag modue out of experimental folder to a new operations folder. New operations directory can host similar modules for internal implementations going

[GitHub] [airflow] uranusjr commented on a change in pull request #18755: Support mysql to s3 in parquet format

2021-10-05 Thread GitBox
uranusjr commented on a change in pull request #18755: URL: https://github.com/apache/airflow/pull/18755#discussion_r722751676 ## File path: airflow/providers/amazon/aws/transfers/mysql_to_s3.py ## @@ -82,6 +86,8 @@ def __init__( pd_csv_kwargs: Optional[dict] = None,

[GitHub] [airflow] uranusjr commented on issue #14396: Make context less nebulous

2021-10-05 Thread GitBox
uranusjr commented on issue #14396: URL: https://github.com/apache/airflow/issues/14396#issuecomment-934943925 Maybe a `typing.TypedDict` definition would be useful though. It requires much less change (basically just write a type definition and use it in some function annotations) and do

[GitHub] [airflow] uranusjr commented on a change in pull request #18757: Make REST API patch user endpoint work the same way as the UI

2021-10-05 Thread GitBox
uranusjr commented on a change in pull request #18757: URL: https://github.com/apache/airflow/pull/18757#discussion_r722748881 ## File path: airflow/api_connexion/endpoints/user_endpoint.py ## @@ -124,9 +124,21 @@ def patch_user(username, update_mask=None): if user is

[GitHub] [airflow] Brooke-white commented on a change in pull request #18447: Add RedshiftSQLHook, RedshiftSQLOperator

2021-10-05 Thread GitBox
Brooke-white commented on a change in pull request #18447: URL: https://github.com/apache/airflow/pull/18447#discussion_r722744704 ## File path: airflow/providers/amazon/aws/operators/redshift.py ## @@ -0,0 +1,73 @@ +# +# Licensed to the Apache Software Foundation (ASF) under

[airflow-site] branch gh-pages updated: Deploying to gh-pages from @ e04b11e761c4b706cdd217a719aa955d01874dcb 

2021-10-05 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch gh-pages in repository https://gitbox.apache.org/repos/asf/airflow-site.git The following commit(s) were added to refs/heads/gh-pages by this push: new a6b7aef Deploying to gh-pages

[airflow-site] branch main updated: Add Influx DB to list of providers (#492)

2021-10-05 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow-site.git The following commit(s) were added to refs/heads/main by this push: new e04b11e Add Influx DB to list of providers

[GitHub] [airflow-site] potiuk merged pull request #492: Add Influx DB to list of providers

2021-10-05 Thread GitBox
potiuk merged pull request #492: URL: https://github.com/apache/airflow-site/pull/492 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] potiuk commented on pull request #18161: Duplicate Connection: Added logic to query if a connection id exists before creating one

2021-10-05 Thread GitBox
potiuk commented on pull request #18161: URL: https://github.com/apache/airflow/pull/18161#issuecomment-934927282 > There is a test in test_views_connection.py that tests the duplication logic, I can add one more to test the logic of erroring out when 10 connections already exists.

[GitHub] [airflow] kaxil commented on pull request #18522: Coerce datetime to pendulum for timetable

2021-10-05 Thread GitBox
kaxil commented on pull request #18522: URL: https://github.com/apache/airflow/pull/18522#issuecomment-934922929 Some API tests are failing -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [airflow] uranusjr opened a new issue #18758: Deprecate 'airflow dags test' and merge the use case into 'airflow dags backfill' and 'airflow dags trigger'

2021-10-05 Thread GitBox
uranusjr opened a new issue #18758: URL: https://github.com/apache/airflow/issues/18758 ### Description Currently (in versions up to 2.1.4), `airflow dags test ` creates a *backfill* run at the specified datetime. This, however, applies regardless of whether the DAG can actually

[GitHub] [airflow] kaxil commented on issue #18713: /roles stable API endpoint PATCH verb needs better documentation in the openapi spec

2021-10-05 Thread GitBox
kaxil commented on issue #18713: URL: https://github.com/apache/airflow/issues/18713#issuecomment-934909441 Assigning it to you Alex. Feel free to pair up with @ephraimbuddy if you need any help -- This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [airflow] potiuk commented on pull request #18733: ECSOperator: airflow exception on edge case when cloudwatch log stream is not found

2021-10-05 Thread GitBox
potiuk commented on pull request #18733: URL: https://github.com/apache/airflow/pull/18733#issuecomment-934907431 Or check-out main version nd run it there (with Breeze providers are by default installed from local sources) -- This is an automated message from the Apache Git Service. To

[GitHub] [airflow] potiuk commented on issue #18598: SerializedDagNotFound - when create 1000 dags

2021-10-05 Thread GitBox
potiuk commented on issue #18598: URL: https://github.com/apache/airflow/issues/18598#issuecomment-934906238 As I understand travelling to Hashicorp seems to be the culprit. As mentioned in the best practices - you should not rech out from DAGs while parsing (i.e. in top-level-code).

[GitHub] [airflow] zachliu commented on pull request #18733: ECSOperator: airflow exception on edge case when cloudwatch log stream is not found

2021-10-05 Thread GitBox
zachliu commented on pull request #18733: URL: https://github.com/apache/airflow/pull/18733#issuecomment-934904278 dumb question: how do i test this out in my airflow? i suppose i just wait for `providers-amazon/2.3.0rc2`? -- This is an automated message from the Apache Git Service.

[GitHub] [airflow] subkanthi commented on pull request #18161: Duplicate Connection: Added logic to query if a connection id exists before creating one

2021-10-05 Thread GitBox
subkanthi commented on pull request #18161: URL: https://github.com/apache/airflow/pull/18161#issuecomment-934892074 > OK. Ping us when there are tests :) There is a test in test_views_connection.py that tests the duplication logic, I can add one more to test the logic of erroring

[GitHub] [airflow] kaxil commented on issue #14396: Make context less nebulous

2021-10-05 Thread GitBox
kaxil commented on issue #14396: URL: https://github.com/apache/airflow/issues/14396#issuecomment-934891462 Exactly what Jarek said -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [airflow] kaxil edited a comment on issue #18721: Add a setter to on_failure_callback and on_success_callback DAG properties

2021-10-05 Thread GitBox
kaxil edited a comment on issue #18721: URL: https://github.com/apache/airflow/issues/18721#issuecomment-934880860 Using `@property` for `has_*_callback` should fix it @vilozio similar to

[GitHub] [airflow] kaxil commented on issue #18721: Add a setter to on_failure_callback and on_success_callback DAG properties

2021-10-05 Thread GitBox
kaxil commented on issue #18721: URL: https://github.com/apache/airflow/issues/18721#issuecomment-934880860 Using `@property` for `has_*_callback` should fix it @vilozio -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [airflow] potiuk commented on pull request #18161: Duplicate Connection: Added logic to query if a connection id exists before creating one

2021-10-05 Thread GitBox
potiuk commented on pull request #18161: URL: https://github.com/apache/airflow/pull/18161#issuecomment-934879885 OK. Ping us when there are tests :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[airflow] branch main updated (a192b4a -> 767a4f5)

2021-10-05 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git. from a192b4a Chart: Mount DAGs in triggerer (#18753) add 767a4f5 ECSOperator: airflow exception on edge case when

[GitHub] [airflow] potiuk merged pull request #18733: ECSOperator: airflow exception on edge case when cloudwatch log stream is not found

2021-10-05 Thread GitBox
potiuk merged pull request #18733: URL: https://github.com/apache/airflow/pull/18733 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] potiuk closed issue #15000: When an ECS Task fails to start, ECS Operator raises a CloudWatch exception

2021-10-05 Thread GitBox
potiuk closed issue #15000: URL: https://github.com/apache/airflow/issues/15000 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] github-actions[bot] commented on pull request #18757: Make REST API patch user endpoint work the same way as the UI

2021-10-05 Thread GitBox
github-actions[bot] commented on pull request #18757: URL: https://github.com/apache/airflow/pull/18757#issuecomment-934877987 The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not

[GitHub] [airflow] denysivanov commented on issue #18598: SerializedDagNotFound - when create 1000 dags

2021-10-05 Thread GitBox
denysivanov commented on issue #18598: URL: https://github.com/apache/airflow/issues/18598#issuecomment-934877646 @potiuk I can if you are talking about my set up :) But I am still do not have real answer why it behaves differently in my environment. -- This is an

[airflow] branch main updated (9eae83c -> a192b4a)

2021-10-05 Thread jedcunningham
This is an automated email from the ASF dual-hosted git repository. jedcunningham pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git. from 9eae83c Remove eagerloading when querying for TI (#18706) add a192b4a Chart: Mount DAGs in triggerer

[GitHub] [airflow] potiuk commented on issue #18598: SerializedDagNotFound - when create 1000 dags

2021-10-05 Thread GitBox
potiuk commented on issue #18598: URL: https://github.com/apache/airflow/issues/18598#issuecomment-934865022 This is fantastic! Maybe you'd like to write a blog post on it - we could publish it at https://medium.com/apache-airflow? -- This is an automated message from the Apache Git

[GitHub] [airflow] jedcunningham merged pull request #18753: Chart: Mount DAGs in triggerer

2021-10-05 Thread GitBox
jedcunningham merged pull request #18753: URL: https://github.com/apache/airflow/pull/18753 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[airflow] branch main updated (958860f -> 9eae83c)

2021-10-05 Thread ephraimanierobi
This is an automated email from the ASF dual-hosted git repository. ephraimanierobi pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git. from 958860f Remove AIRFLOW_GID from Docker images (#18747) add 9eae83c Remove eagerloading when querying

[GitHub] [airflow] ephraimbuddy merged pull request #18706: Remove eagerloading when querying for TI

2021-10-05 Thread GitBox
ephraimbuddy merged pull request #18706: URL: https://github.com/apache/airflow/pull/18706 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] jedcunningham commented on a change in pull request #18542: Allow ingress multiple hostnames w/diff secrets

2021-10-05 Thread GitBox
jedcunningham commented on a change in pull request #18542: URL: https://github.com/apache/airflow/pull/18542#discussion_r722710148 ## File path: chart/templates/webserver/webserver-ingress.yaml ## @@ -99,8 +114,12 @@ spec: servicePort: {{ .servicePort }}

[GitHub] [airflow] ferruzzi commented on a change in pull request #18645: Add AWS Fargate profile support

2021-10-05 Thread GitBox
ferruzzi commented on a change in pull request #18645: URL: https://github.com/apache/airflow/pull/18645#discussion_r722712868 ## File path: airflow/providers/amazon/aws/operators/eks.py ## @@ -23,25 +23,41 @@ from airflow import AirflowException from airflow.models import

[GitHub] [airflow] dstandish commented on a change in pull request #18447: Add RedshiftSQLHook, RedshiftSQLOperator

2021-10-05 Thread GitBox
dstandish commented on a change in pull request #18447: URL: https://github.com/apache/airflow/pull/18447#discussion_r722690470 ## File path: docs/apache-airflow-providers-amazon/connections/redshift.rst ## @@ -0,0 +1,82 @@ +.. Licensed to the Apache Software Foundation (ASF)

[GitHub] [airflow] ferruzzi commented on a change in pull request #18645: Add AWS Fargate profile support

2021-10-05 Thread GitBox
ferruzzi commented on a change in pull request #18645: URL: https://github.com/apache/airflow/pull/18645#discussion_r722712632 ## File path: airflow/providers/amazon/aws/operators/eks.py ## @@ -93,21 +122,42 @@ def __init__( cluster_name: str,

[GitHub] [airflow] ephraimbuddy opened a new pull request #18757: Make REST API patch user endpoint work the same way as the UI

2021-10-05 Thread GitBox
ephraimbuddy opened a new pull request #18757: URL: https://github.com/apache/airflow/pull/18757 Username is updateable in the UI and already exists error is also raised when trying to update with username/email that already exists. This PR ensures that the REST API works the same way

[GitHub] [airflow] ferruzzi commented on a change in pull request #18645: Add AWS Fargate profile support

2021-10-05 Thread GitBox
ferruzzi commented on a change in pull request #18645: URL: https://github.com/apache/airflow/pull/18645#discussion_r722712363 ## File path: airflow/providers/amazon/aws/operators/eks.py ## @@ -227,6 +284,78 @@ def execute(self, context): ) +class

[GitHub] [airflow] ferruzzi commented on a change in pull request #18645: Add AWS Fargate profile support

2021-10-05 Thread GitBox
ferruzzi commented on a change in pull request #18645: URL: https://github.com/apache/airflow/pull/18645#discussion_r722712072 ## File path: airflow/providers/amazon/aws/operators/eks.py ## @@ -280,41 +409,83 @@ def execute(self, context): ) if

[GitHub] [airflow] ferruzzi commented on a change in pull request #18645: Add AWS Fargate profile support

2021-10-05 Thread GitBox
ferruzzi commented on a change in pull request #18645: URL: https://github.com/apache/airflow/pull/18645#discussion_r722712001 ## File path: airflow/providers/amazon/aws/operators/eks.py ## @@ -280,41 +409,83 @@ def execute(self, context): ) if

[GitHub] [airflow] dstandish commented on a change in pull request #18447: Add RedshiftSQLHook, RedshiftSQLOperator

2021-10-05 Thread GitBox
dstandish commented on a change in pull request #18447: URL: https://github.com/apache/airflow/pull/18447#discussion_r722690470 ## File path: docs/apache-airflow-providers-amazon/connections/redshift.rst ## @@ -0,0 +1,82 @@ +.. Licensed to the Apache Software Foundation (ASF)

[GitHub] [airflow] dstandish commented on a change in pull request #18447: Add RedshiftSQLHook, RedshiftSQLOperator

2021-10-05 Thread GitBox
dstandish commented on a change in pull request #18447: URL: https://github.com/apache/airflow/pull/18447#discussion_r722690470 ## File path: docs/apache-airflow-providers-amazon/connections/redshift.rst ## @@ -0,0 +1,82 @@ +.. Licensed to the Apache Software Foundation (ASF)

[GitHub] [airflow] ferruzzi commented on a change in pull request #18645: Add AWS Fargate profile support

2021-10-05 Thread GitBox
ferruzzi commented on a change in pull request #18645: URL: https://github.com/apache/airflow/pull/18645#discussion_r722711594 ## File path: setup.py ## @@ -513,7 +513,7 @@ def write_version(filename: str = os.path.join(*[my_dir, "airflow", "git_version 'jira',

[GitHub] [airflow] ferruzzi commented on a change in pull request #18645: Add AWS Fargate profile support

2021-10-05 Thread GitBox
ferruzzi commented on a change in pull request #18645: URL: https://github.com/apache/airflow/pull/18645#discussion_r722711354 ## File path: airflow/providers/amazon/aws/example_dags/example_eks_with_fargate_profile.py ## @@ -0,0 +1,124 @@ +# Licensed to the Apache Software

[GitHub] [airflow] ferruzzi commented on a change in pull request #18645: Add AWS Fargate profile support

2021-10-05 Thread GitBox
ferruzzi commented on a change in pull request #18645: URL: https://github.com/apache/airflow/pull/18645#discussion_r722711186 ## File path: airflow/providers/amazon/aws/example_dags/example_eks_with_nodegroups.py ## @@ -40,11 +39,15 @@ 'endpointPrivateAccess': False, }

[GitHub] [airflow] dstandish commented on a change in pull request #18447: Add RedshiftSQLHook, RedshiftSQLOperator

2021-10-05 Thread GitBox
dstandish commented on a change in pull request #18447: URL: https://github.com/apache/airflow/pull/18447#discussion_r722654639 ## File path: airflow/providers/amazon/aws/operators/redshift.py ## @@ -0,0 +1,73 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one

[GitHub] [airflow] denysivanov commented on issue #18598: SerializedDagNotFound - when create 1000 dags

2021-10-05 Thread GitBox
denysivanov commented on issue #18598: URL: https://github.com/apache/airflow/issues/18598#issuecomment-934834141 @potiuk Currently I have in PROD 186 dags. I would have around 500 by new year + 500 - 1000 dags during 2022. We set up Grafana in prod and I see that DAG

[GitHub] [airflow] jedcunningham commented on pull request #18615: fix template string passed to env_vars when using render_template_as_native_obj for Kubernetes Pod Operator and add Tests

2021-10-05 Thread GitBox
jedcunningham commented on pull request #18615: URL: https://github.com/apache/airflow/pull/18615#issuecomment-934825317 My main concern here is that `env_vars` intentionally takes `List[V1EnvVar]` - taking a `dict` is currently for backward compatibility only at this point. It's not that

[GitHub] [airflow] codenamestif commented on pull request #18733: ECSOperator: airflow exception on edge case when cloudwatch log stream is not found

2021-10-05 Thread GitBox
codenamestif commented on pull request #18733: URL: https://github.com/apache/airflow/pull/18733#issuecomment-934817443 It should be good to go now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [airflow] guotongfei commented on a change in pull request #18755: Support mysql to s3 in parquet format

2021-10-05 Thread GitBox
guotongfei commented on a change in pull request #18755: URL: https://github.com/apache/airflow/pull/18755#discussion_r722687206 ## File path: airflow/providers/amazon/aws/transfers/mysql_to_s3.py ## @@ -60,6 +60,10 @@ class MySQLToS3Operator(BaseOperator): :type index:

[GitHub] [airflow] potiuk commented on a change in pull request #18747: Remove AIRFLOW_GID from Docker images

2021-10-05 Thread GitBox
potiuk commented on a change in pull request #18747: URL: https://github.com/apache/airflow/pull/18747#discussion_r722683344 ## File path: Dockerfile ## @@ -434,9 +431,9 @@ RUN chmod a+x /scripts/docker/install_mysql.sh && \ /scripts/docker/install_mysql.sh prod && \

[airflow] branch main updated (2c2bbb5 -> 958860f)

2021-10-05 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git. from 2c2bbb5 Fix "Test" connection button when app not mounted at `/` (#18750) add 958860f Remove AIRFLOW_GID from

[GitHub] [airflow] potiuk merged pull request #18747: Remove AIRFLOW_GID from Docker images

2021-10-05 Thread GitBox
potiuk merged pull request #18747: URL: https://github.com/apache/airflow/pull/18747 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] potiuk closed issue #18709: Default Value for AIRFLOW_GID. 0 vs. 50000

2021-10-05 Thread GitBox
potiuk closed issue #18709: URL: https://github.com/apache/airflow/issues/18709 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] potiuk commented on a change in pull request #18747: Remove AIRFLOW_GID from Docker images

2021-10-05 Thread GitBox
potiuk commented on a change in pull request #18747: URL: https://github.com/apache/airflow/pull/18747#discussion_r722675707 ## File path: Dockerfile ## @@ -434,9 +431,9 @@ RUN chmod a+x /scripts/docker/install_mysql.sh && \ /scripts/docker/install_mysql.sh prod && \

[GitHub] [airflow] jedcunningham commented on a change in pull request #18249: Add support for securityContext per deployment

2021-10-05 Thread GitBox
jedcunningham commented on a change in pull request #18249: URL: https://github.com/apache/airflow/pull/18249#discussion_r722672775 ## File path: chart/templates/_helpers.yaml ## @@ -610,3 +610,62 @@ Create the name of the cleanup service account to use {{- end -}} {{-

[GitHub] [airflow] zachliu commented on a change in pull request #18733: ECSOperator: airflow exception on edge case when cloudwatch log stream is not found

2021-10-05 Thread GitBox
zachliu commented on a change in pull request #18733: URL: https://github.com/apache/airflow/pull/18733#discussion_r722668059 ## File path: airflow/providers/amazon/aws/operators/ecs.py ## @@ -452,6 +460,11 @@ def _check_success_task(self) -> None: if

[GitHub] [airflow] eladkal commented on a change in pull request #18755: Support mysql to s3 in parquet format

2021-10-05 Thread GitBox
eladkal commented on a change in pull request #18755: URL: https://github.com/apache/airflow/pull/18755#discussion_r722651040 ## File path: airflow/providers/amazon/aws/transfers/mysql_to_s3.py ## @@ -60,6 +60,10 @@ class MySQLToS3Operator(BaseOperator): :type index: str

[GitHub] [airflow] eladkal commented on a change in pull request #18755: Support mysql to s3 in parquet format

2021-10-05 Thread GitBox
eladkal commented on a change in pull request #18755: URL: https://github.com/apache/airflow/pull/18755#discussion_r722651040 ## File path: airflow/providers/amazon/aws/transfers/mysql_to_s3.py ## @@ -60,6 +60,10 @@ class MySQLToS3Operator(BaseOperator): :type index: str

[GitHub] [airflow] codenamestif commented on a change in pull request #18733: ECSOperator: airflow exception on edge case when cloudwatch log stream is not found

2021-10-05 Thread GitBox
codenamestif commented on a change in pull request #18733: URL: https://github.com/apache/airflow/pull/18733#discussion_r722661411 ## File path: airflow/providers/amazon/aws/operators/ecs.py ## @@ -452,6 +460,11 @@ def _check_success_task(self) -> None: if

[GitHub] [airflow] zachliu commented on a change in pull request #18733: ECSOperator: airflow exception on edge case when cloudwatch log stream is not found

2021-10-05 Thread GitBox
zachliu commented on a change in pull request #18733: URL: https://github.com/apache/airflow/pull/18733#discussion_r722639966 ## File path: airflow/providers/amazon/aws/operators/ecs.py ## @@ -452,6 +460,11 @@ def _check_success_task(self) -> None: if

[airflow-site] 01/01: Add Influx DB to list of providers

2021-10-05 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch add-influxdb-to-providers-list in repository https://gitbox.apache.org/repos/asf/airflow-site.git commit 8625af29432c505c1aebdf32a918432b759b701f Author: Jarek Potiuk AuthorDate: Tue Oct 5

[airflow-site] branch add-influxdb-to-providers-list created (now 8625af2)

2021-10-05 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch add-influxdb-to-providers-list in repository https://gitbox.apache.org/repos/asf/airflow-site.git. at 8625af2 Add Influx DB to list of providers This branch includes the following new

[GitHub] [airflow] zachliu commented on a change in pull request #18733: ECSOperator: airflow exception on edge case when cloudwatch log stream is not found

2021-10-05 Thread GitBox
zachliu commented on a change in pull request #18733: URL: https://github.com/apache/airflow/pull/18733#discussion_r722621084 ## File path: airflow/providers/amazon/aws/operators/ecs.py ## @@ -452,6 +460,11 @@ def _check_success_task(self) -> None: if

[GitHub] [airflow] codenamestif commented on pull request #18733: ECSOperator: airflow exception on edge case when cloudwatch log stream is not found

2021-10-05 Thread GitBox
codenamestif commented on pull request #18733: URL: https://github.com/apache/airflow/pull/18733#issuecomment-934742891 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [airflow] zachliu commented on pull request #18733: ECSOperator: airflow exception on edge case when cloudwatch log stream is not found

2021-10-05 Thread GitBox
zachliu commented on pull request #18733: URL: https://github.com/apache/airflow/pull/18733#issuecomment-934735657 > > > @zachliu i'm double-checking the issue #15000. So would it be better to check `stopCode: TaskFailedToStart` rather than existence of the log stream of the task in order

[GitHub] [airflow] boring-cyborg[bot] commented on issue #18756: Kubernete executor: fetch log failed, no retries on task when task pod is DELETED while running

2021-10-05 Thread GitBox
boring-cyborg[bot] commented on issue #18756: URL: https://github.com/apache/airflow/issues/18756#issuecomment-934706696 Thanks for opening your first issue here! Be sure to follow the issue template! -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [airflow] enniomorricone123 opened a new issue #18756: Kubernete executor: fetch log failed, no retries on task when task pod is DELETED while running

2021-10-05 Thread GitBox
enniomorricone123 opened a new issue #18756: URL: https://github.com/apache/airflow/issues/18756 ### Apache Airflow version 2.1.3 ### Operating System linux ### Versions of Apache Airflow Providers _No response_ ### Deployment Official Apache

[GitHub] [airflow] nikochiko edited a comment on issue #18664: [Oracle] Oracle Hook - make it possible to define a schema in the connection parameters

2021-10-05 Thread GitBox
nikochiko edited a comment on issue #18664: URL: https://github.com/apache/airflow/issues/18664#issuecomment-934694084 @mehmax i don't know what a "schema" is in oracle terminology, but in airflow's case this parameter is used to refer to the `conntype://login:password@host:port/` schema

[GitHub] [airflow] nikochiko edited a comment on issue #18664: [Oracle] Oracle Hook - make it possible to define a schema in the connection parameters

2021-10-05 Thread GitBox
nikochiko edited a comment on issue #18664: URL: https://github.com/apache/airflow/issues/18664#issuecomment-934694084 @mehmax i don't know what a "schema" is in oracle terminology, but in airflow's case this parameter is used to refer to the `conntype://login:password@host:port/` schema

[GitHub] [airflow] nikochiko edited a comment on issue #18664: [Oracle] Oracle Hook - make it possible to define a schema in the connection parameters

2021-10-05 Thread GitBox
nikochiko edited a comment on issue #18664: URL: https://github.com/apache/airflow/issues/18664#issuecomment-934694084 @mehmax i don't know what a "schema" is in oracle terminology, but in airflow's case this parameter is used to refer to the `conntype://login:password@host:port/` schema

  1   2   3   4   >