[jira] [Created] (AIRFLOW-3559) Add missing options to DatadogHook
Josh Carp created AIRFLOW-3559: -- Summary: Add missing options to DatadogHook Key: AIRFLOW-3559 URL: https://issues.apache.org/jira/browse/AIRFLOW-3559 Project: Apache Airflow Issue Type: Improvement Reporter: Josh Carp Assignee: Josh Carp The DataDog hook is missing a few options for creating events and metrics. I'll add those options and backfill unit tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3540) Presence of ~/airflow/airflow.cfg shouldn't override config file settings
Josh Carp created AIRFLOW-3540: -- Summary: Presence of ~/airflow/airflow.cfg shouldn't override config file settings Key: AIRFLOW-3540 URL: https://issues.apache.org/jira/browse/AIRFLOW-3540 Project: Apache Airflow Issue Type: Improvement Reporter: Josh Carp Assignee: Josh Carp If `~/airflow/airflow.cfg` exists, airflow uses that file instead of `$AIRFLOW_HOME/airflow.cfg`. This behavior is confusing. If `$AIRFLOW_HOME` is set and `$AIRFLOW_CONFIG` is not, we should only check `$AIRFLOW_HOME/airflow.cfg`. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3531) Add gcs to gcs transfer operator
Josh Carp created AIRFLOW-3531: -- Summary: Add gcs to gcs transfer operator Key: AIRFLOW-3531 URL: https://issues.apache.org/jira/browse/AIRFLOW-3531 Project: Apache Airflow Issue Type: Improvement Reporter: Josh Carp Assignee: Josh Carp We already support copying from s3 to gcs with the gcp transfer service. It would be useful to support transfer from gcs to gcs as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3444) Expand templated fields in gcs transfer service operator
Josh Carp created AIRFLOW-3444: -- Summary: Expand templated fields in gcs transfer service operator Key: AIRFLOW-3444 URL: https://issues.apache.org/jira/browse/AIRFLOW-3444 Project: Apache Airflow Issue Type: Improvement Reporter: Josh Carp Assignee: Josh Carp The `S3ToGoogleCloudStorageTransferOperator` should support an explicit `description` parameter and allow that parameter to be templated. This will make it easier to set job descriptions, and we'll be able to drop the `job_kwargs` attribute, which doesn't have any other use. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3391) Upgrade pendulum to latest major version
Josh Carp created AIRFLOW-3391: -- Summary: Upgrade pendulum to latest major version Key: AIRFLOW-3391 URL: https://issues.apache.org/jira/browse/AIRFLOW-3391 Project: Apache Airflow Issue Type: Improvement Reporter: Josh Carp We're a major version behind on pendulum; we should upgrade to >=2.0. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3390) Add connections to api
Josh Carp created AIRFLOW-3390: -- Summary: Add connections to api Key: AIRFLOW-3390 URL: https://issues.apache.org/jira/browse/AIRFLOW-3390 Project: Apache Airflow Issue Type: Improvement Reporter: Josh Carp It would be useful to be able to read and write connections via the api. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3383) Simplify fernet key rotation
Josh Carp created AIRFLOW-3383: -- Summary: Simplify fernet key rotation Key: AIRFLOW-3383 URL: https://issues.apache.org/jira/browse/AIRFLOW-3383 Project: Apache Airflow Issue Type: Improvement Reporter: Josh Carp As far as I can tell, it's not straightforward to rotate the fernet key for encrypted passwords and extras. A user would have to generate a new key, restart airflow, and manually re-enter each value to be encrypted via the web interface. It should be possible to specify multiple fernet keys at once, and to easily re-encrypt values with a new key. The cryptography package provides a MultiFernet class with a rotate method that handles this use case, so I wrote up a patch that uses MultiFernet to support multiple keys and rotation via the command line. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3367) Test celery with redis broker
Josh Carp created AIRFLOW-3367: -- Summary: Test celery with redis broker Key: AIRFLOW-3367 URL: https://issues.apache.org/jira/browse/AIRFLOW-3367 Project: Apache Airflow Issue Type: Test Reporter: Josh Carp Current integration tests use celery with the rabbitmq broker, but not the redis broker. We should test with both brokers to avoid regressions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3346) Add gcp transfer service hook and operators
Josh Carp created AIRFLOW-3346: -- Summary: Add gcp transfer service hook and operators Key: AIRFLOW-3346 URL: https://issues.apache.org/jira/browse/AIRFLOW-3346 Project: Apache Airflow Issue Type: Bug Reporter: Josh Carp Add a hook and operator(s) to connect to gcp storage transfer service and transfer files from s3 to gcp (and gcp to gcp) without copying to local disk. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3307) Update insecure node dependencies
Josh Carp created AIRFLOW-3307: -- Summary: Update insecure node dependencies Key: AIRFLOW-3307 URL: https://issues.apache.org/jira/browse/AIRFLOW-3307 Project: Apache Airflow Issue Type: Bug Reporter: Josh Carp Assignee: Josh Carp `npm audit` shows some node dependencies that are out of date and potentially insecure. We should update them with `npm audit fix`. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3306) Disable unused flask-sqlalchemy modification tracking
Josh Carp created AIRFLOW-3306: -- Summary: Disable unused flask-sqlalchemy modification tracking Key: AIRFLOW-3306 URL: https://issues.apache.org/jira/browse/AIRFLOW-3306 Project: Apache Airflow Issue Type: Bug Reporter: Josh Carp Assignee: Josh Carp By default, flask-sqlalchemy tracks model changes for its event system, which adds some overhead. Since I don't think we're using the flask-sqlalchemy event system, we should be able to turn off modification tracking and improve performance. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3141) Fix 500 on duration view when dag doesn't exist
Josh Carp created AIRFLOW-3141: -- Summary: Fix 500 on duration view when dag doesn't exist Key: AIRFLOW-3141 URL: https://issues.apache.org/jira/browse/AIRFLOW-3141 Project: Apache Airflow Issue Type: Bug Reporter: Josh Carp Assignee: Josh Carp Loading the duration view for a dag that doesn't exist throws a 500. Based on the behavior of other dag views, this should redirect to the admin view and flash an error message instead. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3137) Make ProxyFix middleware optional
Josh Carp created AIRFLOW-3137: -- Summary: Make ProxyFix middleware optional Key: AIRFLOW-3137 URL: https://issues.apache.org/jira/browse/AIRFLOW-3137 Project: Apache Airflow Issue Type: Improvement Reporter: Josh Carp Assignee: Josh Carp The werkzeug ProxyFix middleware should only be used when the app is run behind a trusted proxy. We should enable ProxyFix via a configuration flag, like in superset. >From the werkzeug docs: "Do not use this middleware in non-proxy setups for >security reasons." -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3129) Improve test coverage
Josh Carp created AIRFLOW-3129: -- Summary: Improve test coverage Key: AIRFLOW-3129 URL: https://issues.apache.org/jira/browse/AIRFLOW-3129 Project: Apache Airflow Issue Type: Improvement Reporter: Josh Carp Overall test coverage is about 75%. It would be great to improve coverage. I'll start by backfilling some missing tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3103) Update flask-login
Josh Carp created AIRFLOW-3103: -- Summary: Update flask-login Key: AIRFLOW-3103 URL: https://issues.apache.org/jira/browse/AIRFLOW-3103 Project: Apache Airflow Issue Type: Improvement Components: authentication Reporter: Josh Carp Airflow uses a release of flask-login from 2014. Flask-login has fixed some bugs and added some features since then, so we should upgrade. Note: flask-appbuilder also pins to an old version of flask-login, so we'll have to update that library as well; PR submitted at https://github.com/dpgaspar/Flask-AppBuilder/pull/811. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3089) Google auth doesn't work under http
Josh Carp created AIRFLOW-3089: -- Summary: Google auth doesn't work under http Key: AIRFLOW-3089 URL: https://issues.apache.org/jira/browse/AIRFLOW-3089 Project: Apache Airflow Issue Type: Bug Components: authentication Affects Versions: 1.10.0 Reporter: Josh Carp Assignee: Josh Carp The google auth backend hard-codes the https scheme into its redirect urls and so doesn't work with http. This happens because the redirect url sets the `_scheme` parameter to `https`. I I see that this change was made to allow google auth to work when airflow is run behind a proxy, but I think this fix isn't correct. Instead, proxies should be configured to set `X-Forwarded-Proto` to `https`, and Flask will set the scheme correctly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3086) Add extras group in setup.py for google oauth
Josh Carp created AIRFLOW-3086: -- Summary: Add extras group in setup.py for google oauth Key: AIRFLOW-3086 URL: https://issues.apache.org/jira/browse/AIRFLOW-3086 Project: Apache Airflow Issue Type: Bug Reporter: Josh Carp Since the google auth backend requires Flask-OAuthlib, it would be helpful to add an extras group to setup.py for google auth that installs this dependency. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3074) Missing options in ECS operator
Josh Carp created AIRFLOW-3074: -- Summary: Missing options in ECS operator Key: AIRFLOW-3074 URL: https://issues.apache.org/jira/browse/AIRFLOW-3074 Project: Apache Airflow Issue Type: Bug Components: operators Affects Versions: 1.10.0 Reporter: Josh Carp The ECS operator exposes some options passed to the boto3 run_task method, but omits a few other useful options. For example, the networkConfiguration option, which is useful for setting security groups on a Fargate container, isn't exposed. The ECS operator should expose all run_task options that are relevant. I wrote up a quick patch for this issue; I'll amend the pull request to reference this report. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
