This is an automated email from the ASF dual-hosted git repository.
ash pushed a change to branch v2-0-stable
in repository https://gitbox.apache.org/repos/asf/airflow.git.
from 94ba200 Bump version to 2.0.0b3 (#12462)
add a3dfd04 Webserver: Further Sanitize values passed to origin param
(#12459)
add 4b59ce8 Fix case of GitHub in comment (#12474)
add 3241719 Fix setup.py to install the right provider for mysql (#12476)
add 5ffb7f4 Fix failing test on Py3.8 (#12481)
add 13128f4 Fix Python docstring parameter (#12483)
add bc01907 Improve UI file naming/patterns (#12486)
add 9e3b2c5 GCP Secrets Optional Lookup (#12360)
add 8b95e51 Improve www.security.get_accessible_dags() and webserver
performance (#12458)
add fedf633 Remove unused/uncompiled JS file (#12490)
add 93d64e5 Update tag color to be neutral (and match DAGs index view)
(#12493)
add 9e089ab Fix Kube tests (#12479)
add de9d2fa ensure Moment date is valid before attempting to render it
(#12492)
add 502e883 Make kubernetes requirement optional for Example DAGs (#12494)
add e93b7e3 Improvements for transfer operators references (#12482)
add e9cfa39 Turn off foreign keys before altering table to prevent sqlite
issue. (#12487)
add 4428235 Fixes taskInstances API endpoint when start_date, end_date or
state are None(null) (#12453)
add 7d55d45 Reorder Migrations to make it 1.10.13 compatible (#12496)
add c3cf695 Unquarantine test_cli_webserver_background (#12501)
add c34ef85 Separate out documentation building per provider (#12444)
add 20843ff Add missing file_token field to get dag details API endpoint
(#12463)
add 36a9b0f Fix the default value for VaultBackend's config_path (#12518)
add 4495685 Temporarily allow force-push on v1-10-stable (#12524)
add fd62f60 Add Energy Solutions to INTHEWILD.md (#12523)
add f0b9aae Enable v1-10-stable branch protection (#12525)
add 234d689 Fix S3ToSnowflakeOperator docstring (#12504)
add 287ed49 Fix git archive command in Release Management guide (#12526)
add 3188b13 Fix wait-for-migrations command in helm chart (#12522)
add be8f1ac Fix build on RTD (#12529)
add 370e7d0 Fix Python Docstring parameters (#12513)
add 37548f0 Fixes unneeded docker-context-files added in CI (#12534)
add 397d912 Return nonzero exit codes on pool import errors. (#12095)
add 597717d update broken link to cli-and-env-variables-ref (#12540)
add 9eb92e7 Support installing providers with no dependencies via extras
(#12497)
add 0d796e7 Fix line breaks in CeleryKubernetesExecutor docs (#12538)
add f2569de Add example DAGs to provider docs (#12528)
add ef4af21 Move providers docs to separate package + Spell-check in a
common job with docs-build (#12527)
add 919e1d8 Fix sed command on MacOS (#12549)
add de15aa3 Deprecate Read the Docs (#12541)
add ff990f2 Add capability to specify gunicorn access log format (#10261)
add ed09915 [AIRFLOW-5115] Bugfix for S3KeySensor failing to accept
template_fields (#12389)
add 8e1a6e1 Fix build on RTD (#12551)
add c133df8 Troubleshooting moved to the installation page (#12533)
add c02a3f5 Spark-on-k8s sensor logs - properly pass defined namespace to
pod log call (#11199)
add ef0e34d Fix use of `a` vs `an` (#12542)
add dd3a8df Move doc around Manual Trigger Visual diff to Tree View page
(#12565)
add 753f53f Housekeeping for www/security.py (#12516)
add 47bc530 Add MTsolutions to companies using Airflow (#12566)
add c33d86f Quarantine test_cli_webserver_background (#12570)
add 2846e03 add Vestiaire Collective to INTHEWILD.md (#12572)
add f6ba8b5 Doc Fix around Secret/Connection/Variable (#12571)
add 01ff088 Fix Dag Serialization crash caused by preset DagContext
(#12530)
add 6d0dcd2 Use html urls instead of onclick for dags view links. (#12539)
add 3fa51f9 Add check for duplicates in provider.yaml files (#12578)
add 74ed92b Drop random.choice() in BaseHook.get_connection() (#12573)
add b57b932 Improve code quality of ExternalTaskSensor (#12574)
add 936566c limit table of content at the main Airflow doc page (#12561)
add a66d29f Dev documentation uses sphinx-airflow-theme (#12582)
add ce91991 Hide ToC from the Apache Airflow doc main page (#12589)
add 91af0dd Add link to docs index to table of contents (#12594)
add 6caf260 Don't set child tasks to schedulable in test runs (#12595)
add 9f28e41 Allow webserver to read pod logs directly (#12598)
add 58e21ed Adds missing licence headers (#12593)
add 663259d Fix AWS DataSync tests failing (#11020)
add c6467ba Update logging & doc for LocalFilesystem Secrets Backend
(#12597)
add 08251c1 Remove foreign key constraint on SerializedDagModel's
dag_runs field (#12586)
add c457c97 Use AIRFLOW_CONSTRAINTS_LOCATION when passed during docker
build (#12604)
add 4861344 Rename `[scheduler] max_threads` to `[scheduler]
parsing_processes` (#12605)
add 950d80b Revert "Hide ToC from the Apache Airflow doc main page
(#12589)" (#12607)
add cdaaff1 Fix Connection.description migration for MySQL8 (#12596)
add 6d6ca14 Fixes inconsistent behaviour of utf8mb4 encoding on Mysql
5.7/8 (#12614)
add 333c871 Fixes tests that was not compatible with MySQL8 (#12615)
add 0f10207 Add docs about tagging and pushing constraints file (#12625)
add c5700a5 Sync Airflow 1.10.13 Updating.md with Master (#12624)
add 324bc6f Make AzureKeyVaultBackend backwards-compatible (#12626)
add 4f4714f Fix session_lifetime_minutes config docs (#12628)
add 54adda5 Actually run against the version of the DB we select in the
matrix. (#12591)
add 8f29c6d fix db migration downgrade actions (#12608)
add eacf40d Ensure that the `prohibit_commit` guard only applies to _one_
session. (#12575)
add 8b9d52f Adds possibility of forcing upgrade constraint by setting a
label (#12635)
add f16fa09 Clean-up airflow/kubernetes/kube_config.py (#12627)
add 3a8c344 Add Changelog for Airflow 1.10.13 (#12623)
add cf718db Improved breeze messages for initialize-local-virtualenv and
static-check --help (#12640)
add b769209 Add Corsearch to in the wild (#12641)
add 6e9c110 Housekeeping: Remove 'dirty_ids' in www/views.py (#12645)
add c084393 Allows mounting local sources for github run-id images
(#12650)
add e1ebfa6 Add DataflowJobMessagesSensor and
DataflowAutoscalingEventsSensor (#12249)
add 5fafd98 Replace foreign key constraints with foreign annotation
(#12603)
No new revisions were added by this update.
Summary of changes:
.github/workflows/build-images-workflow-run.yml | 18 +-
.github/workflows/ci.yml | 49 +-
.github/workflows/codeql-analysis.yml | 4 +-
.../workflows/label_when_reviewed_workflow_run.yml | 4 +-
.gitignore | 2 +
.pre-commit-config.yaml | 4 +-
.readthedocs.yml | 2 +
BREEZE.rst | 4 +
CHANGELOG.txt | 134 +++-
CI.rst | 15 +-
CONTRIBUTING.rst | 64 +-
Dockerfile | 3 +-
Dockerfile.ci | 11 +-
IMAGES.rst | 11 +-
INSTALL | 15 +-
INTHEWILD.md | 4 +
README.md | 7 +-
UPDATING.md | 70 +-
airflow/api_connexion/exceptions.py | 5 +-
airflow/api_connexion/openapi/v1.yaml | 9 +-
airflow/api_connexion/schemas/dag_schema.py | 9 +
airflow/cli/cli_parser.py | 10 +-
airflow/cli/commands/pool_command.py | 40 +-
airflow/cli/commands/webserver_command.py | 6 +
airflow/config_templates/config.yml | 19 +-
airflow/config_templates/default_airflow.cfg | 13 +-
airflow/config_templates/default_test.cfg | 2 +-
airflow/configuration.py | 1 +
.../secrets/azure_key_vault.py} | 28 +-
.../example_kubernetes_executor_config.py | 253 +++---
airflow/hooks/base_hook.py | 7 +-
airflow/jobs/base_job.py | 8 +-
airflow/kubernetes/kube_config.py | 16 +-
.../versions/3c20cacc0044_add_dagrun_run_type.py | 2 +-
...73d9401f_add_description_field_to_connection.py | 10 +-
...7aae_fix_description_field_in_connection_to_.py | 78 ++
.../849da589634d_prefix_dag_permissions.py | 4 +-
.../versions/92c57b58940d_add_fab_tables.py | 4 +-
...606e2_add_scheduling_decision_to_dagrun_and_.py | 17 +
.../bbf4a7ad0465_remove_id_column_from_xcom.py | 1 -
.../bef4f3d11e8b_drop_kuberesourceversion_and_.py | 2 +-
.../e38be357a868_update_schema_for_smart_sensor.py | 4 +-
airflow/models/baseoperator.py | 8 +
airflow/models/connection.py | 4 +-
airflow/models/serialized_dag.py | 5 +-
airflow/models/taskinstance.py | 3 +-
airflow/provider.yaml.schema.json | 5 +
airflow/providers/amazon/aws/hooks/base_aws.py | 2 +-
airflow/providers/amazon/aws/hooks/datasync.py | 8 +-
.../amazon/aws/log/cloudwatch_task_handler.py | 8 +-
airflow/providers/amazon/aws/operators/datasync.py | 14 +-
airflow/providers/amazon/aws/operators/ecs.py | 2 +-
airflow/providers/amazon/aws/operators/glacier.py | 2 +-
airflow/providers/amazon/aws/sensors/glacier.py | 4 +
airflow/providers/amazon/aws/sensors/s3_key.py | 31 +-
.../amazon/aws/transfers/glacier_to_gcs.py | 2 +-
.../amazon/aws/transfers/imap_attachment_to_s3.py | 2 +-
.../amazon/aws/transfers/s3_to_redshift.py | 2 +-
airflow/providers/amazon/provider.yaml | 4 +-
airflow/providers/apache/cassandra/provider.yaml | 1 +
.../providers/apache/cassandra/sensors/record.py | 2 +-
.../providers/apache/cassandra/sensors/table.py | 2 +-
airflow/providers/apache/druid/provider.yaml | 1 +
airflow/providers/apache/hdfs/hooks/webhdfs.py | 2 +-
airflow/providers/apache/hdfs/provider.yaml | 1 +
airflow/providers/apache/hive/provider.yaml | 1 +
airflow/providers/apache/kylin/provider.yaml | 1 +
airflow/providers/apache/livy/provider.yaml | 1 +
airflow/providers/apache/pig/provider.yaml | 1 +
airflow/providers/apache/pinot/provider.yaml | 1 +
.../providers/apache/spark/operators/spark_jdbc.py | 2 +-
.../providers/apache/spark/operators/spark_sql.py | 2 +-
.../apache/spark/operators/spark_submit.py | 2 +-
airflow/providers/apache/spark/provider.yaml | 1 +
airflow/providers/apache/sqoop/provider.yaml | 1 +
airflow/providers/celery/provider.yaml | 1 +
airflow/providers/cloudant/provider.yaml | 1 +
.../backcompat/backwards_compat_converters.py | 5 -
.../providers/cncf/kubernetes/hooks/kubernetes.py | 2 +-
.../cncf/kubernetes/operators/kubernetes_pod.py | 11 +-
airflow/providers/cncf/kubernetes/provider.yaml | 1 +
.../cncf/kubernetes/sensors/spark_kubernetes.py | 15 +-
.../providers/databricks/operators/databricks.py | 4 +
airflow/providers/databricks/provider.yaml | 1 +
airflow/providers/datadog/provider.yaml | 1 +
.../dingding/example_dags/example_dingding.py | 6 +-
airflow/providers/dingding/provider.yaml | 1 +
airflow/providers/discord/provider.yaml | 1 +
airflow/providers/docker/provider.yaml | 1 +
airflow/providers/elasticsearch/provider.yaml | 1 +
airflow/providers/exasol/provider.yaml | 1 +
airflow/providers/facebook/provider.yaml | 1 +
airflow/providers/ftp/provider.yaml | 1 +
airflow/providers/google/__init__.py | 5 +-
.../google/cloud/example_dags/example_dataflow.py | 37 +-
.../cloud/hooks/cloud_storage_transfer_service.py | 2 +-
airflow/providers/google/cloud/hooks/dataflow.py | 118 ++-
.../providers/google/cloud/operators/cloud_sql.py | 2 +-
.../google/cloud/secrets/secret_manager.py | 23 +-
airflow/providers/google/cloud/sensors/dataflow.py | 198 +++++
.../google/cloud/utils/field_validator.py | 2 +-
.../google/cloud/utils/mlengine_operator_utils.py | 17 +-
.../cloud/utils/mlengine_prediction_summary.py | 4 +-
airflow/providers/google/provider.yaml | 112 ++-
airflow/providers/grpc/provider.yaml | 1 +
airflow/providers/hashicorp/provider.yaml | 1 +
airflow/providers/hashicorp/secrets/vault.py | 2 +-
airflow/providers/http/hooks/http.py | 2 +-
airflow/providers/http/operators/http.py | 2 +-
airflow/providers/http/provider.yaml | 1 +
airflow/providers/http/sensors/http.py | 2 +-
airflow/providers/imap/provider.yaml | 1 +
airflow/providers/jdbc/operators/jdbc.py | 4 +
airflow/providers/jdbc/provider.yaml | 1 +
airflow/providers/jenkins/provider.yaml | 1 +
airflow/providers/jira/provider.yaml | 1 +
airflow/providers/microsoft/azure/provider.yaml | 1 +
.../microsoft/azure/transfers/azure_blob_to_gcs.py | 2 +-
.../microsoft/azure/transfers/local_to_adls.py | 2 +-
airflow/providers/microsoft/mssql/provider.yaml | 1 +
airflow/providers/microsoft/winrm/provider.yaml | 1 +
airflow/providers/mongo/provider.yaml | 1 +
airflow/providers/mysql/operators/mysql.py | 4 +
airflow/providers/mysql/provider.yaml | 1 +
airflow/providers/odbc/hooks/odbc.py | 2 +-
airflow/providers/odbc/provider.yaml | 1 +
airflow/providers/openfaas/provider.yaml | 1 +
airflow/providers/opsgenie/provider.yaml | 1 +
airflow/providers/oracle/provider.yaml | 1 +
airflow/providers/pagerduty/provider.yaml | 1 +
airflow/providers/papermill/provider.yaml | 1 +
airflow/providers/plexus/provider.yaml | 1 +
airflow/providers/postgres/provider.yaml | 1 +
airflow/providers/presto/provider.yaml | 1 +
airflow/providers/qubole/provider.yaml | 1 +
airflow/providers/redis/provider.yaml | 1 +
airflow/providers/salesforce/provider.yaml | 1 +
airflow/providers/samba/provider.yaml | 1 +
airflow/providers/segment/provider.yaml | 1 +
airflow/providers/sendgrid/provider.yaml | 1 +
airflow/providers/sendgrid/utils/emailer.py | 2 +-
airflow/providers/sftp/provider.yaml | 1 +
airflow/providers/singularity/provider.yaml | 1 +
airflow/providers/slack/provider.yaml | 1 +
airflow/providers/snowflake/operators/snowflake.py | 6 +-
airflow/providers/snowflake/provider.yaml | 3 +-
.../snowflake/transfers/s3_to_snowflake.py | 8 +-
.../snowflake/transfers/snowflake_to_slack.py | 4 +
airflow/providers/sqlite/provider.yaml | 1 +
airflow/providers/ssh/provider.yaml | 1 +
airflow/providers/vertica/provider.yaml | 1 +
.../yandex/operators/yandexcloud_dataproc.py | 4 -
airflow/providers/yandex/provider.yaml | 7 +-
airflow/providers/zendesk/provider.yaml | 1 +
airflow/secrets/base_secrets.py | 5 +-
airflow/secrets/environment_variables.py | 3 +-
airflow/secrets/local_filesystem.py | 10 +-
airflow/secrets/metastore.py | 3 +-
airflow/sensors/external_task_sensor.py | 81 +-
airflow/utils/dag_processing.py | 4 +-
airflow/utils/docs.py | 4 +-
airflow/utils/sqlalchemy.py | 6 +-
airflow/www/extensions/init_appbuilder.py | 2 +-
airflow/www/security.py | 26 +-
airflow/www/static/js/graph.js | 63 --
airflow/www/static/js/{base.js => main.js} | 0
airflow/www/static/js/task-instances.js | 12 +-
airflow/www/templates/airflow/dag_details.html | 2 +-
airflow/www/templates/airflow/dags.html | 28 +-
airflow/www/templates/airflow/graph.html | 1 -
.../templates/airflow/{master.html => main.html} | 4 +-
airflow/www/views.py | 38 +-
airflow/www/webpack.config.js | 3 +-
breeze | 7 +-
chart/templates/_helpers.yaml | 3 +-
chart/templates/rbac/pod-log-reader-role.yaml | 56 ++
.../templates/rbac/pod-log-reader-rolebinding.yaml | 53 ++
chart/tests/test_basic_helm_chart.py | 4 +-
chart/values.schema.json | 4 +
chart/values.yaml | 1 +
dev/README.md | 2 +-
dev/README_RELEASE_AIRFLOW.md | 14 +-
docs/README.rst | 126 +++
.../index.rst | 18 +-
.../index.rst | 18 +-
.../index.rst | 12 +-
.../index.rst | 14 +-
.../index.rst | 19 +-
.../index.rst | 19 +-
.../index.rst | 18 +-
.../index.rst | 18 +-
.../index.rst | 12 +-
.../index.rst | 18 +-
.../index.rst | 12 +-
.../index.rst | 16 +-
.../index.rst | 16 +-
.../index.rst | 19 +-
.../index.rst | 19 +-
.../index.rst | 16 +-
.../index.rst | 19 +-
.../index.rst | 16 +-
.../index.rst | 19 +-
.../index.rst | 16 +-
.../index.rst | 16 +-
.../index.rst | 16 +-
.../index.rst | 16 +-
.../api-auth-backend/google-openid.rst | 69 ++
.../configurations-ref.rst | 71 ++
.../connections}/gcp.rst | 3 +-
.../connections}/gcp_sql.rst | 0
.../connections}/gcp_ssh.rst | 2 +-
.../connections}/index.rst | 7 +-
.../example-dags.rst} | 21 +-
.../index.rst | 27 +-
.../logging/gcs.rst | 57 ++
.../logging}/index.rst | 7 +-
.../logging/stackdriver.rst | 63 ++
.../operators}/_partials/prerequisite_tasks.rst | 4 +-
.../operators}/ads.rst | 6 +-
.../operators}/cloud/automl.rst | 28 +-
.../operators}/cloud/bigquery.rst | 44 +-
.../operators}/cloud/bigquery_dts.rst | 12 +-
.../operators}/cloud/bigtable.rst | 16 +-
.../operators}/cloud/cloud_build.rst | 18 +-
.../operators}/cloud/cloud_memorystore.rst | 28 +-
.../cloud/cloud_memorystore_memcached.rst | 16 +-
.../operators}/cloud/cloud_sql.rst | 64 +-
.../cloud/cloud_storage_transfer_service.rst | 46 +-
.../operators}/cloud/compute.rst | 36 +-
.../operators}/cloud/compute_ssh.rst | 8 +-
.../operators/cloud/data_loss_prevention.rst} | 20 +-
.../operators}/cloud/datacatalog.rst | 76 +-
.../operators}/cloud/datafusion.rst | 22 +-
.../operators}/cloud/dataprep.rst | 10 +-
.../operators}/cloud/dataproc.rst | 28 +-
.../operators}/cloud/datastore.rst | 24 +-
.../operators}/cloud/functions.rst | 18 +-
.../operators}/cloud/gcs.rst | 14 +-
.../operators}/cloud/index.rst | 0
.../operators}/cloud/kubernetes_engine.rst | 12 +-
.../operators}/cloud/life_sciences.rst | 8 +-
.../operators}/cloud/mlengine.rst | 30 +-
.../operators}/cloud/natural_language.rst | 22 +-
.../operators}/cloud/pubsub.rst | 20 +-
.../operators}/cloud/spanner.rst | 28 +-
.../operators}/cloud/speech_to_text.rst | 10 +-
.../operators}/cloud/stackdriver.rst | 22 +-
.../operators}/cloud/text_to_speech.rst | 10 +-
.../operators}/cloud/translate.rst | 8 +-
.../operators}/cloud/translate_speech.rst | 8 +-
.../operators}/cloud/video_intelligence.rst | 26 +-
.../operators}/cloud/vision.rst | 160 ++--
.../operators}/firebase/firestore.rst | 4 +-
.../operators}/index.rst | 0
.../operators}/marketing_platform/analytics.rst | 8 +-
.../marketing_platform/campaign_manager.rst | 16 +-
.../marketing_platform/display_video.rst | 22 +-
.../operators}/marketing_platform/index.rst | 0
.../operators}/marketing_platform/search_ads.rst | 10 +-
.../operators}/suite/sheets.rst | 6 +-
.../operators}/transfer/azure_fileshare_to_gcs.rst | 2 +-
.../operators}/transfer/facebook_ads_to_gcs.rst | 4 +-
.../operators}/transfer/gcs_to_gcs.rst | 24 +-
.../operators}/transfer/gcs_to_gdrive.rst | 8 +-
.../operators}/transfer/gcs_to_local.rst | 4 +-
.../operators}/transfer/gcs_to_sftp.rst | 10 +-
.../operators}/transfer/gcs_to_sheets.rst | 4 +-
.../operators}/transfer/index.rst | 0
.../operators}/transfer/local_to_gcs.rst | 4 +-
.../operators}/transfer/mysql_to_gcs.rst | 2 +-
.../operators}/transfer/presto_to_gcs.rst | 10 +-
.../operators}/transfer/s3_to_gcs.rst | 0
.../operators}/transfer/salesforce_to_gcs.rst | 4 +-
.../operators}/transfer/sftp_to_gcs.rst | 10 +-
.../operators}/transfer/sheets_to_gcs.rst | 4 +-
.../google-cloud-secret-manager-backend.rst | 6 +-
.../index.rst | 16 +-
.../index.rst | 16 +-
.../index.rst | 19 +-
.../index.rst | 16 +-
.../index.rst | 19 +-
.../index.rst | 19 +-
.../index.rst | 16 +-
.../index.rst | 19 +-
.../index.rst | 16 +-
.../index.rst | 19 +-
.../index.rst | 16 +-
.../index.rst | 19 +-
.../index.rst | 16 +-
.../index.rst | 16 +-
.../index.rst | 16 +-
.../index.rst | 16 +-
.../index.rst | 16 +-
.../index.rst | 19 +-
.../index.rst | 19 +-
.../index.rst | 16 +-
.../index.rst | 16 +-
.../index.rst | 19 +-
.../index.rst | 16 +-
.../index.rst | 19 +-
.../index.rst | 16 +-
.../index.rst | 16 +-
.../index.rst | 16 +-
.../index.rst | 16 +-
.../index.rst | 19 +-
.../index.rst | 16 +-
.../index.rst | 19 +-
.../index.rst | 16 +-
.../index.rst | 16 +-
.../index.rst | 16 +-
.../index.rst | 19 +-
.../index.rst | 16 +-
.../index.rst} | 11 +
.../operators-and-hooks-ref/apache.rst} | 53 +-
.../operators-and-hooks-ref/aws.rst} | 54 +-
.../operators-and-hooks-ref/azure.rst} | 53 +-
.../operators-and-hooks-ref/google.rst | 83 ++
.../operators-and-hooks-ref}/index.rst | 10 +-
.../operators-and-hooks-ref/protocol.rst} | 24 +-
.../operators-and-hooks-ref/services.rst} | 24 +-
.../operators-and-hooks-ref/software.rst} | 24 +-
.../packages-ref.rst} | 2 +-
docs/autoapi_templates/index.rst | 365 ---------
docs/build_docs.py | 889 +++++++--------------
docs/concepts.rst | 16 +-
docs/conf.py | 397 +++++----
docs/dag-run.rst | 2 +-
docs/executor/celery_kubernetes.rst | 2 +-
docs/extra-packages-ref.rst | 36 +-
docs/exts/airflow_intersphinx.py | 198 +++++
docs/exts/{ => docs_build}/__init__.py | 0
docs/exts/docs_build/code_utils.py | 62 ++
docs/exts/docs_build/dev_index_generator.py | 80 ++
.../exts/docs_build/dev_index_template.html.jinja2 | 54 ++
docs/exts/docs_build/errors.py | 104 +++
docs/exts/docs_build/lint_checks.py | 228 ++++++
docs/exts/docs_build/spelling_checks.py | 172 ++++
.../operators_and_hooks_ref-transfers.rst.jinja2 | 2 +
docs/exts/operators_and_hooks_ref.py | 26 +-
docs/exts/provider_yaml_utils.py | 3 +-
docs/exts/redirects.py | 3 -
docs/exts/sphinx_script_update.py | 5 -
docs/faq.rst | 33 +-
docs/howto/connection/azure.rst | 2 +-
docs/howto/connection/index.rst | 2 +
docs/howto/initialize-database.rst | 4 +-
docs/howto/operator/amazon/aws/emr.rst | 2 +-
docs/howto/operator/index.rst | 1 -
docs/howto/operator/kubernetes.rst | 2 +-
.../microsoft/transfer/blob_storage_to_gcs.rst | 2 +-
docs/howto/variable.rst | 2 +-
docs/index.rst | 5 +-
docs/installation.rst | 37 +-
docs/{start_doc_server.sh => list-roles.sh} | 7 +-
docs/logging-monitoring/logging-tasks.rst | 89 +--
docs/operators-and-hooks-ref.rst | 263 +-----
docs/rest-api-ref.rst | 4 +-
docs/rtd-deprecation/404.html | 27 +
.../provider.yaml => docs/rtd-deprecation/conf.py | 8 +-
.../google/transfer => rtd-deprecation}/index.rst | 13 +-
docs/security/api.rst | 53 --
.../aws-secrets-manaager-backend.rst | 15 +
.../aws-ssm-parameter-store-secrets-backend.rst | 16 +
.../azure-key-vault-secrets-backend.rst | 15 +
.../google-cloud-secret-manager-backend.rst | 16 +
.../hashicorp-vault-secrets-backend.rst | 15 +
.../local-filesystem-secrets-backend.rst | 2 +-
docs/spelling_wordlist.txt | 9 +
docs/start_doc_server.sh | 2 +-
docs/templates/layout.html | 37 -
docs/tutorial.rst | 2 +-
docs/ui.rst | 9 +
kubernetes_tests/test_kubernetes_pod_operator.py | 1 -
.../test_kubernetes_pod_operator_backcompat.py | 1 -
scripts/ci/docker-compose/backend-mysql.yml | 2 +
scripts/ci/dockerfiles/bats/Dockerfile | 17 +
scripts/ci/dockerfiles/stress/Dockerfile | 17 +
scripts/ci/libraries/_build_images.sh | 35 +-
scripts/ci/libraries/_initialization.sh | 4 +-
scripts/ci/openapi/client_codegen_diff.sh | 2 +-
.../pre_commit_check_provider_yaml_files.py | 62 ++
.../pre_commit_check_setup_extra_packages_ref.py | 5 +-
scripts/ci/pre_commit/pre_commit_sync_version.sh | 3 +-
scripts/ci/selective_ci_checks.sh | 56 +-
scripts/in_container/airflow_ci.cfg | 2 +-
scripts/in_container/run_ci_tests.sh | 8 +-
scripts/in_container/run_docs_build.sh | 7 +-
setup.cfg | 1 +
setup.py | 41 +-
tests/api_connexion/endpoints/test_dag_endpoint.py | 39 +-
.../endpoints/test_task_instance_endpoint.py | 40 +
tests/api_connexion/schemas/test_dag_schema.py | 9 +
tests/cli/commands/test_pool_command.py | 19 +
tests/cli/commands/test_task_command.py | 5 +-
tests/cli/commands/test_webserver_command.py | 52 ++
tests/models/test_dagrun.py | 43 +-
tests/operators/test_generic_transfer.py | 4 +-
.../amazon/aws/hooks/test_batch_waiters.py | 1 +
.../amazon/aws/hooks/test_cloud_formation.py | 10 +-
tests/providers/amazon/aws/hooks/test_datasync.py | 21 +-
.../amazon/aws/log/test_cloudwatch_task_handler.py | 10 +-
.../amazon/aws/log/test_s3_task_handler.py | 4 +-
.../amazon/aws/operators/test_datasync.py | 63 +-
tests/providers/amazon/aws/sensors/test_s3_key.py | 47 +-
.../kubernetes/operators/test_kubernetes_pod.py | 60 +-
.../kubernetes/sensors/test_spark_kubernetes.py | 4 +-
tests/providers/dingding/hooks/test_dingding.py | 6 +-
.../providers/google/cloud/hooks/test_dataflow.py | 106 +++
.../google/cloud/secrets/test_secret_manager.py | 43 +
.../google/cloud/sensors/test_dataflow.py | 171 +++-
.../azure/secrets/test_azure_key_vault.py | 6 +-
tests/serialization/test_dag_serialization.py | 7 +-
tests/utils/test_dag_processing.py | 2 +-
tests/utils/test_docs.py | 14 +-
tests/utils/test_sqlalchemy.py | 38 +-
tests/www/test_security.py | 27 +
tests/www/test_views.py | 33 +-
417 files changed, 5745 insertions(+), 3794 deletions(-)
copy airflow/{providers/grpc/provider.yaml =>
contrib/secrets/azure_key_vault.py} (65%)
create mode 100644
airflow/migrations/versions/64a7d6477aae_fix_description_field_in_connection_to_.py
delete mode 100644 airflow/www/static/js/graph.js
rename airflow/www/static/js/{base.js => main.js} (100%)
rename airflow/www/templates/airflow/{master.html => main.html} (97%)
create mode 100644 chart/templates/rbac/pod-log-reader-role.yaml
create mode 100644 chart/templates/rbac/pod-log-reader-rolebinding.yaml
create mode 100644 docs/README.rst
copy docs/{howto/operator/google/marketing_platform =>
apache-airflow-providers-amazon}/index.rst (75%)
copy docs/{howto/operator/google/cloud =>
apache-airflow-providers-apache-cassandra}/index.rst (70%)
copy docs/{howto/operator/google/transfer =>
apache-airflow-providers-apache-druid}/index.rst (82%)
copy docs/{howto/operator/google =>
apache-airflow-providers-apache-hdfs}/index.rst (83%)
copy docs/{howto/operator/google/cloud =>
apache-airflow-providers-apache-hive}/index.rst (71%)
copy docs/{howto/operator/google/cloud =>
apache-airflow-providers-apache-kylin}/index.rst (70%)
copy docs/{howto/operator/google/cloud =>
apache-airflow-providers-apache-livy}/index.rst (71%)
copy docs/{howto/operator/google/cloud =>
apache-airflow-providers-apache-pig}/index.rst (71%)
copy docs/{howto/operator/google/transfer =>
apache-airflow-providers-apache-pinot}/index.rst (82%)
copy docs/{howto/operator/google/cloud =>
apache-airflow-providers-apache-spark}/index.rst (70%)
copy docs/{howto/operator/google/transfer =>
apache-airflow-providers-apache-sqoop}/index.rst (82%)
copy docs/{howto/operator/google => apache-airflow-providers-celery}/index.rst
(83%)
copy docs/{howto/operator/google =>
apache-airflow-providers-cloudant}/index.rst (83%)
copy docs/{howto/operator/google/cloud =>
apache-airflow-providers-cncf-kubernetes}/index.rst (70%)
copy docs/{howto/operator/google/cloud =>
apache-airflow-providers-databricks}/index.rst (71%)
copy docs/{howto/operator/google =>
apache-airflow-providers-datadog}/index.rst (83%)
copy docs/{howto/operator/google/cloud =>
apache-airflow-providers-dingding}/index.rst (71%)
copy docs/{howto/operator/google =>
apache-airflow-providers-discord}/index.rst (83%)
copy docs/{howto/operator/google/marketing_platform =>
apache-airflow-providers-docker}/index.rst (75%)
copy docs/{howto/operator/google =>
apache-airflow-providers-elasticsearch}/index.rst (81%)
copy docs/{howto/operator/google => apache-airflow-providers-exasol}/index.rst
(83%)
copy docs/{howto/operator/google =>
apache-airflow-providers-facebook}/index.rst (83%)
copy docs/{howto/operator/google => apache-airflow-providers-ftp}/index.rst
(84%)
create mode 100644
docs/apache-airflow-providers-google/api-auth-backend/google-openid.rst
create mode 100644 docs/apache-airflow-providers-google/configurations-ref.rst
rename docs/{howto/connection =>
apache-airflow-providers-google/connections}/gcp.rst (99%)
rename docs/{howto/connection =>
apache-airflow-providers-google/connections}/gcp_sql.rst (100%)
rename docs/{howto/connection =>
apache-airflow-providers-google/connections}/gcp_ssh.rst (97%)
copy docs/{howto/operator/google/transfer =>
apache-airflow-providers-google/connections}/index.rst (93%)
copy docs/{howto/operator/google/marketing_platform/index.rst =>
apache-airflow-providers-google/example-dags.rst} (52%)
copy docs/{howto/operator/google/marketing_platform =>
apache-airflow-providers-google}/index.rst (61%)
create mode 100644 docs/apache-airflow-providers-google/logging/gcs.rst
copy docs/{howto/operator/google/transfer =>
apache-airflow-providers-google/logging}/index.rst (93%)
create mode 100644 docs/apache-airflow-providers-google/logging/stackdriver.rst
rename docs/{howto/operator/google =>
apache-airflow-providers-google/operators}/_partials/prerequisite_tasks.rst
(90%)
rename docs/{howto/operator/google =>
apache-airflow-providers-google/operators}/ads.rst (91%)
rename docs/{howto/operator/google =>
apache-airflow-providers-google/operators}/cloud/automl.rst (83%)
rename docs/{howto/operator/google =>
apache-airflow-providers-google/operators}/cloud/bigquery.rst (84%)
rename docs/{howto/operator/google =>
apache-airflow-providers-google/operators}/cloud/bigquery_dts.rst (91%)
rename docs/{howto/operator/google =>
apache-airflow-providers-google/operators}/cloud/bigtable.rst (90%)
rename docs/{howto/operator/google =>
apache-airflow-providers-google/operators}/cloud/cloud_build.rst (84%)
rename docs/{howto/operator/google =>
apache-airflow-providers-google/operators}/cloud/cloud_memorystore.rst (88%)
rename docs/{howto/operator/google =>
apache-airflow-providers-google/operators}/cloud/cloud_memorystore_memcached.rst
(86%)
rename docs/{howto/operator/google =>
apache-airflow-providers-google/operators}/cloud/cloud_sql.rst (86%)
rename docs/{howto/operator/google =>
apache-airflow-providers-google/operators}/cloud/cloud_storage_transfer_service.rst
(81%)
rename docs/{howto/operator/google =>
apache-airflow-providers-google/operators}/cloud/compute.rst (84%)
rename docs/{howto/operator/google =>
apache-airflow-providers-google/operators}/cloud/compute_ssh.rst (89%)
rename docs/{howto/operator/google/cloud/dlp.rst =>
apache-airflow-providers-google/operators/cloud/data_loss_prevention.rst} (92%)
rename docs/{howto/operator/google =>
apache-airflow-providers-google/operators}/cloud/datacatalog.rst (86%)
rename docs/{howto/operator/google =>
apache-airflow-providers-google/operators}/cloud/datafusion.rst (90%)
rename docs/{howto/operator/google =>
apache-airflow-providers-google/operators}/cloud/dataprep.rst (89%)
rename docs/{howto/operator/google =>
apache-airflow-providers-google/operators}/cloud/dataproc.rst (84%)
rename docs/{howto/operator/google =>
apache-airflow-providers-google/operators}/cloud/datastore.rst (81%)
rename docs/{howto/operator/google =>
apache-airflow-providers-google/operators}/cloud/functions.rst (88%)
rename docs/{howto/operator/google =>
apache-airflow-providers-google/operators}/cloud/gcs.rst (88%)
copy docs/{howto/operator/google =>
apache-airflow-providers-google/operators}/cloud/index.rst (100%)
rename docs/{howto/operator/google =>
apache-airflow-providers-google/operators}/cloud/kubernetes_engine.rst (91%)
rename docs/{howto/operator/google =>
apache-airflow-providers-google/operators}/cloud/life_sciences.rst (88%)
rename docs/{howto/operator/google =>
apache-airflow-providers-google/operators}/cloud/mlengine.rst (87%)
rename docs/{howto/operator/google =>
apache-airflow-providers-google/operators}/cloud/natural_language.rst (87%)
rename docs/{howto/operator/google =>
apache-airflow-providers-google/operators}/cloud/pubsub.rst (85%)
rename docs/{howto/operator/google =>
apache-airflow-providers-google/operators}/cloud/spanner.rst (88%)
rename docs/{howto/operator/google =>
apache-airflow-providers-google/operators}/cloud/speech_to_text.rst (85%)
rename docs/{howto/operator/google =>
apache-airflow-providers-google/operators}/cloud/stackdriver.rst (89%)
rename docs/{howto/operator/google =>
apache-airflow-providers-google/operators}/cloud/text_to_speech.rst (85%)
rename docs/{howto/operator/google =>
apache-airflow-providers-google/operators}/cloud/translate.rst (87%)
rename docs/{howto/operator/google =>
apache-airflow-providers-google/operators}/cloud/translate_speech.rst (88%)
rename docs/{howto/operator/google =>
apache-airflow-providers-google/operators}/cloud/video_intelligence.rst (84%)
rename docs/{howto/operator/google =>
apache-airflow-providers-google/operators}/cloud/vision.rst (81%)
rename docs/{howto/operator/google =>
apache-airflow-providers-google/operators}/firebase/firestore.rst (93%)
copy docs/{howto/operator/google =>
apache-airflow-providers-google/operators}/index.rst (100%)
rename docs/{howto/operator/google =>
apache-airflow-providers-google/operators}/marketing_platform/analytics.rst
(89%)
rename docs/{howto/operator/google =>
apache-airflow-providers-google/operators}/marketing_platform/campaign_manager.rst
(89%)
rename docs/{howto/operator/google =>
apache-airflow-providers-google/operators}/marketing_platform/display_video.rst
(88%)
copy docs/{howto/operator/google =>
apache-airflow-providers-google/operators}/marketing_platform/index.rst (100%)
rename docs/{howto/operator/google =>
apache-airflow-providers-google/operators}/marketing_platform/search_ads.rst
(88%)
rename docs/{howto/operator/google =>
apache-airflow-providers-google/operators}/suite/sheets.rst (89%)
rename docs/{howto/operator/google =>
apache-airflow-providers-google/operators}/transfer/azure_fileshare_to_gcs.rst
(95%)
rename docs/{howto/operator/google =>
apache-airflow-providers-google/operators}/transfer/facebook_ads_to_gcs.rst
(90%)
rename docs/{howto/operator/google =>
apache-airflow-providers-google/operators}/transfer/gcs_to_gcs.rst (90%)
rename docs/{howto/operator/google =>
apache-airflow-providers-google/operators}/transfer/gcs_to_gdrive.rst (89%)
rename docs/{howto/operator/google =>
apache-airflow-providers-google/operators}/transfer/gcs_to_local.rst (91%)
rename docs/{howto/operator/google =>
apache-airflow-providers-google/operators}/transfer/gcs_to_sftp.rst (88%)
rename docs/{howto/operator/google =>
apache-airflow-providers-google/operators}/transfer/gcs_to_sheets.rst (92%)
copy docs/{howto/operator/google =>
apache-airflow-providers-google/operators}/transfer/index.rst (100%)
rename docs/{howto/operator/google =>
apache-airflow-providers-google/operators}/transfer/local_to_gcs.rst (92%)
rename docs/{howto/operator/google =>
apache-airflow-providers-google/operators}/transfer/mysql_to_gcs.rst (95%)
rename docs/{howto/operator/google =>
apache-airflow-providers-google/operators}/transfer/presto_to_gcs.rst (92%)
rename docs/{howto/operator/google =>
apache-airflow-providers-google/operators}/transfer/s3_to_gcs.rst (100%)
rename docs/{howto/operator/google =>
apache-airflow-providers-google/operators}/transfer/salesforce_to_gcs.rst (90%)
rename docs/{howto/operator/google =>
apache-airflow-providers-google/operators}/transfer/sftp_to_gcs.rst (89%)
rename docs/{howto/operator/google =>
apache-airflow-providers-google/operators}/transfer/sheets_to_gcs.rst (92%)
copy docs/{security/secrets/secrets-backend =>
apache-airflow-providers-google/secrets-backends}/google-cloud-secret-manager-backend.rst
(98%)
copy docs/{howto/operator/google => apache-airflow-providers-grpc}/index.rst
(84%)
copy docs/{howto/operator/google =>
apache-airflow-providers-hashicorp}/index.rst (82%)
copy docs/{howto/operator/google/cloud =>
apache-airflow-providers-http}/index.rst (72%)
copy docs/{howto/operator/google => apache-airflow-providers-imap}/index.rst
(84%)
copy docs/{howto/operator/google/cloud =>
apache-airflow-providers-jdbc}/index.rst (72%)
copy docs/{howto/operator/google/cloud =>
apache-airflow-providers-jenkins}/index.rst (72%)
copy docs/{howto/operator/google => apache-airflow-providers-jira}/index.rst
(84%)
copy docs/{howto/operator/google/cloud =>
apache-airflow-providers-microsoft-azure}/index.rst (70%)
copy docs/{howto/operator/google =>
apache-airflow-providers-microsoft-mssql}/index.rst (81%)
copy docs/{howto/operator/google/cloud =>
apache-airflow-providers-microsoft-winrm}/index.rst (70%)
copy docs/{howto/operator/google => apache-airflow-providers-mongo}/index.rst
(83%)
copy docs/{howto/operator/google/cloud =>
apache-airflow-providers-mysql}/index.rst (72%)
copy docs/{howto/operator/google => apache-airflow-providers-odbc}/index.rst
(84%)
copy docs/{howto/operator/google =>
apache-airflow-providers-openfaas}/index.rst (83%)
copy docs/{howto/operator/google =>
apache-airflow-providers-opsgenie}/index.rst (83%)
copy docs/{howto/operator/google => apache-airflow-providers-oracle}/index.rst
(83%)
copy docs/{howto/operator/google =>
apache-airflow-providers-pagerduty}/index.rst (82%)
copy docs/{howto/operator/google/cloud =>
apache-airflow-providers-papermill}/index.rst (71%)
copy docs/{howto/operator/google/marketing_platform =>
apache-airflow-providers-plexus}/index.rst (75%)
copy docs/{howto/operator/google =>
apache-airflow-providers-postgres}/index.rst (83%)
copy docs/{howto/operator/google => apache-airflow-providers-presto}/index.rst
(83%)
copy docs/{howto/operator/google/marketing_platform =>
apache-airflow-providers-qubole}/index.rst (75%)
copy docs/{howto/operator/google => apache-airflow-providers-redis}/index.rst
(83%)
copy docs/{howto/operator/google/cloud =>
apache-airflow-providers-salesforce}/index.rst (71%)
copy docs/{howto/operator/google => apache-airflow-providers-samba}/index.rst
(83%)
copy docs/{howto/operator/google =>
apache-airflow-providers-segment}/index.rst (83%)
copy docs/{howto/operator/google =>
apache-airflow-providers-sendgrid}/index.rst (83%)
copy docs/{howto/operator/google => apache-airflow-providers-sftp}/index.rst
(84%)
copy docs/{howto/operator/google/cloud =>
apache-airflow-providers-singularity}/index.rst (71%)
copy docs/{howto/operator/google => apache-airflow-providers-slack}/index.rst
(83%)
copy docs/{howto/operator/google/cloud =>
apache-airflow-providers-snowflake}/index.rst (71%)
copy docs/{howto/operator/google => apache-airflow-providers-sqlite}/index.rst
(83%)
copy docs/{howto/operator/google => apache-airflow-providers-ssh}/index.rst
(84%)
copy docs/{howto/operator/google =>
apache-airflow-providers-vertica}/index.rst (83%)
copy docs/{howto/operator/google/marketing_platform =>
apache-airflow-providers-yandex}/index.rst (75%)
rename docs/{howto/operator/google =>
apache-airflow-providers-zendesk}/index.rst (83%)
rename docs/{provider-packages.rst => apache-airflow-providers/index.rst} (98%)
copy docs/{howto/operator/index.rst =>
apache-airflow-providers/operators-and-hooks-ref/apache.rst} (56%)
copy docs/{howto/operator/index.rst =>
apache-airflow-providers/operators-and-hooks-ref/aws.rst} (57%)
copy docs/{howto/operator/index.rst =>
apache-airflow-providers/operators-and-hooks-ref/azure.rst} (55%)
create mode 100644
docs/apache-airflow-providers/operators-and-hooks-ref/google.rst
copy docs/{howto/operator/google/transfer =>
apache-airflow-providers/operators-and-hooks-ref}/index.rst (91%)
copy docs/{howto/operator/google/marketing_platform/index.rst =>
apache-airflow-providers/operators-and-hooks-ref/protocol.rst} (63%)
rename docs/{howto/operator/google/marketing_platform/index.rst =>
apache-airflow-providers/operators-and-hooks-ref/services.rst} (65%)
rename docs/{howto/operator/google/cloud/index.rst =>
apache-airflow-providers/operators-and-hooks-ref/software.rst} (66%)
rename docs/{provider-packages-ref.rst =>
apache-airflow-providers/packages-ref.rst} (95%)
create mode 100644 docs/exts/airflow_intersphinx.py
copy docs/exts/{ => docs_build}/__init__.py (100%)
create mode 100644 docs/exts/docs_build/code_utils.py
create mode 100644 docs/exts/docs_build/dev_index_generator.py
create mode 100644 docs/exts/docs_build/dev_index_template.html.jinja2
create mode 100644 docs/exts/docs_build/errors.py
create mode 100644 docs/exts/docs_build/lint_checks.py
create mode 100644 docs/exts/docs_build/spelling_checks.py
copy docs/{start_doc_server.sh => list-roles.sh} (90%)
mode change 100755 => 100644
create mode 100644 docs/rtd-deprecation/404.html
copy airflow/providers/sendgrid/provider.yaml => docs/rtd-deprecation/conf.py
(85%)
rename docs/{howto/operator/google/transfer => rtd-deprecation}/index.rst (74%)
delete mode 100644 docs/templates/layout.html
