This is an automated email from the ASF dual-hosted git repository.
eladkal pushed a change to branch v2-6-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
from 27ec3fb419 Add description in a few more places about adding airflow
to pip install (#31448)
new f6cda8fb63 Mark `[secrets] backend_kwargs` as a sensitive config
(#31788)
new aeb4ef3679 Fix overriding `default_args` in nested task groups (#31608)
new 8f343975a1 Fix wrong update for nested group default args (#31776)
new cc3cd20dce Fix typing for POST user endpoint (#31767)
new 5b41ed8209 Fix failing get_safe_url tests for latest Python 3.8 and
3.9 (#31766)
new b9f531697a Few doc fixes - links, grammer and wording (#31719)
new b87c9b1424 Add description on security issue handling in Airflow
(#31160)
new 3563023554 Add severity rating explanation to our Security Policy
(#31702)
new e39ca99398 Add an example for dynamic task mapping with non-TaskFlow
operator (#29762)
new d73cc40435 Add fullname to owner on logging (#30185)
new fd0719ddf5 Fix DAG.access_control can't sync when clean access_control
(#30340)
new 99241948e8 Fix ExternalTaskSensor to work correctly with task groups
(#30742)
new bb86b468ae Add broad-strokes 'security model' (#30843)
new bd7cd41bfa Add Python 3.11 support (#27264)
new 3594da07e9 apply static checks
new e972144197 Save scheduler execution time during search for queued
dag_runs (#30699)
new 945c20e37d Optimize scheduler by skipping "non-scheduable" DAGs
(#30706)
new 1a8d05d7d9 Executor events are not always "exited" here (#30859)
new 71bcb669ae Include rediss to the list of supported URL schemes (#31028)
new 9682a125a4 Update callbacks documentation (errors and context) (#31116)
new f03324137f Validate connection IDs (#31140)
new 9d28825e98 Added spinner activity while the logs load (#31165)
new 78bf9dd6b0 Use only one line for tmp dir log (#31170)
new a880168494 FIPS environments: Mark uses of md5 as
"not-used-for-security" (#31171)
new 5a45bb600f Don't present an undefined execution date (#31196)
new 60f5bdf59d Fix `url_for_asset` fallback and 404 on DAG Audit Log
(#31233)
new d6878b9247 Support for sorting DAGs by Last Run Date in the web UI
(#31234)
new 0d17377343 Fix task.branch error caused by incorrect initial parameter
(#31265)
new 2fbc196de3 Make connection id validation consistent across interface
(#31282)
new 955500e8d9 Make BaseJobRunner a generic on the job class (#31287)
new 76cee6ca13 Added metavar and choices to --state flag in airflow dags
list-jobs CLI for suggesting valid state arguments. (#31308)
new 537991de3d Upgrade FAB to 4.3.1 (#31203)
new d0e8a9eccf Rephrase comment in setup.py (#31312)
new 754624be30 Convert dask upper-binding into exclusion (#31329)
new 12c3a08179 Add an example for sql_alchemy_connect_args conf (#31332)
new 7a17372741 Update the output of airflow info command in the doc
(#31336)
new f57b87d940 Fix crash when clearing run with task from normal to mapped
(#31352)
new 3f36a478df Skip auto-document things with :meta private: (#31380)
new 607773250a Fix dropdown default and adjust tutorial to use 42 as
default for proof (#31400)
new ec4f2cadba Fix error handling when pre-importing modules in DAGs
(#31401)
new c777b3fccb Fix `max_active_tis_per_dagrun` for Dynamic Task Mapping
(#31406)
new 0d27b05ee3 Ensure min backoff in base sensor is at least 1 (#31412)
new 3e5f877b84 Add missing BashOperator import to documentation example
(#31436)
new e29343060e Wording the next and last run DAG columns better (#31467)
new f192f2dd9b Fix version support information (#31468)
new 194ae50607 Use kube_client over default CoreV1Api for deleting pods
(#31477)
new 0f7ef27f69 Remove note about triggerer being 3.7+ only (#31483)
new 38215514cd Fix typing in external task triggers (#31490)
new 65473140fe Remove dependency already registered for this task warning
(#31502)
new e11ef3cd0f Remove typo in docs serializers.rst (#31507)
new 07dd11365e Include triggerer health status in Airflow /health endpoint
(#31529)
new a891a90a59 fix airflow users delete CLI command (#31539)
new 643035f329 Add the missing `map_index` to the xcom key when skipping
downstream tasks (#31541)
new c1c096e6a7 Fix sorting of tags (#31553)
new da3f6539ff Remove found_descendents param from get_flat_relative_ids
(#31559)
new e650760d88 Resolving issue where Grid won't un-collapse when Details
is collapsed (#31561)
new 445264420c Workaround mypy confusing default with necessary arguments
(#31562)
new 0757e2bec6 Updating UI screenshots in docs (#31577)
new e8c9e6bd70 Add TriggererStatus to OpenAPI spec (#31579)
new 2e1a06a543 Trigger Button - Implement Part 2 of AIP-50 (#31583)
new 3ea0e7508e Hide old alert box when testing conn with different value
(#31606)
new 0556a2344d fixing broken links in openapi/v1.yaml (#31619)
new e917954eb4 Add docstring and signature for _read_remote_logs (#31623)
new 09789f71f5 Small cleanups / optimizations in DagRun.update_state
(#31628)
new 6c83f627fb 📝 Update release documentation (#31631)
new eca2f3cc77 remove ducplicate entry in the metrics doc (#31654)
new 8f69bf3041 Fix wrong cron syntax (#31666)
new 6ad08f6284 Retain on_failure_fail_dagrun attr when other task attrs
are overriden (#31678)
new 521b23073d update path of KubernetesPodOperator in core docs (#31683)
new 85747f6a22 Replace unicodecsv with standard csv library (#31693)
new 20a821abe3 Add back accidentally removed `create_session` in
utils/db.py (#31705)
new fda3b46949 Control permissibility of driver config in extra from
airflow.cfg (#31754)
new 12f17cc4e7 Use keyword paramaters for migration methods for mssql
(#31309)
new 2433784fee Fix Kubernetes executor set wrong task status (#31274)
The 74 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/SECURITY.rst | 88 +++-
BREEZE.rst | 3 +-
CI.rst | 8 +-
COMMITTERS.rst | 6 +
LOCAL_VIRTUALENV.rst | 12 +-
README.md | 135 +-----
airflow/__init__.py | 1 +
.../common/airflow_health.py} | 30 +-
airflow/api/common/mark_tasks.py | 4 +-
.../api_connexion/endpoints/connection_endpoint.py | 5 +
airflow/api_connexion/endpoints/health_endpoint.py | 49 +--
airflow/api_connexion/endpoints/user_endpoint.py | 2 +-
airflow/api_connexion/openapi/v1.yaml | 29 +-
airflow/cli/cli_config.py | 9 +-
airflow/cli/commands/connection_command.py | 14 +-
airflow/cli/commands/kubernetes_command.py | 4 +-
airflow/cli/commands/user_command.py | 3 +
airflow/cli/commands/webserver_command.py | 4 +-
airflow/config_templates/config.yml | 2 +-
airflow/config_templates/default_airflow.cfg | 1 +
airflow/config_templates/default_celery.py | 17 +-
airflow/configuration.py | 3 +-
airflow/decorators/base.py | 6 +-
...amic_task_mapping_with_no_taskflow_operators.py | 62 +++
airflow/example_dags/example_params_ui_tutorial.py | 4 +-
airflow/executors/kubernetes_executor.py | 19 +-
airflow/hooks/subprocess.py | 2 +-
airflow/jobs/backfill_job_runner.py | 4 +-
airflow/jobs/base_job_runner.py | 13 +-
airflow/jobs/dag_processor_job_runner.py | 2 +-
airflow/jobs/local_task_job_runner.py | 2 +-
airflow/jobs/scheduler_job_runner.py | 39 +-
airflow/jobs/triggerer_job_runner.py | 2 +-
airflow/kubernetes/pod_generator.py | 4 +-
airflow/kubernetes/pod_generator_deprecated.py | 5 +-
...3_1_10_4_make_taskinstance_pool_not_nullable.py | 12 +-
.../versions/0054_1_10_10_add_dag_code_table.py | 6 +-
.../0060_2_0_0_remove_id_column_from_xcom.py | 2 +-
...89_2_2_0_make_xcom_pkey_columns_non_nullable.py | 4 +-
.../0102_2_3_0_switch_xcom_table_to_use_run_id.py | 6 +-
airflow/models/abstractoperator.py | 25 +-
airflow/models/dag.py | 6 +-
airflow/models/dagbag.py | 28 +-
airflow/models/dagrun.py | 12 +-
airflow/models/serialized_dag.py | 2 +-
airflow/models/skipmixin.py | 3 +
airflow/models/taskinstance.py | 10 +-
airflow/models/taskmixin.py | 15 +-
airflow/operators/python.py | 9 +-
airflow/operators/subdag.py | 7 +-
airflow/providers/amazon/aws/hooks/dms.py | 2 +-
.../providers/amazon/aws/log/s3_task_handler.py | 2 +-
airflow/providers/apache/hive/hooks/hive.py | 6 +-
airflow/providers/apache/hive/provider.yaml | 5 +
.../apache/hive/transfers/mssql_to_hive.py | 6 +-
.../apache/hive/transfers/mysql_to_hive.py | 41 +-
.../apache/hive/transfers/vertica_to_hive.py | 7 +-
airflow/providers/google/cloud/hooks/bigquery.py | 11 +-
.../providers/google/cloud/operators/workflows.py | 8 +-
.../providers/google/cloud/transfers/sql_to_gcs.py | 18 +-
.../microsoft/azure/log/wasb_task_handler.py | 2 +-
.../azure/transfers/oracle_to_azure_data_lake.py | 10 +-
.../providers/mysql/transfers/vertica_to_mysql.py | 6 +-
airflow/providers/odbc/hooks/odbc.py | 30 ++
airflow/sensors/base.py | 3 +-
airflow/sensors/bash.py | 2 +-
airflow/sensors/external_task.py | 17 +-
airflow/timetables/base.py | 22 +-
airflow/timetables/simple.py | 12 +-
airflow/triggers/external_task.py | 17 +-
airflow/utils/file.py | 2 +-
airflow/utils/hashlib_wrapper.py | 21 +-
airflow/utils/log/file_task_handler.py | 14 +-
airflow/utils/task_group.py | 5 +
airflow/www/decorators.py | 2 +-
airflow/www/extensions/init_manifest_files.py | 2 +-
airflow/www/forms.py | 3 +-
airflow/www/security.py | 17 +
airflow/www/static/js/api/useTaskLog.ts | 1 -
airflow/www/static/js/connection_form.js | 6 +
airflow/www/static/js/dag/Main.tsx | 6 +-
.../js/dag/details/taskInstance/Logs/index.tsx | 19 +-
airflow/www/static/js/dag/grid/index.tsx | 38 +-
airflow/www/static/js/task_instances.js | 4 +-
airflow/www/static/js/types/api-generated.ts | 35 +-
airflow/www/templates/airflow/dag.html | 43 +-
airflow/www/templates/airflow/dag_audit_log.html | 1 -
airflow/www/templates/airflow/dags.html | 44 +-
airflow/www/templates/airflow/trigger.html | 9 +-
airflow/www/validators.py | 21 +
airflow/www/views.py | 97 +++--
dev/README_RELEASE_AIRFLOW.md | 17 +-
dev/README_RELEASE_HELM_CHART.md | 15 +
dev/README_RELEASE_PROVIDER_PACKAGES.md | 32 ++
dev/breeze/src/airflow_breeze/global_constants.py | 4 +-
dev/breeze/tests/test_cache.py | 4 +-
dev/breeze/tests/test_selective_checks.py | 458 +++++++++++++++++----
dev/provider_packages/prepare_provider_packages.py | 2 +-
dev/retag_docker_images.py | 2 +-
.../connections/odbc.rst | 9 +
.../logging-monitoring/callbacks.rst | 9 +
.../logging-monitoring/logging-tasks.rst | 22 +-
.../logging-monitoring/metrics.rst | 1 -
.../authoring-and-scheduling/deferring.rst | 5 -
.../dynamic-task-mapping.rst | 5 +-
.../authoring-and-scheduling/serializers.rst | 2 +-
docs/apache-airflow/best-practices.rst | 4 +-
docs/apache-airflow/core-concepts/dags.rst | 4 +-
docs/apache-airflow/howto/custom-operator.rst | 2 +-
docs/apache-airflow/howto/notifications.rst | 7 +-
docs/apache-airflow/howto/set-config.rst | 13 +
.../howto/upgrading-from-1-10/index.rst | 2 +-
docs/apache-airflow/img/airflow_erd.sha256 | 2 +-
docs/apache-airflow/img/grid.png | Bin 347155 -> 249561 bytes
docs/apache-airflow/img/grid_instance_details.png | Bin 287820 -> 178129 bytes
docs/apache-airflow/img/grid_mapped_task.png | Bin 334409 -> 135753 bytes
docs/apache-airflow/img/grid_run_details.png | Bin 243661 -> 202106 bytes
docs/apache-airflow/installation/prerequisites.rst | 6 +-
docs/apache-airflow/release-process.rst | 3 +
docs/apache-airflow/ui.rst | 14 +-
docs/conf.py | 6 +-
docs/docker-stack/build.rst | 19 +-
docs/spelling_wordlist.txt | 2 +
generated/provider_dependencies.json | 237 +++++++----
images/breeze/output-commands-hash.txt | 89 ++--
images/breeze/output-commands.svg | 92 +++--
images/breeze/output_ci-image_build.svg | 4 +-
images/breeze/output_ci-image_pull.svg | 60 +--
images/breeze/output_ci-image_verify.svg | 32 +-
images/breeze/output_k8s_build-k8s-image.svg | 52 +--
images/breeze/output_k8s_configure-cluster.svg | 60 +--
images/breeze/output_k8s_create-cluster.svg | 4 +-
images/breeze/output_k8s_delete-cluster.svg | 34 +-
images/breeze/output_k8s_deploy-airflow.svg | 70 ++--
images/breeze/output_k8s_k9s.svg | 32 +-
images/breeze/output_k8s_logs.svg | 34 +-
images/breeze/output_k8s_run-complete-tests.svg | 68 +--
images/breeze/output_k8s_shell.svg | 40 +-
images/breeze/output_k8s_status.svg | 36 +-
images/breeze/output_k8s_tests.svg | 68 +--
images/breeze/output_k8s_upload-k8s-image.svg | 58 +--
images/breeze/output_prod-image_build.svg | 4 +-
images/breeze/output_prod-image_pull.svg | 60 +--
images/breeze/output_prod-image_verify.svg | 34 +-
...put_release-management_generate-constraints.svg | 4 +-
...tput_release-management_release-prod-images.svg | 2 +-
images/breeze/output_setup_config.svg | 2 +-
images/breeze/output_shell.svg | 116 +++---
images/breeze/output_start-airflow.svg | 2 +-
.../breeze/output_testing_docker-compose-tests.svg | 54 ++-
images/breeze/output_testing_integration-tests.svg | 70 ++--
images/breeze/output_testing_tests.svg | 108 ++---
.../pre_commit_update_providers_dependencies.py | 18 +-
setup.cfg | 4 +-
setup.py | 32 +-
{airflow/api_connexion => tests/api}/__init__.py | 0
.../api_connexion => tests/api/common}/__init__.py | 0
tests/api/common/test_airflow_health.py | 100 +++++
.../endpoints/test_connection_endpoint.py | 14 +
.../api_connexion/endpoints/test_user_endpoint.py | 14 +
tests/cli/commands/test_connection_command.py | 10 +
tests/cli/commands/test_kubernetes_command.py | 4 +-
tests/core/test_configuration.py | 19 +-
tests/dag_processing/test_processor.py | 115 ++++--
tests/decorators/test_python.py | 7 +-
tests/decorators/test_setup_teardown.py | 12 +
tests/decorators/test_task_group.py | 74 ++++
tests/executors/test_kubernetes_executor.py | 2 +
tests/jobs/test_scheduler_job.py | 40 +-
tests/models/test_dagbag.py | 3 +-
tests/models/test_dagrun.py | 26 +-
tests/models/test_param.py | 39 +-
tests/models/test_taskinstance.py | 60 ++-
tests/operators/test_python.py | 50 ++-
tests/operators/test_subdag_operator.py | 11 +-
tests/providers/apache/hive/hooks/test_hive.py | 11 +-
.../apache/hive/transfers/test_mssql_to_hive.py | 5 +-
.../apache/hive/transfers/test_mysql_to_hive.py | 5 +-
.../providers/google/cloud/hooks/test_bigquery.py | 2 +-
.../google/cloud/transfers/test_sql_to_gcs.py | 124 +++---
.../cloud/utils/test_mlengine_operator_utils.py | 16 +-
.../transfers/test_oracle_to_azure_data_lake.py | 7 +-
tests/providers/odbc/hooks/test_odbc.py | 24 +-
tests/sensors/test_base.py | 35 ++
tests/sensors/test_external_task_sensor.py | 61 ++-
tests/serialization/test_dag_serialization.py | 4 +-
tests/utils/test_db.py | 3 +
tests/utils/test_task_group.py | 80 ++++
tests/www/test_validators.py | 35 ++
tests/www/views/test_views.py | 2 +-
tests/www/views/test_views_acl.py | 2 +
tests/www/views/test_views_connection.py | 11 +
tests/www/views/test_views_home.py | 54 ++-
tests/www/views/test_views_tasks.py | 12 +-
tests/www/views/test_views_trigger_dag.py | 37 +-
195 files changed, 3198 insertions(+), 1540 deletions(-)
copy airflow/{api_connexion/endpoints/health_endpoint.py =>
api/common/airflow_health.py} (72%)
create mode 100644
airflow/example_dags/example_dynamic_task_mapping_with_no_taskflow_operators.py
copy {airflow/api_connexion => tests/api}/__init__.py (100%)
copy {airflow/api_connexion => tests/api/common}/__init__.py (100%)
create mode 100644 tests/api/common/test_airflow_health.py
