This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a change to branch utilize-disk-space-workaround-for-doc-publishng
in repository https://gitbox.apache.org/repos/asf/airflow.git
omit e20d846bd5b Bring back documentation publishing to Canary builds
add c0be4023a97 Run only ui tests for PR with new UI only changes. (#43503)
add 6641d78703b Add latest dag run info to dags list and dag details
(#43489)
add dc59212b852 update issue triage process to improve Dosu bot quality
(#43511)
add 64d8f4f6dbb Migrate the public endpoint Get DAG Source to FastAPI
(#43084)
add d8c7d28411b Start porting DAG definition code to the Task SDK (#43076)
add b94b1a19837 Skip integration tests for new UI only PR. Run docs build
only when needed. (#43512)
add da265d92cad Task SDK: Fix typing for `autoregistered_dags` (#43522)
add a2cbaecf625 Keycloak Breeze integration (#43240)
add 51f9e835bf4 Move `uncompress_file` function from `airflow.utils` to
Hive provider (#43526)
add 10c6464d86a Remove unused functions from `airflow.utils.dates` (#43527)
add 31ee7cbe5d6 Remove `task_fail` from `airflow db cleanup` (#43528)
add 15ae7ce9724 Remove deprecated functions from
`airflow/configuration.py` (#43530)
add a923a83338a Bump `uv` to `0.4.29` (#43531)
add d3e18121521 Remove `TaskInstanceKey.reduced` (#43532)
add c42b1d3acd5 Update providers metadata 2024-10-31 (#43539)
add 568c0352f11 Fix main: ignore spell check Uncompress (#43538)
add db4afd793a6 DOC fix documentation error in
`apache-airflow-providers-fab/access-control.rst` (#43495)
add 8308dad6ee0 Include doc-only commit in provider commit history (#43543)
add b3e902dd249 Fix testing tests command to allow passing test as extra
arg (#43529)
add b985b2579bd AIP-84 Get Task Instance (#43485)
add 5886016243d Prepare docs for Oct 2nd wave of providers RC2 (#43540)
add 06088a3abcb Standard provider python operator (#42081)
add 80074fc03b5 Bring back documentation publishing to Canary builds
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (e20d846bd5b)
\
N -- N -- N
refs/heads/utilize-disk-space-workaround-for-doc-publishng (80074fc03b5)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.github/workflows/ci.yml | 1 +
.github/workflows/prod-image-build.yml | 5 +
.github/workflows/static-checks-mypy-docs.yml | 9 +-
.pre-commit-config.yaml | 6 +-
Dockerfile | 35 +-
Dockerfile.ci | 4 +-
ISSUE_TRIAGE_PROCESS.rst | 7 +-
.../api_connexion/endpoints/dag_source_endpoint.py | 2 +
.../endpoints/task_instance_endpoint.py | 2 +
airflow/api_connexion/schemas/dag_schema.py | 1 -
airflow/api_fastapi/app.py | 18 +-
airflow/api_fastapi/core_api/app.py | 20 +
.../api_fastapi/core_api/openapi/v1-generated.yaml | 1331 +++++++++++++-------
.../api_fastapi/core_api/routes/public/__init__.py | 13 +-
.../core_api/routes/public/dag_sources.py | 71 ++
.../core_api/routes/public/task_instances.py | 58 +
.../core_api/serializers/dag_sources.py | 8 +-
airflow/api_fastapi/core_api/serializers/dags.py | 1 -
.../api_fastapi/core_api/serializers/job.py | 21 +-
.../core_api/serializers/task_instances.py | 71 ++
.../api_fastapi/core_api/serializers/trigger.py | 17 +-
airflow/cli/commands/dag_command.py | 1 -
airflow/configuration.py | 141 ---
airflow/dag_processing/collection.py | 5 +-
airflow/decorators/base.py | 44 +-
airflow/decorators/bash.py | 4 +-
airflow/decorators/branch_external_python.py | 2 +-
airflow/decorators/branch_python.py | 2 +-
airflow/decorators/branch_virtualenv.py | 2 +-
airflow/decorators/external_python.py | 2 +-
airflow/decorators/python.py | 2 +-
airflow/decorators/python_virtualenv.py | 2 +-
airflow/decorators/sensor.py | 6 +-
airflow/decorators/short_circuit.py | 2 +-
airflow/decorators/task_group.py | 2 +-
.../example_asset_alias_with_no_taskflow.py | 2 +-
airflow/example_dags/example_branch_operator.py | 4 +-
.../example_branch_operator_decorator.py | 2 +-
.../example_python_context_decorator.py | 6 +-
.../example_python_context_operator.py | 12 +-
airflow/example_dags/example_python_decorator.py | 2 +-
airflow/example_dags/example_python_operator.py | 2 +-
airflow/example_dags/example_sensors.py | 2 +-
.../example_dags/example_short_circuit_operator.py | 2 +-
airflow/example_dags/tutorial_dag.py | 2 +-
.../tutorial_taskflow_api_virtualenv.py | 2 +-
.../example_dags/tutorial_taskflow_templates.py | 2 +-
airflow/exceptions.py | 6 +-
airflow/models/abstractoperator.py | 355 ++----
airflow/models/baseoperator.py | 1190 ++++-------------
airflow/models/dag.py | 1053 +---------------
airflow/models/dagbag.py | 13 +-
airflow/models/mappedoperator.py | 7 +-
airflow/models/param.py | 2 +-
airflow/models/skipmixin.py | 2 +-
airflow/models/taskinstance.py | 12 +-
airflow/models/taskinstancekey.py | 8 -
airflow/models/taskmixin.py | 270 +---
airflow/models/xcom_arg.py | 21 +-
airflow/sensors/external_task.py | 4 +-
airflow/serialization/schema.json | 20 +-
airflow/serialization/serialized_objects.py | 61 +-
airflow/task/priority_strategy.py | 4 +-
airflow/template/templater.py | 17 +-
airflow/typing_compat.py | 5 +-
airflow/ui/openapi-gen/queries/common.ts | 300 +++--
airflow/ui/openapi-gen/queries/prefetch.ts | 305 +++--
airflow/ui/openapi-gen/queries/queries.ts | 653 +++++-----
airflow/ui/openapi-gen/queries/suspense.ts | 385 +++---
airflow/ui/openapi-gen/requests/schemas.gen.ts | 577 ++++++++-
airflow/ui/openapi-gen/requests/services.gen.ts | 676 +++++-----
airflow/ui/openapi-gen/requests/types.gen.ts | 658 ++++++----
airflow/ui/src/layouts/Nav/Nav.tsx | 7 -
airflow/ui/src/pages/DagsList/Dag/Dag.tsx | 23 +-
airflow/ui/src/pages/DagsList/Dag/Header.tsx | 7 +-
airflow/ui/src/pages/DagsList/DagCard.test.tsx | 10 +-
airflow/ui/src/pages/DagsList/DagCard.tsx | 34 +-
airflow/ui/src/pages/DagsList/DagTags.tsx | 5 +-
airflow/ui/src/pages/DagsList/DagsFilters.tsx | 2 +-
airflow/ui/src/pages/DagsList/DagsList.tsx | 99 +-
airflow/ui/src/pages/DagsList/LatestRun.tsx | 41 +
airflow/ui/src/pages/DagsList/RecentRuns.tsx | 83 ++
airflow/ui/src/pages/DagsList/Schedule.tsx | 36 +
airflow/ui/src/queries/useDags.tsx | 81 ++
airflow/ui/src/utils/stateColor.ts | 36 +
airflow/utils/compression.py | 40 -
airflow/utils/dates.py | 38 +-
airflow/utils/db_cleanup.py | 1 -
airflow/utils/decorators.py | 9 +-
airflow/utils/edgemodifier.py | 154 +--
airflow/utils/log/logging_mixin.py | 1 +
airflow/utils/task_group.py | 624 +--------
airflow/utils/types.py | 32 +-
contributing-docs/testing/integration_tests.rst | 68 +-
dev/breeze/doc/ci/02_images.md | 2 +-
dev/breeze/doc/images/output-commands.svg | 4 +-
dev/breeze/doc/images/output_shell.svg | 4 +-
dev/breeze/doc/images/output_shell.txt | 2 +-
dev/breeze/doc/images/output_start-airflow.svg | 4 +-
dev/breeze/doc/images/output_start-airflow.txt | 2 +-
.../images/output_testing_integration-tests.svg | 4 +-
.../images/output_testing_integration-tests.txt | 2 +-
dev/breeze/doc/images/output_testing_tests.svg | 4 +-
dev/breeze/doc/images/output_testing_tests.txt | 2 +-
.../commands/release_management_commands.py | 2 +-
.../airflow_breeze/commands/testing_commands.py | 12 +-
dev/breeze/src/airflow_breeze/global_constants.py | 18 +-
.../src/airflow_breeze/params/shell_params.py | 13 +
.../prepare_providers/provider_documentation.py | 1 +
dev/breeze/src/airflow_breeze/utils/run_tests.py | 16 +-
.../src/airflow_breeze/utils/selective_checks.py | 48 +-
.../tests/test_pytest_args_for_test_types.py | 10 +-
dev/breeze/tests/test_selective_checks.py | 69 +-
dev/mypy/plugin/outputs.py | 1 +
dev/perf/dags/sql_perf_dag.py | 2 +-
docs/apache-airflow-providers-amazon/commits.rst | 7 +-
.../auth-manager/access-control.rst | 2 +-
docs/apache-airflow-providers-fab/commits.rst | 5 +-
.../guides/user.rst | 4 +-
.../dynamic-task-mapping.rst | 2 +-
docs/apache-airflow/best-practices.rst | 16 +-
docs/apache-airflow/core-concepts/dags.rst | 2 +-
docs/apache-airflow/core-concepts/operators.rst | 4 +-
docs/apache-airflow/howto/operator/python.rst | 24 +-
docs/apache-airflow/operators-and-hooks-ref.rst | 4 +-
docs/apache-airflow/tutorial/taskflow.rst | 2 +-
docs/exts/templates/openlineage.rst.jinja2 | 2 +-
docs/spelling_wordlist.txt | 1 +
generated/provider_dependencies.json | 15 +-
generated/provider_metadata.json | 84 ++
hatch_build.py | 3 +-
newsfragments/43530.significant.rst | 21 +
.../src/airflow/providers/amazon/CHANGELOG.rst | 10 +-
.../providers/amazon/aws/operators/appflow.py | 2 +-
.../providers/amazon/aws/operators/comprehend.py | 6 +-
.../airflow/providers/amazon/aws/operators/dms.py | 6 +-
.../amazon/aws/operators/kinesis_analytics.py | 6 +-
.../providers/amazon/aws/operators/sagemaker.py | 4 +-
.../airflow/providers/apache/beam/hooks/beam.py | 2 +-
.../providers/apache/drill/operators/drill.py | 4 +-
.../providers/apache/hive/transfers/s3_to_hive.py | 19 +-
.../providers/apache/spark/decorators/pyspark.py | 2 +-
.../celery/executors/celery_executor_utils.py | 5 +-
.../providers/cncf/kubernetes/operators/pod.py | 4 +-
.../cncf/kubernetes/operators/spark_kubernetes.py | 3 +-
.../providers/common/compat/standard}/__init__.py | 0
.../providers/common/compat/standard/operators.py | 54 +
.../providers/common/compat/standard/utils.py | 14 +-
.../airflow/providers/common/sql/operators/sql.py | 18 +-
.../airflow/providers/common/sql/operators/sql.pyi | 18 +-
.../databricks/operators/databricks_sql.py | 4 +-
.../airflow/providers/docker/decorators/docker.py | 4 +-
.../edge/example_dags/integration_test.py | 2 +-
.../airflow/providers/exasol/operators/exasol.py | 4 +-
providers/src/airflow/providers/fab/CHANGELOG.rst | 7 +-
.../fab/auth_manager/security_manager/override.py | 9 +-
.../google/cloud/utils/mlengine_operator_utils.py | 2 +-
.../src/airflow/providers/jdbc/operators/jdbc.py | 4 +-
.../providers/microsoft/mssql/operators/mssql.py | 4 +-
.../src/airflow/providers/mysql/operators/mysql.py | 4 +-
.../airflow/providers/openlineage/provider.yaml | 2 +-
.../airflow/providers/oracle/operators/oracle.py | 4 +-
.../providers/postgres/operators/postgres.py | 7 +-
.../providers/snowflake/operators/snowflake.py | 4 +-
.../providers/snowflake/operators/snowpark.py | 2 +-
.../airflow/providers/sqlite/operators/sqlite.py | 4 +-
.../providers/standard}/operators/python.py | 86 +-
.../src/airflow/providers/standard/provider.yaml | 3 +-
.../airflow/providers/standard}/sensors/python.py | 0
.../airflow/providers/standard/utils}/__init__.py | 0
.../providers/standard}/utils/python_virtualenv.py | 11 -
.../utils/python_virtualenv_script.jinja2 | 4 +-
.../providers/teradata/operators/teradata.py | 4 +-
.../src/airflow/providers/trino/operators/trino.py | 4 +-
.../airflow/providers/vertica/operators/vertica.py | 4 +-
providers/tests/amazon/aws/operators/test_batch.py | 2 +-
.../tests/apache/hive/transfers/test_s3_to_hive.py | 20 +-
.../kubernetes/log_handlers/test_log_handlers.py | 3 +-
.../tests/google/cloud/operators/test_bigquery.py | 4 +-
.../google/cloud/operators/test_cloud_build.py | 4 +-
.../tests/google/cloud/operators/test_compute.py | 10 +-
.../tests/google/cloud/operators/test_dataflow.py | 2 +-
.../tests/google/cloud/operators/test_dataproc.py | 6 +-
.../cloud/operators/test_kubernetes_engine.py | 16 +-
.../google/cloud/operators/test_speech_to_text.py | 10 +-
.../tests/google/cloud/sensors/test_dataproc.py | 2 +-
.../google/cloud/transfers/test_gcs_to_bigquery.py | 7 +-
.../cloud/utils/test_mlengine_operator_utils.py | 3 +-
.../tests/openlineage/extractors/test_base.py | 4 +-
.../tests/openlineage/extractors/test_manager.py | 3 +-
.../tests/openlineage/extractors/test_python.py | 3 +-
.../tests/openlineage/plugins/test_listener.py | 3 +-
providers/tests/openlineage/plugins/test_utils.py | 4 +-
providers/tests/openlineage/utils/test_utils.py | 13 +-
providers/tests/salesforce/operators/test_bulk.py | 4 +-
.../tests/standard}/operators/test_python.py | 44 +-
providers/tests/standard/operators/test_weekday.py | 2 +-
.../tests/standard}/sensors/test_python.py | 4 +-
.../tests/standard/utils}/__init__.py | 0
.../standard}/utils/test_python_virtualenv.py | 10 +-
providers/tests/system/amazon/aws/example_s3.py | 2 +-
.../tests/system/amazon/aws/example_sagemaker.py | 2 +-
.../apache/kafka/example_dag_event_listener.py | 6 +-
.../system/apache/kafka/example_dag_hello_kafka.py | 8 +-
.../system/docker/example_docker_copy_data.py | 2 +-
.../elasticsearch/example_elasticsearch_query.py | 2 +-
.../cloud/cloud_batch/example_cloud_batch.py | 2 +-
.../google/cloud/cloud_run/example_cloud_run.py | 2 +-
.../system/google/cloud/gcs/example_gcs_to_gcs.py | 2 +-
pyproject.toml | 1 +
scripts/ci/docker-compose/integration-keycloak.yml | 62 +
.../ci/docker-compose/keycloak/init-keycloak-db.sh | 14 +-
.../docker-compose/keycloak/keycloak-entrypoint.sh | 45 +
.../pre_commit/base_operator_partial_arguments.py | 80 +-
scripts/ci/pre_commit/sync_init_decorator.py | 201 +--
scripts/docker/entrypoint_ci.sh | 2 +-
.../docker/install_from_docker_context_files.sh | 33 +-
task_sdk/pyproject.toml | 35 +-
task_sdk/src/airflow/sdk/__init__.py | 39 +-
.../src/airflow/sdk/definitions}/__init__.py | 0
.../airflow/sdk/definitions/abstractoperator.py | 261 ++++
.../src/airflow/sdk/definitions/baseoperator.py | 1226 ++++++++++++++++++
.../src/airflow/sdk/definitions/contextmanager.py | 124 ++
task_sdk/src/airflow/sdk/definitions/dag.py | 1119 ++++++++++++++++
.../src/airflow/sdk/definitions/decorators.py | 37 +-
.../src/airflow/sdk/definitions/edges.py | 29 +-
task_sdk/src/airflow/sdk/definitions/mixins.py | 121 ++
.../src/airflow/sdk/definitions/node.py | 174 +--
.../src/airflow/sdk/definitions/taskgroup.py | 290 ++---
.../src/airflow/sdk/exceptions.py | 0
task_sdk/src/airflow/sdk/types.py | 75 ++
task_sdk/tests/defintions/test_baseoperator.py | 343 +++++
task_sdk/tests/defintions/test_dag.py | 419 ++++++
tests/api_connexion/endpoints/test_dag_endpoint.py | 32 -
tests/api_connexion/schemas/test_dag_schema.py | 5 -
.../core_api/routes/public/test_dag_sources.py | 110 ++
.../core_api/routes/public/test_dags.py | 1 -
.../core_api/routes/public/test_task_instances.py | 396 ++++++
tests/core/test_configuration.py | 17 -
tests/core/test_core.py | 2 +-
tests/core/test_example_dags_system.py | 2 +-
tests/core/test_sentry.py | 2 +-
tests/dags/test_assets.py | 2 +-
tests/dags/test_cli_triggered_dags.py | 2 +-
tests/dags/test_dag_xcom_openlineage.py | 2 +-
tests/dags/test_dagrun_fast_follow.py | 2 +-
tests/dags/test_future_start_date.py | 2 +-
tests/dags/test_invalid_param.py | 2 +-
tests/dags/test_invalid_param2.py | 2 +-
tests/dags/test_invalid_param3.py | 2 +-
tests/dags/test_invalid_param4.py | 2 +-
tests/dags/test_logging_in_dag.py | 2 +-
tests/dags/test_mapped_classic.py | 2 +-
tests/dags/test_mark_state.py | 2 +-
tests/dags/test_on_failure_callback.py | 2 +-
tests/dags/test_task_view_type_check.py | 2 +-
tests/dags/test_valid_param.py | 2 +-
tests/dags/test_valid_param2.py | 2 +-
tests/dags_corrupted/test_impersonation_custom.py | 2 +-
tests/decorators/test_python.py | 2 +-
tests/jobs/test_local_task_job.py | 2 +-
tests/jobs/test_triggerer_job.py | 2 +-
tests/models/test_backfill.py | 2 +-
tests/models/test_baseoperator.py | 301 +----
tests/models/test_baseoperatormeta.py | 2 +-
tests/models/test_cleartasks.py | 2 +-
tests/models/test_dag.py | 437 +------
tests/models/test_dagbag.py | 4 +-
tests/models/test_dagrun.py | 4 +-
tests/models/test_mappedoperator.py | 6 +-
tests/models/test_renderedtifields.py | 2 +-
tests/models/test_taskinstance.py | 16 +-
tests/models/test_xcom_arg.py | 2 +-
tests/sensors/test_external_task_sensor.py | 2 +-
tests/serialization/test_dag_serialization.py | 95 +-
tests/serialization/test_pydantic_models.py | 16 +-
tests/serialization/test_serialized_objects.py | 2 +-
.../deps/test_not_previously_skipped_dep.py | 2 +-
tests/utils/log/test_log_reader.py | 2 +-
tests/utils/test_compression.py | 87 --
tests/utils/test_dates.py | 53 -
tests/utils/test_db_cleanup.py | 2 +-
tests/utils/test_dot_renderer.py | 6 +-
tests/utils/test_edgemodifier.py | 2 +-
tests/utils/test_log_handlers.py | 2 +-
tests/utils/test_task_group.py | 20 +-
tests/www/views/test_views_rendered.py | 3 +-
tests_common/test_utils/compat.py | 4 +
tests_common/test_utils/mock_operators.py | 15 +-
289 files changed, 9949 insertions(+), 7463 deletions(-)
create mode 100644 airflow/api_fastapi/core_api/routes/public/dag_sources.py
create mode 100644 airflow/api_fastapi/core_api/routes/public/task_instances.py
copy task_sdk/tests/test_hello.py =>
airflow/api_fastapi/core_api/serializers/dag_sources.py (85%)
copy task_sdk/tests/test_hello.py =>
airflow/api_fastapi/core_api/serializers/job.py (63%)
create mode 100644 airflow/api_fastapi/core_api/serializers/task_instances.py
copy task_sdk/tests/test_hello.py =>
airflow/api_fastapi/core_api/serializers/trigger.py (67%)
create mode 100644 airflow/ui/src/pages/DagsList/LatestRun.tsx
create mode 100644 airflow/ui/src/pages/DagsList/RecentRuns.tsx
create mode 100644 airflow/ui/src/pages/DagsList/Schedule.tsx
create mode 100644 airflow/ui/src/queries/useDags.tsx
create mode 100644 airflow/ui/src/utils/stateColor.ts
delete mode 100644 airflow/utils/compression.py
create mode 100644 newsfragments/43530.significant.rst
copy {airflow/api_connexion =>
providers/src/airflow/providers/common/compat/standard}/__init__.py (100%)
create mode 100644
providers/src/airflow/providers/common/compat/standard/operators.py
copy task_sdk/tests/test_hello.py =>
providers/src/airflow/providers/common/compat/standard/utils.py (63%)
rename {airflow =>
providers/src/airflow/providers/standard}/operators/python.py (95%)
rename {airflow => providers/src/airflow/providers/standard}/sensors/python.py
(100%)
copy {airflow/api_connexion =>
providers/src/airflow/providers/standard/utils}/__init__.py (100%)
rename {airflow =>
providers/src/airflow/providers/standard}/utils/python_virtualenv.py (93%)
rename {airflow =>
providers/src/airflow/providers/standard}/utils/python_virtualenv_script.jinja2
(95%)
rename {tests => providers/tests/standard}/operators/test_python.py (98%)
rename {tests => providers/tests/standard}/sensors/test_python.py (95%)
copy {airflow/api_connexion => providers/tests/standard/utils}/__init__.py
(100%)
rename {tests => providers/tests/standard}/utils/test_python_virtualenv.py
(92%)
create mode 100644 scripts/ci/docker-compose/integration-keycloak.yml
rename task_sdk/tests/test_hello.py =>
scripts/ci/docker-compose/keycloak/init-keycloak-db.sh (70%)
mode change 100644 => 100755
create mode 100755 scripts/ci/docker-compose/keycloak/keycloak-entrypoint.sh
copy {airflow/api_connexion =>
task_sdk/src/airflow/sdk/definitions}/__init__.py (100%)
create mode 100644 task_sdk/src/airflow/sdk/definitions/abstractoperator.py
create mode 100644 task_sdk/src/airflow/sdk/definitions/baseoperator.py
create mode 100644 task_sdk/src/airflow/sdk/definitions/contextmanager.py
create mode 100644 task_sdk/src/airflow/sdk/definitions/dag.py
copy tests/dags/test_logging_in_dag.py =>
task_sdk/src/airflow/sdk/definitions/decorators.py (51%)
copy airflow/utils/edgemodifier.py =>
task_sdk/src/airflow/sdk/definitions/edges.py (88%)
create mode 100644 task_sdk/src/airflow/sdk/definitions/mixins.py
copy airflow/models/taskmixin.py =>
task_sdk/src/airflow/sdk/definitions/node.py (57%)
copy airflow/utils/task_group.py =>
task_sdk/src/airflow/sdk/definitions/taskgroup.py (73%)
copy airflow/api_connexion/__init__.py =>
task_sdk/src/airflow/sdk/exceptions.py (100%)
create mode 100644 task_sdk/src/airflow/sdk/types.py
create mode 100644 task_sdk/tests/defintions/test_baseoperator.py
create mode 100644 task_sdk/tests/defintions/test_dag.py
create mode 100644 tests/api_fastapi/core_api/routes/public/test_dag_sources.py
create mode 100644
tests/api_fastapi/core_api/routes/public/test_task_instances.py
delete mode 100644 tests/utils/test_compression.py
