This is an automated email from the ASF dual-hosted git repository.
ash pushed a change to branch execution-time-code-in-task-sdk
in repository https://gitbox.apache.org/repos/asf/airflow.git
discard fa4d908515 WIP
discard 518348b1c2 Start building the replacement task runner for AIP-72
add 23dfe54bc3 Fix `HttpToS3Operator` throws exception if s3_bucket
parameter is not passed (#43828)
add 0b59ca372f bump packaging version to 24.2 (#43827)
add 9a5a80e6df Bump UV to 0.5.0 (#43829)
add 053c8a5516 AIP-84 remove remnants of async def (#43831)
add 8906f127fe Bump apache-airflow from 2.10.1 to 2.10.3 in /performance
(#43834)
add a212bf86ab Add basic support for git and local dag bundles (#42689)
add 6c30fc5bff Remove `async` keyword for Get Connection (#43836)
add 01302a1822 AIP-72: Add "Get Variable" endpoint for Execution API
(#43832)
add 2a9bded5ac Bump `uv` to `0.5.1` (#43838)
add 340a70bfe7 Added condition to check if it is a scheduled save or rerun
(#43453)
add 74bc8728fb Fix logs with leading spaces in the Docker operator
(#33692) (#43840)
add 14aa545ad8 chore: add leading underscore (#43843)
add 9bbf6f16f7 Test standard provider with Airflow 2.8 and 2.9 (#43556)
add 229750d8bb Fix EdgeExecutor breeze call (#43842)
add 63b2bbd167 Add random_name_suffix to SparkKubernetesOperator (#43800)
(#43847)
add 614f75db70 Contribution Documentation Updates (#43848)
add 9a5f6d6155 fix: replace \s with space (#43849)
add de88182700 #43252 Disable extra links button if link is null or empty
(#43844)
add b9513cc9b9 Fix Power BI trigger testcase (#43494)
add d4541015ef pydocstyle check: add leading underscore (PT004) (#43852)
add 96cd5a91a9 Correct provider path details in docs (#43850)
add b823f944c3 Additional Contribution Documentation Updates (#43856)
add 6d85a0466d Remove deprecations cncf.kubernetes (#43689)
add 02217ccf5c Fix hatchling to 1.25.0 (#43858)
add cf3f9c7640 bump hatchling to 1.26.1 (#43863)
add c89ab99ace Recommend and use ``uv`` instead of ``pipx`` in a few
remaining places (#43866)
add 19b73982d8 fix(scheulder_job_runner): remove the to-write warnings
that already exists in the db instead of those that does not exist (#43693)
add 12c7dd429c [Edge]Worker UI link beautification (#43869)
add 3f6f04868d Further make sure graphviz is truly optional feature on
MacOS (#43867)
add 084e29e9e2 Add documentation to standard provider operators (#43716)
add c82a76e2e3 Remove deprecated trigger rule (#43349)
add 606ef453a1 Provide option to `force_delete` for
`GCSToBigQueryOperator` (#43785)
add 564139d616 ci(github-actions): add uv to news-fragment action (#43878)
add bec090ceb8 Include unit test to Legacy UI/API Selective Check in
Breeze (#43719)
add ddbdf2e353 Add global events page to browse along with support to
display only events for the dag. (#43793)
add c269be90c9 Change check for SQLite for FastAPI tests (#43884)
add 02003219d6 Use updated data from Dags API for Dag with recent DagRun.
(#43857)
add f757b87581 update how to guide section with doc references (#43889)
new 36a56cce45 Start building the replacement task runner for Task
Execution SDK
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (fa4d908515)
\
N -- N -- N refs/heads/execution-time-code-in-task-sdk
(36a56cce45)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/workflows/basic-tests.yml | 5 +-
.github/workflows/check-providers.yml | 8 +-
.github/workflows/news-fragment.yml | 3 +-
.github/workflows/task-sdk-tests.yml | 4 +-
.pre-commit-config.yaml | 9 +-
Dockerfile | 2 +-
Dockerfile.ci | 36 +--
INSTALL | 142 ++++-----
README.md | 4 +-
airflow/api_fastapi/common/parameters.py | 8 +-
.../core_api/routes/public/event_logs.py | 1 +
.../core_api/routes/public/task_instances.py | 6 +-
airflow/api_fastapi/execution_api/datamodels.py | 9 +
.../api_fastapi/execution_api/routes/__init__.py | 11 +-
.../execution_api/routes/connections.py | 8 +-
airflow/api_fastapi/execution_api/routes/health.py | 4 +-
.../execution_api/routes/task_instance.py | 9 +-
.../routes/{connections.py => variables.py} | 45 ++-
airflow/cli/commands/kubernetes_command.py | 4 +-
airflow/config_templates/config.yml | 8 +
.../bundles}/__init__.py | 0
airflow/dag_processing/bundles/base.py | 82 ++++++
airflow/dag_processing/bundles/git.py | 115 ++++++++
.../dag_processing/bundles/local.py | 36 ++-
airflow/jobs/scheduler_job_runner.py | 2 +-
airflow/settings.py | 4 +-
airflow/ui/src/layouts/Nav/BrowseButton.tsx | 48 ++++
airflow/ui/src/layouts/Nav/Nav.tsx | 16 +-
airflow/ui/src/pages/Events/Events.tsx | 143 +++++++++
airflow/ui/src/pages/Events/index.tsx | 20 ++
airflow/ui/src/queries/useDags.tsx | 6 +-
airflow/ui/src/router.tsx | 7 +-
airflow/utils/trigger_rule.py | 1 -
.../js/dag/details/taskInstance/ExtraLinks.tsx | 2 +-
clients/python/pyproject.toml | 2 +-
contributing-docs/03_contributors_quick_start.rst | 103 +++----
contributing-docs/09_testing.rst | 9 +-
contributing-docs/10_working_with_git.rst | 19 +-
contributing-docs/16_contribution_workflow.rst | 8 +-
contributing-docs/README.rst | 60 ++--
.../contributors_quick_start_gitpod.rst | 18 +-
dev/README_RELEASE_AIRFLOW.md | 42 +--
dev/breeze/README.md | 18 +-
dev/breeze/doc/01_installation.rst | 61 ++--
dev/breeze/doc/04_troubleshooting.rst | 19 +-
dev/breeze/doc/10_advanced_breeze_topics.rst | 13 +-
.../doc/adr/0016-use-uv-tool-to-install-breeze.md | 56 ++++
dev/breeze/doc/ci/02_images.md | 2 +-
dev/breeze/pyproject.toml | 1 -
.../commands/release_management_commands.py | 4 +-
dev/breeze/src/airflow_breeze/global_constants.py | 6 +-
.../src/airflow_breeze/params/shell_params.py | 3 +
dev/breeze/src/airflow_breeze/utils/path_utils.py | 8 +-
dev/breeze/src/airflow_breeze/utils/run_utils.py | 4 +-
dev/breeze/tests/test_run_test_args.py | 2 +-
dev/breeze/tests/test_selective_checks.py | 118 ++++++++
docker_tests/requirements.txt | 2 +-
.../operators.rst | 2 +-
docs/apache-airflow-providers-standard/index.rst | 4 +-
.../operators}/bash.rst | 15 -
.../{operators.rst => operators/datetime.rst} | 16 +-
.../operators}/index.rst | 20 +-
.../operators}/python.rst | 36 +--
.../sensors/bash.rst} | 23 +-
.../{sensors.rst => sensors/datetime.rst} | 16 +-
.../sensors}/index.rst | 20 +-
.../sensors/python.rst | 50 ++++
docs/apache-airflow/faq.rst | 2 +-
docs/apache-airflow/howto/operator/index.rst | 2 -
.../installation/installing-from-pypi.rst | 4 +-
docs/apache-airflow/operators-and-hooks-ref.rst | 15 -
docs/docker-stack/changelog.rst | 3 +
generated/PYPI_README.md | 4 +-
hatch_build.py | 10 +-
kubernetes_tests/test_kubernetes_pod_operator.py | 18 --
newsfragments/43368.significant.rst | 5 +
performance/requirements.txt | 2 +-
providers/pyproject.toml | 6 -
.../providers/MANAGING_PROVIDERS_LIFECYCLE.rst | 81 +++---
.../src/airflow/providers/amazon/aws/hooks/eks.py | 4 +-
.../src/airflow/providers/amazon/aws/hooks/s3.py | 4 +-
.../airflow/providers/amazon/aws/operators/eks.py | 6 +-
.../providers/cncf/kubernetes/CHANGELOG.rst | 28 ++
.../cncf/kubernetes/kubernetes_helper_functions.py | 41 ---
.../providers/cncf/kubernetes/operators/pod.py | 32 +--
.../cncf/kubernetes/operators/spark_kubernetes.py | 11 +-
.../providers/cncf/kubernetes/pod_generator.py | 118 --------
.../cncf/kubernetes/pod_launcher_deprecated.py | 320 ---------------------
.../cncf/kubernetes/triggers/kubernetes_pod.py | 31 --
.../providers/cncf/kubernetes/triggers/pod.py | 23 +-
.../providers/cncf/kubernetes/utils/pod_manager.py | 23 +-
.../airflow/providers/dbt/cloud/operators/dbt.py | 4 +-
.../airflow/providers/docker/operators/docker.py | 21 +-
providers/src/airflow/providers/edge/CHANGELOG.rst | 8 +
providers/src/airflow/providers/edge/__init__.py | 2 +-
.../edge/plugins/templates/edge_worker_hosts.html | 12 +-
.../edge/plugins/templates/edge_worker_jobs.html | 14 +-
providers/src/airflow/providers/edge/provider.yaml | 2 +-
.../google/cloud/transfers/gcs_to_bigquery.py | 9 +-
.../providers/microsoft/azure/triggers/powerbi.py | 19 +-
.../src/airflow/providers/standard/__init__.py | 9 +
.../airflow/providers/standard/operators/python.py | 84 ++++--
.../src/airflow/providers/standard/provider.yaml | 4 +-
.../providers/standard/sensors/date_time.py | 23 +-
.../src/airflow/providers/standard/sensors/time.py | 23 +-
.../providers/standard/sensors/time_delta.py | 10 +-
.../aws/executors/batch/test_batch_executor.py | 4 +-
.../amazon/aws/executors/ecs/test_ecs_executor.py | 4 +-
providers/tests/amazon/aws/hooks/test_s3.py | 6 +
.../tests/amazon/aws/operators/test_athena.py | 2 +-
providers/tests/amazon/aws/operators/test_ecs.py | 2 +-
.../tests/amazon/aws/operators/test_glacier.py | 4 +-
providers/tests/amazon/aws/operators/test_sns.py | 2 +-
providers/tests/amazon/aws/operators/test_sqs.py | 2 +-
.../amazon/aws/operators/test_step_function.py | 4 +-
providers/tests/amazon/aws/sensors/test_ecs.py | 2 +-
providers/tests/amazon/aws/sensors/test_eks.py | 6 +-
providers/tests/amazon/aws/sensors/test_sqs.py | 2 +-
providers/tests/amazon/aws/triggers/test_sqs.py | 2 +-
providers/tests/amazon/aws/waiters/test_batch.py | 2 +-
providers/tests/amazon/aws/waiters/test_dynamo.py | 2 +-
providers/tests/amazon/aws/waiters/test_ecs.py | 2 +-
providers/tests/amazon/aws/waiters/test_emr.py | 2 +-
.../tests/amazon/aws/waiters/test_glue_databrew.py | 2 +-
providers/tests/amazon/aws/waiters/test_neptune.py | 2 +-
.../executors/test_kubernetes_executor.py | 7 +-
.../tests/cncf/kubernetes/models/test_secret.py | 5 +-
.../tests/cncf/kubernetes/operators/test_pod.py | 35 ---
.../kubernetes/operators/test_spark_kubernetes.py | 26 ++
.../kubernetes/test_kubernetes_helper_functions.py | 21 +-
.../tests/cncf/kubernetes/test_pod_generator.py | 105 +------
.../tests/cncf/kubernetes/triggers/test_pod.py | 1 -
.../cncf/kubernetes/utils/test_pod_manager.py | 16 --
providers/tests/common/sql/operators/test_sql.py | 1 -
providers/tests/dbt/cloud/operators/test_dbt.py | 78 +++++
providers/tests/docker/operators/test_docker.py | 32 ++-
.../google/cloud/transfers/test_gcs_to_bigquery.py | 25 ++
.../cloud/triggers/test_kubernetes_engine.py | 8 +-
.../tests/microsoft/azure/triggers/test_powerbi.py | 19 +-
providers/tests/openlineage/plugins/test_utils.py | 12 +-
providers/tests/openlineage/utils/test_utils.py | 5 +-
providers/tests/standard/operators/test_python.py | 182 ++++++++----
.../tests/standard/utils/test_python_virtualenv.py | 6 +-
pyproject.toml | 4 +-
scripts/ci/install_breeze.sh | 10 +-
scripts/ci/pre_commit/generate_airflow_diagrams.py | 14 +-
scripts/ci/pre_commit/update_installers.py | 9 +
scripts/docker/install_pipx_tools.sh | 43 ---
scripts/in_container/run_prepare_er_diagram.py | 15 +-
.../in_container/run_provider_yaml_files_check.py | 2 -
scripts/tools/setup_breeze | 30 +-
task_sdk/pyproject.toml | 5 +-
task_sdk/src/airflow/sdk/api/client.py | 76 ++++-
.../src/airflow/sdk/api/datamodels/_generated.py | 29 +-
task_sdk/src/airflow/sdk/api/datamodels/dagrun.py | 25 --
task_sdk/src/airflow/sdk/api/datamodels/ti.py | 5 +-
task_sdk/src/airflow/sdk/execution_time/comms.py | 6 +-
.../src/airflow/sdk/execution_time/supervisor.py | 65 +++--
.../src/airflow/sdk/execution_time/task_runner.py | 5 +-
task_sdk/tests/execution_time/test_supervisor.py | 88 ++++--
task_sdk/tests/execution_time/test_task_runner.py | 5 +-
tests/always/test_pandas.py | 4 +-
tests/always/test_project_structure.py | 4 -
.../{test_connection.py => test_connections.py} | 8 +-
.../execution_api/routes/test_variables.py | 77 +++++
tests/core/test_sqlalchemy_config.py | 5 +-
tests/dag_processing/test_dag_bundles.py | 178 ++++++++++++
tests/utils/test_trigger_rule.py | 3 +-
168 files changed, 2241 insertions(+), 1737 deletions(-)
copy airflow/api_fastapi/execution_api/routes/{connections.py => variables.py}
(64%)
copy airflow/{api_connexion => dag_processing/bundles}/__init__.py (100%)
create mode 100644 airflow/dag_processing/bundles/base.py
create mode 100644 airflow/dag_processing/bundles/git.py
rename
providers/src/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py =>
airflow/dag_processing/bundles/local.py (52%)
create mode 100644 airflow/ui/src/layouts/Nav/BrowseButton.tsx
create mode 100644 airflow/ui/src/pages/Events/Events.tsx
create mode 100644 airflow/ui/src/pages/Events/index.tsx
create mode 100644 dev/breeze/doc/adr/0016-use-uv-tool-to-install-breeze.md
rename docs/{apache-airflow/howto/operator =>
apache-airflow-providers-standard/operators}/bash.rst (96%)
rename docs/apache-airflow-providers-standard/{operators.rst =>
operators/datetime.rst} (89%)
copy docs/{apache-airflow/howto/operator =>
apache-airflow-providers-standard/operators}/index.rst (72%)
rename docs/{apache-airflow/howto/operator =>
apache-airflow-providers-standard/operators}/python.rst (95%)
copy docs/{apache-airflow/howto/operator/index.rst =>
apache-airflow-providers-standard/sensors/bash.rst} (65%)
rename docs/apache-airflow-providers-standard/{sensors.rst =>
sensors/datetime.rst} (85%)
copy docs/{apache-airflow/howto/operator =>
apache-airflow-providers-standard/sensors}/index.rst (72%)
create mode 100644 docs/apache-airflow-providers-standard/sensors/python.rst
create mode 100644 newsfragments/43368.significant.rst
delete mode 100644
providers/src/airflow/providers/cncf/kubernetes/pod_launcher_deprecated.py
delete mode 100644
providers/src/airflow/providers/cncf/kubernetes/triggers/kubernetes_pod.py
delete mode 100644 scripts/docker/install_pipx_tools.sh
delete mode 100644 task_sdk/src/airflow/sdk/api/datamodels/dagrun.py
rename tests/api_fastapi/execution_api/routes/{test_connection.py =>
test_connections.py} (92%)
create mode 100644 tests/api_fastapi/execution_api/routes/test_variables.py
create mode 100644 tests/dag_processing/test_dag_bundles.py
