This is an automated email from the ASF dual-hosted git repository.
jason810496 pushed a change to branch
jason/lang-sdk-e2e/03c-coordinator-dag-parsing
in repository https://gitbox.apache.org/repos/asf/airflow.git
omit 1e177e60233 Break the new dag_bundle_name docs only at sentence ends
omit 122154d8737 Say which failures fall back to a Python file in ADR-0010
omit eafec25b77c Add a newsfragment for the removed coordinator artifact
roots
omit 8364bb2737d Describe how a Dag importer finds its coordinator in
ADR-0010
omit fb6c4bfa6a8 Say dag_bundle_name only names the stub task artifact
bundle
omit 58aade928ed Find a Dag importer's coordinator from its class
omit 0022eedf09d Explain dag_bundle_name by the coordinator that parses a
Dag file
omit 95a9ed9f3d1 Hand out a coordinator's Dag importer through
get_dag_importer alone
omit ce8aac3144b Say dag_bundle_name is only needed for two coordinators of
one class
omit d38901621ad Drop serves_bundle so get_parsed_bundles is the only answer
omit 68a619133ec Check Dag file claims per bundle on the parse path
omit 90e3aad2a39 Import the coordinator manager at the top of the importer
base
omit 82fa8626f8d Let a coordinator parse the Dag files of the bundles it
serves
omit a7efc7c851c Report a schema error's path the same way under every
jsonschema version
omit 2bcf0508832 Check a serialized Dag as the SDK Dag is checked and
return it
omit e2b4e2972c7 Sync the Go SDK copy of the Dag schema
omit e840e922684 Sync the Java SDK copy of the Dag schema
omit ac2244adbfb Serialize the SDK Dag in tests that write a dag_maker Dag
omit 973cf03f119 Note in the parsing ADR that runtimes may omit
config-backed fields
omit 6158ada8ab3 Check SDK Dags with the Dag processor's validation in
conformance
omit 8636b093e38 Reject malformed task entries in a serialized Dag
omit 9ef13d3ca8c Fill a serialized Dag's unset settings from the Airflow
config
omit b5e58c96bb4 Check that a serialized Dag can be stored and loaded
add 803cc82370d Update supported Kubernetes versions (#73798)
add 4a73cb7d5fa Keep builtin annotations from resolving to project
attributes in Sphinx 9 (#74168)
add 79bc6439a9c Always clear the session cookie on logout (#73788)
add a1b02e26f3d Support Python patch exclusions in provider metadata
(#73598)
add bdfc5555305 Add configurable request timeout for the Airbyte API
(#71092)
add b2c215cc037 Fix Beam async hook launching pipelines through a shell
(#72510)
add d2883eead5c Fix Cloud SQL 409 operationInProgress on import/export
operations (#68361)
add c688a6f0abc Add breeze verify to list local checks for changed files
(#73241)
add 07c9ceba527 Fail provider DB test jobs when their tests fail (#74172)
add 043f97c2b00 Test get_async_extra_dejson against the real supervisor
comms (#74166)
add 592cf3ffdd2 Use the current user in the HITL detail tests instead of
hardcoding (#74184)
add 8331bb485c5 Read the msgraph connection extra with
get_async_extra_dejson (#74161)
add b156582855c Fix OpenSearch provider log pagination (#73947)
add c8b136358ce Allow OpensearchTaskHandler and OpensearchRemoteLogIO to
take empty username and password (#71692)
add cc94831e213 Fix docstring cross-references that Sphinx 9 cannot
resolve (#74169)
add b58cfa912a9 Add an API endpoint and UI tab comparing two stored Dag
versions (#73322)
add d1d5b1af26e Adapt GCP BigtableHook to run in private cloud (#73181)
add b3674d43be0 `SparkSubmitHook`: Mask `_mask_cmd` secrets in linear time
(#71711)
add 8d5814c09d5 Bump the react group across 1 directory with 4 updates
(#73334)
add 7e448f70af5 Use async database access for Variable key listing (#73407)
add 523567e47d0 Add tests for Google Go module subprocess utility (#73268)
add 4a766db2b03 Bump the typescript group across 1 directory with 4
updates (#72994)
add 7fad35d9c99 Add deadline alert coverage to Dag endpoint tests (#72296)
add 6f5d5d06b17 Align standard provider tests with source paths (#73267)
add c2f600f6ec6 Add airflowctl tasks state command (#71206)
add 86b8a521837 Bind the AWS auth manager SAML response to the browser
that started the login (#73698)
add cefc0c3239d Bump the java-sdk-dependency-updates group across 1
directory with 3 updates (#74108)
add 2a274ab3c93 Refuse nested secret ids looked up with no team in
multi-team mode (#73789)
add 8055f6a8717 Skip creating an empty hook lineage collector for
OpenLineage events (#74190)
add 38fefc1ca95 Fix invalid cron expression in OpenLineage system test Dag
(#74187)
add 27f827fcdce Honor the polling interval for deferred Kubernetes jobs
(#74143)
add 3bd4df6b2da Fix Snowflake SQL API wait_for_query timing out on
finished queries (#73761)
add f74a685978c Use configured default queue for Celery executor callbacks
(#73552)
add ad3bb1f676c Replace coordinator artifact roots with
task_handler_bundle_name (#73970)
add 1c3f78e1d57 Validate a serialized Dag and fill its unset settings from
the Airflow config (#74041)
add e004a3b2173 Let a Dag importer find the coordinator that parses its
files
add aef23367721 Describe how a Dag importer finds its coordinator in
ADR-0010
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (1e177e60233)
\
N -- N -- N
refs/heads/jason/lang-sdk-e2e/03c-coordinator-dag-parsing (aef23367721)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.agents/skills/airflow-java-sdk/SKILL.md | 10 +-
.agents/skills/airflow-publish-changes/SKILL.md | 2 +-
AGENTS.md | 1 +
README.md | 16 +-
.../adr/lang-sdk/0010-native-dag-processing.md | 11 +-
.../0013-persisted-task-handler-bindings.md | 2 +-
.../authoring-and-scheduling/language-sdks/go.rst | 93 ++--
.../language-sdks/index.rst | 6 +-
.../language-sdks/java.rst | 113 ++--
.../language-sdks/typescript.rst | 88 +--
airflow-core/docs/installation/prerequisites.rst | 2 +-
airflow-core/docs/security/api_permissions_ref.rst | 4 +
airflow-core/docs/security/security_model.rst | 9 +
airflow-core/newsfragments/73970.significant.rst | 5 +
airflow-core/newsfragments/74042.significant.rst | 8 -
.../src/airflow/api_fastapi/auth/tokens.py | 24 +-
.../core_api/datamodels/dag_versions.py | 158 ++++++
.../core_api/openapi/v2-rest-api-generated.yaml | 273 ++++++++++
.../api_fastapi/core_api/routes/public/auth.py | 9 +-
.../core_api/routes/public/dag_versions.py | 98 +++-
.../api_fastapi/execution_api/routes/variables.py | 10 +-
.../src/airflow/config_templates/config.yml | 19 +-
airflow-core/src/airflow/provider.yaml.schema.json | 5 +-
.../src/airflow/serialization/dag_version_diff.py | 3 +-
.../src/airflow/ui/openapi-gen/queries/common.ts | 18 +
.../ui/openapi-gen/queries/ensureQueryData.ts | 38 ++
.../src/airflow/ui/openapi-gen/queries/prefetch.ts | 38 ++
.../src/airflow/ui/openapi-gen/queries/queries.ts | 38 ++
.../src/airflow/ui/openapi-gen/queries/suspense.ts | 38 ++
.../airflow/ui/openapi-gen/requests/schemas.gen.ts | 188 +++++++
.../ui/openapi-gen/requests/services.gen.ts | 72 ++-
.../airflow/ui/openapi-gen/requests/types.gen.ts | 143 +++++
airflow-core/src/airflow/ui/package.json | 16 +-
airflow-core/src/airflow/ui/pnpm-lock.yaml | 591 +++++++++------------
.../ui/public/i18n/locales/en/components.json | 1 +
.../src/airflow/ui/public/i18n/locales/en/dag.json | 63 +++
.../src/components/VersionCompareSelect.test.tsx | 82 +++
.../Code => components}/VersionCompareSelect.tsx | 4 +-
.../src/airflow/ui/src/constants/searchParams.ts | 3 +
airflow-core/src/airflow/ui/src/constants/tab.ts | 1 +
.../src/airflow/ui/src/pages/Dag/Code/Code.tsx | 2 +-
airflow-core/src/airflow/ui/src/pages/Dag/Dag.tsx | 3 +-
.../pages/Dag/Versions/VersionDiff.i18n.test.tsx | 83 +++
.../ui/src/pages/Dag/Versions/VersionDiff.test.tsx | 275 ++++++++++
.../ui/src/pages/Dag/Versions/VersionDiff.tsx | 197 +++++++
.../ui/src/pages/Dag/Versions/Versions.test.tsx | 227 ++++++++
.../airflow/ui/src/pages/Dag/Versions/Versions.tsx | 182 +++++++
.../Details/Gantt => pages/Dag/Versions}/index.ts | 3 +-
airflow-core/src/airflow/ui/src/router.tsx | 6 +
.../tests/unit/always/test_project_structure.py | 3 -
.../tests/unit/always/test_provider_yaml_schema.py | 10 +
.../tests/unit/api_fastapi/auth/test_tokens.py | 43 ++
.../core_api/routes/public/test_auth.py | 45 +-
.../core_api/routes/public/test_dag_versions.py | 296 +++++++++++
.../core_api/routes/public/test_dags.py | 175 +++++-
.../core_api/routes/public/test_hitl.py | 3 +-
.../airflowctl_tests/test_airflowctl_commands.py | 2 +
airflow-ctl/docs/images/command_hashes.txt | 2 +-
airflow-ctl/docs/images/output_tasks.svg | 78 +--
.../src/airflowctl/api/datamodels/generated.py | 157 ++++++
airflow-ctl/src/airflowctl/ctl/cli_config.py | 17 +
.../src/airflowctl/ctl/commands/task_command.py | 53 +-
.../airflow_ctl/ctl/commands/test_task_command.py | 275 +++++++++-
airflow-e2e-tests/docker/go.yml | 9 +-
airflow-e2e-tests/docker/java.yml | 13 +-
airflow-e2e-tests/docker/ts.yml | 9 +-
.../tests/airflow_e2e_tests/conftest.py | 84 ++-
chart/README.md | 2 +-
contributing-docs/08_static_code_checks.rst | 1 +
contributing-docs/30_new_language_sdk.rst | 7 +-
dev/breeze/doc/03_developer_tasks.rst | 55 ++
dev/breeze/doc/ci/04_selective_checks.md | 5 +
dev/breeze/doc/images/output-commands.svg | 68 +--
.../doc/images/output_k8s_configure-cluster.svg | 8 +-
.../doc/images/output_k8s_configure-cluster.txt | 2 +-
.../doc/images/output_k8s_create-cluster.svg | 8 +-
.../doc/images/output_k8s_create-cluster.txt | 2 +-
.../doc/images/output_k8s_delete-cluster.svg | 4 +-
.../doc/images/output_k8s_delete-cluster.txt | 2 +-
.../doc/images/output_k8s_deploy-airflow.svg | 8 +-
.../doc/images/output_k8s_deploy-airflow.txt | 2 +-
.../doc/images/output_k8s_deploy-cluster.svg | 4 +-
.../doc/images/output_k8s_deploy-cluster.txt | 2 +-
dev/breeze/doc/images/output_k8s_dev.svg | 4 +-
dev/breeze/doc/images/output_k8s_dev.txt | 2 +-
dev/breeze/doc/images/output_k8s_k9s.svg | 4 +-
dev/breeze/doc/images/output_k8s_k9s.txt | 2 +-
dev/breeze/doc/images/output_k8s_logs.svg | 4 +-
dev/breeze/doc/images/output_k8s_logs.txt | 2 +-
.../doc/images/output_k8s_run-complete-tests.svg | 8 +-
.../doc/images/output_k8s_run-complete-tests.txt | 2 +-
.../doc/images/output_k8s_setup-lang-sdk-test.svg | 4 +-
.../doc/images/output_k8s_setup-lang-sdk-test.txt | 2 +-
dev/breeze/doc/images/output_k8s_shell.svg | 4 +-
dev/breeze/doc/images/output_k8s_shell.txt | 2 +-
.../doc/images/output_k8s_smoke-test-overlay.svg | 4 +-
.../doc/images/output_k8s_smoke-test-overlay.txt | 2 +-
dev/breeze/doc/images/output_k8s_status.svg | 4 +-
dev/breeze/doc/images/output_k8s_status.txt | 2 +-
dev/breeze/doc/images/output_k8s_tests.svg | 8 +-
dev/breeze/doc/images/output_k8s_tests.txt | 2 +-
.../doc/images/output_k8s_upload-k8s-image.svg | 8 +-
.../doc/images/output_k8s_upload-k8s-image.txt | 2 +-
.../output_setup_check-all-params-in-groups.svg | 4 +-
.../output_setup_check-all-params-in-groups.txt | 2 +-
.../output_setup_regenerate-command-images.svg | 4 +-
.../output_setup_regenerate-command-images.txt | 2 +-
.../doc/images/output_testing_helm-tests.svg | 4 +-
.../doc/images/output_testing_helm-tests.txt | 2 +-
dev/breeze/doc/images/output_verify.svg | 128 +++++
dev/breeze/doc/images/output_verify.txt | 1 +
dev/breeze/src/airflow_breeze/breeze.py | 1 +
.../commands/developer_commands_config.py | 11 +
.../airflow_breeze/commands/testing_commands.py | 10 +
.../src/airflow_breeze/commands/verify_commands.py | 180 +++++++
dev/breeze/src/airflow_breeze/global_constants.py | 16 +-
.../src/airflow_breeze/utils/reproduce_ci.py | 10 +-
.../src/airflow_breeze/utils/verification_plan.py | 170 ++++++
dev/breeze/tests/test_packages.py | 27 +
dev/breeze/tests/test_reproduce_ci.py | 12 +
dev/breeze/tests/test_run_test_args.py | 35 +-
dev/breeze/tests/test_selective_checks.py | 15 +-
dev/breeze/tests/test_verification_plan.py | 244 +++++++++
dev/breeze/tests/test_verify_commands.py | 327 ++++++++++++
dev/skill-evals/last-eval-hash.txt | 2 +-
devel-common/src/docs/utils/conf_constants.py | 1 +
.../src/sphinx_exts/python_builtin_xrefs.py | 66 +++
.../unit/sphinx_exts/test_python_builtin_xrefs.py | 92 ++++
docs/spelling_wordlist.txt | 3 +
generated/PYPI_README.md | 16 +-
generated/known_airflow_exceptions.txt | 4 +-
go-sdk/README.md | 37 +-
go-sdk/example/bundle/Justfile | 3 +-
java-sdk/README.md | 31 +-
java-sdk/buildSrc/build.gradle.kts | 2 +-
java-sdk/gradle/verification-metadata.xml | 410 +++++++-------
.../apache/airflow/sdk/plugin/AirflowSdkPlugin.kt | 4 +-
java-sdk/scala_spark_example/README.md | 17 +-
java-sdk/sdk/build.gradle.kts | 4 +-
kubernetes-tests/lang_sdk/README.md | 14 +-
kubernetes-tests/lang_sdk/config/values.yaml | 12 +-
.../lang_sdk/pod_templates/lang_sdk_golang.yaml | 3 +-
.../lang_sdk/pod_templates/lang_sdk_java.yaml | 5 +-
kubernetes-tests/lang_sdk/stage_artifacts.py | 8 +-
providers/MANAGING_PROVIDERS_LIFECYCLE.rst | 20 +
providers/airbyte/docs/connections.rst | 10 +-
.../src/airflow/providers/airbyte/hooks/airbyte.py | 37 +-
.../tests/unit/airbyte/hooks/test_airbyte.py | 61 +++
providers/akeyless/docs/changelog.rst | 7 +
.../airflow/providers/akeyless/secrets/akeyless.py | 28 +-
.../tests/unit/akeyless/secrets/test_akeyless.py | 27 +-
.../docs/auth-manager/setup/identity-center.rst | 36 ++
providers/amazon/provider.yaml | 15 +
.../providers/amazon/aws/auth_manager/constants.py | 1 +
.../amazon/aws/auth_manager/routes/login.py | 200 ++++++-
.../providers/amazon/aws/executors/batch/utils.py | 6 +-
.../airflow/providers/amazon/get_provider_info.py | 7 +
.../amazon/aws/auth_manager/routes/test_login.py | 359 +++++++++++--
.../airflow/providers/apache/beam/hooks/beam.py | 45 +-
.../beam/tests/unit/apache/beam/hooks/test_beam.py | 42 ++
.../providers/apache/spark/hooks/spark_submit.py | 98 +++-
.../unit/apache/spark/hooks/test_spark_submit.py | 118 ++++
.../providers/celery/executors/celery_executor.py | 4 +-
.../unit/celery/executors/test_celery_executor.py | 29 +-
.../providers/cncf/kubernetes/triggers/job.py | 4 +-
.../unit/cncf/kubernetes/triggers/test_job.py | 13 +-
.../connection/test_connection_supervisor.py | 214 ++++++++
providers/fab/docs/auth-manager/access-control.rst | 4 +
.../providers/google/cloud/hooks/bigtable.py | 24 +
.../providers/google/cloud/hooks/cloud_sql.py | 57 +-
.../providers/google/cloud/hooks/compute.py | 3 -
.../google/cloud/hooks/vertex_ai/auto_ml.py | 2 +-
.../tests/unit/google/cloud/hooks/test_bigtable.py | 26 +
.../unit/google/cloud/hooks/test_cloud_sql.py | 78 ++-
.../tests/unit/google/test_go_module_utils.py | 73 +++
providers/hashicorp/docs/changelog.rst | 7 +
.../airflow/providers/hashicorp/secrets/vault.py | 31 +-
.../tests/unit/hashicorp/secrets/test_vault.py | 55 ++
providers/microsoft/azure/pyproject.toml | 2 +-
.../providers/microsoft/azure/hooks/msgraph.py | 13 +-
.../unit/microsoft/azure/hooks/test_msgraph.py | 17 +
.../providers/openlineage/extractors/manager.py | 31 ++
...schedule_multiple_cron_trigger_timetable_dag.py | 2 +-
...hedule_multiple_cron_trigger_timetable_dag.json | 12 +-
.../unit/openlineage/extractors/test_manager.py | 71 ++-
.../providers/opensearch/log/os_task_handler.py | 32 +-
.../opensearch/log/test_os_remote_log_io.py | 18 +
.../unit/opensearch/log/test_os_task_handler.py | 102 +++-
.../providers/snowflake/hooks/snowflake_sql_api.py | 12 +-
.../unit/snowflake/hooks/test_snowflake_sql_api.py | 35 +-
.../tests/unit/standard/operators/test_empty.py | 56 ++
...ternal_task_sensor.py => test_external_task.py} | 32 ++
.../prek/check_contextmanager_class_decorators.py | 1 +
scripts/ci/prek/check_excluded_provider_markers.py | 36 +-
scripts/ci/prek/update_airflow_pyproject_toml.py | 35 +-
scripts/ci/testing/run_unit_tests.sh | 7 +-
.../test_check_contextmanager_class_decorators.py | 80 +++
.../prek/test_check_excluded_provider_markers.py | 56 ++
.../ci/prek/test_update_airflow_pyproject_toml.py | 58 ++
scripts/tests/ci/testing/test_run_unit_tests.py | 15 +
task-sdk/docs/deferred-vs-async-operators.rst | 2 +-
task-sdk/docs/executable-bundle-spec.rst | 31 +-
task-sdk/docs/ts-bundle-spec.rst | 25 +-
.../src/airflow/sdk/coordinators/_subprocess.py | 57 +-
.../sdk/coordinators/executable/coordinator.py | 12 +-
.../airflow/sdk/coordinators/java/coordinator.py | 17 +-
.../airflow/sdk/coordinators/node/coordinator.py | 16 +-
.../src/airflow/sdk/execution_time/coordinator.py | 21 +-
.../coordinators/executable/test_coordinator.py | 49 +-
.../task_sdk/coordinators/java/test_coordinator.py | 78 +--
.../task_sdk/coordinators/node/test_coordinator.py | 15 +-
.../tests/task_sdk/coordinators/test_subprocess.py | 140 ++---
.../task_sdk/execution_time/test_coordinator.py | 85 ++-
ts-sdk/README.md | 24 +-
ts-sdk/example/README.md | 17 +-
215 files changed, 8261 insertions(+), 1538 deletions(-)
create mode 100644 airflow-core/newsfragments/73970.significant.rst
delete mode 100644 airflow-core/newsfragments/74042.significant.rst
create mode 100644
airflow-core/src/airflow/ui/src/components/VersionCompareSelect.test.tsx
rename airflow-core/src/airflow/ui/src/{pages/Dag/Code =>
components}/VersionCompareSelect.tsx (97%)
create mode 100644
airflow-core/src/airflow/ui/src/pages/Dag/Versions/VersionDiff.i18n.test.tsx
create mode 100644
airflow-core/src/airflow/ui/src/pages/Dag/Versions/VersionDiff.test.tsx
create mode 100644
airflow-core/src/airflow/ui/src/pages/Dag/Versions/VersionDiff.tsx
create mode 100644
airflow-core/src/airflow/ui/src/pages/Dag/Versions/Versions.test.tsx
create mode 100644
airflow-core/src/airflow/ui/src/pages/Dag/Versions/Versions.tsx
copy airflow-core/src/airflow/ui/src/{layouts/Details/Gantt =>
pages/Dag/Versions}/index.ts (95%)
create mode 100644 dev/breeze/doc/images/output_verify.svg
create mode 100644 dev/breeze/doc/images/output_verify.txt
create mode 100644 dev/breeze/src/airflow_breeze/commands/verify_commands.py
create mode 100644 dev/breeze/src/airflow_breeze/utils/verification_plan.py
create mode 100644 dev/breeze/tests/test_verification_plan.py
create mode 100644 dev/breeze/tests/test_verify_commands.py
create mode 100644 devel-common/src/sphinx_exts/python_builtin_xrefs.py
create mode 100644
devel-common/tests/unit/sphinx_exts/test_python_builtin_xrefs.py
create mode 100644
providers/common/compat/tests/unit/common/compat/connection/test_connection_supervisor.py
create mode 100644 providers/google/tests/unit/google/test_go_module_utils.py
create mode 100644
providers/standard/tests/unit/standard/operators/test_empty.py
rename
providers/standard/tests/unit/standard/sensors/{test_external_task_sensor.py =>
test_external_task.py} (98%)
create mode 100644
scripts/tests/ci/prek/test_check_contextmanager_class_decorators.py
create mode 100644 scripts/tests/ci/prek/test_update_airflow_pyproject_toml.py
