This is an automated email from the ASF dual-hosted git repository.
jason810496 pushed a change to branch jason/lang-sdk-e2e/06-ts-sdk
in repository https://gitbox.apache.org/repos/asf/airflow.git
omit e90bb77c47c TS SDK: add triggerDagRun to the native Dag surface
omit 000db486bf0 TS SDK: add multi-way branching with dag.switch
omit 027941fcfff TS SDK: add conditional branching with dag.if
omit f46373341c6 TS SDK: answer the Dag-parsing request from bundle.serve
omit 70c3ba277d3 Skip the Python-only parse checks for non-Python Dag files
omit c407fabe7d0 Queue the Dag files of non-Python Dag importers
omit c7a9cf1c7ef Share the bundle section mutation helper between Node tests
omit 6f1e4654d65 Document parsing native TypeScript Dags
omit 0fe02d2a986 Parse native TypeScript Dags with the Node coordinator
omit fce2457872a Name the JAR manifest attributes the Java coordinator reads
omit d5472efdd1b Test the Java Dag source placeholder for a JAR without a
manifest
omit 06614930fdf Treat an unreadable JAR like one without a manifest
omit 431f4fb1411 Document native Java Dag parsing
omit c5a3fda12a1 Parse native Java Dags with their coordinator
omit 8e4f054dab6 Embed the Dag source in Java bundle JARs
omit 179b4d0bcd0 Build the JVM command that parses a Dag JAR
omit dd102276a33 Read JAR manifests per the JAR specification
omit e0f332c645f Parse coordinator-claimed Dag files with their runtime in
a Dag bag
omit 1610ac78b41 Report a native Dag's task that reaches a Python worker
omit e07e9050515 Report a Lang-SDK parse past its import timeout as an
import error
omit fa2b6523f71 Parse coordinator-claimed Dag files with their runtime in
the Dag processor
omit 66fe7d5fc96 Let a coordinator parse the Dag files of the bundles it
serves
omit e923e5d0aa7 Fill a serialized Dag's unset settings from the Airflow
config
omit d79ced881bf Check that a serialized Dag can be stored and loaded
omit cef0df530ff Move the Dag file processor's shared plumbing into a base
class
omit 5c2018283cf Share Dag cycle detection between the Task SDK and core
add e9c552cc827 Add a vendor-neutral managed-agent hook contract to Common
AI (#73532)
add a33fe27cace Add async client to `AnthropicHook` (#73967)
add e6a9de5d832 Go SDK: bring the coordinator-protocol models back to the
supervisor schema (#73963)
add 27cd88ca74c Cache repeated agent prompts by default in AgentOperator
(#73994)
add 529f429400a Go SDK: reject a task handler and a Dag that share a
dag_id (#73876)
add bc9ca5b1227 Correctly shutdown async sessions on exit. (#73838)
add f1b36db6093 Back off between GitDagBundle bare clone attempts so a
fresh GitHub App token can propagate (#73878)
add 6b847411de1 Stop rebuilding the FAB app on every users and roles API
call (#73104)
add 56817294cdd Go SDK: add airflow.TriggerDagRun, a task that triggers a
Dag run (#74003)
add b8475094703 Fix go-sdk generated drift (#74028)
add 8f3e8466c67 Support Airflow 2.11 in the Common AI provider (#73991)
add f0cde671827 Use "Airflow versions" in user-facing docs, not "cores"
(#74045)
add 0bb09cb6ba5 Close the providers testing issue right after announcing
the release (#73890)
add 43377466a06 Release the DuckDB provider as 0.2.0 (#73888)
add 53e43f29130 Speed up CI image restores with zstd-compressed stashes
(#73997)
add 541001b9ee1 Share Dag cycle detection between the Task SDK and core
add 0b47cba3704 Move the Dag file processor's shared plumbing into a base
class
add d67fee105db Check that a serialized Dag can be stored and loaded
add 35704801174 Fill a serialized Dag's unset settings from the Airflow
config
add 107f1eb6dc0 Let a coordinator parse the Dag files of the bundles it
serves
add d2e4f162d06 Parse coordinator-claimed Dag files with their runtime in
the Dag processor
add 5bb5f71ce57 Report a Lang-SDK parse past its import timeout as an
import error
add cd38d3392b0 Report a native Dag's task that reaches a Python worker
add 64b7effa3c6 Parse coordinator-claimed Dag files with their runtime in
a Dag bag
add 5c0e91834a0 Read JAR manifests per the JAR specification
add 9b2dce5c692 Build the JVM command that parses a Dag JAR
add 60079e4c40b Embed the Dag source in Java bundle JARs
add 7436b6119fb Parse native Java Dags with their coordinator
add d2327ded717 Document native Java Dag parsing
add 55376f47ed9 Treat an unreadable JAR like one without a manifest
add dc36a40e453 Test the Java Dag source placeholder for a JAR without a
manifest
add 5a8f7c401db Name the JAR manifest attributes the Java coordinator reads
add a380e5ce397 Parse native TypeScript Dags with the Node coordinator
add 20a2b6c234f Document parsing native TypeScript Dags
add 9cbd149a395 Share the bundle section mutation helper between Node tests
add b2cc2e3daf8 Queue the Dag files of non-Python Dag importers
add e780de44a40 Skip the Python-only parse checks for non-Python Dag files
add b6b049b9b04 TS SDK: answer the Dag-parsing request from bundle.serve
add 5d20a5532ae TS SDK: add conditional branching with dag.if
add 028de33a23a TS SDK: add multi-way branching with dag.switch
add 6009f31fa08 TS SDK: add triggerDagRun to the native Dag surface
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (e90bb77c47c)
\
N -- N -- N refs/heads/jason/lang-sdk-e2e/06-ts-sdk (6009f31fa08)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.agents/skills/airflow-java-sdk/SKILL.md | 11 +-
.github/workflows/ci-image-build.yml | 8 +
.pre-commit-config.yaml | 12 +-
.../authoring-and-scheduling/language-sdks/go.rst | 61 ++--
.../language-sdks/java.rst | 80 ++--
.../language-sdks/typescript.rst | 76 ++--
airflow-core/newsfragments/68496.significant.rst | 4 +-
airflow-core/src/airflow/api_fastapi/app.py | 2 +
.../src/airflow/api_fastapi/execution_api/app.py | 2 +
.../src/airflow/config_templates/config.yml | 10 +-
airflow-core/src/airflow/settings.py | 14 +-
.../api_fastapi/auth/managers/simple/conftest.py | 3 +-
airflow-core/tests/unit/api_fastapi/conftest.py | 27 +-
.../core_api/routes/public/test_auth.py | 28 +-
.../core_api/routes/public/test_backfills.py | 20 +-
.../core_api/routes/public/test_connections.py | 9 +-
.../core_api/routes/public/test_dag_bundles.py | 43 ++-
.../core_api/routes/public/test_dag_parsing.py | 18 +-
.../core_api/routes/public/test_dag_run.py | 78 ++--
.../core_api/routes/public/test_task_instances.py | 111 +++---
.../unit/api_fastapi/execution_api/test_app.py | 84 ++++-
airflow-core/tests/unit/api_fastapi/test_app.py | 108 +++++-
airflow-core/tests/unit/core/test_settings.py | 120 ++++++
airflow-core/tests/unit/state/test_metastore.py | 9 +
airflow-core/tests/unit/utils/test_session.py | 18 +-
airflow-e2e-tests/docker/go.yml | 9 +-
airflow-e2e-tests/docker/java.yml | 13 +-
airflow-e2e-tests/docker/ts.yml | 9 +-
.../tests/airflow_e2e_tests/conftest.py | 54 ++-
.../airflow_e2e_tests/openlineage_tests/README.md | 2 +-
.../dags_extra/e2e_versioned_bundle.py | 4 +-
contributing-docs/30_new_language_sdk.rst | 7 +-
dev/README_RELEASE_PROVIDERS.md | 25 +-
.../airflow_breeze/commands/testing_commands.py | 2 +-
dev/breeze/src/airflow_breeze/global_constants.py | 2 +-
dev/breeze/tests/test_selective_checks.py | 16 +-
docs/spelling_wordlist.txt | 2 +
go-sdk/README.md | 31 +-
go-sdk/airflow/bundle.go | 46 ++-
go-sdk/airflow/bundle_test.go | 111 ++++++
go-sdk/airflow/dag.go | 79 +++-
go-sdk/airflow/dag_test.go | 6 +
go-sdk/airflow/inputs.go | 6 +
go-sdk/airflow/spec.gen.go | 3 +-
go-sdk/airflow/trigger_dag_run.go | 177 +++++++++
go-sdk/airflow/trigger_dag_run_test.go | 272 ++++++++++++++
go-sdk/example/bundle/Justfile | 4 +-
go-sdk/internal/genspec/authoring.go | 2 +-
.../pkg/execution/genmodels/discriminators.gen.go | 4 +
go-sdk/pkg/execution/genmodels/models.gen.go | 171 +++++----
java-sdk/README.md | 18 +-
.../apache/airflow/sdk/plugin/AirflowSdkPlugin.kt | 4 +-
java-sdk/scala_spark_example/README.md | 15 +-
kubernetes-tests/lang_sdk/README.md | 2 +-
kubernetes-tests/lang_sdk/config/values.yaml | 12 +-
.../lang_sdk/pod_templates/lang_sdk_golang.yaml | 2 +-
.../lang_sdk/pod_templates/lang_sdk_java.yaml | 4 +-
kubernetes-tests/lang_sdk/stage_artifacts.py | 6 +-
providers/amazon/docs/index.rst | 2 +
providers/amazon/docs/operators/bedrock.rst | 40 ++
providers/amazon/pyproject.toml | 4 +
.../airflow/providers/amazon/aws/hooks/bedrock.py | 229 +++++++++++-
.../tests/unit/amazon/aws/hooks/test_bedrock.py | 280 ++++++++++++++
providers/anthropic/pyproject.toml | 2 +-
.../airflow/providers/anthropic/hooks/anthropic.py | 99 ++++-
.../tests/unit/anthropic/hooks/test_anthropic.py | 141 ++++++-
providers/common/ai/README.rst | 3 +-
providers/common/ai/docs/approval_gates.rst | 2 +-
providers/common/ai/docs/index.rst | 5 +-
providers/common/ai/docs/installation.rst | 33 +-
providers/common/ai/docs/observability.rst | 3 +
providers/common/ai/docs/operators/agent.rst | 98 ++++-
providers/common/ai/docs/operators/llm_batch.rst | 4 +-
providers/common/ai/docs/operators/llm_branch.rst | 2 +-
providers/common/ai/docs/quickstart.rst | 3 +-
providers/common/ai/docs/self_hosted_models.rst | 3 +-
providers/common/ai/docs/toolsets/index.rst | 35 +-
.../common/ai/docs/toolsets/managed_agent.rst | 337 +++++++++--------
providers/common/ai/docs/troubleshooting.rst | 8 +-
providers/common/ai/pyproject.toml | 7 +-
.../ai/src/airflow/providers/common/ai/__init__.py | 4 +-
.../airflow/providers/common/ai/batch/anthropic.py | 5 +-
.../airflow/providers/common/ai/batch/openai.py | 5 +-
.../airflow/providers/common/ai/batch/results.py | 5 +-
.../providers/common/ai/decorators/agent.py | 2 +-
.../airflow/providers/common/ai/decorators/llm.py | 2 +-
.../providers/common/ai/decorators/llm_batch.py | 2 +-
.../providers/common/ai/decorators/llm_branch.py | 2 +-
.../common/ai/decorators/llm_file_analysis.py | 2 +-
.../common/ai/decorators/llm_schema_compare.py | 2 +-
.../providers/common/ai/decorators/llm_sql.py | 2 +-
.../providers/common/ai/durable/caching_model.py | 4 +-
.../providers/common/ai/durable/caching_toolset.py | 17 +-
.../providers/common/ai/durable/fingerprint.py | 12 +-
.../airflow/providers/common/ai/durable/storage.py | 7 +-
.../common/ai/durable/task_state_store.py | 4 +-
.../src/airflow/providers/common/ai/exceptions.py | 15 +-
.../providers/common/ai/hooks/pydantic_ai.py | 10 +
.../providers/common/ai/managed_agents/__init__.py | 49 +++
.../providers/common/ai/managed_agents/base.py | 251 +++++++++++++
.../providers/common/ai/managed_agents/failover.py | 145 ++++++++
.../airflow/providers/common/ai/mixins/approval.py | 6 +-
.../airflow/providers/common/ai/observability.py | 19 +-
.../airflow/providers/common/ai/operators/agent.py | 54 ++-
.../common/ai/operators/llamaindex_embedding.py | 2 +-
.../common/ai/operators/llamaindex_retrieval.py | 2 +-
.../airflow/providers/common/ai/operators/llm.py | 17 +-
.../providers/common/ai/toolsets/__init__.py | 4 +-
.../providers/common/ai/toolsets/managed_agent.py | 354 +++++++++---------
.../airflow/providers/common/ai/utils/logging.py | 18 +-
.../providers/common/ai/utils/prompt_cache.py | 140 +++++++
.../providers/common/ai/utils/task_logger.py | 67 ++++
.../providers/common/ai/utils/usage_budget.py | 11 +-
.../ai/tests/unit/common/ai/batch/test_dispatch.py | 2 +-
.../ai/tests/unit/common/ai/batch/test_results.py | 2 +-
.../ai/tests/unit/common/ai/batch/test_state.py | 2 +-
.../common/ai/tests/unit/common/ai/conftest.py | 11 +-
.../tests/unit/common/ai/decorators/test_agent.py | 5 +-
.../unit/common/ai/decorators/test_llm_batch.py | 3 +-
.../common/ai/decorators/test_llm_file_analysis.py | 5 +-
.../ai/tests/unit/common/ai/durable/test_base.py | 2 +-
.../unit/common/ai/durable/test_caching_toolset.py | 55 +++
.../unit/common/ai/durable/test_fingerprint.py | 20 +
.../unit/common/ai/durable/test_replay_cost.py | 2 +-
.../common/ai/durable/test_replay_verification.py | 2 +-
.../tests/unit/common/ai/durable/test_storage.py | 2 +-
.../common/ai/durable/test_task_state_store.py | 2 +-
.../tests/unit/common/ai/hooks/test_pydantic_ai.py | 15 +
.../unit/common/ai/managed_agents}/__init__.py | 0
.../unit/common/ai/managed_agents/test_base.py | 137 +++++++
.../unit/common/ai/managed_agents/test_failover.py | 205 +++++++++++
.../tests/unit/common/ai/mixins/test_approval.py | 6 +-
.../tests/unit/common/ai/operators/test_agent.py | 119 +++++-
.../common/ai/operators/test_document_loader.py | 12 +-
.../ai/operators/test_llamaindex_embedding.py | 2 +-
.../ai/operators/test_llamaindex_retrieval.py | 4 +-
.../ai/tests/unit/common/ai/operators/test_llm.py | 24 +-
.../unit/common/ai/operators/test_llm_batch.py | 3 +-
.../unit/common/ai/operators/test_llm_branch.py | 4 +-
.../common/ai/operators/test_llm_file_analysis.py | 11 +-
.../tests/unit/common/ai/operators/test_llm_sql.py | 4 +-
.../ai/tests/unit/common/ai/test_observability.py | 19 +
.../unit/common/ai/toolsets/test_managed_agent.py | 403 ++++++++-------------
.../unit/common/ai/toolsets/test_object_storage.py | 8 +-
.../ai/tests/unit/common/ai/utils/test_logging.py | 55 ++-
.../unit/common/ai/utils/test_prompt_cache.py | 209 +++++++++++
.../tests/unit/common/ai/utils/test_task_logger.py | 82 +++++
.../common/compat/_set_during_execution.py | 40 ++
.../src/airflow/providers/common/compat/sdk.py | 14 +-
.../common/compat/security/test_access_view.py | 8 +-
.../common/compat/test__set_during_execution.py} | 22 +-
.../compat/tests/unit/common/compat/test_sdk.py | 18 +-
providers/duckdb/README.rst | 6 +-
providers/duckdb/docs/changelog.rst | 9 +-
providers/duckdb/docs/index.rst | 6 +-
providers/duckdb/provider.yaml | 2 +-
providers/duckdb/pyproject.toml | 6 +-
.../src/airflow/providers/duckdb/__init__.py | 2 +-
providers/fab/pyproject.toml | 2 +-
.../fab/auth_manager/api_fastapi/routes/roles.py | 14 +-
.../fab/auth_manager/api_fastapi/routes/users.py | 12 +-
.../providers/fab/auth_manager/fab_auth_manager.py | 4 +-
.../unit/fab/auth_manager/api_fastapi/conftest.py | 21 ++
.../auth_manager/api_fastapi/routes/test_roles.py | 161 ++++----
.../auth_manager/api_fastapi/routes/test_users.py | 128 ++++---
.../git/src/airflow/providers/git/bundles/git.py | 31 +-
.../git/src/airflow/providers/git/hooks/git.py | 7 +-
providers/git/tests/unit/git/bundles/test_git.py | 44 +++
providers/git/tests/unit/git/hooks/test_git.py | 10 +
providers/google/docs/index.rst | 2 +
.../google/docs/operators/cloud/vertex_ai.rst | 24 ++
providers/google/pyproject.toml | 4 +
.../google/cloud/hooks/vertex_ai/agent_engine.py | 159 +++++++-
.../cloud/hooks/vertex_ai/test_agent_engine.py | 145 ++++++++
.../smtp/src/airflow/providers/smtp/hooks/smtp.py | 2 +-
.../airflow/providers/standard/operators/hitl.py | 6 +-
.../tests/unit/standard/operators/test_hitl.py | 4 +-
pyproject.toml | 4 +-
scripts/ci/prek/check_go_sdk_generated_drift.py | 187 ++++++++++
scripts/ci/prek/check_go_sdk_spec_drift.py | 140 -------
...ift.py => test_check_go_sdk_generated_drift.py} | 83 +++--
.../observability/metrics/metrics_template.yaml | 20 +-
task-sdk/docs/executable-bundle-spec.rst | 8 +-
task-sdk/docs/ts-bundle-spec.rst | 21 +-
.../airflow/sdk/coordinators/_bundle_metadata.py | 45 +--
.../src/airflow/sdk/coordinators/_subprocess.py | 140 ++-----
.../sdk/coordinators/executable/coordinator.py | 28 +-
.../airflow/sdk/coordinators/java/coordinator.py | 24 +-
.../airflow/sdk/coordinators/node/coordinator.py | 29 +-
task-sdk/src/airflow/sdk/serde/__init__.py | 7 +-
.../coordinators/executable/test_coordinator.py | 44 +--
.../task_sdk/coordinators/java/test_coordinator.py | 82 ++---
.../coordinators/java/test_dag_importer.py | 21 +-
.../task_sdk/coordinators/node/test_coordinator.py | 57 +--
.../tests/task_sdk/coordinators/test_subprocess.py | 140 ++-----
.../task_sdk/execution_time/test_coordinator.py | 4 +-
ts-sdk/README.md | 8 +-
ts-sdk/example/README.md | 18 +-
uv.lock | 20 +-
199 files changed, 5998 insertions(+), 2139 deletions(-)
create mode 100644 go-sdk/airflow/trigger_dag_run.go
create mode 100644 go-sdk/airflow/trigger_dag_run_test.go
create mode 100644
providers/common/ai/src/airflow/providers/common/ai/managed_agents/__init__.py
create mode 100644
providers/common/ai/src/airflow/providers/common/ai/managed_agents/base.py
create mode 100644
providers/common/ai/src/airflow/providers/common/ai/managed_agents/failover.py
create mode 100644
providers/common/ai/src/airflow/providers/common/ai/utils/prompt_cache.py
create mode 100644
providers/common/ai/src/airflow/providers/common/ai/utils/task_logger.py
copy {airflow-core/src/airflow/_shared =>
providers/common/ai/tests/unit/common/ai/managed_agents}/__init__.py (100%)
create mode 100644
providers/common/ai/tests/unit/common/ai/managed_agents/test_base.py
create mode 100644
providers/common/ai/tests/unit/common/ai/managed_agents/test_failover.py
create mode 100644
providers/common/ai/tests/unit/common/ai/utils/test_prompt_cache.py
create mode 100644
providers/common/ai/tests/unit/common/ai/utils/test_task_logger.py
create mode 100644
providers/common/compat/src/airflow/providers/common/compat/_set_during_execution.py
copy providers/{amazon/tests/unit/amazon/aws/cli/test_definition.py =>
common/compat/tests/unit/common/compat/test__set_during_execution.py} (52%)
create mode 100755 scripts/ci/prek/check_go_sdk_generated_drift.py
delete mode 100755 scripts/ci/prek/check_go_sdk_spec_drift.py
rename scripts/tests/ci/prek/{test_check_go_sdk_spec_drift.py =>
test_check_go_sdk_generated_drift.py} (52%)
