This is an automated email from the ASF dual-hosted git repository.
jason810496 pushed a change to branch jason/lang-sdk-e2e/06-ts-sdk
in repository https://gitbox.apache.org/repos/asf/airflow.git
discard 6009f31fa08 TS SDK: add triggerDagRun to the native Dag surface
discard 028de33a23a TS SDK: add multi-way branching with dag.switch
discard 5d20a5532ae TS SDK: add conditional branching with dag.if
discard b6b049b9b04 TS SDK: answer the Dag-parsing request from bundle.serve
discard e780de44a40 Skip the Python-only parse checks for non-Python Dag files
discard b2cc2e3daf8 Queue the Dag files of non-Python Dag importers
discard 9cbd149a395 Share the bundle section mutation helper between Node tests
discard 20a2b6c234f Document parsing native TypeScript Dags
discard a380e5ce397 Parse native TypeScript Dags with the Node coordinator
discard 5a8f7c401db Name the JAR manifest attributes the Java coordinator reads
discard dc36a40e453 Test the Java Dag source placeholder for a JAR without a
manifest
discard 55376f47ed9 Treat an unreadable JAR like one without a manifest
discard d2327ded717 Document native Java Dag parsing
discard 7436b6119fb Parse native Java Dags with their coordinator
discard 60079e4c40b Embed the Dag source in Java bundle JARs
discard 9b2dce5c692 Build the JVM command that parses a Dag JAR
discard 5c0e91834a0 Read JAR manifests per the JAR specification
discard 64b7effa3c6 Parse coordinator-claimed Dag files with their runtime in
a Dag bag
discard cd38d3392b0 Report a native Dag's task that reaches a Python worker
discard 5bb5f71ce57 Report a Lang-SDK parse past its import timeout as an
import error
discard d2e4f162d06 Parse coordinator-claimed Dag files with their runtime in
the Dag processor
discard 107f1eb6dc0 Let a coordinator parse the Dag files of the bundles it
serves
discard 35704801174 Fill a serialized Dag's unset settings from the Airflow
config
discard d67fee105db Check that a serialized Dag can be stored and loaded
discard 0b47cba3704 Move the Dag file processor's shared plumbing into a base
class
discard 541001b9ee1 Share Dag cycle detection between the Task SDK and core
add e4d397f021e Bound get_schema results for very wide tables in the SQL
toolsets (#74017)
add 7d6244bf191 Clarify `max_db_retries` doc wording to avoid off-by-one
confusion (#73722)
add bb463bf9230 Correlate executor task events by attempt UUID (#73916)
add c19faf0c5ae Render the HITL review workflow as a Mermaid diagram
(#74024)
add 2aa31ae7bff Use airflow.sdk imports in Airflow 3 upgrade guide example
(#74015)
add f3532c60eda Use Task SDK TaskInstanceState in OpenSearch log handler
(#72444)
add 0ec26723858 Add GKEPodExecOperator for existing Pods (#72577)
add 740986adfce Add missing cycle assertion to test_semi_complex (#71973)
add fca1c01c73e Fix Windows UnicodeDecodeError during pytest collection
(#73536)
add 1787ef7cbba Run the deferred AWS hook configuration check as a prek
hook (#73681)
add 413ae46a045 Align timetable docs with Airflow 3 data interval and
catchup behaviour (#70658)
add 57e89c0a676 Share Dag cycle detection between the Task SDK and core
add e5ae872c9a1 Pin the empty task id case in the Dag cycle detection tests
add 3ff4e27860a Move the Dag file processor's shared plumbing into a base
class
add 1153e7eecc9 Check that a serialized Dag can be stored and loaded
add d996d4f88ad Fill a serialized Dag's unset settings from the Airflow
config
add aefc2d92e62 Reject malformed task entries in a serialized Dag
add 0f3006a6769 Check SDK Dags with the Dag processor's validation in
conformance
add ad14011af01 Note in the parsing ADR that runtimes may omit
config-backed fields
add 7bf33a97660 Let a coordinator parse the Dag files of the bundles it
serves
add 16e4118bb3a Import the coordinator manager at the top of the importer
base
add 90e99e3cd7f Check Dag file claims per bundle on the parse path
add e3fc4402ad7 Drop serves_bundle so get_parsed_bundles is the only answer
add 8e0e58455bd Parse coordinator-claimed Dag files with their runtime in
the Dag processor
add 97b61d1076d Report a Lang-SDK parse past its import timeout as an
import error
add c94553e7f90 Report a native Dag's task that reaches a Python worker
add 0f933aa2b05 Drop callbacks for Lang-SDK Dag files before queuing the
file
add 11346702645 Start each Dag file processor with explicit arguments
add 273be37c009 Return a Dag importer from every SubprocessCoordinator call
add 6e7fbd6958e Find the claiming coordinator in the Task SDK and wrap it
in core
add f5134f7ca3a Fail a native Dag's task on a Python worker without retries
add b70d91a5584 Build each bundle's Dag importer registry before freezing
the heap
add 6ec3b9d872a Match the schema error's JSON path in a Lang-SDK parse test
add 5e9b6c34007 Kill what a Lang-SDK runtime leaves in its process group
add 9d2b068fc24 Name the setting in a Lang-SDK parse timeout error
add 6dceaa7f7b4 Spec the mocks of a hand-built Lang-SDK processor in tests
add 5f7f31b6776 Apply the multi-team task checks to Lang-SDK Dags
add 2a3e14ba73e Clear import errors after the native-file Dag bag sync test
add 5b9cd9c3664 Note in ADR-0010 that the manager routes claimed files for
now
add b2ce95c7167 Move the client-less Lang-SDK parse out of this change
add 3ff6be22d01 Parse coordinator-claimed Dag files with their runtime in
a Dag bag
add 243a2e3fbf0 Let a Dag file processor run without an API client
add 0d5cd08ff4c Bag native Lang-SDK Dags in core, only when asked
add bd89a180966 List the tasks of a native Lang-SDK Dag
add a88ed9d8bbe Read JAR manifests per the JAR specification
add 4673b378e83 Build the JVM command that parses a Dag JAR
add 7e4ae847f12 Embed the Dag source in Java bundle JARs
add be296473c5b Parse native Java Dags with their coordinator
add d5ad8e33dbf Treat an unreadable JAR like one without a manifest
add 490ea209115 Test the Java Dag source placeholder for a JAR without a
manifest
add 24e049de385 Name the JAR manifest attributes the Java coordinator reads
add 7ad9b106dcd Document native Java Dag parsing
add 1a7c6d45a51 Run a task on its run's version when dag_bundle_name names
its bundle
add ecc6cc2ac64 Pass the task's Dag file to the coordinator's command
builder
add d17430948f3 Report why a Java Dag JAR cannot be read
add 08cc9a263ff Run the JAR a native Java Dag was parsed from
add e206f8cfda6 Reject a Java Dag JAR whose SDK cannot answer a parse
add 5586c0a9781 Stop promising that Java dependency JARs are not parsed
add 47500af0539 Keep the Java parse_dag test from changing the pytest
worker
add c1c1f7a06b1 Parse native TypeScript Dags with the Node coordinator
add 693ce34f01b Document parsing native TypeScript Dags
add 34ecf5e6c9c Share the bundle section mutation helper between Node tests
add 1d487fc7fe1 Run a native TypeScript Dag's task from its own bundle
add 3f53f6c6c26 Say when the Dag processor needs Node.js for TypeScript
bundles
add 4e61ea98cae Align the TypeScript build docs with the Code view
add 4cb0b86546a Warn against defining a Dag in both TypeScript and Python
add 15a5db748a3 Import NodeDagImporter at the top of the Node coordinator
add be3f73ef961 Let NodeDagImporter source read errors reach the caller
add caa8462fc23 Pin the Code view to the entry module in Node bundle tests
add 493d06911ba Migrating from filepath based Dag to DagDefinition
discovery
add 0d12101ce76 Handle Dags nested at any depth in Dag archives
add 660b696fb3e Stub Dag file discovery with DagFileInfo in the Lang-SDK
manager tests
add 42b64dd4c33 TS SDK: answer the Dag-parsing request from bundle.serve
add ee348dbebba TS SDK: add conditional branching with dag.if
add 8415bbd3882 TS SDK: add multi-way branching with dag.switch
add b5978974419 TS SDK: add triggerDagRun to the native Dag surface
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (6009f31fa08)
\
N -- N -- N refs/heads/jason/lang-sdk-e2e/06-ts-sdk (b5978974419)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
airflow-core/adr/lang-sdk/0004-dag-parsing.md | 4 +-
.../adr/lang-sdk/0010-native-dag-processing.md | 25 +-
.../authoring-and-scheduling/language-sdks/go.rst | 3 +-
.../language-sdks/java.rst | 19 +-
.../language-sdks/typescript.rst | 48 +--
.../docs/authoring-and-scheduling/timetable.rst | 154 ++++++---
airflow-core/docs/core-concepts/executor/index.rst | 23 +-
.../docs/installation/upgrading_to_airflow3.rst | 2 +-
.../src/airflow/cli/commands/task_command.py | 5 +-
.../src/airflow/config_templates/config.yml | 6 +-
airflow-core/src/airflow/dag_processing/dagbag.py | 196 ++++++-----
.../src/airflow/dag_processing/importer_routing.py | 122 +------
.../airflow/dag_processing/lang_sdk_processor.py | 123 +++++--
airflow-core/src/airflow/dag_processing/manager.py | 172 +++++-----
.../src/airflow/dag_processing/processor.py | 78 +++--
.../src/airflow/executors/base_executor.py | 123 ++++++-
.../src/airflow/executors/local_executor.py | 13 +-
.../src/airflow/executors/workloads/task.py | 3 +-
.../src/airflow/executors/workloads/types.py | 17 +-
.../src/airflow/jobs/scheduler_job_runner.py | 49 +--
airflow-core/src/airflow/models/taskinstance.py | 2 +-
.../src/airflow/serialization/definitions/dag.py | 6 +-
airflow-core/src/airflow/serialization/schema.json | 13 +-
.../airflow/serialization/serialized_objects.py | 15 +-
airflow-core/src/airflow/utils/cli.py | 45 ++-
airflow-core/src/airflow/utils/file.py | 44 ---
.../versions/head/test_task_instances.py | 5 +-
.../tests/unit/cli/commands/test_dag_command.py | 3 +
.../tests/unit/cli/commands/test_task_command.py | 26 ++
.../tests/unit/dag_processing/fake_importers.py | 199 -----------
.../lang_sdk_fixtures}/java_native.json | 0
.../lang_sdk_fixtures}/ts_conformance.json | 0
.../tests/unit/dag_processing/test_dagbag.py | 154 ++++++++-
.../unit/dag_processing/test_importer_routing.py | 80 ++---
.../unit/dag_processing/test_lang_sdk_processor.py | 152 ++++++++-
.../tests/unit/dag_processing/test_manager.py | 373 ++++++++++++---------
.../tests/unit/dag_processing/test_processor.py | 59 ++--
.../tests/unit/executors/test_base_executor.py | 278 +++++++++++++--
.../tests/unit/executors/test_local_executor.py | 10 +-
.../tests/unit/executors/test_workloads.py | 16 +-
airflow-core/tests/unit/jobs/test_scheduler_job.py | 181 +++++++---
airflow-core/tests/unit/models/test_dag.py | 3 +-
.../tests/unit/models/test_taskinstance.py | 12 +
.../unit/serialization/test_dag_serialization.py | 37 +-
.../unit/serialization/test_dag_version_diff.py | 1 -
airflow-core/tests/unit/utils/test_cli_util.py | 37 +-
airflow-core/tests/unit/utils/test_file.py | 33 --
devel-common/src/tests_common/pytest_plugin.py | 9 +-
.../src/tests_common/test_utils/mock_executor.py | 5 +-
.../src/tests_common/test_utils/providers.py | 2 +-
generated/known_sdk_imports_in_core.txt | 5 +-
providers/.pre-commit-config.yaml | 7 +
.../amazon/aws/test_deferred_hook_configuration.py | 242 +------------
.../unit/celery/executors/test_celery_executor.py | 4 +
.../executors/test_kubernetes_executor.py | 2 +
providers/common/ai/docs/changelog.rst | 12 +
providers/common/ai/docs/hitl_review.rst | 63 ++--
providers/common/ai/docs/toolsets/datafusion.rst | 11 +-
providers/common/ai/docs/toolsets/sql.rst | 41 ++-
.../providers/common/ai/toolsets/datafusion.py | 35 +-
.../airflow/providers/common/ai/toolsets/sql.py | 33 +-
.../providers/common/ai/utils/query_results.py | 202 ++++++++++-
.../unit/common/ai/toolsets/test_datafusion.py | 55 ++-
.../ai/tests/unit/common/ai/toolsets/test_sql.py | 62 +++-
.../unit/common/ai/utils/test_query_results.py | 167 ++++++++-
providers/google/docs/index.rst | 8 +-
.../docs/operators/cloud/kubernetes_engine.rst | 20 ++
providers/google/pyproject.toml | 2 +-
.../google/cloud/operators/kubernetes_engine.py | 114 +++++++
.../kubernetes_engine/example_kubernetes_engine.py | 105 +++++-
.../cloud/operators/test_kubernetes_engine.py | 146 +++++++-
.../providers/opensearch/log/os_task_handler.py | 3 +-
.../ci/lang_sdk_serialization/serialize_python.py | 23 +-
.../ci/prek/check_deferred_hook_configuration.py | 294 ++++++++++++++++
scripts/ci/prek/common_prek_utils.py | 26 +-
scripts/ci/prek/update_providers_dependencies.py | 14 +-
.../prek/test_check_deferred_hook_configuration.py | 373 +++++++++++++++++++++
shared/dagnode/tests/dagnode/test_cycle.py | 10 +-
.../src/airflow/sdk/coordinators/_dag_importer.py | 60 ++--
.../src/airflow/sdk/coordinators/_subprocess.py | 40 ++-
.../sdk/coordinators/executable/coordinator.py | 4 +-
.../airflow/sdk/coordinators/java/_dag_importer.py | 4 +-
.../airflow/sdk/coordinators/java/coordinator.py | 101 +++++-
.../airflow/sdk/coordinators/node/_dag_importer.py | 16 +-
.../airflow/sdk/coordinators/node/coordinator.py | 35 +-
.../src/airflow/sdk/execution_time/coordinator.py | 125 +++----
.../src/airflow/sdk/execution_time/task_runner.py | 52 ++-
task-sdk/src/airflow/sdk/importers/base.py | 23 +-
.../src/airflow/sdk/importers/python_importer.py | 4 +-
task-sdk/src/airflow/sdk/importers/zip_importer.py | 6 +-
.../task_sdk/coordinators/java/test_coordinator.py | 117 ++++++-
.../coordinators/node/test_bundle_reader.py | 5 +-
.../task_sdk/coordinators/node/test_coordinator.py | 46 +++
.../coordinators/node/test_dag_importer.py | 22 +-
.../task_sdk/coordinators/test_dag_importer.py | 93 +----
.../tests/task_sdk/coordinators/test_subprocess.py | 64 +++-
task-sdk/tests/task_sdk/definitions/test_dag.py | 2 +
.../task_sdk/execution_time/test_coordinator.py | 196 +++++++----
.../task_sdk/execution_time/test_task_runner.py | 119 ++++---
task-sdk/tests/task_sdk/importers/test_registry.py | 10 +-
.../tests/task_sdk/importers/test_zip_importer.py | 3 +-
ts-sdk/schema/dag-schema.json | 13 +-
102 files changed, 4322 insertions(+), 1902 deletions(-)
delete mode 100644 airflow-core/tests/unit/dag_processing/fake_importers.py
rename {task-sdk/tests/task_sdk/coordinators/fixtures =>
airflow-core/tests/unit/dag_processing/lang_sdk_fixtures}/java_native.json
(100%)
rename {task-sdk/tests/task_sdk/coordinators/fixtures =>
airflow-core/tests/unit/dag_processing/lang_sdk_fixtures}/ts_conformance.json
(100%)
create mode 100755 scripts/ci/prek/check_deferred_hook_configuration.py
create mode 100644
scripts/tests/ci/prek/test_check_deferred_hook_configuration.py
