This is an automated email from the ASF dual-hosted git repository.
jason810496 pushed a change to branch jason/lang-sdk-e2e/06-ts-sdk
in repository https://gitbox.apache.org/repos/asf/airflow.git
discard 556bc9a7ff7 TS SDK: add triggerDagRun to the native Dag surface
discard d6219421085 TS SDK: add multi-way branching with dag.switch
discard d86ba516b66 TS SDK: add conditional branching with dag.if
discard f4ecb133d52 TS SDK: answer the Dag-parsing request from bundle.serve
discard 4ff4171bd7f Stub Dag file discovery with DagFileInfo in the Lang-SDK
manager tests
discard 17776f430b6 Handle Dags nested at any depth in Dag archives
discard 24eb53e370e Migrating from filepath based Dag to DagDefinition
discovery
discard d9cc6b1791a Pin the Code view to the entry module in Node bundle tests
discard 73a18f9cd30 Let NodeDagImporter source read errors reach the caller
discard b2f7c17adca Import NodeDagImporter at the top of the Node coordinator
discard 116c8a2dd82 Warn against defining a Dag in both TypeScript and Python
discard cc949b3316c Align the TypeScript build docs with the Code view
discard e2076b8e900 Say when the Dag processor needs Node.js for TypeScript
bundles
discard 7702d8420b9 Run a native TypeScript Dag's task from its own bundle
discard 01316f8d0a7 Share the bundle section mutation helper between Node tests
discard d98a2540a4b Document parsing native TypeScript Dags
discard 064db1cce39 Parse native TypeScript Dags with the Node coordinator
discard b045547d080 Keep the Java parse_dag test from changing the pytest
worker
discard 1a6e3ea86e1 Stop promising that Java dependency JARs are not parsed
discard f3f6bea154b Reject a Java Dag JAR whose SDK cannot answer a parse
discard ee9521355b1 Run the JAR a native Java Dag was parsed from
discard 1aac2d02868 Report why a Java Dag JAR cannot be read
discard ac38e820b06 Pass the task's Dag file to the coordinator's command
builder
discard c179527fedc Run a task on its run's version when dag_bundle_name names
its bundle
discard 97f60a3322e Document native Java Dag parsing
discard bac857dec40 Name the JAR manifest attributes the Java coordinator reads
discard 2ca56a5effe Test the Java Dag source placeholder for a JAR without a
manifest
discard 0b55a3dd2a6 Treat an unreadable JAR like one without a manifest
discard 2eb08b4308c Parse native Java Dags with their coordinator
discard a7be34b19ff Embed the Dag source in Java bundle JARs
discard 02bc2bc0794 Build the JVM command that parses a Dag JAR
discard bd060ceb11b Read JAR manifests per the JAR specification
discard 96f2076b96d Ignore the bagged Dag union in the external task sensor
test
discard fd0a742a2d7 List the tasks of a native Lang-SDK Dag
discard a8797750305 Bag native Lang-SDK Dags in core, only when asked
discard cab27a25c52 Let a Dag file processor run without an API client
discard c53c8c723e5 Parse coordinator-claimed Dag files with their runtime in
a Dag bag
discard e072db64742 Wait for the killed leftover in the Lang-SDK processor test
discard 57a417b164d Move the client-less Lang-SDK parse out of this change
discard 2ee5ead4c9a Note in ADR-0010 that the manager routes claimed files for
now
discard 04706bdc0b5 Clear import errors after the native-file Dag bag sync test
discard 55311fe73c1 Apply the multi-team task checks to Lang-SDK Dags
discard 8b91affcf79 Spec the mocks of a hand-built Lang-SDK processor in tests
discard 56c6820d7c3 Name the setting in a Lang-SDK parse timeout error
discard f632602f806 Kill what a Lang-SDK runtime leaves in its process group
discard 280fb07887a Match the schema error's JSON path in a Lang-SDK parse test
discard 3c0b4ba6181 Build each bundle's Dag importer registry before freezing
the heap
discard 001b6d90f29 Fail a native Dag's task on a Python worker without retries
discard fd2a897053d Find the claiming coordinator in the Task SDK and wrap it
in core
discard 4f54283d0de Return a Dag importer from every SubprocessCoordinator call
discard ed8cbad9fa0 Start each Dag file processor with explicit arguments
discard 0cb6bf43a66 Drop callbacks for Lang-SDK Dag files before queuing the
file
discard 7c38262692d Report a native Dag's task that reaches a Python worker
discard 50981761a38 Report a Lang-SDK parse past its import timeout as an
import error
discard 9ef4d490013 Parse coordinator-claimed Dag files with their runtime in
the Dag processor
discard 30187220d54 Say dag_bundle_name is only needed for two coordinators of
one class
discard 52b88ba0c85 Drop serves_bundle so get_parsed_bundles is the only answer
discard 75c77d508be Check Dag file claims per bundle on the parse path
discard 9b98c04bd91 Import the coordinator manager at the top of the importer
base
discard 60d7af4cad9 Let a coordinator parse the Dag files of the bundles it
serves
discard f9fd8e01be2 Sync the Java SDK copy of the Dag schema
discard 464c9273c0c Serialize the SDK Dag in tests that write a dag_maker Dag
discard e5d1a71ca9c Note in the parsing ADR that runtimes may omit
config-backed fields
discard 76389a9ffa2 Check SDK Dags with the Dag processor's validation in
conformance
discard d8cf5805ec7 Reject malformed task entries in a serialized Dag
discard c1226408285 Fill a serialized Dag's unset settings from the Airflow
config
discard adf872d0ac6 Check that a serialized Dag can be stored and loaded
discard 0649a43f468 Move the Dag file processor's shared plumbing into a base
class
omit 20603491841 Pin the empty task id case in the Dag cycle detection tests
omit 737097a0dfe Share Dag cycle detection between the Task SDK and core
add 383a9a87a98 Stop wiping the uv cache before every CI image build
(#74052)
add 58e987defec Fix Non-DB test collection failure in Google Stackdriver
hook test (#74066)
add 498724fc13a Go SDK: enumerate every user-facing enum in the airflow
package (#74064)
add 5e476d7cf89 TS SDK: answer the Dag-parsing request from bundle.serve
(#73442)
add dd271599939 Go SDK: vendor the schemas it generates from (#74047)
add 77927390c17 Move Dag cycle detection from the Task SDK to shared
(#74034)
add a3e07159a0b Add Dag ID filtering to Assets search (#70971)
add d3b9a0dd2b3 Add KafkaSharedStreamProducer and KafkaSharedStreamTrigger
(#68625)
add ae6139d58a8 Fix DecreasingPriorityStrategy example by initializing
try_number before weight evaluation (#62148)
add 402d7885194 Java SDK: Resolve a task's arguments from the Dag's own
wiring (#73596)
add ca0d7328288 Move the Dag file processor's shared plumbing into a base
class
add 235be2d6bda Check that a serialized Dag can be stored and loaded
add bcba1cbf97f Fill a serialized Dag's unset settings from the Airflow
config
add 2f770c94c9a Reject malformed task entries in a serialized Dag
add fc73b8cc582 Check SDK Dags with the Dag processor's validation in
conformance
add 8cfd06019f4 Note in the parsing ADR that runtimes may omit
config-backed fields
add a63b9f1c091 Serialize the SDK Dag in tests that write a dag_maker Dag
add 0afbeb1e11c Sync the Java SDK copy of the Dag schema
add fd5eb37bd70 Let a coordinator parse the Dag files of the bundles it
serves
add 2cb5c904df8 Import the coordinator manager at the top of the importer
base
add 266131caf2a Check Dag file claims per bundle on the parse path
add 13d82cc57ec Drop serves_bundle so get_parsed_bundles is the only answer
add 6afe857c7e0 Say dag_bundle_name is only needed for two coordinators of
one class
add e610aa76488 Hand out a coordinator's Dag importer through
get_dag_importer alone
add 42c00d465ae Explain dag_bundle_name by the coordinator that parses a
Dag file
add f72fb46b402 Parse coordinator-claimed Dag files with their runtime in
the Dag processor
add 426d3bb8956 Report a Lang-SDK parse past its import timeout as an
import error
add 3dc7c020ac7 Report a native Dag's task that reaches a Python worker
add 76b203ed127 Drop callbacks for Lang-SDK Dag files before queuing the
file
add fe4b272b1ae Start each Dag file processor with explicit arguments
add 77bd5b757d4 Find the claiming coordinator in the Task SDK and wrap it
in core
add f13c43fa99b Fail a native Dag's task on a Python worker without retries
add 2a253c147f6 Build each bundle's Dag importer registry before freezing
the heap
add 985271f2515 Match the schema error's JSON path in a Lang-SDK parse test
add 1697091e40a Kill what a Lang-SDK runtime leaves in its process group
add 4c5f7e2572f Name the setting in a Lang-SDK parse timeout error
add bfcc16ccfca Spec the mocks of a hand-built Lang-SDK processor in tests
add a68f51b07cd Apply the multi-team task checks to Lang-SDK Dags
add 85c86080632 Clear import errors after the native-file Dag bag sync test
add e3bd6f0b0b3 Note in ADR-0010 that the manager routes claimed files for
now
add 778193ac00d Move the client-less Lang-SDK parse out of this change
add 917db98d4a1 Wait for the killed leftover in the Lang-SDK processor test
add a70eb430b82 Find a native Dag's coordinator key through its importer
add e8fbf760526 Parse coordinator-claimed Dag files with their runtime in
a Dag bag
add ff48b1f19aa Let a Dag file processor run without an API client
add dca01a5a0c6 Bag native Lang-SDK Dags in core, only when asked
add 897a3ef4c63 List the tasks of a native Lang-SDK Dag
add 6c3ae5d7c33 Ignore the bagged Dag union in the external task sensor
test
add 447eea4b249 Read JAR manifests per the JAR specification
add 00eaafa1752 Build the JVM command that parses a Dag JAR
add 4f8762224ec Embed the Dag source in Java bundle JARs
add a7093859961 Parse native Java Dags with their coordinator
add aab428b0bf7 Treat an unreadable JAR like one without a manifest
add 2bfbd658b47 Test the Java Dag source placeholder for a JAR without a
manifest
add cbeab5bb190 Name the JAR manifest attributes the Java coordinator reads
add f08c7823a97 Document native Java Dag parsing
add 0d10d21b2f1 Run a task on its run's version when dag_bundle_name names
its bundle
add 3ad83f79caf Pass the task's Dag file to the coordinator's command
builder
add cf32a1492cb Report why a Java Dag JAR cannot be read
add c44b01925f3 Run the JAR a native Java Dag was parsed from
add e02d9fb1f39 Reject a Java Dag JAR whose SDK cannot answer a parse
add ae5ebe24c46 Stop promising that Java dependency JARs are not parsed
add 9998e1ea102 Keep the Java parse_dag test from changing the pytest
worker
add 51198c6125c Hand out the Java Dag importer through get_dag_importer
add 4f3c85f0dc1 Parse native TypeScript Dags with the Node coordinator
add 6b8fd989bf6 Document parsing native TypeScript Dags
add 55d17f96ece Share the bundle section mutation helper between Node tests
add 42d6dd45114 Run a native TypeScript Dag's task from its own bundle
add d73eaf8b39e Say when the Dag processor needs Node.js for TypeScript
bundles
add 82d5a7ea320 Align the TypeScript build docs with the Code view
add 9869a1e4f3a Warn against defining a Dag in both TypeScript and Python
add 8a8c43e4344 Import NodeDagImporter at the top of the Node coordinator
add 47a97f0f08d Let NodeDagImporter source read errors reach the caller
add 4d6f5242f0f Pin the Code view to the entry module in Node bundle tests
add 44e11868ff9 Hand out the Node Dag importer through get_dag_importer
add 8eab80a0098 Migrating from filepath based Dag to DagDefinition
discovery
add 1af68883a10 Handle Dags nested at any depth in Dag archives
add 4c33164da25 Stub Dag file discovery with DagFileInfo in the Lang-SDK
manager tests
add 3c041acca23 TS SDK: add conditional branching with dag.if
add 4542ce9778e TS SDK: add multi-way branching with dag.switch
add 32c2b5c80a1 TS SDK: add triggerDagRun to the native Dag surface
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (556bc9a7ff7)
\
N -- N -- N refs/heads/jason/lang-sdk-e2e/06-ts-sdk (32c2b5c80a1)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.pre-commit-config.yaml | 25 +-
.../adr/lang-sdk/0010-native-dag-processing.md | 25 +-
.../authoring-and-scheduling/language-sdks/go.rst | 27 +-
.../language-sdks/java.rst | 10 +-
.../language-sdks/typescript.rst | 12 +-
.../airflow/dag_processing/lang_sdk_processor.py | 4 +-
airflow-core/src/airflow/models/taskinstance.py | 2 +-
.../ui/src/components/FilterBar/FilterBar.test.tsx | 21 +
.../ui/src/components/FilterBar/FilterBar.tsx | 31 +-
.../ui/src/pages/AssetsList/AssetsList.test.tsx | 85 +++-
.../airflow/ui/src/pages/AssetsList/AssetsList.tsx | 10 +-
.../tests/unit/dag_processing/fake_lang_sdk.py | 5 +-
airflow-core/tests/unit/models/test_dag.py | 2 +
.../airflow_breeze/commands/ci_image_commands.py | 4 +-
dev/breeze/tests/test_ci_image_commands.py | 20 +
go-sdk/README.md | 59 ++-
go-sdk/adr/0008-native-dag-interface.md | 5 +
go-sdk/airflow/dag.go | 5 +
go-sdk/airflow/enums.go | 156 +++++++
go-sdk/airflow/enums_test.go | 221 ++++++++++
go-sdk/airflow/spec.go | 33 +-
go-sdk/airflow/spec_test.go | 71 ---
go-sdk/airflow/trigger_dag_run.go | 36 +-
go-sdk/airflow/trigger_dag_run_test.go | 24 +-
go-sdk/internal/genspec/authoring.go | 6 +-
go-sdk/internal/genspec/main.go | 9 +-
go-sdk/internal/genspec/normalize_test.go | 5 +-
go-sdk/pkg/execution/genmodels/gen.go | 13 +-
go-sdk/pkg/execution/messages_test.go | 23 +-
{ts-sdk => go-sdk}/schema/dag-schema.json | 13 +-
.../schema/supervisor-schema.json | 0
java-sdk/README.md | 6 +-
.../org/apache/airflow/sdk/BuilderProcessor.kt | 2 +-
.../kotlin/org/apache/airflow/sdk/BuilderTest.kt | 4 +-
.../src/main/kotlin/org/apache/airflow/sdk/Arg.kt | 11 +-
.../main/kotlin/org/apache/airflow/sdk/Context.kt | 10 +-
.../main/kotlin/org/apache/airflow/sdk/DagDef.kt | 1 +
.../kotlin/org/apache/airflow/sdk/InputTask.kt | 2 +-
.../org/apache/airflow/sdk/execution/Task.kt | 7 +-
.../org/apache/airflow/sdk/internal/ArgValues.kt | 124 ++++++
.../org/apache/airflow/sdk/internal/TaskArgs.kt | 42 +-
.../org/apache/airflow/sdk/ArgTestSupport.kt | 15 +
.../kotlin/org/apache/airflow/sdk/ArgValuesTest.kt | 2 +-
.../kotlin/org/apache/airflow/sdk/InputTaskTest.kt | 90 ++++
.../org/apache/airflow/sdk/execution/TaskTest.kt | 24 +
.../apache/airflow/sdk/internal/ArgValuesTest.kt | 266 ++++++++++++
providers/apache/kafka/docs/triggers.rst | 33 ++
providers/apache/kafka/provider.yaml | 1 +
.../providers/apache/kafka/get_provider_info.py | 1 +
.../apache/kafka/triggers/shared_stream.py | 407 +++++++++++++++++
.../providers/apache/kafka/version_compat.py | 2 +
.../apache/kafka/triggers/test_shared_stream.py | 259 +++++++++++
.../apache/kafka/triggers/test_shared_stream.py | 481 +++++++++++++++++++++
.../unit/google/cloud/hooks/test_stackdriver.py | 2 +-
scripts/ci/prek/check_go_sdk_generated_drift.py | 25 +-
scripts/ci/prek/sync_go_sdk_schemas.py | 142 ++++++
.../ci/prek/test_check_go_sdk_generated_drift.py | 13 +-
scripts/tests/ci/prek/test_sync_go_sdk_schemas.py | 127 ++++++
.../src/airflow/sdk/coordinators/_subprocess.py | 21 +-
.../airflow/sdk/coordinators/java/coordinator.py | 5 +-
.../airflow/sdk/coordinators/node/coordinator.py | 5 +-
.../src/airflow/sdk/execution_time/coordinator.py | 93 ++--
.../src/airflow/sdk/execution_time/task_runner.py | 7 +-
task-sdk/src/airflow/sdk/importers/base.py | 18 +-
.../src/airflow/sdk/importers/python_importer.py | 4 +-
task-sdk/src/airflow/sdk/importers/zip_importer.py | 6 +-
.../coordinators/java/test_dag_importer.py | 1 -
.../task_sdk/coordinators/node/test_coordinator.py | 1 -
.../tests/task_sdk/coordinators/test_subprocess.py | 25 --
.../task_sdk/execution_time/test_coordinator.py | 119 +++--
.../task_sdk/execution_time/test_task_runner.py | 5 +-
task-sdk/tests/task_sdk/importers/test_registry.py | 18 +-
ts-sdk/README.md | 8 +-
ts-sdk/example/README.md | 6 +-
ts-sdk/src/coordinator/runtime.ts | 46 +-
ts-sdk/src/sdk/bundle.ts | 18 -
ts-sdk/tests/coordinator/integration.test.ts | 6 +-
77 files changed, 2905 insertions(+), 568 deletions(-)
create mode 100644 go-sdk/airflow/enums.go
create mode 100644 go-sdk/airflow/enums_test.go
delete mode 100644 go-sdk/airflow/spec_test.go
copy {ts-sdk => go-sdk}/schema/dag-schema.json (97%)
copy task-sdk/src/airflow/sdk/execution_time/schema/schema.json =>
go-sdk/schema/supervisor-schema.json (100%)
create mode 100644
java-sdk/sdk/src/test/kotlin/org/apache/airflow/sdk/internal/ArgValuesTest.kt
create mode 100644
providers/apache/kafka/src/airflow/providers/apache/kafka/triggers/shared_stream.py
create mode 100644
providers/apache/kafka/tests/integration/apache/kafka/triggers/test_shared_stream.py
create mode 100644
providers/apache/kafka/tests/unit/apache/kafka/triggers/test_shared_stream.py
create mode 100755 scripts/ci/prek/sync_go_sdk_schemas.py
create mode 100644 scripts/tests/ci/prek/test_sync_go_sdk_schemas.py
