jason810496 opened a new pull request, #73127: URL: https://github.com/apache/airflow/pull/73127
- related: #69905 - follow-up to review comments on #72046 - **Depends on https://github.com/apache/airflow/pull/73126 get merged first** (only the last commit is new here). - **Diff for early review**: https://github.com/jason810496/airflow/compare/refactor/ts-sdk/bundle-discovery-and-minify...feature/ts-sdk/bundle-entrypoint-source ## Why The shipped code region is minified, so it is not the code anyone wrote, and there was nothing readable for Airflow to display for a natively authored TypeScript Dag. ## How - A packed bundle now carries the entry module as its author wrote it, in a `/*# airflowSource ... #*/` region with its own byte range and SHA-256 digest in the layout header: ``` //# airflowBundle={"code":{...},"metadata":{...},"source":{...}} //# airflowMetadata={...} /*# airflowSource <the entrypoint, readable> #*/ <minified code> ``` - Only the entry module, not the module graph behind it, mirroring the Java SDK's single `Airflow-Java-SDK-Dag-Code` attribute. ADR-0006 declined multi-file source display for mixed-language Dags. - A block comment rather than the line comments the layout and metadata use, because the entrypoint spans the lines it was written on. Line terminators are therefore legal inside the region and need no escaping, which leaves `*/` as the one sequence that must not appear: - It would end the comment where Node reads the file, putting the rest of the payload into executable position while the layout still calls those bytes source and both digests still match. - The packer inserts a `\` between the two characters, and escapes `*\` the same way so the transformation is reversible. - The reader reverses it, and rejects a bundle carrying the sequence unescaped rather than vouch for one. - The source range's length is declared rather than derivable from a newline, so the reader pins it by checking that the prescribed opener and closer sit exactly where the declared range implies. - `airflow_bundle_metadata_version` stays at `1.0`, redefined in place because the TypeScript packing workflow is unreleased. A source-less bundle still fails closed, on the now-required `source` layout section, before any version check is reached. ## What - `ts-sdk/src/cli/bundle-encoder.ts`: third region, its header section, and the escaping. - `ts-sdk/src/cli/pack.ts`: pass the entry file through to the encoder. - `coordinators/node/_bundle_reader.py`: parse and digest-verify the `source` section, re-derive the framing, reject an unescaped terminator, and add `read_bundle_source`. - `coordinators/node/coordinator.py`: `NodeCoordinator.get_source_code` returns the verified source. Nothing calls it yet; it is shaped for `AbstractDagImporter.get_source_code`, but a TypeScript importer also has to produce `DAG` objects from a bundle, which needs native TypeScript Dag serialization that has not landed. - Rename the metadata `dags` mapping to `task_handlers`, which is what a TypeScript bundle actually provides: handlers for Dags declared elsewhere, not Dag definitions of its own. The mapping stays keyed by Dag ID. The Go/executable `AFBNDL01` metadata is a separate contract and is untouched. - Regenerate the cross-language golden fixture as `bundle-v1.min.mjs`. - Docs: `ts-bundle-spec.rst` gains a "Source" section, plus `typescript.rst` and `ts-sdk/README.md`. ## Verification - `pytest task-sdk/tests/task_sdk/coordinators/` 193 passed, including 17 new source-region cases: offset mismatch, missing opener, bad closer, range past EOF, oversized range, unescaped `*/`, non-UTF-8, digest mismatch, and byte-exact round-trips through `*/`, `*\/`, `*\\`, non-ASCII, and multi-line input. - `pnpm test` 249 passed, plus `typecheck`, `lint`, `format:check` - `prek --stage pre-commit` and `--stage manual` clean, `mypy-task-sdk` passed - End to end on an entrypoint containing a JSDoc terminator and `*\/` / `*\\` literals: packs, runs under `node`, is found by recursive discovery, and its source comes back byte-exact. --- ##### Was generative AI tooling used to co-author this PR? - [x] Yes, with help of Claude Code Opus 5 following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
