pierrejeambrun commented on code in PR #73723:
URL: https://github.com/apache/airflow/pull/73723#discussion_r4105489050
##########
ts-sdk/src/cli/bundle-encoder.ts:
##########
@@ -139,26 +180,47 @@ function encodeHeader(regions: { metadata: Buffer;
source: Buffer; executable: B
}
/**
- * Wrap the entrypoint as written in a block comment.
+ * Wrap each source in its own block comment, one per author-owned Dag file.
*
* A comment terminator would splice the rest of the payload into executable
position, so it is
* escaped. `*\\` is escaped too, which keeps the transformation reversible.
+ *
+ * The returned regions are keyed by path and hold byte offsets within the
+ * concatenated sources buffer (not the final bundle); the header adds the
+ * bundle-relative base offset when it renders.
*/
-function encodeSource(entrypointSource: string): Buffer {
- const payload = Buffer.from(escapeBlockComment(entrypointSource), "utf-8");
- const source = Buffer.concat([
- Buffer.from(EMBEDDED_SOURCE_OPEN, "ascii"),
- payload,
- Buffer.from(EMBEDDED_SOURCE_CLOSE, "ascii"),
- ]);
- if (source.length > EMBEDDED_SOURCE_MAX_BYTES) {
+function encodeSources(sources: Record<string, string>): EncodedSources {
+ const chunks: Buffer[] = [];
+ const regions: SourceRegion[] = [];
+ let offset = 0;
+
+ for (const [path, content] of Object.entries(sources)) {
+ const openMarker =
`${EMBEDDED_SOURCE_OPEN_PREFIX}${path}${EMBEDDED_SOURCE_OPEN_SUFFIX}`;
Review Comment:
Fixed in 709a42f1c30 — a source path containing `*/` (or a newline) now
rejects at pack time. Real filesystem paths don't hold either, and rejecting
keeps the marker line trivially readable and offsets valid.
##########
ts-sdk/src/cli/pack.ts:
##########
@@ -202,11 +216,82 @@ async function loadEsbuild(): Promise<typeof
import("esbuild")> {
}
}
+/** Author source files esbuild's onLoad tags with their path. Same filter set
the previous
+ * task-id plugin used, which matched every language a Dag can be declared
in. */
+const AUTHOR_SOURCE_FILTER = /\.[cm]?[jt]sx?$/;
+
+/** Skip anything under node_modules: an installed dependency is not a Dag
file, and tagging it
+ * would only cost bundle bytes while overwriting the slot with the wrong
path. */
+const DEPENDENCY_PATH = /[\\/]node_modules[\\/]/;
+
+/**
+ * esbuild plugin: prepend a single line to each author-owned source file that
+ * writes the file's path into the SDK's module-source slot. `Dag`'s
+ * constructor reads that slot, so a Dag declared in `src/dags/reports.ts`
+ * carries that path even though esbuild will soon inline every module into
+ * one bundle.
+ *
+ * The prepend is text-only — no parsing, no AST — so it can never mis-identify
+ * a construction site. ES modules hoist imports above non-import statements
+ * at execution, so the tag runs *after* imported modules have written their
+ * own slots and *before* this module's own top-level statements, which is
+ * exactly when its `new Dag(...)` calls fire.
+ */
+function moduleSourceTagPlugin(cwd: string): import("esbuild").Plugin {
+ const slotKey = JSON.stringify(MODULE_SOURCE_SLOT_KEY);
+ return {
+ name: "airflow-module-source-tag",
+ setup(build) {
+ build.onLoad({ filter: AUTHOR_SOURCE_FILTER }, async ({ path: file,
namespace }) => {
+ // Only the `file` namespace has a path on disk to attribute a Dag to;
+ // a virtual module from another plugin has nothing to tag.
+ if (namespace !== "file" || DEPENDENCY_PATH.test(file)) return
undefined;
+ const source = await readFileAsync(file, "utf-8");
+ // Project-relative so the bundle is portable and readable (no host
+ // filesystem prefix), matching how esbuild's own metafile keys
sources.
+ const relative = path.relative(cwd, file);
+ const tag =
`globalThis[Symbol.for(${slotKey})]=${JSON.stringify(relative)};\n`;
Review Comment:
Partly addressed in 9c0fb20fea7 — the tag now saves the previous slot value
on entry and restores it after the module's top-level code returns. Nested
imports (A imports B) and top-level `await` within a single module hand the
slot back on the way out, so a `new Dag(...)` after the import is still
attributed to A.
Two limits remain and are documented in module-source.ts: (a) a Dag
constructed in a scheduled callback (setTimeout, microtask, .then) after the
module's epilog has restored lands under whichever module touched the slot most
recently; (b) two modules genuinely evaluated in parallel through
`Promise.all([import, import])` can interleave. Both are non-standard patterns
for Dag files; the documented invariant is 'declare Dags at module top level'.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
