kaxil commented on code in PR #67217:
URL: https://github.com/apache/airflow/pull/67217#discussion_r3455817069
##########
airflow-core/src/airflow/executors/workloads/base.py:
##########
@@ -66,6 +66,13 @@ class BundleInfo(BaseModel):
name: str
version: str | None = None
+ version_data: dict[str, Any] | None = None
+ """Optional structured metadata for this bundle version (e.g., an S3
object manifest).
+
+ This field is serialized on every workload payload sent through executor
channels
+ (Celery/Redis, SQS, K8s pod annotations, etc.). Keep payloads small —
ideally under
Review Comment:
Small accuracy fix on the transports: K8s/ECS/Batch/Lambda workloads ship as
`--json-string` argv, not as pod annotations. Annotations only carry routing
metadata. The relevant caps are `ARG_MAX` (~128 KB on Linux) and the etcd
PodSpec ceiling (~1.5 MB), not the 256 KB annotation cap. Suggest replacing the
parenthetical with "executor command-line argv for K8s/ECS/Batch/Lambda,
message body for Celery/SQS" so future debugging looks in the right place.
(That's on me from the prior review, I asked for the docstring without
specifying the transports.)
Separate design question while we're on this field: the wire shape is
`dict[str, Any]`, which means the manifest content rides every workload payload
forever. An opaque pointer (e.g., `s3://bucket/manifest.json` + a content
digest) plus lazy worker-side fetch through the bundle credentials side-steps
the size cap entirely. Worth deciding before the first consumer ships, because
the dict-on-wire shape is the consumer contract, wdyt? Not a blocker for the
PR, though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
