github-actions[bot] opened a new pull request, #75115:
URL: https://github.com/apache/airflow/pull/75115
* Only rebuild built-in value types when converting next_kwargs for older
workers
When serving a task instance to a worker on an older Execution API
version, the version converter decodes the stored serde next_kwargs and
re-encodes them in the legacy format. Decoding reconstructs every value
in the API server, including instances of arbitrary classes, although
only the worker ever needs those objects.
Only decode payloads whose values are built-in value types - the types
the legacy format can carry anyway - and pass anything else through as
stored.
Generated-by: Claude Opus 5
* Keep converting Asset values in next_kwargs for older workers
Task SDK 1.1.x always decodes next_kwargs with BaseSerialization.deserialize,
so passing a serde payload through unchanged makes the deferred task crash
on resume with KeyError('__var'). That broke resuming with Asset or
AssetAlias values, which the legacy format supports natively.
Assets and asset aliases are now translated structurally from their
encoded fields into the legacy envelope, so nothing beyond built-in value
types is instantiated in the API server. Any other class makes the task
resume through the trigger-failure path, so older workers fail it with a
TaskDeferralError naming the type instead of an opaque KeyError. Payloads
already in the legacy format are still passed through.
Generated-by: Claude Opus 5
* Translate tuples and sets of assets in next_kwargs for older workers
A tuple or set holding an Asset or AssetAlias failed the built-in value
check because of the nested asset, so its serde envelope was treated as
unconvertible and the task resumed through the trigger-failure path, even
though the legacy format carries tuples and sets of assets.
Tuple, set and frozenset envelopes are now translated structurally into
the legacy tuple/set encoding, with their items converted recursively and
set items sorted when comparable, as BaseSerialization.serialize does. The
legacy format has no frozenset, so it is sent as a set. No class beyond
the built-in value types is constructed.
Generated-by: Claude Opus 5
* Only treat exact legacy envelopes as legacy and fail clearly on frozensets
of assets
A dict was taken for a legacy BaseSerialization envelope as soon as it had
a "__type" key, so user data such as {"__type": "record", "asset": ...}
resumed the task through the failure path, and a top-level dict with both
"__type" and "__var" plus other keys was passed through unconverted. Only
a dict whose keys are exactly "__type" and "__var" is now treated as an
envelope; any other dict is converted recursively.
Frozensets holding assets were sent as legacy sets, which older workers
cannot decode when the frozenset sits inside a set. The legacy format has
no frozenset, so these now resume through the trigger-failure path with a
clear message. Frozensets of built-in values keep the built-in path.
Generated-by: Claude Opus 5
* Never deserialize non-builtin classes behind legacy-looking dicts in
next_kwargs
The built-in value check followed only "__var" of any dict holding
"__type" and "__var", while serde's decoder unwraps such a dict and then
decodes the inner dict as a class envelope without re-checking it. A
class envelope placed inside a legacy dict envelope was therefore passed
as built-in and constructed in the API server, and other keys of such a
dict were silently dropped.
The check now mirrors the decoder and fails safe: only exact envelopes
count as envelopes, a dict envelope's content must be a plain dict, class
envelopes must have exactly the serde keys, and any other dict holding a
reserved key is left to the structural translation. A top-level payload
already in the legacy format is passed through only when every dict in it
is an envelope the legacy decoder can read; otherwise the task resumes
through the trigger-failure path. Legacy payloads are no longer decoded
in the API server at all.
Generated-by: Claude Opus 5
* Keep asset access control when translating next_kwargs for older workers
The structural Asset translation copied only name, uri, group and extra,
so a non-default access_control was dropped, while encode_asset_like()
emits it whenever producer_teams is set, consumer_teams is not None or
allow_global is false.
The translation now builds the access_control dict exactly as
encode_asset_like() does, from the serde data of AssetAccessControl, with
nothing constructed. An Asset or AssetAlias whose serde data holds a field
or shape that cannot be mapped with confidence (unknown keys, watchers,
class envelopes in extra, malformed access control) resumes through the
trigger-failure path rather than losing data silently.
Generated-by: Claude Opus 5
* Add newsfragment for older-worker next_kwargs conversion
Generated-by: Claude Opus 5
* Tell users how to avoid next_kwargs failures on older workers
Generated-by: Claude Opus 5
* Name Task SDK 1.1.x as the affected worker versions
(cherry picked from commit b0239d215e1f7c7f244a8e3fa7c3bc36fc729fe0)
Co-authored-by: Jarek Potiuk <[email protected]>
Generated-by: Claude Opus 5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
