SEPURI-SAI-KRISHNA commented on issue #76805:
URL: https://github.com/apache/airflow/issues/76805#issuecomment-6789706284
Verified amazon 9.40.0rc2 for #74843 on Airflow 3.4.0, Python 3.12.3,
botocore 1.44.0.
One script run unchanged against 9.39.0 and 9.40.0rc2, with
`MwaaTriggerDagRunOperator(deferrable=True, wait_for_completion=True,
botocore_config=Config(read_timeout=120, retries={"max_attempts": 5}))` and
only `invoke_rest_api` mocked, so this covers the defer boundary rather than a
live MWAA environment. The Dag run is created before the task defers in both.
- 9.39.0: the `Config` object reaches the trigger, and
`airflow.sdk.serde.serialize` on the trigger arguments, which is what
`Trigger.encrypt_kwargs` uses to store them, raises `TypeError: Cannot
serialize object of type <class 'botocore.config.Config'>`. The Dag run already
exists at that point, so it is orphaned and a retry triggers it again.
- 9.40.0rc2: it is reduced to `{"read_timeout": 120, "retries":
{"max_attempts": 5}}`, the trigger arguments serialize, and after a store and
reload the trigger's hook has `read_timeout` 120, `retries`
`{"total_max_attempts": 6, "mode": "standard"}` and region `eu-west-1`.
Both forms build the same client, with the resolved configuration equal on
`connect_timeout`, `read_timeout`, `retries`, `max_pool_connections`,
`region_name`, `signature_version`, `user_agent_extra` and `tcp_keepalive`.
Dict and tuple option values, a `Config` subclass that injects its own default,
and the sensor half of the mixin behave the same. One visible change:
`operator.botocore_config` is now a dict where a passed `Config` used to be
kept as given.
One case is not covered. An option value that is not a JSON type survives
the reduction and still fails: `Config(signature_version=botocore.UNSIGNED)`
reduces to `{"signature_version": <botocore.UNSIGNED object>}` and raises
`TypeError: Cannot serialize object of type <class 'botocore.UNSIGNED'>` at
defer time. This is not a regression, since 9.39.0 fails on every `Config`, and
the client the mapping builds is still unsigned, so only the trigger arguments
fail. This is older than the change and wider than it: S3 sensors have passed
`botocore_config` to their trigger since #47321. Anonymous access configured on
the connection is unaffected, since `AwsConnectionWrapper` translates a
`signature_version` of `"unsigned"` to the sentinel, so I would not hold the
release for it.
`aws/utils/mixins.py` in the rc2 wheel is byte identical by sha256 to that
file at main `d021d1e535`, and is the only source file #74843 changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
