SEPURI-SAI-KRISHNA opened a new pull request, #74843:
URL: https://github.com/apache/airflow/pull/74843
Passing a `botocore.config.Config` object as `botocore_config` works for an
Amazon operator or sensor that runs to completion, and fails the task once it
defers.
Reproduced on current main with `SsmRunCommandOperator`:
```python
from botocore.config import Config
SsmRunCommandOperator(
task_id="run",
document_name="AWS-RunShellScript",
run_command_kwargs={"InstanceIds": ["i-1234567890abcdef0"]},
botocore_config=Config(read_timeout=120, retries={"max_attempts": 5}),
deferrable=True,
)
```
`execute` calls `send_command` first and defers afterwards, so by the time
the trigger argument is serialized the command is already running on the
instance. Serialization then raises `TypeError: Cannot serialize object of type
<class 'botocore.config.Config'>` and the task fails, leaving the command
running with nothing watching it or reporting its result. The non deferrable
form of the same task succeeds, and so does the same task with
`botocore_config={"read_timeout": 120, "retries": {"max_attempts": 5}}`.
The mapping is what the interface declares. Every operator and sensor
documents the parameter as `botocore_config: dict | None` and as a
"Configuration dictionary (key-values)", and the trigger signatures repeat it.
Only the hooks widen it, declaring `config: Config | None`, which is how a
`Config` gets this far at all. Both forms produce the same client, verified by
building an SSM client each way and reading back `read_timeout=120`.
This resolves the parameter to the declared mapping where the hook
parameters are assembled, so the value a trigger receives is one it can carry.
Nothing that works today changes behaviour: a dictionary and `None` are
returned untouched, and a `Config` yields the options it was constructed with,
which the hook rebuilds into an equivalent `Config`.
All 39 operator and sensor modules that forward `botocore_config` to a
trigger derive from `AwsBaseOperator` or `AwsBaseSensor`, and all three
constructors that resolve these parameters route through
`AwsHookParams.from_constructor`, so the single place covers the whole surface.
Only the options the caller actually passed are carried over, so the result
does not pin whichever defaults the installed botocore happens to carry. If a
future botocore stops exposing them, the value is passed through unchanged and
the behaviour is the one in place today rather than a new failure.
Verified locally on Python 3.12 against SQLite:
- The new tests pass, and with the production change reverted the two that
cover it fail, the operator-level one reporting a `Config` object where the
dictionary is expected.
- `providers/amazon/tests/unit/amazon/aws/utils/test_mixins.py`,
`operators/test_base_aws.py`, `sensors/test_base_aws.py` and
`utils/test_connection_wrapper.py`: 126 passed.
- `hooks/test_base_aws.py`, `operators/test_ssm.py`, `sensors/test_ssm.py`
and the whole `triggers/` directory: 599 passed, with 1 failure and 2 errors
that are present without this change as well and come from provider-version and
waiter-config fixtures unrelated to it.
- `scripts/ci/prek/check_deferred_hook_configuration.py`: clean.
- `ruff format` and `ruff check` on both changed files: clean.
mypy was not run; it needs the Breeze image, which this machine cannot start.
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes - Claude Code (Opus 5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
