gocoolp opened a new issue, #69051:
URL: https://github.com/apache/airflow/issues/69051
### Description
DynamoDBValueSensor currently only supports blocking poll mode — it calls
table.get_item() on a fixed poke_interval and holds an Airflow worker slot for
the entire wait duration. There is no deferrable mode, no DynamoDBValueTrigger,
and no event-driven path.
### Use case/motivation
A common Airflow pattern is using DynamoDB as a lightweight approval or
job-state store — write metadata on submission, wait for a status field to flip
to a target value (e.g. APPROVED, COMPLETED) before proceeding. The wait window
can be hours (human approval workflows) or unpredictable (async external jobs).
With blocking poll mode:
- A worker slot is held for the entire wait duration
- Approval-to-execution latency equals up to one full poke_interval even
after the value changes
- At scale, long-running sensors starve the worker pool
Proposed: add a deferrable mode to DynamoDBValueSensor backed by a new
DynamoDBValueTrigger, using DynamoDB Streams → SQS as the event source:
1. DynamoDB Streams captures item changes (stream view: NEW_IMAGE)
2. EventBridge Pipe or Lambda forwards records to an SQS queue
3. DynamoDBValueTrigger consumes SQS asynchronously (following the
existing SqsSensorTrigger pattern in triggers/sqs.py), filtering for records
where the target attribute matches the expected value
4. On match, fires a TriggerEvent and the sensor resumes
Sensor interface (backwards compatible):
DynamoDBValueSensor(
...,
deferrable=True,
stream_sqs_queue_url="https://sqs.us-east-1.amazonaws.com/...";,
waiter_delay=30,
)
When deferrable=False (default), existing polling behaviour is unchanged.
### Related issues
- Existing SqsSensorTrigger:
providers/amazon/src/airflow/providers/amazon/aws/triggers/sqs.py
- Existing DynamoDBValueSensor:
providers/amazon/src/airflow/providers/amazon/aws/sensors/dynamodb.py
- Related CRUD hook PR: #69050
### Are you willing to submit a PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
