tardunge opened a new pull request, #63937: URL: https://github.com/apache/airflow/pull/63937
Adds durable list-based messaging to the Redis provider using LPUSH/BRPOP, complementing the existing pub/sub support. **Motivation:** Redis pub/sub is fire-and-forget — messages are lost if no subscriber is listening. List-based queues (LPUSH + BRPOP) provide message persistence until consumed, exactly-once delivery via atomic BRPOP, and priority queue patterns via ordered multi-list BRPOP. We've been running this in production for ~8 months. **New components:** - `RedisLPushOperator` — push messages to Redis lists (FIFO queue producer) - `AwaitMessageFromListTrigger` — async trigger using BRPOP with priority queue support (multiple lists checked in order) - `RedisListMessageQueueProvider` — common-messaging integration with `redis+list://` scheme - `RedisHook.get_async_conn()` — async context manager using `redis.asyncio` for truly non-blocking trigger operations (the existing hook is sync-only, which requires `sync_to_async` wrapping in triggers) **No new dependencies** — `redis.asyncio` is part of `redis>=4.2.0`, and the provider already requires `redis>=4.5.2`. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code (Opus 4.6) Generated-by: Claude Code (Opus 4.6) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
