> Can you please explain why it should be a provider managed by Airflow rather than on your own?
Just to explain it Elad - and yes, I know I am usually the one that is sceptical -> valley is truly community driven replacement for redis https://valkey.io/participants/ - you see a lot of well-known community recognised entities, and there is no single entity behind it that could take "responsibility" - while there are enough stakeholders to make sure that it will be maintained here in Airflow community. Also it's a "drop-in" replacement for redis generally - and I do not expect a lot of problems when we start with both redis provider and redis celery integration. I've heard and saw a lot of praise from the community for Valkey being good redis replacement, and - even we at the PMC and devlist discussions discussed about the dangers Redis licence creates for that - even now we limited Redis support in Helm chart of ours to the "really free" version and we discussed that Valkey might be a good way forward - especially if someone wants to contribute support for it. so if someone wants to contribute to Airflow AND make sure that Redis broker support in Celery is covered - I am all for it. On Tue, Jul 15, 2025 at 9:37 PM Elad Kalif <elad...@gmail.com> wrote: > > * Helm Chart support to run valley as Celery Broker instead of Redis > > Oh yes! didn't consider its usage as a broker but is it related to the > provider package? > Amazon SQS can be used as broker but it's not related to the SQS > integrations in the Airflow amazon provider (hook,operator,sensor) > I also don't see ValKey listed as option for brokers in > > https://docs.celeryq.dev/en/latest/getting-started/backends-and-brokers/index.html#broker-overview > > On Tue, Jul 15, 2025 at 10:27 PM Jarek Potiuk <ja...@potiuk.com> wrote: > > > Yes. Valkey is a good replacement/parallel implementation to Redis. One > > thing however is important - I would see this even more useful if it also > > allows for the Celery Redis replacement for broker - means that it is > fully > > supported in our: > > > > * unit tests (of course) > > * integration tests with ("valkey") integration - including running > > docker-compose integration to run in CI > > * Helm Chart support to run valley as Celery Broker instead of Redis > > > > I think I'd only consider valley as "successful" provider integration if > > all that above is completed. > > > > J. > > > > On Tue, Jul 15, 2025 at 9:03 PM Ghaeli, Sean <gha...@amazon.com.invalid> > > wrote: > > > > > Hi all, > > > > > > > > > > > > I’d like to propose the addition of a new provider to support Valkey< > > > https://valkey.io/>: a fully open-source, community-governed fork of > > > Redis 7.2, now maintained under the Linux Foundation. > > > > > > > > > > > > Valkey has been gaining traction recently, in part due to changes by > > Redis > > > moving their license to one unrecognized by the Open Source Initiative. > > > Valkey represents a future-proof alternative. The proposed provider > would > > > closely mirror the existing Redis provider in functionality and serve > as > > a > > > drop-in, OSS-aligned option for workflows that depend on key-value > > storage, > > > pub/sub, or caching patterns. > > > > > > > > > > > > Tentatively, here are the components I plan on implementing (mirroring > > the > > > Redis provider): > > > > > > > > > > > > * ValkeyHook – Manage connections and interactions with the Valkey > > > server, enabling reuse across operators and sensors. > > > * ValkeyPublishOperator – Allows tasks to publish messages to > Valkey > > > pub/sub channels, useful for event-driven workflows or inter-task > > signaling. > > > * ValkeyKeySensor – Waits for a specific key to exist or reach a > > > value, supporting synchronization patterns where one task's output > > controls > > > downstream logic. > > > * ValkeyPubSubSensor – Subscribes to a Valkey channel and triggers > on > > > incoming messages, enabling real-time DAG triggering. > > > > > > > > > > > > Here's an example of how we could use the components to publish a > message > > > and then trigger a DAG: > > > > > > > > > > > > ` > > > > > > publish_message = ValkeyPublishOperator( > > > > > > task_id="publish_ready_signal", > > > > > > channel="processing_status", > > > > > > message="READY", > > > > > > valkey_conn_id="valkey_default", > > > > > > ) > > > > > > ` > > > > > > > > > > > > This task publishes the message "READY" to the processing_status > channel > > > on a Valkey server. Other processes listening to this channel will > > receive > > > the message and proceed with their execution. > > > > > > > > > > > > ` > > > > > > wait_for_ready = ValkeyPubSubSensor( > > > > > > task_id="wait_for_ready_message", > > > > > > channel="processing_status", > > > > > > valkey_conn_id="valkey_default", > > > > > > timeout=600, > > > > > > ) > > > > > > ` > > > > > > > > > > > > Here, a sensor waits for the message on the same channel before > > > continuing. Once the message is received, the DAG can continue with > > further > > > tasks—such as triggering another DAG—in an event-driven manner. > > > > > > > > > > > > Given the community's interest in event-driven scheduling (AIP 82< > > > https://github.com/apache/airflow/issues/52712>) as well as concerns > > > about limited provider optionality (PR #52712< > > > https://github.com/apache/airflow/issues/52712>), a Valkey provider > > would > > > expand user choice. > > > > > > > > > > > > I’d be happy to begin working on the initial PR and documentation. > > > > > > > > > > > > Looking forward to feedback. > > > > > > Sean > > > > > > > > >
