Valkey checks off many boxes that align with Airflow’s provider criteria:
* Truly open-source * Dedicated maintainers responsive to community questions * Not a commercial service that could maintain their own provider * Very lean dependencies: https://github.com/valkey-io/valkey/blob/unstable/src/CMakeLists.txt * Popular and actively maintained with broad community support: https://github.com/valkey-io/valkey Given all of these, I believe the Valkey provider makes sense as an official Airflow provider. Sean On 2025-07-15, 1:22 PM, "Jarek Potiuk" <ja...@potiuk.com <mailto:ja...@potiuk.com>> wrote: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le contenu ne présente aucun risque. On Tue, Jul 15, 2025 at 9:52 PM Dominik Heilbock <dheilb...@gmx.net.inva <mailto:dheilb...@gmx.net.inva>lid> wrote: > As of last year, the redis Python client was still fully compatible with > Valkey, has this changed? This would make it even easier. I think they also > forked redis-py Yep. I think that's part of the solution to make the dependency tree for the provider independent of redis-controlled dependencies. Governance, decision making and release process of valley should be separate from redis (the company). That's for me one of the criteria and requirements for having a separate valley provider. Very much like "opensearch" is separate from "elasticsearch". > > Gesendet: Dienstag, den 15.07.2025 um 21:38 Uhr > > Von: "Jarek Potiuk" <ja...@potiuk.com <mailto:ja...@potiuk.com>> > > An: dev@airflow.apache.org <mailto:dev@airflow.apache.org> > > Cc: gha...@amazon.com.inva <mailto:gha...@amazon.com.inva>lid > > Betreff: Re: [DISCUSSION] New Provider: Valkey (Redis OSS Fork) > > > > > Can you please explain why it should be a provider managed by Airflow > > rather than on your own? > > > > Just to explain it Elad - and yes, I know I am usually the one that is > > sceptical -> valley is truly community driven replacement for redis > > https://valkey.io/participants/ <https://valkey.io/participants/> - you see > > a lot of well-known community > > recognised entities, and there is no single entity behind it that could > > take "responsibility" - while there are enough stakeholders to make sure > > that it will be maintained here in Airflow community. Also it's a > "drop-in" > > replacement for redis generally - and I do not expect a lot of problems > > when we start with both redis provider and redis celery integration. > > > > I've heard and saw a lot of praise from the community for Valkey being > good > > redis replacement, and - even we at the PMC and devlist discussions > > discussed about the dangers Redis licence creates for that - even now we > > limited Redis support in Helm chart of ours to the "really free" version > > and we discussed that Valkey might be a good way forward - especially if > > someone wants to contribute support for it. so if someone wants to > > contribute to Airflow AND make sure that Redis broker support in Celery > is > > covered - I am all for it. > > > > > > On Tue, Jul 15, 2025 at 9:37 PM Elad Kalif <elad...@gmail.com > > <mailto:elad...@gmail.com>> wrote: > > > > > > * Helm Chart support to run valley as Celery Broker instead of Redis > > > > > > Oh yes! didn't consider its usage as a broker but is it related to the > > > provider package? > > > Amazon SQS can be used as broker but it's not related to the SQS > > > integrations in the Airflow amazon provider (hook,operator,sensor) > > > I also don't see ValKey listed as option for brokers in > > > > > > > https://docs.celeryq.dev/en/latest/getting-started/backends-and-brokers/index.html#broker-overview > > <https://docs.celeryq.dev/en/latest/getting-started/backends-and-brokers/index.html#broker-overview> > > > > > > On Tue, Jul 15, 2025 at 10:27 PM Jarek Potiuk <ja...@potiuk.com > > > <mailto:ja...@potiuk.com>> > wrote: > > > > > > > Yes. Valkey is a good replacement/parallel implementation to Redis. > One > > > > thing however is important - I would see this even more useful if it > also > > > > allows for the Celery Redis replacement for broker - means that it is > > > fully > > > > supported in our: > > > > > > > > * unit tests (of course) > > > > * integration tests with ("valkey") integration - including running > > > > docker-compose integration to run in CI > > > > * Helm Chart support to run valley as Celery Broker instead of Redis > > > > > > > > I think I'd only consider valley as "successful" provider > integration if > > > > all that above is completed. > > > > > > > > J. > > > > > > > > On Tue, Jul 15, 2025 at 9:03 PM Ghaeli, Sean > <gha...@amazon.com.inva <mailto:gha...@amazon.com.inva>lid> > > > > wrote: > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > > > I’d like to propose the addition of a new provider to support > Valkey< > > > > > https://valkey.io/> <https://valkey.io/>>: a fully open-source, > > > > > community-governed fork > of > > > > > Redis 7.2, now maintained under the Linux Foundation. > > > > > > > > > > > > > > > > > > > > Valkey has been gaining traction recently, in part due to changes > by > > > > Redis > > > > > moving their license to one unrecognized by the Open Source > Initiative. > > > > > Valkey represents a future-proof alternative. The proposed provider > > > would > > > > > closely mirror the existing Redis provider in functionality and > serve > > > as > > > > a > > > > > drop-in, OSS-aligned option for workflows that depend on key-value > > > > storage, > > > > > pub/sub, or caching patterns. > > > > > > > > > > > > > > > > > > > > Tentatively, here are the components I plan on implementing > (mirroring > > > > the > > > > > Redis provider): > > > > > > > > > > > > > > > > > > > > * ValkeyHook – Manage connections and interactions with the > Valkey > > > > > server, enabling reuse across operators and sensors. > > > > > * ValkeyPublishOperator – Allows tasks to publish messages to > > > Valkey > > > > > pub/sub channels, useful for event-driven workflows or inter-task > > > > signaling. > > > > > * ValkeyKeySensor – Waits for a specific key to exist or reach > a > > > > > value, supporting synchronization patterns where one task's output > > > > controls > > > > > downstream logic. > > > > > * ValkeyPubSubSensor – Subscribes to a Valkey channel and > triggers > > > on > > > > > incoming messages, enabling real-time DAG triggering. > > > > > > > > > > > > > > > > > > > > Here's an example of how we could use the components to publish a > > > message > > > > > and then trigger a DAG: > > > > > > > > > > > > > > > > > > > > ` > > > > > > > > > > publish_message = ValkeyPublishOperator( > > > > > > > > > > task_id="publish_ready_signal", > > > > > > > > > > channel="processing_status", > > > > > > > > > > message="READY", > > > > > > > > > > valkey_conn_id="valkey_default", > > > > > > > > > > ) > > > > > > > > > > ` > > > > > > > > > > > > > > > > > > > > This task publishes the message "READY" to the processing_status > > > channel > > > > > on a Valkey server. Other processes listening to this channel will > > > > receive > > > > > the message and proceed with their execution. > > > > > > > > > > > > > > > > > > > > ` > > > > > > > > > > wait_for_ready = ValkeyPubSubSensor( > > > > > > > > > > task_id="wait_for_ready_message", > > > > > > > > > > channel="processing_status", > > > > > > > > > > valkey_conn_id="valkey_default", > > > > > > > > > > timeout=600, > > > > > > > > > > ) > > > > > > > > > > ` > > > > > > > > > > > > > > > > > > > > Here, a sensor waits for the message on the same channel before > > > > > continuing. Once the message is received, the DAG can continue with > > > > further > > > > > tasks—such as triggering another DAG—in an event-driven manner. > > > > > > > > > > > > > > > > > > > > Given the community's interest in event-driven scheduling (AIP 82< > > > > > https://github.com/apache/airflow/issues/52712> > > > > > <https://github.com/apache/airflow/issues/52712>>) as well as > concerns > > > > > about limited provider optionality (PR #52712< > > > > > https://github.com/apache/airflow/issues/52712> > > > > > <https://github.com/apache/airflow/issues/52712>>), a Valkey > provider > > > > would > > > > > expand user choice. > > > > > > > > > > > > > > > > > > > > I’d be happy to begin working on the initial PR and documentation. > > > > > > > > > > > > > > > > > > > > Looking forward to feedback. > > > > > > > > > > Sean > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > <mailto:dev-unsubscr...@airflow.apache.org> > For additional commands, e-mail: dev-h...@airflow.apache.org > <mailto:dev-h...@airflow.apache.org> > >
