Just to clarify - for the "Open Governance" argument part - it is important to me, mostly because there is no single entity that can exclusively benefit from us maintaining the provider. So I simply do not buy the argument that "some 3rd party provider can release it". I just think this argument could ONLY be used when such a 3rd-party provider actually exists. And I suggest that in future discussions we simply take into account the "governance" of such proposed providers, and do not use the argument when it is not really valid. That was my main "argument" and "disagreement" here.
On the other argument - popularity, and whether we actually "need" a separate provider for it - indeed it would be great to have some information from people who use "redis" already (and how). And have some numbers - though - we know currently, having reliable numbers is not easy - or even not possible and currently even asking for such numbers is really asking someone to complete an impossible task. Yes. We can "ask" for numbers, but if we know that getting those numbers cannot be provided, this is really equivalent to "NO". And I am also not strongly for "yes we should accept it now" - I think we should consider (and this is more of an intuition, hearing though grapevines and leap of faith than hard data) that we - as a community - can make by believing that it **might** be useful. To be honest - of course redis/valkey support is not useful for ETL - for sure. But at least my intuition tells me that in-memory key value storage might be very useful to keep context for all kinds of agentic workflows in the near future. We currently do not provide any "real" solution for that (we discuss state management for watermarks [1] but we don't provide our workflows a way to do so. Having a provider, where people can easily use Valkey Hooks for such state management in their task-flow tasks seems like awfully good idea I think. And from "grapevine" discussions i know many companies turn their back to redis and replace it with Valkey as we speak due to the licensing changes of Redis. Even we ourselves decided that we stick to the old version of redis in our helm chart because of that [2]. So I think this discussion is more about the "future" than the past and what we "believe" might be useful. And yes - we can decide to wait, but also we have to remember that we also have great powers of influencing data engineering space. If **we** decide that Valkey is first-class-citizen in Airflow - people might follow. And I think we should carefully and deliberately use that power as a community. So I would love to hear what others think about it as well ? J [1] Watermarks state AIP draft https://cwiki.apache.org/confluence/display/AIRFLOW/%5BDRAFT%5D+AIP-93+Watermarks [2] Discussion on our redis approach https://lists.apache.org/thread/4mjs2mqcgxswk96wwvw1rjc969lq5rl7 On Thu, Aug 14, 2025 at 12:02 PM Ash Berlin-Taylor <a...@apache.org> wrote: > I’m with Elad on this, -1. The open nature/source/core or not of the > project isn’t the key aspect for this, but simply the usefulness of it at > all > > > 1. Do we need a separate provider for it? It already bugs me that we have > separate ElasticSearch and OpenSearch providers and this feels like the > same thing all over — we’re having to maintain two copies of a provider > that is otherwise identical due to upstream OSS vs commercial spats. > > 2. Is it even worth having a redis provider? I have a feeling the > popularity of the redis provider is a side-effect of using reds for Celery > broker `apache-airflow[celery,redis]` being installed. We have no way of > knowing but I would make the case that 99% of cases don’t use the hooks, > operators or sensors. > > -ash > > > > On 13 Aug 2025, at 11:06, Jarek Potiuk <ja...@potiuk.com> wrote: > > > >> Since Valkey 8.1.x (latest serie) is still 100% compatible with Redis, > >> wouldn't be easier to just state in docs Redis provider is also > >> compatible with Valkey backend? > > > > Yep, that's one of the options. We discussed it above - that we could > have > > valley/redis switch for example in the current redis provider - it will > > make it more coupled and future features might diverge, but yes, this is > > indeed a valid option, I think it's more of the question of name in this > > case. If for example we rename redis provider to "kv-storage" or > > "common-kv-storage" for example or something similar, that might be > > actually quite a viable option for us. > > > > J. > > > > > > On Wed, Aug 13, 2025 at 11:44 AM Josef Šimánek <josef.sima...@gmail.com> > > wrote: > > > >> st 13. 8. 2025 v 0:38 odesílatel Ghaeli, Sean > >> <gha...@amazon.com.invalid> napsal: > >>> > >>> Valkey checks off many boxes that align with Airflow’s provider > criteria: > >>> > >>> * Truly open-source > >>> * Dedicated maintainers responsive to community questions > >>> * Not a commercial service that could maintain their own provider > >>> * Very lean dependencies: > >> https://github.com/valkey-io/valkey/blob/unstable/src/CMakeLists.txt > >>> * Popular and actively maintained with broad community support: > >> https://github.com/valkey-io/valkey > >>> > >>> Given all of these, I believe the Valkey provider makes sense as an > >> official Airflow provider. > >> > >> Since Valkey 8.1.x (latest serie) is still 100% compatible with Redis, > >> wouldn't be easier to just state in docs Redis provider is also > >> compatible with Valkey backend? > >> > >>> Sean > >>> > >>> On 2025-07-15, 1:22 PM, "Jarek Potiuk" <ja...@potiuk.com <mailto: > >> ja...@potiuk.com>> wrote: > >>> > >>> > >>> CAUTION: This email originated from outside of the organization. Do not > >> click links or open attachments unless you can confirm the sender and > know > >> the content is safe. > >>> > >>> > >>> > >>> > >>> > >>> > >>> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur > >> externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si > vous > >> ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas > >> certain que le contenu ne présente aucun risque. > >>> > >>> > >>> > >>> > >>> > >>> > >>> On Tue, Jul 15, 2025 at 9:52 PM Dominik Heilbock > <dheilb...@gmx.net.inva > >> <mailto:dheilb...@gmx.net.inva>lid> > >>> wrote: > >>> > >>> > >>>> As of last year, the redis Python client was still fully compatible > >> with > >>>> Valkey, has this changed? This would make it even easier. I think they > >> also > >>>> forked redis-py > >>> > >>> > >>> > >>> > >>> Yep. I think that's part of the solution to make the dependency tree > for > >>> the provider independent of redis-controlled dependencies. > >>> > >>> > >>> Governance, decision making and release process of valley should be > >>> separate from redis (the company). That's for me one of the criteria > and > >>> requirements for having a separate valley provider. Very much like > >>> "opensearch" is separate from "elasticsearch". > >>> > >>> > >>> > >>> > >>> > >>> > >>>> > >>> > >>> > >>>> Gesendet: Dienstag, den 15.07.2025 um 21:38 Uhr > >>>>> Von: "Jarek Potiuk" <ja...@potiuk.com <mailto:ja...@potiuk.com>> > >>>>> An: dev@airflow.apache.org <mailto:dev@airflow.apache.org> > >>>>> Cc: gha...@amazon.com.inva <mailto:gha...@amazon.com.inva>lid > >>>>> Betreff: Re: [DISCUSSION] New Provider: Valkey (Redis OSS Fork) > >>>>> > >>>>>> Can you please explain why it should be a provider managed by > >> Airflow > >>>>> rather than on your own? > >>>>> > >>>>> Just to explain it Elad - and yes, I know I am usually the one that > >> is > >>>>> sceptical -> valley is truly community driven replacement for redis > >>>>> https://valkey.io/participants/ <https://valkey.io/participants/> - > >> you see a lot of well-known community > >>>>> recognised entities, and there is no single entity behind it that > >> could > >>>>> take "responsibility" - while there are enough stakeholders to make > >> sure > >>>>> that it will be maintained here in Airflow community. Also it's a > >>>> "drop-in" > >>>>> replacement for redis generally - and I do not expect a lot of > >> problems > >>>>> when we start with both redis provider and redis celery integration. > >>>>> > >>>>> I've heard and saw a lot of praise from the community for Valkey > >> being > >>>> good > >>>>> redis replacement, and - even we at the PMC and devlist discussions > >>>>> discussed about the dangers Redis licence creates for that - even > >> now we > >>>>> limited Redis support in Helm chart of ours to the "really free" > >> version > >>>>> and we discussed that Valkey might be a good way forward - > >> especially if > >>>>> someone wants to contribute support for it. so if someone wants to > >>>>> contribute to Airflow AND make sure that Redis broker support in > >> Celery > >>>> is > >>>>> covered - I am all for it. > >>>>> > >>>>> > >>>>> On Tue, Jul 15, 2025 at 9:37 PM Elad Kalif <elad...@gmail.com > >> <mailto:elad...@gmail.com>> wrote: > >>>>> > >>>>>>> * Helm Chart support to run valley as Celery Broker instead of > >> Redis > >>>>>> > >>>>>> Oh yes! didn't consider its usage as a broker but is it related to > >> the > >>>>>> provider package? > >>>>>> Amazon SQS can be used as broker but it's not related to the SQS > >>>>>> integrations in the Airflow amazon provider (hook,operator,sensor) > >>>>>> I also don't see ValKey listed as option for brokers in > >>>>>> > >>>>>> > >>>> > >> > https://docs.celeryq.dev/en/latest/getting-started/backends-and-brokers/index.html#broker-overview > >> < > >> > https://docs.celeryq.dev/en/latest/getting-started/backends-and-brokers/index.html#broker-overview > >>> > >>>>>> > >>>>>> On Tue, Jul 15, 2025 at 10:27 PM Jarek Potiuk <ja...@potiuk.com > >> <mailto:ja...@potiuk.com>> > >>>> wrote: > >>>>>> > >>>>>>> Yes. Valkey is a good replacement/parallel implementation to > >> Redis. > >>>> One > >>>>>>> thing however is important - I would see this even more useful > >> if it > >>>> also > >>>>>>> allows for the Celery Redis replacement for broker - means that > >> it is > >>>>>> fully > >>>>>>> supported in our: > >>>>>>> > >>>>>>> * unit tests (of course) > >>>>>>> * integration tests with ("valkey") integration - including > >> running > >>>>>>> docker-compose integration to run in CI > >>>>>>> * Helm Chart support to run valley as Celery Broker instead of > >> Redis > >>>>>>> > >>>>>>> I think I'd only consider valley as "successful" provider > >>>> integration if > >>>>>>> all that above is completed. > >>>>>>> > >>>>>>> J. > >>>>>>> > >>>>>>> On Tue, Jul 15, 2025 at 9:03 PM Ghaeli, Sean > >>>> <gha...@amazon.com.inva <mailto:gha...@amazon.com.inva>lid> > >>>>>>> wrote: > >>>>>>> > >>>>>>>> Hi all, > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> I’d like to propose the addition of a new provider to support > >>>> Valkey< > >>>>>>>> https://valkey.io/> <https://valkey.io/>>: a fully > >> open-source, community-governed fork > >>>> of > >>>>>>>> Redis 7.2, now maintained under the Linux Foundation. > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> Valkey has been gaining traction recently, in part due to > >> changes > >>>> by > >>>>>>> Redis > >>>>>>>> moving their license to one unrecognized by the Open Source > >>>> Initiative. > >>>>>>>> Valkey represents a future-proof alternative. The proposed > >> provider > >>>>>> would > >>>>>>>> closely mirror the existing Redis provider in functionality and > >>>> serve > >>>>>> as > >>>>>>> a > >>>>>>>> drop-in, OSS-aligned option for workflows that depend on > >> key-value > >>>>>>> storage, > >>>>>>>> pub/sub, or caching patterns. > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> Tentatively, here are the components I plan on implementing > >>>> (mirroring > >>>>>>> the > >>>>>>>> Redis provider): > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> * ValkeyHook – Manage connections and interactions with the > >>>> Valkey > >>>>>>>> server, enabling reuse across operators and sensors. > >>>>>>>> * ValkeyPublishOperator – Allows tasks to publish messages to > >>>>>> Valkey > >>>>>>>> pub/sub channels, useful for event-driven workflows or > >> inter-task > >>>>>>> signaling. > >>>>>>>> * ValkeyKeySensor – Waits for a specific key to exist or reach > >>>> a > >>>>>>>> value, supporting synchronization patterns where one task's > >> output > >>>>>>> controls > >>>>>>>> downstream logic. > >>>>>>>> * ValkeyPubSubSensor – Subscribes to a Valkey channel and > >>>> triggers > >>>>>> on > >>>>>>>> incoming messages, enabling real-time DAG triggering. > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> Here's an example of how we could use the components to > >> publish a > >>>>>> message > >>>>>>>> and then trigger a DAG: > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> ` > >>>>>>>> > >>>>>>>> publish_message = ValkeyPublishOperator( > >>>>>>>> > >>>>>>>> task_id="publish_ready_signal", > >>>>>>>> > >>>>>>>> channel="processing_status", > >>>>>>>> > >>>>>>>> message="READY", > >>>>>>>> > >>>>>>>> valkey_conn_id="valkey_default", > >>>>>>>> > >>>>>>>> ) > >>>>>>>> > >>>>>>>> ` > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> This task publishes the message "READY" to the > >> processing_status > >>>>>> channel > >>>>>>>> on a Valkey server. Other processes listening to this channel > >> will > >>>>>>> receive > >>>>>>>> the message and proceed with their execution. > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> ` > >>>>>>>> > >>>>>>>> wait_for_ready = ValkeyPubSubSensor( > >>>>>>>> > >>>>>>>> task_id="wait_for_ready_message", > >>>>>>>> > >>>>>>>> channel="processing_status", > >>>>>>>> > >>>>>>>> valkey_conn_id="valkey_default", > >>>>>>>> > >>>>>>>> timeout=600, > >>>>>>>> > >>>>>>>> ) > >>>>>>>> > >>>>>>>> ` > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> Here, a sensor waits for the message on the same channel before > >>>>>>>> continuing. Once the message is received, the DAG can continue > >> with > >>>>>>> further > >>>>>>>> tasks—such as triggering another DAG—in an event-driven manner. > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> Given the community's interest in event-driven scheduling (AIP > >> 82< > >>>>>>>> https://github.com/apache/airflow/issues/52712> < > >> https://github.com/apache/airflow/issues/52712>>) as well as > >>>> concerns > >>>>>>>> about limited provider optionality (PR #52712< > >>>>>>>> https://github.com/apache/airflow/issues/52712> < > >> https://github.com/apache/airflow/issues/52712>>), a Valkey > >>>> provider > >>>>>>> would > >>>>>>>> expand user choice. > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> I’d be happy to begin working on the initial PR and > >> documentation. > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> Looking forward to feedback. > >>>>>>>> > >>>>>>>> Sean > >>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>> > >>>> --------------------------------------------------------------------- > >>>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org <mailto: > >> dev-unsubscr...@airflow.apache.org> > >>>> For additional commands, e-mail: dev-h...@airflow.apache.org <mailto: > >> dev-h...@airflow.apache.org> > >>>> > >>>> > >>> > >>> > >>> > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > >> For additional commands, e-mail: dev-h...@airflow.apache.org > >> > >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > For additional commands, e-mail: dev-h...@airflow.apache.org > >
