I already stated my position but I will mention the key aspects. I don't think the fact of ValKey being a "true" open source is not about that. Great Expectation is also true open source and they maintained their own provider till they donated it to Astronomer. I still haven't heard why ValKey provider can be hosted on ValKey repo or under AWS.
I am waiting for any users on this mailing list to present a real life use case for using ValKey with offline processing. My stand is that it's very very low demand (similar to Redis!) ValKey can be extremely useful as a broker for Celery and I welcome AWS (or anyone else) to add integration for this in Helm Chart (That doesn't require a provider!) The three answers I am seeking are: 1. How popular we expect it to be with dag authors and explain the reasoning for the number/estimation. 2. Is the popularity cross the threshold of accepting it as official Airflow provider or leave it as 3rd party and I'm saying again... Great Expectation is not yet an official provider so... 3. Maintainability commitment that includes also issue triage and on-going support for users. On Wed, Aug 13, 2025 at 12:04 PM Jarek Potiuk <ja...@potiuk.com> wrote: > To make it easier -> https://valkey.io/participants/ here is the list of > companies to participate in valley development. Similarly to Airflow and > other ASF projects - Valkey seem to be "truly" community driven, and when I > look at the list, it's hard to pin-point that single entity who would be > taking it on (at least it's hard for me). > > On Wed, Aug 13, 2025 at 11:01 AM Jarek Potiuk <ja...@potiuk.com> wrote: > > > > at this point I am still -1 > > > > > What is the downside of running it as a 3rd party provider and if/when > it > > > gains popularity donate it to airflow? > > Just out of curiosity - since Valkey is a community project, not > > controlled by a single entity nor having money or team to actually > release > > and manage Airflow integration. Who do you think is the best 3rd-party to > > release the provider? > > Usually those arguments about the 3rd-party we raise when there is - > > clearly - such a 3rd-party - but I fail to see who that 3rd-party would > be > > - what do you think Elad? Can you point such a 3rd-party? > > > > J. > > > > > > On Wed, Aug 13, 2025 at 10:55 AM Elad Kalif <elad...@apache.org> wrote: > > > >> at this point I am still -1 > >> > >> On Wed, Aug 13, 2025 at 1:43 AM Jarek Potiuk <ja...@potiuk.com> wrote: > >> > >> > If you would like to contribute - start a PR and get a VOTE on it. > >> That's > >> > what most people do when they want to add a new provider or call for > >> LAZY > >> > CONSENSUS if you think consensus is reached and you can get it by the > >> lazy > >> > consensus https://www.apache.org/foundation/voting.html > >> > > >> > On Wed, Aug 13, 2025 at 12:38 AM Ghaeli, Sean > <gha...@amazon.com.invalid > >> > > >> > wrote: > >> > > >> > > Valkey checks off many boxes that align with Airflow’s provider > >> criteria: > >> > > > >> > > * Truly open-source > >> > > * Dedicated maintainers responsive to community questions > >> > > * Not a commercial service that could maintain their own provider > >> > > * Very lean dependencies: > >> > > > https://github.com/valkey-io/valkey/blob/unstable/src/CMakeLists.txt > >> > > * Popular and actively maintained with broad community support: > >> > > https://github.com/valkey-io/valkey > >> > > > >> > > Given all of these, I believe the Valkey provider makes sense as an > >> > > official Airflow provider. > >> > > > >> > > Sean > >> > > > >> > > On 2025-07-15, 1:22 PM, "Jarek Potiuk" <ja...@potiuk.com <mailto: > >> > > ja...@potiuk.com>> wrote: > >> > > > >> > > > >> > > CAUTION: This email originated from outside of the organization. Do > >> not > >> > > click links or open attachments unless you can confirm the sender > and > >> > know > >> > > the content is safe. > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur > >> externe. > >> > > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne > >> > pouvez > >> > > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas > certain > >> > que > >> > > le contenu ne présente aucun risque. > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > On Tue, Jul 15, 2025 at 9:52 PM Dominik Heilbock > >> <dheilb...@gmx.net.inva > >> > > <mailto:dheilb...@gmx.net.inva>lid> > >> > > wrote: > >> > > > >> > > > >> > > > As of last year, the redis Python client was still fully > compatible > >> > with > >> > > > Valkey, has this changed? This would make it even easier. I think > >> they > >> > > also > >> > > > forked redis-py > >> > > > >> > > > >> > > > >> > > > >> > > Yep. I think that's part of the solution to make the dependency tree > >> for > >> > > the provider independent of redis-controlled dependencies. > >> > > > >> > > > >> > > Governance, decision making and release process of valley should be > >> > > separate from redis (the company). That's for me one of the criteria > >> and > >> > > requirements for having a separate valley provider. Very much like > >> > > "opensearch" is separate from "elasticsearch". > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > > >> > > > >> > > > >> > > > Gesendet: Dienstag, den 15.07.2025 um 21:38 Uhr > >> > > > > Von: "Jarek Potiuk" <ja...@potiuk.com <mailto:ja...@potiuk.com > >> > >> > > > > An: dev@airflow.apache.org <mailto:dev@airflow.apache.org> > >> > > > > Cc: gha...@amazon.com.inva <mailto:gha...@amazon.com.inva>lid > >> > > > > Betreff: Re: [DISCUSSION] New Provider: Valkey (Redis OSS Fork) > >> > > > > > >> > > > > > Can you please explain why it should be a provider managed by > >> > Airflow > >> > > > > rather than on your own? > >> > > > > > >> > > > > Just to explain it Elad - and yes, I know I am usually the one > >> that > >> > is > >> > > > > sceptical -> valley is truly community driven replacement for > >> redis > >> > > > > https://valkey.io/participants/ < > https://valkey.io/participants/> > >> - > >> > > you see a lot of well-known community > >> > > > > recognised entities, and there is no single entity behind it > that > >> > could > >> > > > > take "responsibility" - while there are enough stakeholders to > >> make > >> > > sure > >> > > > > that it will be maintained here in Airflow community. Also it's > a > >> > > > "drop-in" > >> > > > > replacement for redis generally - and I do not expect a lot of > >> > problems > >> > > > > when we start with both redis provider and redis celery > >> integration. > >> > > > > > >> > > > > I've heard and saw a lot of praise from the community for Valkey > >> > being > >> > > > good > >> > > > > redis replacement, and - even we at the PMC and devlist > >> discussions > >> > > > > discussed about the dangers Redis licence creates for that - > even > >> now > >> > > we > >> > > > > limited Redis support in Helm chart of ours to the "really free" > >> > > version > >> > > > > and we discussed that Valkey might be a good way forward - > >> especially > >> > > if > >> > > > > someone wants to contribute support for it. so if someone wants > to > >> > > > > contribute to Airflow AND make sure that Redis broker support in > >> > Celery > >> > > > is > >> > > > > covered - I am all for it. > >> > > > > > >> > > > > > >> > > > > On Tue, Jul 15, 2025 at 9:37 PM Elad Kalif <elad...@gmail.com > >> > <mailto: > >> > > elad...@gmail.com>> wrote: > >> > > > > > >> > > > > > > * Helm Chart support to run valley as Celery Broker instead > of > >> > > Redis > >> > > > > > > >> > > > > > Oh yes! didn't consider its usage as a broker but is it > related > >> to > >> > > the > >> > > > > > provider package? > >> > > > > > Amazon SQS can be used as broker but it's not related to the > SQS > >> > > > > > integrations in the Airflow amazon provider > >> (hook,operator,sensor) > >> > > > > > I also don't see ValKey listed as option for brokers in > >> > > > > > > >> > > > > > > >> > > > > >> > > > >> > > >> > https://docs.celeryq.dev/en/latest/getting-started/backends-and-brokers/index.html#broker-overview > >> > > < > >> > > > >> > > >> > https://docs.celeryq.dev/en/latest/getting-started/backends-and-brokers/index.html#broker-overview > >> > > > > >> > > > > > > >> > > > > > On Tue, Jul 15, 2025 at 10:27 PM Jarek Potiuk < > ja...@potiuk.com > >> > > <mailto:ja...@potiuk.com>> > >> > > > wrote: > >> > > > > > > >> > > > > > > Yes. Valkey is a good replacement/parallel implementation to > >> > Redis. > >> > > > One > >> > > > > > > thing however is important - I would see this even more > >> useful if > >> > > it > >> > > > also > >> > > > > > > allows for the Celery Redis replacement for broker - means > >> that > >> > it > >> > > is > >> > > > > > fully > >> > > > > > > supported in our: > >> > > > > > > > >> > > > > > > * unit tests (of course) > >> > > > > > > * integration tests with ("valkey") integration - including > >> > running > >> > > > > > > docker-compose integration to run in CI > >> > > > > > > * Helm Chart support to run valley as Celery Broker instead > of > >> > > Redis > >> > > > > > > > >> > > > > > > I think I'd only consider valley as "successful" provider > >> > > > integration if > >> > > > > > > all that above is completed. > >> > > > > > > > >> > > > > > > J. > >> > > > > > > > >> > > > > > > On Tue, Jul 15, 2025 at 9:03 PM Ghaeli, Sean > >> > > > <gha...@amazon.com.inva <mailto:gha...@amazon.com.inva>lid> > >> > > > > > > wrote: > >> > > > > > > > >> > > > > > > > Hi all, > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > I’d like to propose the addition of a new provider to > >> support > >> > > > Valkey< > >> > > > > > > > https://valkey.io/> <https://valkey.io/>>: a fully > >> > > open-source, community-governed fork > >> > > > of > >> > > > > > > > Redis 7.2, now maintained under the Linux Foundation. > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > Valkey has been gaining traction recently, in part due to > >> > changes > >> > > > by > >> > > > > > > Redis > >> > > > > > > > moving their license to one unrecognized by the Open > Source > >> > > > Initiative. > >> > > > > > > > Valkey represents a future-proof alternative. The proposed > >> > > provider > >> > > > > > would > >> > > > > > > > closely mirror the existing Redis provider in > functionality > >> and > >> > > > serve > >> > > > > > as > >> > > > > > > a > >> > > > > > > > drop-in, OSS-aligned option for workflows that depend on > >> > > key-value > >> > > > > > > storage, > >> > > > > > > > pub/sub, or caching patterns. > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > Tentatively, here are the components I plan on > implementing > >> > > > (mirroring > >> > > > > > > the > >> > > > > > > > Redis provider): > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > * ValkeyHook – Manage connections and interactions with > the > >> > > > Valkey > >> > > > > > > > server, enabling reuse across operators and sensors. > >> > > > > > > > * ValkeyPublishOperator – Allows tasks to publish messages > >> to > >> > > > > > Valkey > >> > > > > > > > pub/sub channels, useful for event-driven workflows or > >> > inter-task > >> > > > > > > signaling. > >> > > > > > > > * ValkeyKeySensor – Waits for a specific key to exist or > >> reach > >> > > > a > >> > > > > > > > value, supporting synchronization patterns where one > task's > >> > > output > >> > > > > > > controls > >> > > > > > > > downstream logic. > >> > > > > > > > * ValkeyPubSubSensor – Subscribes to a Valkey channel and > >> > > > triggers > >> > > > > > on > >> > > > > > > > incoming messages, enabling real-time DAG triggering. > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > Here's an example of how we could use the components to > >> > publish a > >> > > > > > message > >> > > > > > > > and then trigger a DAG: > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > ` > >> > > > > > > > > >> > > > > > > > publish_message = ValkeyPublishOperator( > >> > > > > > > > > >> > > > > > > > task_id="publish_ready_signal", > >> > > > > > > > > >> > > > > > > > channel="processing_status", > >> > > > > > > > > >> > > > > > > > message="READY", > >> > > > > > > > > >> > > > > > > > valkey_conn_id="valkey_default", > >> > > > > > > > > >> > > > > > > > ) > >> > > > > > > > > >> > > > > > > > ` > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > This task publishes the message "READY" to the > >> > processing_status > >> > > > > > channel > >> > > > > > > > on a Valkey server. Other processes listening to this > >> channel > >> > > will > >> > > > > > > receive > >> > > > > > > > the message and proceed with their execution. > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > ` > >> > > > > > > > > >> > > > > > > > wait_for_ready = ValkeyPubSubSensor( > >> > > > > > > > > >> > > > > > > > task_id="wait_for_ready_message", > >> > > > > > > > > >> > > > > > > > channel="processing_status", > >> > > > > > > > > >> > > > > > > > valkey_conn_id="valkey_default", > >> > > > > > > > > >> > > > > > > > timeout=600, > >> > > > > > > > > >> > > > > > > > ) > >> > > > > > > > > >> > > > > > > > ` > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > Here, a sensor waits for the message on the same channel > >> before > >> > > > > > > > continuing. Once the message is received, the DAG can > >> continue > >> > > with > >> > > > > > > further > >> > > > > > > > tasks—such as triggering another DAG—in an event-driven > >> manner. > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > Given the community's interest in event-driven scheduling > >> (AIP > >> > > 82< > >> > > > > > > > https://github.com/apache/airflow/issues/52712> < > >> > > https://github.com/apache/airflow/issues/52712>>) as well as > >> > > > concerns > >> > > > > > > > about limited provider optionality (PR #52712< > >> > > > > > > > https://github.com/apache/airflow/issues/52712> < > >> > > https://github.com/apache/airflow/issues/52712>>), a Valkey > >> > > > provider > >> > > > > > > would > >> > > > > > > > expand user choice. > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > I’d be happy to begin working on the initial PR and > >> > > documentation. > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > Looking forward to feedback. > >> > > > > > > > > >> > > > > > > > Sean > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > >> > > > > >> --------------------------------------------------------------------- > >> > > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > <mailto: > >> > > dev-unsubscr...@airflow.apache.org> > >> > > > For additional commands, e-mail: dev-h...@airflow.apache.org > >> <mailto: > >> > > dev-h...@airflow.apache.org> > >> > > > > >> > > > > >> > > > >> > > > >> > > > >> > > > >> > > >> > > >
