To make it easier -> https://valkey.io/participants/ here is the list of companies to participate in valley development. Similarly to Airflow and other ASF projects - Valkey seem to be "truly" community driven, and when I look at the list, it's hard to pin-point that single entity who would be taking it on (at least it's hard for me).
On Wed, Aug 13, 2025 at 11:01 AM Jarek Potiuk <ja...@potiuk.com> wrote: > > at this point I am still -1 > > > What is the downside of running it as a 3rd party provider and if/when it > > gains popularity donate it to airflow? > Just out of curiosity - since Valkey is a community project, not > controlled by a single entity nor having money or team to actually release > and manage Airflow integration. Who do you think is the best 3rd-party to > release the provider? > Usually those arguments about the 3rd-party we raise when there is - > clearly - such a 3rd-party - but I fail to see who that 3rd-party would be > - what do you think Elad? Can you point such a 3rd-party? > > J. > > > On Wed, Aug 13, 2025 at 10:55 AM Elad Kalif <elad...@apache.org> wrote: > >> at this point I am still -1 >> >> On Wed, Aug 13, 2025 at 1:43 AM Jarek Potiuk <ja...@potiuk.com> wrote: >> >> > If you would like to contribute - start a PR and get a VOTE on it. >> That's >> > what most people do when they want to add a new provider or call for >> LAZY >> > CONSENSUS if you think consensus is reached and you can get it by the >> lazy >> > consensus https://www.apache.org/foundation/voting.html >> > >> > On Wed, Aug 13, 2025 at 12:38 AM Ghaeli, Sean <gha...@amazon.com.invalid >> > >> > wrote: >> > >> > > Valkey checks off many boxes that align with Airflow’s provider >> criteria: >> > > >> > > * Truly open-source >> > > * Dedicated maintainers responsive to community questions >> > > * Not a commercial service that could maintain their own provider >> > > * Very lean dependencies: >> > > https://github.com/valkey-io/valkey/blob/unstable/src/CMakeLists.txt >> > > * Popular and actively maintained with broad community support: >> > > https://github.com/valkey-io/valkey >> > > >> > > Given all of these, I believe the Valkey provider makes sense as an >> > > official Airflow provider. >> > > >> > > Sean >> > > >> > > On 2025-07-15, 1:22 PM, "Jarek Potiuk" <ja...@potiuk.com <mailto: >> > > ja...@potiuk.com>> wrote: >> > > >> > > >> > > CAUTION: This email originated from outside of the organization. Do >> not >> > > click links or open attachments unless you can confirm the sender and >> > know >> > > the content is safe. >> > > >> > > >> > > >> > > >> > > >> > > >> > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur >> externe. >> > > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne >> > pouvez >> > > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain >> > que >> > > le contenu ne présente aucun risque. >> > > >> > > >> > > >> > > >> > > >> > > >> > > On Tue, Jul 15, 2025 at 9:52 PM Dominik Heilbock >> <dheilb...@gmx.net.inva >> > > <mailto:dheilb...@gmx.net.inva>lid> >> > > wrote: >> > > >> > > >> > > > As of last year, the redis Python client was still fully compatible >> > with >> > > > Valkey, has this changed? This would make it even easier. I think >> they >> > > also >> > > > forked redis-py >> > > >> > > >> > > >> > > >> > > Yep. I think that's part of the solution to make the dependency tree >> for >> > > the provider independent of redis-controlled dependencies. >> > > >> > > >> > > Governance, decision making and release process of valley should be >> > > separate from redis (the company). That's for me one of the criteria >> and >> > > requirements for having a separate valley provider. Very much like >> > > "opensearch" is separate from "elasticsearch". >> > > >> > > >> > > >> > > >> > > >> > > >> > > > >> > > >> > > >> > > > Gesendet: Dienstag, den 15.07.2025 um 21:38 Uhr >> > > > > Von: "Jarek Potiuk" <ja...@potiuk.com <mailto:ja...@potiuk.com>> >> > > > > An: dev@airflow.apache.org <mailto:dev@airflow.apache.org> >> > > > > Cc: gha...@amazon.com.inva <mailto:gha...@amazon.com.inva>lid >> > > > > Betreff: Re: [DISCUSSION] New Provider: Valkey (Redis OSS Fork) >> > > > > >> > > > > > Can you please explain why it should be a provider managed by >> > Airflow >> > > > > rather than on your own? >> > > > > >> > > > > Just to explain it Elad - and yes, I know I am usually the one >> that >> > is >> > > > > sceptical -> valley is truly community driven replacement for >> redis >> > > > > https://valkey.io/participants/ <https://valkey.io/participants/> >> - >> > > you see a lot of well-known community >> > > > > recognised entities, and there is no single entity behind it that >> > could >> > > > > take "responsibility" - while there are enough stakeholders to >> make >> > > sure >> > > > > that it will be maintained here in Airflow community. Also it's a >> > > > "drop-in" >> > > > > replacement for redis generally - and I do not expect a lot of >> > problems >> > > > > when we start with both redis provider and redis celery >> integration. >> > > > > >> > > > > I've heard and saw a lot of praise from the community for Valkey >> > being >> > > > good >> > > > > redis replacement, and - even we at the PMC and devlist >> discussions >> > > > > discussed about the dangers Redis licence creates for that - even >> now >> > > we >> > > > > limited Redis support in Helm chart of ours to the "really free" >> > > version >> > > > > and we discussed that Valkey might be a good way forward - >> especially >> > > if >> > > > > someone wants to contribute support for it. so if someone wants to >> > > > > contribute to Airflow AND make sure that Redis broker support in >> > Celery >> > > > is >> > > > > covered - I am all for it. >> > > > > >> > > > > >> > > > > On Tue, Jul 15, 2025 at 9:37 PM Elad Kalif <elad...@gmail.com >> > <mailto: >> > > elad...@gmail.com>> wrote: >> > > > > >> > > > > > > * Helm Chart support to run valley as Celery Broker instead of >> > > Redis >> > > > > > >> > > > > > Oh yes! didn't consider its usage as a broker but is it related >> to >> > > the >> > > > > > provider package? >> > > > > > Amazon SQS can be used as broker but it's not related to the SQS >> > > > > > integrations in the Airflow amazon provider >> (hook,operator,sensor) >> > > > > > I also don't see ValKey listed as option for brokers in >> > > > > > >> > > > > > >> > > > >> > > >> > >> https://docs.celeryq.dev/en/latest/getting-started/backends-and-brokers/index.html#broker-overview >> > > < >> > > >> > >> https://docs.celeryq.dev/en/latest/getting-started/backends-and-brokers/index.html#broker-overview >> > > > >> > > > > > >> > > > > > On Tue, Jul 15, 2025 at 10:27 PM Jarek Potiuk <ja...@potiuk.com >> > > <mailto:ja...@potiuk.com>> >> > > > wrote: >> > > > > > >> > > > > > > Yes. Valkey is a good replacement/parallel implementation to >> > Redis. >> > > > One >> > > > > > > thing however is important - I would see this even more >> useful if >> > > it >> > > > also >> > > > > > > allows for the Celery Redis replacement for broker - means >> that >> > it >> > > is >> > > > > > fully >> > > > > > > supported in our: >> > > > > > > >> > > > > > > * unit tests (of course) >> > > > > > > * integration tests with ("valkey") integration - including >> > running >> > > > > > > docker-compose integration to run in CI >> > > > > > > * Helm Chart support to run valley as Celery Broker instead of >> > > Redis >> > > > > > > >> > > > > > > I think I'd only consider valley as "successful" provider >> > > > integration if >> > > > > > > all that above is completed. >> > > > > > > >> > > > > > > J. >> > > > > > > >> > > > > > > On Tue, Jul 15, 2025 at 9:03 PM Ghaeli, Sean >> > > > <gha...@amazon.com.inva <mailto:gha...@amazon.com.inva>lid> >> > > > > > > wrote: >> > > > > > > >> > > > > > > > Hi all, >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > I’d like to propose the addition of a new provider to >> support >> > > > Valkey< >> > > > > > > > https://valkey.io/> <https://valkey.io/>>: a fully >> > > open-source, community-governed fork >> > > > of >> > > > > > > > Redis 7.2, now maintained under the Linux Foundation. >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > Valkey has been gaining traction recently, in part due to >> > changes >> > > > by >> > > > > > > Redis >> > > > > > > > moving their license to one unrecognized by the Open Source >> > > > Initiative. >> > > > > > > > Valkey represents a future-proof alternative. The proposed >> > > provider >> > > > > > would >> > > > > > > > closely mirror the existing Redis provider in functionality >> and >> > > > serve >> > > > > > as >> > > > > > > a >> > > > > > > > drop-in, OSS-aligned option for workflows that depend on >> > > key-value >> > > > > > > storage, >> > > > > > > > pub/sub, or caching patterns. >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > Tentatively, here are the components I plan on implementing >> > > > (mirroring >> > > > > > > the >> > > > > > > > Redis provider): >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > * ValkeyHook – Manage connections and interactions with the >> > > > Valkey >> > > > > > > > server, enabling reuse across operators and sensors. >> > > > > > > > * ValkeyPublishOperator – Allows tasks to publish messages >> to >> > > > > > Valkey >> > > > > > > > pub/sub channels, useful for event-driven workflows or >> > inter-task >> > > > > > > signaling. >> > > > > > > > * ValkeyKeySensor – Waits for a specific key to exist or >> reach >> > > > a >> > > > > > > > value, supporting synchronization patterns where one task's >> > > output >> > > > > > > controls >> > > > > > > > downstream logic. >> > > > > > > > * ValkeyPubSubSensor – Subscribes to a Valkey channel and >> > > > triggers >> > > > > > on >> > > > > > > > incoming messages, enabling real-time DAG triggering. >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > Here's an example of how we could use the components to >> > publish a >> > > > > > message >> > > > > > > > and then trigger a DAG: >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > ` >> > > > > > > > >> > > > > > > > publish_message = ValkeyPublishOperator( >> > > > > > > > >> > > > > > > > task_id="publish_ready_signal", >> > > > > > > > >> > > > > > > > channel="processing_status", >> > > > > > > > >> > > > > > > > message="READY", >> > > > > > > > >> > > > > > > > valkey_conn_id="valkey_default", >> > > > > > > > >> > > > > > > > ) >> > > > > > > > >> > > > > > > > ` >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > This task publishes the message "READY" to the >> > processing_status >> > > > > > channel >> > > > > > > > on a Valkey server. Other processes listening to this >> channel >> > > will >> > > > > > > receive >> > > > > > > > the message and proceed with their execution. >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > ` >> > > > > > > > >> > > > > > > > wait_for_ready = ValkeyPubSubSensor( >> > > > > > > > >> > > > > > > > task_id="wait_for_ready_message", >> > > > > > > > >> > > > > > > > channel="processing_status", >> > > > > > > > >> > > > > > > > valkey_conn_id="valkey_default", >> > > > > > > > >> > > > > > > > timeout=600, >> > > > > > > > >> > > > > > > > ) >> > > > > > > > >> > > > > > > > ` >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > Here, a sensor waits for the message on the same channel >> before >> > > > > > > > continuing. Once the message is received, the DAG can >> continue >> > > with >> > > > > > > further >> > > > > > > > tasks—such as triggering another DAG—in an event-driven >> manner. >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > Given the community's interest in event-driven scheduling >> (AIP >> > > 82< >> > > > > > > > https://github.com/apache/airflow/issues/52712> < >> > > https://github.com/apache/airflow/issues/52712>>) as well as >> > > > concerns >> > > > > > > > about limited provider optionality (PR #52712< >> > > > > > > > https://github.com/apache/airflow/issues/52712> < >> > > https://github.com/apache/airflow/issues/52712>>), a Valkey >> > > > provider >> > > > > > > would >> > > > > > > > expand user choice. >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > I’d be happy to begin working on the initial PR and >> > > documentation. >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > Looking forward to feedback. >> > > > > > > > >> > > > > > > > Sean >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > >> > > > >> --------------------------------------------------------------------- >> > > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org <mailto: >> > > dev-unsubscr...@airflow.apache.org> >> > > > For additional commands, e-mail: dev-h...@airflow.apache.org >> <mailto: >> > > dev-h...@airflow.apache.org> >> > > > >> > > > >> > > >> > > >> > > >> > > >> > >> >
