I’m with Elad on this, -1. The open nature/source/core or not of the project isn’t the key aspect for this, but simply the usefulness of it at all
1. Do we need a separate provider for it? It already bugs me that we have separate ElasticSearch and OpenSearch providers and this feels like the same thing all over — we’re having to maintain two copies of a provider that is otherwise identical due to upstream OSS vs commercial spats. 2. Is it even worth having a redis provider? I have a feeling the popularity of the redis provider is a side-effect of using reds for Celery broker `apache-airflow[celery,redis]` being installed. We have no way of knowing but I would make the case that 99% of cases don’t use the hooks, operators or sensors. -ash > On 13 Aug 2025, at 11:06, Jarek Potiuk <ja...@potiuk.com> wrote: > >> Since Valkey 8.1.x (latest serie) is still 100% compatible with Redis, >> wouldn't be easier to just state in docs Redis provider is also >> compatible with Valkey backend? > > Yep, that's one of the options. We discussed it above - that we could have > valley/redis switch for example in the current redis provider - it will > make it more coupled and future features might diverge, but yes, this is > indeed a valid option, I think it's more of the question of name in this > case. If for example we rename redis provider to "kv-storage" or > "common-kv-storage" for example or something similar, that might be > actually quite a viable option for us. > > J. > > > On Wed, Aug 13, 2025 at 11:44 AM Josef Šimánek <josef.sima...@gmail.com> > wrote: > >> st 13. 8. 2025 v 0:38 odesílatel Ghaeli, Sean >> <gha...@amazon.com.invalid> napsal: >>> >>> Valkey checks off many boxes that align with Airflow’s provider criteria: >>> >>> * Truly open-source >>> * Dedicated maintainers responsive to community questions >>> * Not a commercial service that could maintain their own provider >>> * Very lean dependencies: >> https://github.com/valkey-io/valkey/blob/unstable/src/CMakeLists.txt >>> * Popular and actively maintained with broad community support: >> https://github.com/valkey-io/valkey >>> >>> Given all of these, I believe the Valkey provider makes sense as an >> official Airflow provider. >> >> Since Valkey 8.1.x (latest serie) is still 100% compatible with Redis, >> wouldn't be easier to just state in docs Redis provider is also >> compatible with Valkey backend? >> >>> Sean >>> >>> On 2025-07-15, 1:22 PM, "Jarek Potiuk" <ja...@potiuk.com <mailto: >> ja...@potiuk.com>> wrote: >>> >>> >>> CAUTION: This email originated from outside of the organization. Do not >> click links or open attachments unless you can confirm the sender and know >> the content is safe. >>> >>> >>> >>> >>> >>> >>> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur >> externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous >> ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas >> certain que le contenu ne présente aucun risque. >>> >>> >>> >>> >>> >>> >>> On Tue, Jul 15, 2025 at 9:52 PM Dominik Heilbock <dheilb...@gmx.net.inva >> <mailto:dheilb...@gmx.net.inva>lid> >>> wrote: >>> >>> >>>> As of last year, the redis Python client was still fully compatible >> with >>>> Valkey, has this changed? This would make it even easier. I think they >> also >>>> forked redis-py >>> >>> >>> >>> >>> Yep. I think that's part of the solution to make the dependency tree for >>> the provider independent of redis-controlled dependencies. >>> >>> >>> Governance, decision making and release process of valley should be >>> separate from redis (the company). That's for me one of the criteria and >>> requirements for having a separate valley provider. Very much like >>> "opensearch" is separate from "elasticsearch". >>> >>> >>> >>> >>> >>> >>>> >>> >>> >>>> Gesendet: Dienstag, den 15.07.2025 um 21:38 Uhr >>>>> Von: "Jarek Potiuk" <ja...@potiuk.com <mailto:ja...@potiuk.com>> >>>>> An: dev@airflow.apache.org <mailto:dev@airflow.apache.org> >>>>> Cc: gha...@amazon.com.inva <mailto:gha...@amazon.com.inva>lid >>>>> Betreff: Re: [DISCUSSION] New Provider: Valkey (Redis OSS Fork) >>>>> >>>>>> Can you please explain why it should be a provider managed by >> Airflow >>>>> rather than on your own? >>>>> >>>>> Just to explain it Elad - and yes, I know I am usually the one that >> is >>>>> sceptical -> valley is truly community driven replacement for redis >>>>> https://valkey.io/participants/ <https://valkey.io/participants/> - >> you see a lot of well-known community >>>>> recognised entities, and there is no single entity behind it that >> could >>>>> take "responsibility" - while there are enough stakeholders to make >> sure >>>>> that it will be maintained here in Airflow community. Also it's a >>>> "drop-in" >>>>> replacement for redis generally - and I do not expect a lot of >> problems >>>>> when we start with both redis provider and redis celery integration. >>>>> >>>>> I've heard and saw a lot of praise from the community for Valkey >> being >>>> good >>>>> redis replacement, and - even we at the PMC and devlist discussions >>>>> discussed about the dangers Redis licence creates for that - even >> now we >>>>> limited Redis support in Helm chart of ours to the "really free" >> version >>>>> and we discussed that Valkey might be a good way forward - >> especially if >>>>> someone wants to contribute support for it. so if someone wants to >>>>> contribute to Airflow AND make sure that Redis broker support in >> Celery >>>> is >>>>> covered - I am all for it. >>>>> >>>>> >>>>> On Tue, Jul 15, 2025 at 9:37 PM Elad Kalif <elad...@gmail.com >> <mailto:elad...@gmail.com>> wrote: >>>>> >>>>>>> * Helm Chart support to run valley as Celery Broker instead of >> Redis >>>>>> >>>>>> Oh yes! didn't consider its usage as a broker but is it related to >> the >>>>>> provider package? >>>>>> Amazon SQS can be used as broker but it's not related to the SQS >>>>>> integrations in the Airflow amazon provider (hook,operator,sensor) >>>>>> I also don't see ValKey listed as option for brokers in >>>>>> >>>>>> >>>> >> https://docs.celeryq.dev/en/latest/getting-started/backends-and-brokers/index.html#broker-overview >> < >> https://docs.celeryq.dev/en/latest/getting-started/backends-and-brokers/index.html#broker-overview >>> >>>>>> >>>>>> On Tue, Jul 15, 2025 at 10:27 PM Jarek Potiuk <ja...@potiuk.com >> <mailto:ja...@potiuk.com>> >>>> wrote: >>>>>> >>>>>>> Yes. Valkey is a good replacement/parallel implementation to >> Redis. >>>> One >>>>>>> thing however is important - I would see this even more useful >> if it >>>> also >>>>>>> allows for the Celery Redis replacement for broker - means that >> it is >>>>>> fully >>>>>>> supported in our: >>>>>>> >>>>>>> * unit tests (of course) >>>>>>> * integration tests with ("valkey") integration - including >> running >>>>>>> docker-compose integration to run in CI >>>>>>> * Helm Chart support to run valley as Celery Broker instead of >> Redis >>>>>>> >>>>>>> I think I'd only consider valley as "successful" provider >>>> integration if >>>>>>> all that above is completed. >>>>>>> >>>>>>> J. >>>>>>> >>>>>>> On Tue, Jul 15, 2025 at 9:03 PM Ghaeli, Sean >>>> <gha...@amazon.com.inva <mailto:gha...@amazon.com.inva>lid> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi all, >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> I’d like to propose the addition of a new provider to support >>>> Valkey< >>>>>>>> https://valkey.io/> <https://valkey.io/>>: a fully >> open-source, community-governed fork >>>> of >>>>>>>> Redis 7.2, now maintained under the Linux Foundation. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Valkey has been gaining traction recently, in part due to >> changes >>>> by >>>>>>> Redis >>>>>>>> moving their license to one unrecognized by the Open Source >>>> Initiative. >>>>>>>> Valkey represents a future-proof alternative. The proposed >> provider >>>>>> would >>>>>>>> closely mirror the existing Redis provider in functionality and >>>> serve >>>>>> as >>>>>>> a >>>>>>>> drop-in, OSS-aligned option for workflows that depend on >> key-value >>>>>>> storage, >>>>>>>> pub/sub, or caching patterns. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Tentatively, here are the components I plan on implementing >>>> (mirroring >>>>>>> the >>>>>>>> Redis provider): >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> * ValkeyHook – Manage connections and interactions with the >>>> Valkey >>>>>>>> server, enabling reuse across operators and sensors. >>>>>>>> * ValkeyPublishOperator – Allows tasks to publish messages to >>>>>> Valkey >>>>>>>> pub/sub channels, useful for event-driven workflows or >> inter-task >>>>>>> signaling. >>>>>>>> * ValkeyKeySensor – Waits for a specific key to exist or reach >>>> a >>>>>>>> value, supporting synchronization patterns where one task's >> output >>>>>>> controls >>>>>>>> downstream logic. >>>>>>>> * ValkeyPubSubSensor – Subscribes to a Valkey channel and >>>> triggers >>>>>> on >>>>>>>> incoming messages, enabling real-time DAG triggering. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Here's an example of how we could use the components to >> publish a >>>>>> message >>>>>>>> and then trigger a DAG: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> ` >>>>>>>> >>>>>>>> publish_message = ValkeyPublishOperator( >>>>>>>> >>>>>>>> task_id="publish_ready_signal", >>>>>>>> >>>>>>>> channel="processing_status", >>>>>>>> >>>>>>>> message="READY", >>>>>>>> >>>>>>>> valkey_conn_id="valkey_default", >>>>>>>> >>>>>>>> ) >>>>>>>> >>>>>>>> ` >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> This task publishes the message "READY" to the >> processing_status >>>>>> channel >>>>>>>> on a Valkey server. Other processes listening to this channel >> will >>>>>>> receive >>>>>>>> the message and proceed with their execution. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> ` >>>>>>>> >>>>>>>> wait_for_ready = ValkeyPubSubSensor( >>>>>>>> >>>>>>>> task_id="wait_for_ready_message", >>>>>>>> >>>>>>>> channel="processing_status", >>>>>>>> >>>>>>>> valkey_conn_id="valkey_default", >>>>>>>> >>>>>>>> timeout=600, >>>>>>>> >>>>>>>> ) >>>>>>>> >>>>>>>> ` >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Here, a sensor waits for the message on the same channel before >>>>>>>> continuing. Once the message is received, the DAG can continue >> with >>>>>>> further >>>>>>>> tasks—such as triggering another DAG—in an event-driven manner. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Given the community's interest in event-driven scheduling (AIP >> 82< >>>>>>>> https://github.com/apache/airflow/issues/52712> < >> https://github.com/apache/airflow/issues/52712>>) as well as >>>> concerns >>>>>>>> about limited provider optionality (PR #52712< >>>>>>>> https://github.com/apache/airflow/issues/52712> < >> https://github.com/apache/airflow/issues/52712>>), a Valkey >>>> provider >>>>>>> would >>>>>>>> expand user choice. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> I’d be happy to begin working on the initial PR and >> documentation. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Looking forward to feedback. >>>>>>>> >>>>>>>> Sean >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org <mailto: >> dev-unsubscr...@airflow.apache.org> >>>> For additional commands, e-mail: dev-h...@airflow.apache.org <mailto: >> dev-h...@airflow.apache.org> >>>> >>>> >>> >>> >>> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org >> For additional commands, e-mail: dev-h...@airflow.apache.org >> >> --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org For additional commands, e-mail: dev-h...@airflow.apache.org
