> at this point I am still -1 > What is the downside of running it as a 3rd party provider and if/when it > gains popularity donate it to airflow? Just out of curiosity - since Valkey is a community project, not controlled by a single entity nor having money or team to actually release and manage Airflow integration. Who do you think is the best 3rd-party to release the provider? Usually those arguments about the 3rd-party we raise when there is - clearly - such a 3rd-party - but I fail to see who that 3rd-party would be - what do you think Elad? Can you point such a 3rd-party?
J. On Wed, Aug 13, 2025 at 10:55 AM Elad Kalif <elad...@apache.org> wrote: > at this point I am still -1 > > On Wed, Aug 13, 2025 at 1:43 AM Jarek Potiuk <ja...@potiuk.com> wrote: > > > If you would like to contribute - start a PR and get a VOTE on it. That's > > what most people do when they want to add a new provider or call for LAZY > > CONSENSUS if you think consensus is reached and you can get it by the > lazy > > consensus https://www.apache.org/foundation/voting.html > > > > On Wed, Aug 13, 2025 at 12:38 AM Ghaeli, Sean <gha...@amazon.com.invalid > > > > wrote: > > > > > Valkey checks off many boxes that align with Airflow’s provider > criteria: > > > > > > * Truly open-source > > > * Dedicated maintainers responsive to community questions > > > * Not a commercial service that could maintain their own provider > > > * Very lean dependencies: > > > https://github.com/valkey-io/valkey/blob/unstable/src/CMakeLists.txt > > > * Popular and actively maintained with broad community support: > > > https://github.com/valkey-io/valkey > > > > > > Given all of these, I believe the Valkey provider makes sense as an > > > official Airflow provider. > > > > > > Sean > > > > > > On 2025-07-15, 1:22 PM, "Jarek Potiuk" <ja...@potiuk.com <mailto: > > > ja...@potiuk.com>> wrote: > > > > > > > > > CAUTION: This email originated from outside of the organization. Do not > > > click links or open attachments unless you can confirm the sender and > > know > > > the content is safe. > > > > > > > > > > > > > > > > > > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur > externe. > > > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne > > pouvez > > > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain > > que > > > le contenu ne présente aucun risque. > > > > > > > > > > > > > > > > > > > > > On Tue, Jul 15, 2025 at 9:52 PM Dominik Heilbock > <dheilb...@gmx.net.inva > > > <mailto:dheilb...@gmx.net.inva>lid> > > > wrote: > > > > > > > > > > As of last year, the redis Python client was still fully compatible > > with > > > > Valkey, has this changed? This would make it even easier. I think > they > > > also > > > > forked redis-py > > > > > > > > > > > > > > > Yep. I think that's part of the solution to make the dependency tree > for > > > the provider independent of redis-controlled dependencies. > > > > > > > > > Governance, decision making and release process of valley should be > > > separate from redis (the company). That's for me one of the criteria > and > > > requirements for having a separate valley provider. Very much like > > > "opensearch" is separate from "elasticsearch". > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Gesendet: Dienstag, den 15.07.2025 um 21:38 Uhr > > > > > Von: "Jarek Potiuk" <ja...@potiuk.com <mailto:ja...@potiuk.com>> > > > > > An: dev@airflow.apache.org <mailto:dev@airflow.apache.org> > > > > > Cc: gha...@amazon.com.inva <mailto:gha...@amazon.com.inva>lid > > > > > Betreff: Re: [DISCUSSION] New Provider: Valkey (Redis OSS Fork) > > > > > > > > > > > Can you please explain why it should be a provider managed by > > Airflow > > > > > rather than on your own? > > > > > > > > > > Just to explain it Elad - and yes, I know I am usually the one that > > is > > > > > sceptical -> valley is truly community driven replacement for redis > > > > > https://valkey.io/participants/ <https://valkey.io/participants/> > - > > > you see a lot of well-known community > > > > > recognised entities, and there is no single entity behind it that > > could > > > > > take "responsibility" - while there are enough stakeholders to make > > > sure > > > > > that it will be maintained here in Airflow community. Also it's a > > > > "drop-in" > > > > > replacement for redis generally - and I do not expect a lot of > > problems > > > > > when we start with both redis provider and redis celery > integration. > > > > > > > > > > I've heard and saw a lot of praise from the community for Valkey > > being > > > > good > > > > > redis replacement, and - even we at the PMC and devlist discussions > > > > > discussed about the dangers Redis licence creates for that - even > now > > > we > > > > > limited Redis support in Helm chart of ours to the "really free" > > > version > > > > > and we discussed that Valkey might be a good way forward - > especially > > > if > > > > > someone wants to contribute support for it. so if someone wants to > > > > > contribute to Airflow AND make sure that Redis broker support in > > Celery > > > > is > > > > > covered - I am all for it. > > > > > > > > > > > > > > > On Tue, Jul 15, 2025 at 9:37 PM Elad Kalif <elad...@gmail.com > > <mailto: > > > elad...@gmail.com>> wrote: > > > > > > > > > > > > * Helm Chart support to run valley as Celery Broker instead of > > > Redis > > > > > > > > > > > > Oh yes! didn't consider its usage as a broker but is it related > to > > > the > > > > > > provider package? > > > > > > Amazon SQS can be used as broker but it's not related to the SQS > > > > > > integrations in the Airflow amazon provider > (hook,operator,sensor) > > > > > > I also don't see ValKey listed as option for brokers in > > > > > > > > > > > > > > > > > > > > > > https://docs.celeryq.dev/en/latest/getting-started/backends-and-brokers/index.html#broker-overview > > > < > > > > > > https://docs.celeryq.dev/en/latest/getting-started/backends-and-brokers/index.html#broker-overview > > > > > > > > > > > > > > > > On Tue, Jul 15, 2025 at 10:27 PM Jarek Potiuk <ja...@potiuk.com > > > <mailto:ja...@potiuk.com>> > > > > wrote: > > > > > > > > > > > > > Yes. Valkey is a good replacement/parallel implementation to > > Redis. > > > > One > > > > > > > thing however is important - I would see this even more useful > if > > > it > > > > also > > > > > > > allows for the Celery Redis replacement for broker - means that > > it > > > is > > > > > > fully > > > > > > > supported in our: > > > > > > > > > > > > > > * unit tests (of course) > > > > > > > * integration tests with ("valkey") integration - including > > running > > > > > > > docker-compose integration to run in CI > > > > > > > * Helm Chart support to run valley as Celery Broker instead of > > > Redis > > > > > > > > > > > > > > I think I'd only consider valley as "successful" provider > > > > integration if > > > > > > > all that above is completed. > > > > > > > > > > > > > > J. > > > > > > > > > > > > > > On Tue, Jul 15, 2025 at 9:03 PM Ghaeli, Sean > > > > <gha...@amazon.com.inva <mailto:gha...@amazon.com.inva>lid> > > > > > > > wrote: > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I’d like to propose the addition of a new provider to support > > > > Valkey< > > > > > > > > https://valkey.io/> <https://valkey.io/>>: a fully > > > open-source, community-governed fork > > > > of > > > > > > > > Redis 7.2, now maintained under the Linux Foundation. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Valkey has been gaining traction recently, in part due to > > changes > > > > by > > > > > > > Redis > > > > > > > > moving their license to one unrecognized by the Open Source > > > > Initiative. > > > > > > > > Valkey represents a future-proof alternative. The proposed > > > provider > > > > > > would > > > > > > > > closely mirror the existing Redis provider in functionality > and > > > > serve > > > > > > as > > > > > > > a > > > > > > > > drop-in, OSS-aligned option for workflows that depend on > > > key-value > > > > > > > storage, > > > > > > > > pub/sub, or caching patterns. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Tentatively, here are the components I plan on implementing > > > > (mirroring > > > > > > > the > > > > > > > > Redis provider): > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > * ValkeyHook – Manage connections and interactions with the > > > > Valkey > > > > > > > > server, enabling reuse across operators and sensors. > > > > > > > > * ValkeyPublishOperator – Allows tasks to publish messages to > > > > > > Valkey > > > > > > > > pub/sub channels, useful for event-driven workflows or > > inter-task > > > > > > > signaling. > > > > > > > > * ValkeyKeySensor – Waits for a specific key to exist or > reach > > > > a > > > > > > > > value, supporting synchronization patterns where one task's > > > output > > > > > > > controls > > > > > > > > downstream logic. > > > > > > > > * ValkeyPubSubSensor – Subscribes to a Valkey channel and > > > > triggers > > > > > > on > > > > > > > > incoming messages, enabling real-time DAG triggering. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Here's an example of how we could use the components to > > publish a > > > > > > message > > > > > > > > and then trigger a DAG: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ` > > > > > > > > > > > > > > > > publish_message = ValkeyPublishOperator( > > > > > > > > > > > > > > > > task_id="publish_ready_signal", > > > > > > > > > > > > > > > > channel="processing_status", > > > > > > > > > > > > > > > > message="READY", > > > > > > > > > > > > > > > > valkey_conn_id="valkey_default", > > > > > > > > > > > > > > > > ) > > > > > > > > > > > > > > > > ` > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This task publishes the message "READY" to the > > processing_status > > > > > > channel > > > > > > > > on a Valkey server. Other processes listening to this channel > > > will > > > > > > > receive > > > > > > > > the message and proceed with their execution. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ` > > > > > > > > > > > > > > > > wait_for_ready = ValkeyPubSubSensor( > > > > > > > > > > > > > > > > task_id="wait_for_ready_message", > > > > > > > > > > > > > > > > channel="processing_status", > > > > > > > > > > > > > > > > valkey_conn_id="valkey_default", > > > > > > > > > > > > > > > > timeout=600, > > > > > > > > > > > > > > > > ) > > > > > > > > > > > > > > > > ` > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Here, a sensor waits for the message on the same channel > before > > > > > > > > continuing. Once the message is received, the DAG can > continue > > > with > > > > > > > further > > > > > > > > tasks—such as triggering another DAG—in an event-driven > manner. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Given the community's interest in event-driven scheduling > (AIP > > > 82< > > > > > > > > https://github.com/apache/airflow/issues/52712> < > > > https://github.com/apache/airflow/issues/52712>>) as well as > > > > concerns > > > > > > > > about limited provider optionality (PR #52712< > > > > > > > > https://github.com/apache/airflow/issues/52712> < > > > https://github.com/apache/airflow/issues/52712>>), a Valkey > > > > provider > > > > > > > would > > > > > > > > expand user choice. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I’d be happy to begin working on the initial PR and > > > documentation. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Looking forward to feedback. > > > > > > > > > > > > > > > > Sean > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org <mailto: > > > dev-unsubscr...@airflow.apache.org> > > > > For additional commands, e-mail: dev-h...@airflow.apache.org > <mailto: > > > dev-h...@airflow.apache.org> > > > > > > > > > > > > > > > > > > > > > > >
