> * Helm Chart support to run valley as Celery Broker instead of Redis

Oh yes! didn't consider its usage as a broker but is it related to the
provider package?
Amazon SQS can be used as broker but it's not related to the SQS
integrations in the Airflow amazon provider (hook,operator,sensor)
I also don't see ValKey listed as option for brokers in
https://docs.celeryq.dev/en/latest/getting-started/backends-and-brokers/index.html#broker-overview

On Tue, Jul 15, 2025 at 10:27 PM Jarek Potiuk <ja...@potiuk.com> wrote:

> Yes. Valkey is a good replacement/parallel implementation to Redis. One
> thing however is important - I would see this even more useful if it also
> allows for the Celery Redis replacement for broker - means that it is fully
> supported in our:
>
> * unit tests (of course)
> * integration tests with ("valkey") integration - including running
> docker-compose integration to run in CI
> * Helm Chart support to run valley as Celery Broker instead of Redis
>
> I think I'd only consider valley as "successful" provider integration if
> all that above is completed.
>
> J.
>
> On Tue, Jul 15, 2025 at 9:03 PM Ghaeli, Sean <gha...@amazon.com.invalid>
> wrote:
>
> > Hi all,
> >
> >
> >
> > I’d like to propose the addition of a new provider to support Valkey<
> > https://valkey.io/>: a fully open-source, community-governed fork of
> > Redis 7.2, now maintained under the Linux Foundation.
> >
> >
> >
> > Valkey has been gaining traction recently, in part due to changes by
> Redis
> > moving their license to one unrecognized by the Open Source Initiative.
> > Valkey represents a future-proof alternative. The proposed provider would
> > closely mirror the existing Redis provider in functionality and serve as
> a
> > drop-in, OSS-aligned option for workflows that depend on key-value
> storage,
> > pub/sub, or caching patterns.
> >
> >
> >
> > Tentatively, here are the components I plan on implementing (mirroring
> the
> > Redis provider):
> >
> >
> >
> >   *   ValkeyHook – Manage connections and interactions with the Valkey
> > server, enabling reuse across operators and sensors.
> >   *   ValkeyPublishOperator – Allows tasks to publish messages to Valkey
> > pub/sub channels, useful for event-driven workflows or inter-task
> signaling.
> >   *   ValkeyKeySensor – Waits for a specific key to exist or reach a
> > value, supporting synchronization patterns where one task's output
> controls
> > downstream logic.
> >   *   ValkeyPubSubSensor – Subscribes to a Valkey channel and triggers on
> > incoming messages, enabling real-time DAG triggering.
> >
> >
> >
> > Here's an example of how we could use the components to publish a message
> > and then trigger a DAG:
> >
> >
> >
> > `
> >
> > publish_message = ValkeyPublishOperator(
> >
> >         task_id="publish_ready_signal",
> >
> >         channel="processing_status",
> >
> >         message="READY",
> >
> >         valkey_conn_id="valkey_default",
> >
> >     )
> >
> > `
> >
> >
> >
> > This task publishes the message "READY" to the processing_status channel
> > on a Valkey server. Other processes listening to this channel will
> receive
> > the message and proceed with their execution.
> >
> >
> >
> > `
> >
> > wait_for_ready = ValkeyPubSubSensor(
> >
> >     task_id="wait_for_ready_message",
> >
> >     channel="processing_status",
> >
> >     valkey_conn_id="valkey_default",
> >
> >     timeout=600,
> >
> > )
> >
> > `
> >
> >
> >
> > Here, a sensor waits for the message on the same channel before
> > continuing. Once the message is received, the DAG can continue with
> further
> > tasks—such as triggering another DAG—in an event-driven manner.
> >
> >
> >
> > Given the community's interest in event-driven scheduling (AIP 82<
> > https://github.com/apache/airflow/issues/52712>) as well as concerns
> > about limited provider optionality (PR #52712<
> > https://github.com/apache/airflow/issues/52712>), a Valkey provider
> would
> > expand user choice.
> >
> >
> >
> > I’d be happy to begin working on the initial PR and documentation.
> >
> >
> >
> > Looking forward to feedback.
> >
> > Sean
> >
> >
>

Reply via email to