Thanks Sean - I think it's a very good idea how to proceed. Having some more data-points on compatibility and broker replacement would be great.
On Fri, Aug 22, 2025 at 12:20 AM Ghaeli, Sean <gha...@amazon.com.invalid> wrote: > I like the generic "keyval" provider direction. To build on this > discussion, I'm planning two separate experiments: > > 1. Valkey as Celery backend: Testing if Valkey can drop-in replace Redis > as the message broker for Celery executor > 2. Valkey with existing Redis provider: Testing if the current > apache-airflow-providers-redis works when connected to Valkey instead of > Redis > > These should give us concrete data on both use cases to inform whether a > generic "keyval" provider approach makes sense and how seamless the > interchangeability really is. > > On 2025-08-14, 10:54 AM, "Jarek Potiuk" <ja...@potiuk.com <mailto: > ja...@potiuk.com>> wrote: > > > CAUTION: This email originated from outside of the organization. Do not > click links or open attachments unless you can confirm the sender and know > the content is safe. > > > > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que > le contenu ne présente aucun risque. > > > > > > > Ah.. Yes. Sorry - thanks Jed - you are right. I mis-read it.. > > > What my proposal would be is to keep redis to do what it does but have > optional `[redis]' (alongside ['rabbitmq'' on celery provider: > "apache-airflow-providers-celery[redis]` - and it would **not** install > redis but install redis as dependency. And also what we could add is a > combo "celery-redis" that would install > "apache-airflow-providers-celery[redis]". > > > However - I would propose something else to keep the consistency. Why don't > we already rename redis provider to "keyval" or something else and have it > work for both redis and valkey ? Even if we do not want to implement > anything specific for "valkey". For me that would be quite a good solution > for now - no "two" providers separately - just one for now. > > > THEN we could indeed change "redis" dependency to point to just redis > dependencies - as there will be no "redis" provider. > > > J. > > > > > On Thu, Aug 14, 2025 at 7:24 PM Jed Cunningham <jedcunning...@apache.org > <mailto:jedcunning...@apache.org>> > wrote: > > > > It feels like a slippery slope to depart from the "apache-airflow[x]" -> > > "apache-airflow-providers-x" pattern when x is a valid provider name. > Right > > now it's easy to know what it'll do before you run, but if we did this > > you'd either have to look it up or try it. > > > > > >
