Hi Amogh,
Thanks for the feedback! I've updated the PR (#66022) to implement the
lazy iterator pattern you suggested:
for key in Variable.keys(prefix="team_a_config_"):
val = Variable.get(key)
Now, Variable.keys(prefix=None) returns a list[str] of matching keys,
supported by a new GET /variables/keys?prefix= endpoint in the
Execution API. This allows callers to fetch only the specific values
they need via Variable.get(key).
Regarding your point about other Airflow resources (Connections,
XComs, etc.) having the same gap: once this PR is merged and the
pattern is established, I'd be happy to extend this keys() approach to
those resources in follow-up PRs. Would that be a direction you'd
encourage?
The PR is ready for another review whenever you have a moment.
Best regards,
Jun Yeong
2026년 5월 5일 (화) 오후 5:51, Amogh Desai <[email protected]>님이 작성:
>
> Hi Jun Yeong,
>
> Valid gap. There is no GET /variables list endpoint at all, but that is
> true for any
> other Airflow artefacts too. So variables isn't the only one missing it,
> its all of the
> Airflow resources: connections, xcoms, etc.
>
> One suggestion worth considering: rather than Variable.list() returning a
> full list,
>
> a lazy iterator might be a better fit. Something like:
>
>
> for key in Variable.keys(prefix="team_a_config_"):
> val = Variable.get(key)
>
> Happy to review when a PR is ready.
>
> Thanks & Regards,
> Amogh Desai
>
>
> On Fri, May 1, 2026 at 3:02 AM 김준영 <[email protected]> wrote:
>
> > Hi Jens,
> >
> > That is a valid alternative, and using a single JSON variable is
> > indeed a great pattern for centralized or static configurations to
> > reduce I/O.
> >
> > However, for the use cases I'm targeting, the "Single JSON Variable"
> > approach has a few significant drawbacks:
> >
> > 1.Concurrency & Atomic Updates: When multiple external systems or
> > independent CI/CD pipelines need to update their own configurations, a
> > single JSON variable creates a race condition. They would have to
> > "Read-Modify-Write" the entire blob, which risks overwriting each
> > other's changes. Separate variables allow for atomic, independent
> > updates.
> >
> > 2.Integration Complexity: Many users integrate Airflow with external
> > tools that independently push values to the Airflow Metadata DB or
> > Secret Backend. Forcing these decoupled systems to coordinate and
> > maintain a single shared JSON structure adds significant integration
> > overhead.
> >
> > 3.Data Modeling Flexibility: While a JSON blob works for some,
> > Variable.list(prefix=) allows Airflow to be unopinionated about how
> > users model their data. It provides a standard "Key-Value store"
> > experience (similar to AWS SSM or Redis) where prefix-based discovery
> > is a first-class citizen.
> >
> > In short, while the JSON approach is a good workaround for specific
> > cases, Variable.list() provides the necessary flexibility for highly
> > dynamic and decoupled environments without forcing a specific data
> > structure on the user.
> >
> > What do you think?
> >
> > Best regards, Jun Yeong
> >
> > 2026년 5월 1일 (금) 오전 6:19, Jens Scheffler <[email protected]>님이 작성:
> > >
> > > Hi Jun Yeong,
> > >
> > > one thought on this: We had similar.
> > >
> > > Our use case: Implemented a custom archiving (Dag) that needed to take
> > > care of different retention times in different Dags (maybe bad example
> > > because this archival itself accesses the database for archiving...
> > > haha) and we wanted to have different retention times per Dag.
> > >
> > > What we did is we made a JSON structure with all parameters into a
> > > single Variable. Then we did not need to have many IO operations and
> > > Variables but could store all in a single Variable.
> > >
> > > Would this be a viable solution for your case as well?
> > >
> > > Jens
> > >
> > > On 30.04.26 22:47, 김준영 wrote:
> > > > Hi Jens,
> > > >
> > > > Thank you for the thoughtful feedback!
> > > >
> > > > The primary demand for this feature comes from workflows that require
> > > > dynamic configuration discovery. A common pattern is grouping related
> > > > variables under a shared prefix (e.g., team_a_config_ or
> > > > pipeline_x_param_). In many cases, these keys are generated or updated
> > > > dynamically by external systems, meaning the exact set of keys isn't
> > > > known at DAG authoring time.
> > > >
> > > > While users in Airflow 2.x relied on session.query(Variable).all() as
> > > > a workaround, Airflow 3’s move toward the Task SDK aims to abstract
> > > > away direct ORM/DB access for better security and stability.
> > > > Variable.list(prefix=) provides a supported, clean way to achieve this
> > > > discovery within that new architecture.
> > > >
> > > > Regarding the secrets backend limitation, I completely agree. It’s
> > > > important to manage expectations, so I will update the PyDoc to
> > > > explicitly state that this method only lists variables stored in the
> > > > metadata database.
> > > >
> > > > Best regards, Jun Yeong
> > > >
> > > > 2026년 5월 1일 (금) 오전 5:23, Jens Scheffler <[email protected]>님이 작성:
> > > >> Hi!
> > > >>
> > > >> thanks for the discussion. While I am not against this I would say in
> > > >> Airflow 2 it was also not a "public API" but the DB connecton "just
> > > >> used" to list and have a missing API compensated.
> > > >>
> > > >> Can you express what the demand for the missing feature is? What
> > > >> business function did you implement based on listing all Variables?
> > > >>
> > > >> As you already stated and also highlighted in the PR the list() might
> > > >> not tell about all Variables as the list is not provided from secret
> > > >> managers. So it might (small risk thoug) lead to some confusion.
> > Should
> > > >> be explicitly documented in the PyDoc. But this is a nit.
> > > >>
> > > >> Jens
> > > >>
> > > >> On 30.04.26 11:47, 김준영 wrote:
> > > >>> Hi all,
> > > >>>
> > > >>> I'd like to propose adding Variable.list() to the Task SDK to address
> > > >>> the gap left by the removal of direct ORM access in Airflow 3.
> > > >>>
> > > >>> Background:
> > > >>> In Airflow 2.x, users could list all variables via:
> > > >>>
> > > >>> from airflow.models import Variable
> > > >>> from airflow.utils.session import create_session
> > > >>>
> > > >>> with create_session() as session:
> > > >>> variables = session.query(Variable).all()
> > > >>>
> > > >>> In Airflow 3.x, this pattern raises:
> > > >>> RuntimeError: Direct database access via the ORM is not
> > allowed in Airflow
> > > >>> 3.0
> > > >>>
> > > >>> There is currently no supported way to discover variable keys
> > dynamically
> > > >>> when they are not known at DAG authoring time.
> > > >>>
> > > >>> Proposal:
> > > >>> - Add Variable.list(prefix=None) to the Task SDK
> > > >>> - Scope is limited to the metadata database only (same as the
> > old ORM pattern)
> > > >>> - Secrets backend support is intentionally out of scope, as it
> > would
> > > >>> require a broader interface contract change and separate
> > community
> > > >>> discussion
> > > >>>
> > > >>> Related issue: https://github.com/apache/airflow/issues/61166
> > > >>> Draft PR: https://github.com/apache/airflow/pull/66022
> > > >>>
> > > >>> I would appreciate any feedback or concerns from the community
> > before
> > > >>> this moves forward.
> > > >>>
> > > >>> Best Regards,
> > > >>> Jun Yeong Kim
> > > >>>
> > > >>> ---------------------------------------------------------------------
> > > >>> To unsubscribe, e-mail: [email protected]
> > > >>> For additional commands, e-mail: [email protected]
> > > >>>
> > > >> ---------------------------------------------------------------------
> > > >> To unsubscribe, e-mail: [email protected]
> > > >> For additional commands, e-mail: [email protected]
> > > >>
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: [email protected]
> > > > For additional commands, e-mail: [email protected]
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: [email protected]
> > > For additional commands, e-mail: [email protected]
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [email protected]
> > For additional commands, e-mail: [email protected]
> >
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
