Some of these use cases might be better suited to AIP-103 (the State storage AIP)
One possible issue with exposing Variables.list() (and connections) — do the secrets backends support listing variables? -ash > On 5 May 2026, at 11:41, 김준영 <[email protected]> wrote: > > Hi Amogh, > > Thanks for the feedback! I've updated the PR (#66022) to implement the > lazy iterator pattern you suggested: > > for key in Variable.keys(prefix="team_a_config_"): > val = Variable.get(key) > > Now, Variable.keys(prefix=None) returns a list[str] of matching keys, > supported by a new GET /variables/keys?prefix= endpoint in the > Execution API. This allows callers to fetch only the specific values > they need via Variable.get(key). > > Regarding your point about other Airflow resources (Connections, > XComs, etc.) having the same gap: once this PR is merged and the > pattern is established, I'd be happy to extend this keys() approach to > those resources in follow-up PRs. Would that be a direction you'd > encourage? > > The PR is ready for another review whenever you have a moment. > > Best regards, > Jun Yeong > > 2026년 5월 5일 (화) 오후 5:51, Amogh Desai <[email protected]>님이 작성: >> >> Hi Jun Yeong, >> >> Valid gap. There is no GET /variables list endpoint at all, but that is >> true for any >> other Airflow artefacts too. So variables isn't the only one missing it, >> its all of the >> Airflow resources: connections, xcoms, etc. >> >> One suggestion worth considering: rather than Variable.list() returning a >> full list, >> >> a lazy iterator might be a better fit. Something like: >> >> >> for key in Variable.keys(prefix="team_a_config_"): >> val = Variable.get(key) >> >> Happy to review when a PR is ready. >> >> Thanks & Regards, >> Amogh Desai >> >> >> On Fri, May 1, 2026 at 3:02 AM 김준영 <[email protected]> wrote: >> >>> Hi Jens, >>> >>> That is a valid alternative, and using a single JSON variable is >>> indeed a great pattern for centralized or static configurations to >>> reduce I/O. >>> >>> However, for the use cases I'm targeting, the "Single JSON Variable" >>> approach has a few significant drawbacks: >>> >>> 1.Concurrency & Atomic Updates: When multiple external systems or >>> independent CI/CD pipelines need to update their own configurations, a >>> single JSON variable creates a race condition. They would have to >>> "Read-Modify-Write" the entire blob, which risks overwriting each >>> other's changes. Separate variables allow for atomic, independent >>> updates. >>> >>> 2.Integration Complexity: Many users integrate Airflow with external >>> tools that independently push values to the Airflow Metadata DB or >>> Secret Backend. Forcing these decoupled systems to coordinate and >>> maintain a single shared JSON structure adds significant integration >>> overhead. >>> >>> 3.Data Modeling Flexibility: While a JSON blob works for some, >>> Variable.list(prefix=) allows Airflow to be unopinionated about how >>> users model their data. It provides a standard "Key-Value store" >>> experience (similar to AWS SSM or Redis) where prefix-based discovery >>> is a first-class citizen. >>> >>> In short, while the JSON approach is a good workaround for specific >>> cases, Variable.list() provides the necessary flexibility for highly >>> dynamic and decoupled environments without forcing a specific data >>> structure on the user. >>> >>> What do you think? >>> >>> Best regards, Jun Yeong >>> >>> 2026년 5월 1일 (금) 오전 6:19, Jens Scheffler <[email protected]>님이 작성: >>>> >>>> Hi Jun Yeong, >>>> >>>> one thought on this: We had similar. >>>> >>>> Our use case: Implemented a custom archiving (Dag) that needed to take >>>> care of different retention times in different Dags (maybe bad example >>>> because this archival itself accesses the database for archiving... >>>> haha) and we wanted to have different retention times per Dag. >>>> >>>> What we did is we made a JSON structure with all parameters into a >>>> single Variable. Then we did not need to have many IO operations and >>>> Variables but could store all in a single Variable. >>>> >>>> Would this be a viable solution for your case as well? >>>> >>>> Jens >>>> >>>> On 30.04.26 22:47, 김준영 wrote: >>>>> Hi Jens, >>>>> >>>>> Thank you for the thoughtful feedback! >>>>> >>>>> The primary demand for this feature comes from workflows that require >>>>> dynamic configuration discovery. A common pattern is grouping related >>>>> variables under a shared prefix (e.g., team_a_config_ or >>>>> pipeline_x_param_). In many cases, these keys are generated or updated >>>>> dynamically by external systems, meaning the exact set of keys isn't >>>>> known at DAG authoring time. >>>>> >>>>> While users in Airflow 2.x relied on session.query(Variable).all() as >>>>> a workaround, Airflow 3’s move toward the Task SDK aims to abstract >>>>> away direct ORM/DB access for better security and stability. >>>>> Variable.list(prefix=) provides a supported, clean way to achieve this >>>>> discovery within that new architecture. >>>>> >>>>> Regarding the secrets backend limitation, I completely agree. It’s >>>>> important to manage expectations, so I will update the PyDoc to >>>>> explicitly state that this method only lists variables stored in the >>>>> metadata database. >>>>> >>>>> Best regards, Jun Yeong >>>>> >>>>> 2026년 5월 1일 (금) 오전 5:23, Jens Scheffler <[email protected]>님이 작성: >>>>>> Hi! >>>>>> >>>>>> thanks for the discussion. While I am not against this I would say in >>>>>> Airflow 2 it was also not a "public API" but the DB connecton "just >>>>>> used" to list and have a missing API compensated. >>>>>> >>>>>> Can you express what the demand for the missing feature is? What >>>>>> business function did you implement based on listing all Variables? >>>>>> >>>>>> As you already stated and also highlighted in the PR the list() might >>>>>> not tell about all Variables as the list is not provided from secret >>>>>> managers. So it might (small risk thoug) lead to some confusion. >>> Should >>>>>> be explicitly documented in the PyDoc. But this is a nit. >>>>>> >>>>>> Jens >>>>>> >>>>>> On 30.04.26 11:47, 김준영 wrote: >>>>>>> Hi all, >>>>>>> >>>>>>> I'd like to propose adding Variable.list() to the Task SDK to address >>>>>>> the gap left by the removal of direct ORM access in Airflow 3. >>>>>>> >>>>>>> Background: >>>>>>> In Airflow 2.x, users could list all variables via: >>>>>>> >>>>>>> from airflow.models import Variable >>>>>>> from airflow.utils.session import create_session >>>>>>> >>>>>>> with create_session() as session: >>>>>>> variables = session.query(Variable).all() >>>>>>> >>>>>>> In Airflow 3.x, this pattern raises: >>>>>>> RuntimeError: Direct database access via the ORM is not >>> allowed in Airflow >>>>>>> 3.0 >>>>>>> >>>>>>> There is currently no supported way to discover variable keys >>> dynamically >>>>>>> when they are not known at DAG authoring time. >>>>>>> >>>>>>> Proposal: >>>>>>> - Add Variable.list(prefix=None) to the Task SDK >>>>>>> - Scope is limited to the metadata database only (same as the >>> old ORM pattern) >>>>>>> - Secrets backend support is intentionally out of scope, as it >>> would >>>>>>> require a broader interface contract change and separate >>> community >>>>>>> discussion >>>>>>> >>>>>>> Related issue: https://github.com/apache/airflow/issues/61166 >>>>>>> Draft PR: https://github.com/apache/airflow/pull/66022 >>>>>>> >>>>>>> I would appreciate any feedback or concerns from the community >>> before >>>>>>> this moves forward. >>>>>>> >>>>>>> Best Regards, >>>>>>> Jun Yeong Kim >>>>>>> >>>>>>> --------------------------------------------------------------------- >>>>>>> To unsubscribe, e-mail: [email protected] >>>>>>> For additional commands, e-mail: [email protected] >>>>>>> >>>>>> --------------------------------------------------------------------- >>>>>> To unsubscribe, e-mail: [email protected] >>>>>> For additional commands, e-mail: [email protected] >>>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: [email protected] >>>>> For additional commands, e-mail: [email protected] >>>>> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: [email protected] >>>> For additional commands, e-mail: [email protected] >>>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: [email protected] >>> For additional commands, e-mail: [email protected] >>> >>> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
