Hi Amogh, Quick closing note on this — PR #66022 (Variable.keys via lazy proxy) merged yesterday. Thanks again for steering the design toward the lazy iterator shape.
On your wider point that the listing gap exists for connections/xcoms/ etc. too: I scanned GitHub issues for equivalent reports and didn't find concrete user demand for those resources (Variable.all() had a direct report at #61166; Connection / Pool / XCom keys turn up nothing similar). Reading that as a signal that Variable was the unique case driven by a real DAG-author pattern, rather than a generic API symmetry gap. So I'm calling the scope done at Variable.keys for now and happy to revisit if a concrete case shows up for another resource. Thanks, Jun Yeong 2026년 5월 6일 (수) 오전 1:35, 김준영 <[email protected]>님이 작성: > > Agreed with the design direction. I've pushed the change to return a > lazy_object_proxy.Proxy. Would appreciate it if you could take a look. > > Best regards, > Jun Yeong > > 2026년 5월 6일 (수) 오전 12:59, Amogh Desai <[email protected]>님이 작성: > > > > Small correction on the design: Variable.keys() returning list[str] is > > eager. What I had in > > mind was a lazy proxy object: fetch the keys from the execution API once on > > first access, cache > > them, then serve iteration/indexing from that cache. Something like: > > > > keys = Variable.keys(prefix="team_a_config_") > > for key in keys: > > val = Variable.get(key) > > > > Similar to how lazy_object_proxy.Proxy is used in the template context > > today. Keeps the call > > site clean without pulling everything upfront. > > > > Re Ash's point about AIP 103 -- task state is for tasks persisting state > > across retries, not a shared > > config store. So I think you read it right. > > > > And forgot to comment earlier, but re Jens' workaround -- probably better > > suited for a centralized static > > config, but it will likely fall apart when multiple external systems are > > independently writing their own keys. > > The use case you have is legitimate and the workaround is honestly awkward > > for it. > > > > Regarding secrets backend scoping, I think I am ok with it being on the > > metadata db backend for the time > > being as long as we document it well. > > > > Thanks & Regards, > > Amogh Desai > > > > > > On Tue, May 5, 2026 at 7:40 PM 김준영 <[email protected]> wrote: > > > > > Hi Ash, > > > > > > Regarding secrets backends—since they don't currently support listing > > > in Airflow, I’ve scoped this feature to the metastore only and will > > > document that explicitly. As for AIP-103, Task State is designed for > > > tasks to persist their own state across retries, whereas this PR > > > focuses on DAGs fetching externally-managed config at runtime, keeping > > > the boundary between them clean. > > > > > > Best regards, > > > Jun Yeong > > > > > > 2026년 5월 5일 (화) 오후 10:48, Ash Berlin-Taylor <[email protected]>님이 작성: > > > > > > > > Some of these use cases might be better suited to AIP-103 (the State > > > storage AIP) > > > > > > > > One possible issue with exposing Variables.list() (and connections) — do > > > the secrets backends support listing variables? > > > > > > > > -ash > > > > > > > > > On 5 May 2026, at 11:41, 김준영 <[email protected]> wrote: > > > > > > > > > > Hi Amogh, > > > > > > > > > > Thanks for the feedback! I've updated the PR (#66022) to implement the > > > > > lazy iterator pattern you suggested: > > > > > > > > > > for key in Variable.keys(prefix="team_a_config_"): > > > > > val = Variable.get(key) > > > > > > > > > > Now, Variable.keys(prefix=None) returns a list[str] of matching keys, > > > > > supported by a new GET /variables/keys?prefix= endpoint in the > > > > > Execution API. This allows callers to fetch only the specific values > > > > > they need via Variable.get(key). > > > > > > > > > > Regarding your point about other Airflow resources (Connections, > > > > > XComs, etc.) having the same gap: once this PR is merged and the > > > > > pattern is established, I'd be happy to extend this keys() approach to > > > > > those resources in follow-up PRs. Would that be a direction you'd > > > > > encourage? > > > > > > > > > > The PR is ready for another review whenever you have a moment. > > > > > > > > > > Best regards, > > > > > Jun Yeong > > > > > > > > > > 2026년 5월 5일 (화) 오후 5:51, Amogh Desai <[email protected]>님이 작성: > > > > >> > > > > >> Hi Jun Yeong, > > > > >> > > > > >> Valid gap. There is no GET /variables list endpoint at all, but that > > > is > > > > >> true for any > > > > >> other Airflow artefacts too. So variables isn't the only one missing > > > it, > > > > >> its all of the > > > > >> Airflow resources: connections, xcoms, etc. > > > > >> > > > > >> One suggestion worth considering: rather than Variable.list() > > > returning a > > > > >> full list, > > > > >> > > > > >> a lazy iterator might be a better fit. Something like: > > > > >> > > > > >> > > > > >> for key in Variable.keys(prefix="team_a_config_"): > > > > >> val = Variable.get(key) > > > > >> > > > > >> Happy to review when a PR is ready. > > > > >> > > > > >> Thanks & Regards, > > > > >> Amogh Desai > > > > >> > > > > >> > > > > >> On Fri, May 1, 2026 at 3:02 AM 김준영 <[email protected]> wrote: > > > > >> > > > > >>> Hi Jens, > > > > >>> > > > > >>> That is a valid alternative, and using a single JSON variable is > > > > >>> indeed a great pattern for centralized or static configurations to > > > > >>> reduce I/O. > > > > >>> > > > > >>> However, for the use cases I'm targeting, the "Single JSON Variable" > > > > >>> approach has a few significant drawbacks: > > > > >>> > > > > >>> 1.Concurrency & Atomic Updates: When multiple external systems or > > > > >>> independent CI/CD pipelines need to update their own configurations, > > > a > > > > >>> single JSON variable creates a race condition. They would have to > > > > >>> "Read-Modify-Write" the entire blob, which risks overwriting each > > > > >>> other's changes. Separate variables allow for atomic, independent > > > > >>> updates. > > > > >>> > > > > >>> 2.Integration Complexity: Many users integrate Airflow with external > > > > >>> tools that independently push values to the Airflow Metadata DB or > > > > >>> Secret Backend. Forcing these decoupled systems to coordinate and > > > > >>> maintain a single shared JSON structure adds significant integration > > > > >>> overhead. > > > > >>> > > > > >>> 3.Data Modeling Flexibility: While a JSON blob works for some, > > > > >>> Variable.list(prefix=) allows Airflow to be unopinionated about how > > > > >>> users model their data. It provides a standard "Key-Value store" > > > > >>> experience (similar to AWS SSM or Redis) where prefix-based > > > > >>> discovery > > > > >>> is a first-class citizen. > > > > >>> > > > > >>> In short, while the JSON approach is a good workaround for specific > > > > >>> cases, Variable.list() provides the necessary flexibility for highly > > > > >>> dynamic and decoupled environments without forcing a specific data > > > > >>> structure on the user. > > > > >>> > > > > >>> What do you think? > > > > >>> > > > > >>> Best regards, Jun Yeong > > > > >>> > > > > >>> 2026년 5월 1일 (금) 오전 6:19, Jens Scheffler <[email protected]>님이 작성: > > > > >>>> > > > > >>>> Hi Jun Yeong, > > > > >>>> > > > > >>>> one thought on this: We had similar. > > > > >>>> > > > > >>>> Our use case: Implemented a custom archiving (Dag) that needed to > > > take > > > > >>>> care of different retention times in different Dags (maybe bad > > > example > > > > >>>> because this archival itself accesses the database for archiving... > > > > >>>> haha) and we wanted to have different retention times per Dag. > > > > >>>> > > > > >>>> What we did is we made a JSON structure with all parameters into a > > > > >>>> single Variable. Then we did not need to have many IO operations > > > > >>>> and > > > > >>>> Variables but could store all in a single Variable. > > > > >>>> > > > > >>>> Would this be a viable solution for your case as well? > > > > >>>> > > > > >>>> Jens > > > > >>>> > > > > >>>> On 30.04.26 22:47, 김준영 wrote: > > > > >>>>> Hi Jens, > > > > >>>>> > > > > >>>>> Thank you for the thoughtful feedback! > > > > >>>>> > > > > >>>>> The primary demand for this feature comes from workflows that > > > require > > > > >>>>> dynamic configuration discovery. A common pattern is grouping > > > related > > > > >>>>> variables under a shared prefix (e.g., team_a_config_ or > > > > >>>>> pipeline_x_param_). In many cases, these keys are generated or > > > updated > > > > >>>>> dynamically by external systems, meaning the exact set of keys > > > isn't > > > > >>>>> known at DAG authoring time. > > > > >>>>> > > > > >>>>> While users in Airflow 2.x relied on session.query(Variable).all() > > > as > > > > >>>>> a workaround, Airflow 3’s move toward the Task SDK aims to > > > > >>>>> abstract > > > > >>>>> away direct ORM/DB access for better security and stability. > > > > >>>>> Variable.list(prefix=) provides a supported, clean way to achieve > > > this > > > > >>>>> discovery within that new architecture. > > > > >>>>> > > > > >>>>> Regarding the secrets backend limitation, I completely agree. It’s > > > > >>>>> important to manage expectations, so I will update the PyDoc to > > > > >>>>> explicitly state that this method only lists variables stored in > > > the > > > > >>>>> metadata database. > > > > >>>>> > > > > >>>>> Best regards, Jun Yeong > > > > >>>>> > > > > >>>>> 2026년 5월 1일 (금) 오전 5:23, Jens Scheffler <[email protected]>님이 > > > 작성: > > > > >>>>>> Hi! > > > > >>>>>> > > > > >>>>>> thanks for the discussion. While I am not against this I would > > > say in > > > > >>>>>> Airflow 2 it was also not a "public API" but the DB connecton > > > "just > > > > >>>>>> used" to list and have a missing API compensated. > > > > >>>>>> > > > > >>>>>> Can you express what the demand for the missing feature is? What > > > > >>>>>> business function did you implement based on listing all > > > Variables? > > > > >>>>>> > > > > >>>>>> As you already stated and also highlighted in the PR the list() > > > might > > > > >>>>>> not tell about all Variables as the list is not provided from > > > secret > > > > >>>>>> managers. So it might (small risk thoug) lead to some confusion. > > > > >>> Should > > > > >>>>>> be explicitly documented in the PyDoc. But this is a nit. > > > > >>>>>> > > > > >>>>>> Jens > > > > >>>>>> > > > > >>>>>> On 30.04.26 11:47, 김준영 wrote: > > > > >>>>>>> Hi all, > > > > >>>>>>> > > > > >>>>>>> I'd like to propose adding Variable.list() to the Task SDK to > > > address > > > > >>>>>>> the gap left by the removal of direct ORM access in Airflow 3. > > > > >>>>>>> > > > > >>>>>>> Background: > > > > >>>>>>> In Airflow 2.x, users could list all variables via: > > > > >>>>>>> > > > > >>>>>>> from airflow.models import Variable > > > > >>>>>>> from airflow.utils.session import create_session > > > > >>>>>>> > > > > >>>>>>> with create_session() as session: > > > > >>>>>>> variables = session.query(Variable).all() > > > > >>>>>>> > > > > >>>>>>> In Airflow 3.x, this pattern raises: > > > > >>>>>>> RuntimeError: Direct database access via the ORM is not > > > > >>> allowed in Airflow > > > > >>>>>>> 3.0 > > > > >>>>>>> > > > > >>>>>>> There is currently no supported way to discover variable keys > > > > >>> dynamically > > > > >>>>>>> when they are not known at DAG authoring time. > > > > >>>>>>> > > > > >>>>>>> Proposal: > > > > >>>>>>> - Add Variable.list(prefix=None) to the Task SDK > > > > >>>>>>> - Scope is limited to the metadata database only (same as the > > > > >>> old ORM pattern) > > > > >>>>>>> - Secrets backend support is intentionally out of scope, as > > > > >>>>>>> it > > > > >>> would > > > > >>>>>>> require a broader interface contract change and separate > > > > >>> community > > > > >>>>>>> discussion > > > > >>>>>>> > > > > >>>>>>> Related issue: https://github.com/apache/airflow/issues/61166 > > > > >>>>>>> Draft PR: https://github.com/apache/airflow/pull/66022 > > > > >>>>>>> > > > > >>>>>>> I would appreciate any feedback or concerns from the > > > > >>>>>>> community > > > > >>> before > > > > >>>>>>> this moves forward. > > > > >>>>>>> > > > > >>>>>>> Best Regards, > > > > >>>>>>> Jun Yeong Kim > > > > >>>>>>> > > > > >>>>>>> > > > --------------------------------------------------------------------- > > > > >>>>>>> To unsubscribe, e-mail: [email protected] > > > > >>>>>>> For additional commands, e-mail: [email protected] > > > > >>>>>>> > > > > >>>>>> > > > --------------------------------------------------------------------- > > > > >>>>>> To unsubscribe, e-mail: [email protected] > > > > >>>>>> For additional commands, e-mail: [email protected] > > > > >>>>>> > > > > >>>>> > > > --------------------------------------------------------------------- > > > > >>>>> To unsubscribe, e-mail: [email protected] > > > > >>>>> For additional commands, e-mail: [email protected] > > > > >>>>> > > > > >>>> > > > > >>>> > > > --------------------------------------------------------------------- > > > > >>>> To unsubscribe, e-mail: [email protected] > > > > >>>> For additional commands, e-mail: [email protected] > > > > >>>> > > > > >>> > > > > >>> --------------------------------------------------------------------- > > > > >>> To unsubscribe, e-mail: [email protected] > > > > >>> For additional commands, e-mail: [email protected] > > > > >>> > > > > >>> > > > > > > > > > > --------------------------------------------------------------------- > > > > > To unsubscribe, e-mail: [email protected] > > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: [email protected] > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: [email protected] > > > For additional commands, e-mail: [email protected] > > > > > > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
