At a high level I’m good with allowing this to be fully configurable, as long
as we document the possible warts (“Doctor, it hurts when I do this” “well
don’t do that then!” etc) — though as Amogh mentioned it is slightly
complicated by the distinction between API Server/Scheduler and the execution
time on the worker.
(I haven’t looked at the specific implementation yet)
-ash
> On 2 Jul 2025, at 11:56, Amogh Desai <amoghdesai....@gmail.com> wrote:
>
> Hello Anton,
>
> Thanks for kicking off this discussion. I’d love to understand your
> motivations a bit more on this front.
> From your PR, I am seeing that you are just not allowing addition of
> multiple custom backends
> but also changing the *default_backend* order. I am a bit torn on that
> part.
>
> The current design intentionally places the metadata DB backend at the
> lowest precedence in the order,
> since it’s meant to serve as the ultimate fallback source of truth. Any
> additional configured
> backends are prioritized higher than it by design.
>
> With your changes, we now allow configurations like:
>
>
>
> * @conf_vars({("secrets", "backends_order"):
> "metastore,environment_variable,unsupported"}) def
> test_backends_order_unsupported(self): with
> pytest.raises(AirflowConfigException): ensure_secrets_loaded()*
>
> I don’t fully understand the motivation behind supporting this level of
> override, especially since it
> could allow unsupported or unintended configurations. Additionally, with
> Airflow 3.0+, we already support
> a multi layered secret backend resolution capability with the introduction
> of secrets backend for workers.
> Order goes as:
>
> *secrets backend on worker directly (optional) > env vars on worker > *
> *reach out to api server [secrets backend defined here (optional) > env
> vars on api server > metadata DB].*
>
> You will have to consider this angle too.
>
> In my opinion, a more practical and realistic use case would be to have the
> ability to define multiple custom backends
> both on worker or the API server.
>
> Looking forward to hearing more from you.
>
> Thanks & Regards,
> Amogh Desai
>
>
> On Wed, Jul 2, 2025 at 3:59 PM Anton Nitochkin <ant.nitoch...@gmail.com>
> wrote:
>
>> Hello,
>>
>> I'd like to discuss a new option that can be added via this PR:
>> https://github.com/apache/airflow/pull/45931.
>>
>> Recently, I asked developers in Slack for their thoughts on the new
>> variable [secrets]backend_order. Long story short: this option will
>> introduce the ability to configure the backend order and control it using
>> this variable. The default value will remain the same as in the current
>> version, so for users who don't need it, things will stay as they are now.
>>
>> Jarek Potiuk advised starting a conversation and discussing the PR to reach
>> a consensus with the community.
>>
>> Can you please share your thoughts on the option and its implementation?
>>
>> Anton Nitochkin
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org
