Also +100 from my side.
We discussed exactly this in a Airflow 3 dev call, I was looking for the
notes... that was when we discussed about the component split in the
future. Found a reference in
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=308153072#Airflow3Devcall:MeetingNotes-22August2024
```
**Plan for Decoupling Providers's Connections metadata from FAB (Jens
Scheffler <https://cwiki.apache.org/confluence/display/~jscheffl>)**
* Jens created this draft PR
<https://github.com/apache/airflow/pull/41656> with the POC for it
and presented it on the call.
* Jarek <https://cwiki.apache.org/confluence/display/~potiuk> proposed
the idea of dumping the JSON/YAML with connection fields in the
Database or loading it via package metadata so we don't load all the
dependencies on the webserver.
* We will need some plan for external providers on how they can define
connections or register them.
* The POC successfully proved that we can separate the connection
metadata from FAB
* /*Action Item*/: Jens
<https://cwiki.apache.org/confluence/display/~jscheffl> to create a
GitHub issue for decoupling the Connection metadata from FAB
```
Also on Sep 19th 2024 we had an overview which pieces of the providers
are needed where:
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=308153072#Airflow3Devcall:MeetingNotes-19September2024
Follow-up was notes in Github ticket:
https://github.com/apache/airflow/issues/42016
One main thing was assuming that all providers need to be available on
Scheduler (? I think that changed?) that there the connection form
definitons are persisted to DB such that the API server directly can
read from there - no need to install providers on API Server!
Looking forward for the contribution... I assume no VOTE needed :-D
Jens
On 1/15/26 15:52, Ash Berlin-Taylor wrote:
As an idea/structure I think its certainly the right way to go — not needing
the code, not the instantiated widget classes, to (I suspect) throw them away
in the new React UI certainly seems like a silly idea now.
In your POC I don’t think you have got the ability to have the extra fields
that, for instance, Google Cloud connection has yet though.
As for the schema we need to express: I’d say we should look at what the react
UI currently supports?
-ash
On 15 Jan 2026, at 14:07, Amogh Desai<[email protected]> wrote:
Hi All,
I wanted to get feedback on something I have been twiddling with. For
context, the API server has to import
every single hook class from all providers just to render connection forms
in the UI. This is because the UI
metadata (what fields to show, labels, validators, etc.) are living in
python functions like `get_connection_form_widgets()`
and `get_ui_field_behaviour()` which are defined on the hook classes.
This means:
- API server startup imports 100+ hook classes it might not actually need
- Slower startup due to heavier memory footprint
- Poor client-server separation (why does the API server need to know about
pyodbc just to show a UI form?)
My proposal
Moving the UI metadata from python code to something static / declarative
like yaml. I want to add this information
in the provider.yaml file that every provider already has. For example -
class PostgresHook(BaseHook):
@classmethod
def get_ui_field_behaviour(cls) -> dict[str, Any]:
return {
"hidden_fields": [],
"relabeling": {
"schema": "Database",
},
}
Will become:
connection-types:
- connection-type: postgres
hook-class-name: airflow.providers.postgres.hooks.postgres.PostgresHook
ui-field-behaviour:
hidden-fields: []
relabeling:
schema: "Database"
conn-fields:
sslmode:
type: string
label: SSL Mode
enum: ["disable", "prefer", "require"]
default: "prefer"
timeout:
type: integer
label: Timeout
range: [1, 300]
default: 30
The schema will now consist of two new sections:
1. ui-field-behaviour
- Used to customize the standard connection fields (host, port, login, etc.)
- hidden-fields: Hide some fields
- relabeling: Change labels for some fields (like schema -> Database above)
- placeholders: Show hints in the form (port 5432 for example)
2. conn-fields
- Can be used to define custom fields stored in Connection.extra
- You can define inline validators like enum, range, pattern, min-length,
max-length
- Will support the standard wtforms string, integer, boolean, number types
As for why this schema was chosen, check the comparison with alternative in
the PR
desc:https://github.com/apache/airflow/pull/60410
Current Status
I have a POC in:https://github.com/apache/airflow/pull/60410 where I chose
two pilot providers of
varying difficulty: HTTP and SMTP (HTTP is easy, just a vanilla form but
SMTP has some hidden fields).
Benefits this will offer
- Once complete, the API server won't import any hook classes for UI
rendering leading to faster startup
- Provider dependencies don't affect API server
- YAML is easier to read/write than python functions for form metadata
Would love feedback on:
1. Schema design - does it cover your use cases?
2. Any missing field types or validators?
The goal is to get the pilot providers in so we can start migrating
providers incrementally. Old way still
works, so no rush for everyone to migrate at once.
Thoughts?
Thanks & Regards,
Amogh Desai
---------------------------------------------------------------------
To unsubscribe, e-mail:[email protected]
For additional commands, e-mail:[email protected]
