Thanks for the suggestion for using jsonschema!
I updated the implementation to use jsonschema instead of the custom
format. Now the structure looks like this for example:
conn-fields:
timeout:
label: "Connection Timeout"
description: "Timeout in seconds"
schema:
type: integer
minimum: 1
maximum: 300
default: 30
As for the concerns regarding GCP (14 fields including string, int,
boolean, and password), I tested it and it
works well (updated on PR). The code now uses schema object for all
jsonschema validation properties like min, max, pattern,
enum, etc while keeping UI metadata like label, description, sensitive or
not at the top level. This aligns
better with the react UI which already expects this format.
Thanks & Regards,
Amogh Desai
On Fri, Jan 16, 2026 at 12:50 PM Amogh Desai <[email protected]> wrote:
> Ash -
>
> Good catch on the GCP concern. I checked it and this is what it uses:
>
> @classmethod
> def get_connection_form_widgets(cls) -> dict[str, Any]:
> """Return connection widgets to add to connection form."""
> from flask_appbuilder.fieldwidgets import BS3PasswordFieldWidget,
> BS3TextFieldWidget
> from flask_babel import lazy_gettext
> from wtforms import BooleanField, IntegerField, PasswordField,
> StringField
> from wtforms.validators import NumberRange
>
> return {
> "project": StringField(lazy_gettext("Project Id"),
> widget=BS3TextFieldWidget()),
> "key_path": StringField(lazy_gettext("Keyfile Path"),
> widget=BS3TextFieldWidget()),
> "keyfile_dict": PasswordField(lazy_gettext("Keyfile JSON"),
> widget=BS3PasswordFieldWidget()),
> "credential_config_file": StringField(
> lazy_gettext("Credential Configuration File"),
> widget=BS3TextFieldWidget()
> ),
> "scope": StringField(lazy_gettext("Scopes (comma separated)"),
> widget=BS3TextFieldWidget()),
> "key_secret_name": StringField(
> lazy_gettext("Keyfile Secret Name (in GCP Secret
> Manager)"), widget=BS3TextFieldWidget()
> ),
> "key_secret_project_id": StringField(
> lazy_gettext("Keyfile Secret Project Id (in GCP Secret
> Manager)"), widget=BS3TextFieldWidget()
> ),
> "num_retries": IntegerField(
> lazy_gettext("Number of Retries"),
> validators=[NumberRange(min=0)],
> widget=BS3TextFieldWidget(),
> default=5,
> ),
> "impersonation_chain": StringField(
> lazy_gettext("Impersonation Chain"),
> widget=BS3TextFieldWidget()
> ),
> "idp_issuer_url": StringField(
> lazy_gettext("IdP Token Issue URL (Client Credentials
> Grant Flow)"),
> widget=BS3TextFieldWidget(),
> ),
> "client_id": StringField(
> lazy_gettext("Client ID (Client Credentials Grant Flow)"),
> widget=BS3TextFieldWidget()
> ),
> "client_secret": StringField(
> lazy_gettext("Client Secret (Client Credentials Grant
> Flow)"),
> widget=BS3PasswordFieldWidget(),
> ),
> "idp_extra_parameters": StringField(
> lazy_gettext("IdP Extra Request Parameters"),
> widget=BS3TextFieldWidget()
> ),
> "is_anonymous": BooleanField(
> lazy_gettext("Anonymous credentials (ignores all other
> settings)"), default=False
> ),
> }
>
> @classmethod
> def get_ui_field_behaviour(cls) -> dict[str, Any]:
> """Return custom field behaviour."""
> return {
> "hidden_fields": ["host", "schema", "login", "password",
> "port", "extra"],
> "relabeling": {},
> }
>
> All of these are covered by my schema.
>
> Also checked what the react UI supports and:
>
> I checked what the react UI supports as of now and this is what I found:
>
> string - Text input
> integer - Number input
> number - Number input
> boolean - Checkbox
> object - JSON object editor
> array - Array input
>
> String Formats:
> format: "password" - Masked password field
> format: "multiline" - Textarea
> format: "date" - Date picker
> format: "date-time" - DateTime picker
> format: "time" - Time picker
>
> Array Types
>
> This all comes from a field selector logic:
> https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/ui/src/components/FlexibleForm/FieldSelector.tsx#L58-L92
> .
>
> Fields are selected based on
> - `schema.type` (string, integer, boolean, array, object)
> - `schema.format` (password, multiline, date, date-time, time, email, url)
> - `schema.enum` (if present, dropdown select)
>
> So essentially anything with a type, format, and enum defined can be
> handled by react UI. That said, maybe I should
> try and adopt using jsonschema format here.
>
> Thanks & Regards,
> Amogh Desai
>
>
> On Fri, Jan 16, 2026 at 12:36 PM Amogh Desai <[email protected]>
> wrote:
>
>> Jarek -
>>
>> Re backcompat, yeah, I already have the fallback in place in my POC. The
>> discovery code
>> will first try to load the metadata from yaml, and if it fails to do so,
>> it will use the *python method*
>> flow to discover the metadata.
>>
>> Re the bigger vision about API servers without providers, I love where
>> you are going with this, but
>> I think we need to split up the tasks because we aren't there yet. Let me
>> explain -
>>
>> Your idea to discover providers via triggerer, store in DB and API server
>> reads from DB might work
>> for connection forms, but there are a few more reasons why API server
>> will continue to need the
>> ProvidersManager:
>>
>> 1. Auth Managers
>> 2. Secrets Backends
>> 3. Providers List Endpoint: maybe we should get rid of this? IDK who the
>> consumer of this endpoint is
>>
>> So the API server without Providers thing is harder than just connection
>> forms and we aren't there yet
>> until we figure out the 3 points from above.
>>
>> I suggest we do this instead:
>>
>> Phase 1: Connection forms from YAML to establish foundation for the future
>> Phase 2: The DB storage phase - decide if Triggerer / who populates in DB
>> (Maybe not triggerer because we do not want it to have DB access
>> eventually)
>>
>> Does that sound reasonable? What do you think?
>>
>>
>> Thanks & Regards,
>> Amogh Desai
>>
>>
>> On Fri, Jan 16, 2026 at 4:46 AM Jarek Potiuk <[email protected]> wrote:
>>
>>> > One main thing was assuming that all providers need to be available on
>>> > Scheduler (? I think that changed?) that there the connection form
>>> > definitons are persisted to DB such that the API server directly can
>>> > read from there - no need to install providers on API Server!
>>>
>>> I think Triggerer is better than Scheduler to persist connection
>>> definition
>>> to the DB. Essentially Triggerer is the only component that needs DB
>>> access
>>> and also needs to have providers installed. Any of the providers might
>>> implement Triggers and they are very tightly coupled with "Hooks" and
>>> "Operators". Scheduler only really needs **scheduler plugins**
>>> (Timetables
>>> and such) and **executors** (which we eventually want to split-off from
>>> current "worker" providers). It does not need "worker providers".
>>>
>>> IMHO in many discussions of ours this long term plan / vision is most
>>> appealing:
>>>
>>> * api-server: only needs distributions that are "ui plugins" (no
>>> providers)
>>> * scheduler only needs distributions that are "scheduler plugins" (e.g.
>>> timetables) and "executors"
>>> * worker only needs "worker/triggerer providers" (i.e. hooks and
>>> operators
>>> essentially) and "worker plugins" (e.g. macros)
>>> * triggerer only needs "worker/triggerer providers" (as in workers) -
>>> possibly "triggerer plugins" if we ever have a need to have them
>>>
>>> Eventually, optionally, each of those should ("api-server", "scheduler",
>>> "worker", "triggerer") should be a separate distribution. Each with its
>>> own
>>> dependencies. But this one only makes sense if we find that those
>>> dependencies could be very different between those - it's likely this
>>> will
>>> not happen, because dependency-set for each of those "components" will be
>>> very close. when we finalize the current task-sdk isolation work.
>>>
>>> Of course we cannot do it all at once and it will take quite some time to
>>> get there.
>>>
>>> But I think we should have it as a "North Star" that we should look at
>>> when
>>> we make any "architecture" decisions. And every decision we make should
>>> bring us closer to this "North Star".
>>>
>>> Also - just to note - with the "shared" libraries concept we already
>>> have,
>>> and with "uv workspace" in our monorepo - we have ALL the mechanisms
>>> needed
>>> to make it happen. And to do it in a very maintainable way with very
>>> little
>>> overhead and virtually no change in regular development workflow. For
>>> example the shared libraries concept might be used to share common code
>>> for
>>> both: apache-airflow-providers-cncf-kubernetes (worker provider - KPO
>>> essentially - installable for worker and triggerer) and (future)
>>> apache-airflow-executors-cncf-kubernetes (executor installable for
>>> scheduler). Same for amazon worker provider/executor split and edge
>>> worker
>>> provider/executor split. All that is doable.
>>>
>>> J.
>>>
>>>
>>>
>>> On Thu, Jan 15, 2026 at 10:23 PM Jens Scheffler <[email protected]>
>>> wrote:
>>>
>>> > Also +100 from my side.
>>> >
>>> > We discussed exactly this in a Airflow 3 dev call, I was looking for
>>> the
>>> > notes... that was when we discussed about the component split in the
>>> > future. Found a reference in
>>> >
>>> >
>>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=308153072#Airflow3Devcall:MeetingNotes-22August2024
>>> >
>>> > ```
>>> >
>>> > **Plan for Decoupling Providers's Connections metadata from FAB (Jens
>>> > Scheffler <https://cwiki.apache.org/confluence/display/~jscheffl>)**
>>> >
>>> > * Jens created this draft PR
>>> > <https://github.com/apache/airflow/pull/41656> with the POC for it
>>> > and presented it on the call.
>>> > * Jarek <https://cwiki.apache.org/confluence/display/~potiuk>
>>> proposed
>>> > the idea of dumping the JSON/YAML with connection fields in the
>>> > Database or loading it via package metadata so we don't load all
>>> the
>>> > dependencies on the webserver.
>>> > * We will need some plan for external providers on how they can
>>> define
>>> > connections or register them.
>>> > * The POC successfully proved that we can separate the connection
>>> > metadata from FAB
>>> > * /*Action Item*/: Jens
>>> > <https://cwiki.apache.org/confluence/display/~jscheffl> to create
>>> a
>>> > GitHub issue for decoupling the Connection metadata from FAB
>>> >
>>> > ```
>>> >
>>> > Also on Sep 19th 2024 we had an overview which pieces of the providers
>>> > are needed where:
>>> >
>>> >
>>> >
>>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=308153072#Airflow3Devcall:MeetingNotes-19September2024
>>> >
>>> > Follow-up was notes in Github ticket:
>>> > https://github.com/apache/airflow/issues/42016
>>> >
>>> >
>>> > One main thing was assuming that all providers need to be available on
>>> > Scheduler (? I think that changed?) that there the connection form
>>> > definitons are persisted to DB such that the API server directly can
>>> > read from there - no need to install providers on API Server!
>>> >
>>> > Looking forward for the contribution... I assume no VOTE needed :-D
>>> >
>>> > Jens
>>> >
>>> > On 1/15/26 15:52, Ash Berlin-Taylor wrote:
>>> > > As an idea/structure I think its certainly the right way to go — not
>>> > needing the code, not the instantiated widget classes, to (I suspect)
>>> throw
>>> > them away in the new React UI certainly seems like a silly idea now.
>>> > >
>>> > > In your POC I don’t think you have got the ability to have the extra
>>> > fields that, for instance, Google Cloud connection has yet though.
>>> > >
>>> > > As for the schema we need to express: I’d say we should look at what
>>> the
>>> > react UI currently supports?
>>> > >
>>> > > -ash
>>> > >
>>> > >> On 15 Jan 2026, at 14:07, Amogh Desai<[email protected]> wrote:
>>> > >>
>>> > >> Hi All,
>>> > >>
>>> > >> I wanted to get feedback on something I have been twiddling with.
>>> For
>>> > >> context, the API server has to import
>>> > >> every single hook class from all providers just to render connection
>>> > forms
>>> > >> in the UI. This is because the UI
>>> > >> metadata (what fields to show, labels, validators, etc.) are living
>>> in
>>> > >> python functions like `get_connection_form_widgets()`
>>> > >> and `get_ui_field_behaviour()` which are defined on the hook
>>> classes.
>>> > >>
>>> > >> This means:
>>> > >> - API server startup imports 100+ hook classes it might not actually
>>> > need
>>> > >> - Slower startup due to heavier memory footprint
>>> > >> - Poor client-server separation (why does the API server need to
>>> know
>>> > about
>>> > >> pyodbc just to show a UI form?)
>>> > >>
>>> > >> My proposal
>>> > >>
>>> > >> Moving the UI metadata from python code to something static /
>>> > declarative
>>> > >> like yaml. I want to add this information
>>> > >> in the provider.yaml file that every provider already has. For
>>> example -
>>> > >>
>>> > >> class PostgresHook(BaseHook):
>>> > >> @classmethod
>>> > >> def get_ui_field_behaviour(cls) -> dict[str, Any]:
>>> > >> return {
>>> > >> "hidden_fields": [],
>>> > >> "relabeling": {
>>> > >> "schema": "Database",
>>> > >> },
>>> > >> }
>>> > >>
>>> > >> Will become:
>>> > >>
>>> > >> connection-types:
>>> > >> - connection-type: postgres
>>> > >> hook-class-name:
>>> > airflow.providers.postgres.hooks.postgres.PostgresHook
>>> > >>
>>> > >> ui-field-behaviour:
>>> > >> hidden-fields: []
>>> > >> relabeling:
>>> > >> schema: "Database"
>>> > >>
>>> > >> conn-fields:
>>> > >> sslmode:
>>> > >> type: string
>>> > >> label: SSL Mode
>>> > >> enum: ["disable", "prefer", "require"]
>>> > >> default: "prefer"
>>> > >>
>>> > >> timeout:
>>> > >> type: integer
>>> > >> label: Timeout
>>> > >> range: [1, 300]
>>> > >> default: 30
>>> > >>
>>> > >> The schema will now consist of two new sections:
>>> > >>
>>> > >> 1. ui-field-behaviour
>>> > >> - Used to customize the standard connection fields (host, port,
>>> login,
>>> > etc.)
>>> > >> - hidden-fields: Hide some fields
>>> > >> - relabeling: Change labels for some fields (like schema -> Database
>>> > above)
>>> > >> - placeholders: Show hints in the form (port 5432 for example)
>>> > >>
>>> > >> 2. conn-fields
>>> > >> - Can be used to define custom fields stored in Connection.extra
>>> > >> - You can define inline validators like enum, range, pattern,
>>> > min-length,
>>> > >> max-length
>>> > >> - Will support the standard wtforms string, integer, boolean, number
>>> > types
>>> > >>
>>> > >> As for why this schema was chosen, check the comparison with
>>> > alternative in
>>> > >> the PR
>>> > >> desc:https://github.com/apache/airflow/pull/60410
>>> > >>
>>> > >>
>>> > >> Current Status
>>> > >>
>>> > >> I have a POC in:https://github.com/apache/airflow/pull/60410 where
>>> I
>>> > chose
>>> > >> two pilot providers of
>>> > >> varying difficulty: HTTP and SMTP (HTTP is easy, just a vanilla
>>> form but
>>> > >> SMTP has some hidden fields).
>>> > >>
>>> > >>
>>> > >> Benefits this will offer
>>> > >>
>>> > >> - Once complete, the API server won't import any hook classes for UI
>>> > >> rendering leading to faster startup
>>> > >> - Provider dependencies don't affect API server
>>> > >> - YAML is easier to read/write than python functions for form
>>> metadata
>>> > >>
>>> > >> Would love feedback on:
>>> > >> 1. Schema design - does it cover your use cases?
>>> > >> 2. Any missing field types or validators?
>>> > >>
>>> > >> The goal is to get the pilot providers in so we can start migrating
>>> > >> providers incrementally. Old way still
>>> > >> works, so no rush for everyone to migrate at once.
>>> > >>
>>> > >> Thoughts?
>>> > >>
>>> > >> Thanks & Regards,
>>> > >> Amogh Desai
>>> > >
>>> > > ---------------------------------------------------------------------
>>> > > To unsubscribe, e-mail:[email protected]
>>> > > For additional commands, e-mail:[email protected]
>>> > >
>>>
>>
