So -1 on deprecating. I think we should do the opposite and go all in. The suggestion by Jarek makes sense in this case, as it remains and is fully standards compliant. The JSON representation isn't: it is our own crafted format.
My 2 cents, Bolke On Sat, 18 Nov 2023 at 22:15, Jarek Potiuk <ja...@potiuk.com> wrote: > I agree with Ash that it's much easier to define a lot of simpler > connections this way. But it is also very confusing the way now how you > can mix the "real" url with "airflow connection" URL. > And Daniel is very right about the magnitude of breaking change. > > But possibly there is a way to eat cake and have it too. > > Answering Andrey's thinking: 1) I propose to deprecate it (raise warning) > and even remove it from documentation > But following Daniel's "pain": 2) possibly we should not even schedule it > for removal for Airflow 3 > And answering Ash's concern: 3) we should likely figure a non-confusing way > how to define connection in "airlfow way". > > We could be following SQLlAlchemy approach - and define connection with > "airflowconnection+" prefix in the schema > > AIRFLOW_CONN_MY_DB=airflowconnection+postgresql://user:pass@host > > I think this would remove all the confusion, keep an easy way of defining > connection but also avoid "heavily" breaking change. > > J. > > > > On Sat, Nov 18, 2023 at 5:41 PM Daniel Standish > <daniel.stand...@astronomer.io.invalid> wrote: > > > The thing that makes *me* hesitant to deprecate is the sheer magnitude of > > breaking it would bring (even though we're only talking about a > > hypothetical 3.0 release), balanced against the actual pain it causes. > > I.e. it's confusing to use, and takes up space in docs (when, if removed, > > we could just show one good way of doing things), but, it doesn't create > > that much of a maintenance burden anymore, and has certainly become more > > reliable and easy to use over time (e.g. since we added get_uri). > > > > > > On Sat, Nov 18, 2023 at 3:30 AM Ash Berlin-Taylor <a...@apache.org> > wrote: > > > > > -1 — for the simple case the URI format is perfect: > > > > > > AIRFLOW_CONN_MY_DB=postgresql://user:pass@host > > > > > > We can push JSON format for more complex cases, and/or limit URI to > only > > > the simple cases as long as we don’t remove it entirely. > > > > > > -ash > > > > > > > On 17 Nov 2023, at 22:44, Andrey Anshin <andrey.ans...@taragol.is> > > > wrote: > > > > > > > > Greetings! > > > > > > > > I want to propose to deprecate Airflow Connection URI representation > > and > > > > remove it in Airflow 3 in favor of the already existing replacement - > > > JSON > > > > representation. > > > > > > > > In the past URI representation helped to add one of the awesome > > features > > > - > > > > Alternative Secrets Backends: Environment Variables, Files or > Provider > > > > specific Backends. > > > > > > > > However I have a feeling that nowadays it have some sort of > limitations > > > due > > > > to the parser, see: > > > > - https://github.com/apache/airflow/issues/33442 > > > > > > > > Or miss interpretation that Airflow connection it is something more > > than > > > > just representation of Airflow Connection and it is a the replacement > > to > > > > any other type of URIs such as: HTTP, Postgres URIs, SQLAlchemy URI, > > > Spark > > > > URI and etc > > > > > > > > And attempt to fix it > > > > - https://github.com/apache/airflow/issues/10913 > > > > - https://github.com/apache/airflow/pull/27796 > > > > - https://github.com/apache/airflow/pull/35712 > > > > > > > > And attempt to use it in DbApiHooks, for detail see: > > > > - https://lists.apache.org/thread/8rhmz3qh30hvkondct4sfmgk4vd07mn5 > > > > > > > > In addition, complicated Connection in the URI format is not human > > > > readable, requires additional decoding, and it is hard to create it > > > > manually. > > > > > > > > The only one beneficial part of URI representation is ability to > create > > > it > > > > programatically > > > > > > > > > > https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html#generating-a-connection-uri > > > > , there is no such of method for generate JSON, but I don't see the > > > problem > > > > to create such of method in Connection object which return string > > > > representation of JSON connection. > > > > > > > > So my suggestion would be pretty simple: > > > > 1. Deprecate accept URI as parameter to the Connection object > > > > 2. Deprecate uri property > > > > 3. Replace get_uri by get_json > > > > 4. Replace all URI examples in Providers Connections by JSON > > alternatives > > > > > > > > > > > > WDYT? Maybe I miss something and we should keep Airflow Connection as > > URI > > > > in the future Airflow's major versions and support both ways to > provide > > > > connections as JSON and URI. > > > > > > > > ---- > > > > Best Wishes > > > > *Andrey Anshin* > > > > > > > > > -- -- Bolke de Bruin bdbr...@gmail.com
