I would also be a -1 for deprecating URI connection representation, this is one of the defacto standards across Python. I do agree with the suggestion from Jarek that it could be less confusing if it follows SQLAlchemy style formatting. That is a known format that users are likely going to be familiar with.
Sent from my iPhone > On Nov 18, 2023, at 1:51 PM, Bolke de Bruin <bdbr...@gmail.com> wrote: > > So -1 on deprecating. I think we should do the opposite and go all in. The > suggestion by Jarek makes sense in this case, as it remains and is fully > standards compliant. The JSON representation isn't: it is our own crafted > format. > > My 2 cents, > > Bolke > >> On Sat, 18 Nov 2023 at 22:15, Jarek Potiuk <ja...@potiuk.com> wrote: >> >> I agree with Ash that it's much easier to define a lot of simpler >> connections this way. But it is also very confusing the way now how you >> can mix the "real" url with "airflow connection" URL. >> And Daniel is very right about the magnitude of breaking change. >> >> But possibly there is a way to eat cake and have it too. >> >> Answering Andrey's thinking: 1) I propose to deprecate it (raise warning) >> and even remove it from documentation >> But following Daniel's "pain": 2) possibly we should not even schedule it >> for removal for Airflow 3 >> And answering Ash's concern: 3) we should likely figure a non-confusing way >> how to define connection in "airlfow way". >> >> We could be following SQLlAlchemy approach - and define connection with >> "airflowconnection+" prefix in the schema >> >> AIRFLOW_CONN_MY_DB=airflowconnection+postgresql://user:pass@host >> >> I think this would remove all the confusion, keep an easy way of defining >> connection but also avoid "heavily" breaking change. >> >> J. >> >> >> >> On Sat, Nov 18, 2023 at 5:41 PM Daniel Standish >> <daniel.stand...@astronomer.io.invalid> wrote: >> >>> The thing that makes *me* hesitant to deprecate is the sheer magnitude of >>> breaking it would bring (even though we're only talking about a >>> hypothetical 3.0 release), balanced against the actual pain it causes. >>> I.e. it's confusing to use, and takes up space in docs (when, if removed, >>> we could just show one good way of doing things), but, it doesn't create >>> that much of a maintenance burden anymore, and has certainly become more >>> reliable and easy to use over time (e.g. since we added get_uri). >>> >>> >>> On Sat, Nov 18, 2023 at 3:30 AM Ash Berlin-Taylor <a...@apache.org> >> wrote: >>> >>>> -1 — for the simple case the URI format is perfect: >>>> >>>> AIRFLOW_CONN_MY_DB=postgresql://user:pass@host >>>> >>>> We can push JSON format for more complex cases, and/or limit URI to >> only >>>> the simple cases as long as we don’t remove it entirely. >>>> >>>> -ash >>>> >>>>> On 17 Nov 2023, at 22:44, Andrey Anshin <andrey.ans...@taragol.is> >>>> wrote: >>>>> >>>>> Greetings! >>>>> >>>>> I want to propose to deprecate Airflow Connection URI representation >>> and >>>>> remove it in Airflow 3 in favor of the already existing replacement - >>>> JSON >>>>> representation. >>>>> >>>>> In the past URI representation helped to add one of the awesome >>> features >>>> - >>>>> Alternative Secrets Backends: Environment Variables, Files or >> Provider >>>>> specific Backends. >>>>> >>>>> However I have a feeling that nowadays it have some sort of >> limitations >>>> due >>>>> to the parser, see: >>>>> - https://github.com/apache/airflow/issues/33442 >>>>> >>>>> Or miss interpretation that Airflow connection it is something more >>> than >>>>> just representation of Airflow Connection and it is a the replacement >>> to >>>>> any other type of URIs such as: HTTP, Postgres URIs, SQLAlchemy URI, >>>> Spark >>>>> URI and etc >>>>> >>>>> And attempt to fix it >>>>> - https://github.com/apache/airflow/issues/10913 >>>>> - https://github.com/apache/airflow/pull/27796 >>>>> - https://github.com/apache/airflow/pull/35712 >>>>> >>>>> And attempt to use it in DbApiHooks, for detail see: >>>>> - https://lists.apache.org/thread/8rhmz3qh30hvkondct4sfmgk4vd07mn5 >>>>> >>>>> In addition, complicated Connection in the URI format is not human >>>>> readable, requires additional decoding, and it is hard to create it >>>>> manually. >>>>> >>>>> The only one beneficial part of URI representation is ability to >> create >>>> it >>>>> programatically >>>>> >>>> >>> >> https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html#generating-a-connection-uri >>>>> , there is no such of method for generate JSON, but I don't see the >>>> problem >>>>> to create such of method in Connection object which return string >>>>> representation of JSON connection. >>>>> >>>>> So my suggestion would be pretty simple: >>>>> 1. Deprecate accept URI as parameter to the Connection object >>>>> 2. Deprecate uri property >>>>> 3. Replace get_uri by get_json >>>>> 4. Replace all URI examples in Providers Connections by JSON >>> alternatives >>>>> >>>>> >>>>> WDYT? Maybe I miss something and we should keep Airflow Connection as >>> URI >>>>> in the future Airflow's major versions and support both ways to >> provide >>>>> connections as JSON and URI. >>>>> >>>>> ---- >>>>> Best Wishes >>>>> *Andrey Anshin* >>>> >>>> >>> >> > > > -- > > -- > Bolke de Bruin > bdbr...@gmail.com --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org For additional commands, e-mail: dev-h...@airflow.apache.org
