And I think we should make the whole experimenta API deprecated in 1.10.12 possibly ?
On Thu, Aug 13, 2020 at 11:12 AM Jarek Potiuk <jarek.pot...@polidea.com> wrote: > I think no matter what, Maybe we should simply make it deprecated in the > upcoming (today?) release of 1.10.12 ? Then we can decide if in - > potential - 1.10.13 we remove it or leave it as it is. > > On Wed, Aug 12, 2020 at 6:37 PM Kamil Breguła <kamil.breg...@polidea.com> > wrote: > >> I started this thread mainly to discuss what we want to do with this >> remote >> mode prior to the Airflow 2.0 release. This is mainly due to the fact that >> he is using an experimental API which will be deprecated. >> >> In my opinion, we have several solutions. >> a) Delete this mode as unused and not supported. >> b) Rewrite in-core API client to support stable API >> c) Prepare OpenAPI based client and rewrite CLI to use it >> d) Leave as-is >> >> There were various expectations on the mailing list about this mode, but I >> haven't seen anyone actively contribute to it. This brings me further >> questions. Does anyone use this mode in its current form? If no one is >> using it, I think we can take more radical steps to start with a blank >> page. This will make it easier to start work and be able to iterate over >> it >> faster in the future. This looks like a simple task, but if we want to be >> sure that no breaking-changes are made, we should pay off some technical >> debt and increase testing coverage before we can think about making more >> changes. It may not be necessary if we choose a different path of >> development. >> >> I think now is a good time to try to make some decisions if someone is >> actually interested in developing these features. I do not think we need a >> precise vision of the development, if currently this feature is not used >> by >> anyone and no one is really interested in its development. >> >> On Wed, Aug 12, 2020 at 7:16 AM QP Hou <q...@scribd.com> wrote: >> >> > I think it's best to divide the discussion into two separate topics. >> > >> > First one is to replace the existing json_client with the new to be >> created >> > official Airflow Python Client backed by the new RESTful API. This IMHO >> is >> > a must have considering we are to deprecate experimental API going >> forward. >> > >> > The second topic is to create a better CLI experience leveraging the new >> > APIs. This is much more controversial. I remember us having a >> > similar discussion in the dev list a year ago, which didn't get much >> > traction. It's not possible to fit all existing CLI functionalities into >> > REST APIs. DB utils use-case that Ash mentioned is a great example. So I >> > think one potential solution is to split the CLI into two. Keep the >> > existing CLI as the admin/management CLI can communicate directly with >> the >> > DB and taps into airflow core code base. On the other hand, we can >> create a >> > separate user facing CLI that's light weight, fast and remote only. It >> > doesn't even need to be written in python to make it easier to >> distribute >> > as a single binary. >> > >> > Thanks, >> > QP Hou >> > >> > >> > On Tue, Aug 11, 2020 at 12:44 PM Ash Berlin-Taylor <a...@apache.org> >> wrote: >> > >> > > -1 from me without a firm plan how we will replace it. >> > > >> > > I see keeping it and extending to use the new API would ensure that >> > > everything the CLI can do locally (i.e. when airflow webserver isn't >> up >> > > yet, with the ) also works over the API with the exception of db >> > utilities. >> > > >> > > -ash >> > > >> > > On 11 August 2020 20:05:56 BST, QP Hou <q...@scribd.com> wrote: >> > > >+1 for replacing the existing remote mode client with the open api >> > > >based >> > > >client. IMO, we don't really have other options here because the >> > > >experimental API will be deprecated in the future. >> > > > >> > > >For OpenAPI based Airflow REST clients, the current plan is to >> maintain >> > > >all >> > > >the code gen automation within the main source tree [1], then use it >> to >> > > >populate each individual language specific client repo like the go >> > > >client >> > > >mentioned earlier. So far, we have the go client completed and >> > > >validated to >> > > >make sure this development flow will meet our needs. The next step I >> > > >think >> > > >the community should focus on is getting API auth implemented [2] >> > > >before we >> > > >move on to generate the python client. How we do API auth could have >> a >> > > >big >> > > >impact on client code gen automation, so it is worth waiting for. >> > > > >> > > >Once we have authentication implemented in both Airflow core and >> > > >clients, >> > > >we should be all good to start doing version releases for our API >> > > >clients. >> > > > >> > > >That said, adopting open api based clients in the CLI alone won't >> > > >address >> > > >the issue of CLI depending on full airflow installation. Some of the >> > > >cli >> > > >commands like `dags test` depend on a full airflow installation by >> > > >design. >> > > >We will have to either develop a separate CLI intended for remote >> only >> > > >use >> > > >or add a flag in the existing cli so it can run in a pure remote mode >> > > >where >> > > >it would disable loading of code that requires airflow installation >> > > >entirely. >> > > > >> > > >[1]: https://github.com/apache/airflow/tree/master/clients, >> > > >[2]: https://github.com/apache/airflow/issues/8112 >> > > > >> > > >On Tue, Aug 11, 2020 at 11:05 AM Kamil Breguła >> > > ><kamil.breg...@polidea.com> >> > > >wrote: >> > > > >> > > >> Hello, >> > > >> >> > > >> I think we should remove remote mode in CLI and in-core API Client >> > > >> (airflow.api.client package). >> > > >> Here is docs about remote mode: >> > > >> >> > > >> >> > > > >> > > >> > >> https://airflow.readthedocs.io/en/latest/usage-cli.html#set-up-connection-to-a-remote-airflow-instance >> > > >> >> > > >> Since these features were introduced, it has never been actively >> > > >developed >> > > >> and I don't think it's widely used. At the same time, Apache >> Airflow >> > > >is >> > > >> evolving, and this code stands out more and more from the rest. >> > > >> >> > > >> My main reservations about these features: >> > > >> - Remote mode/in-core API Client is rarely used. I asked a few >> people >> > > >and >> > > >> none of them used it in production. Does anyone use it? >> > > >> - A very small number of commands are available (7 pools command >> and >> > > >2 dags >> > > >> command only) >> > > >> - Remote mode/API Client depends on experimental REST API. >> > > >> - Remote mode/API Client is a handwritten code that is difficult to >> > > >> maintain. >> > > >> - No documentation for API client >> > > >> - Remote mode/API Client has low test coverage. >> > > >> - Remote mode does not provide a good level of security, because it >> > > >depends >> > > >> on experimental API. There is the only authentication, but the >> > > >> authenticated user can perform any operation. >> > > >> - Requires full Airflow to be installed along with a large number >> of >> > > >> unnecessary dependencies. Some of them are difficult to install in >> > > >some >> > > >> environments, e.g. setproctitle on Windows >> > > >> - Using this client API changes the logger configuration because it >> > > >> requires importing the airflow package. >> > > >> >> > > >> I think this remote mode in CLI is something valuable, but I think >> we >> > > >can >> > > >> do it in a different way in the future, e.g. generate a CLI/API >> > > >Client >> > > >> based on the OpenAPI specification. >> > > >> >> > > >> Generated API clients can be installed independently of airflow and >> > > >will be >> > > >> easier to maintain. We already have one API client for golang >> > > >implemented >> > > >> in this way, so new languages will only be developing this idea. >> > > >> - https://github.com/apache/airflow-client-go >> > > >> >> > > >> I will be happy to discuss the vision of the development of these >> two >> > > >> things. How do we want to develop these two things? >> > > >> >> > > >> Best regards, >> > > >> Kamil Bregula >> > > >> >> > > >> > >> > > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
