I think it's best to divide the discussion into two separate topics. First one is to replace the existing json_client with the new to be created official Airflow Python Client backed by the new RESTful API. This IMHO is a must have considering we are to deprecate experimental API going forward.
The second topic is to create a better CLI experience leveraging the new APIs. This is much more controversial. I remember us having a similar discussion in the dev list a year ago, which didn't get much traction. It's not possible to fit all existing CLI functionalities into REST APIs. DB utils use-case that Ash mentioned is a great example. So I think one potential solution is to split the CLI into two. Keep the existing CLI as the admin/management CLI can communicate directly with the DB and taps into airflow core code base. On the other hand, we can create a separate user facing CLI that's light weight, fast and remote only. It doesn't even need to be written in python to make it easier to distribute as a single binary. Thanks, QP Hou On Tue, Aug 11, 2020 at 12:44 PM Ash Berlin-Taylor <a...@apache.org> wrote: > -1 from me without a firm plan how we will replace it. > > I see keeping it and extending to use the new API would ensure that > everything the CLI can do locally (i.e. when airflow webserver isn't up > yet, with the ) also works over the API with the exception of db utilities. > > -ash > > On 11 August 2020 20:05:56 BST, QP Hou <q...@scribd.com> wrote: > >+1 for replacing the existing remote mode client with the open api > >based > >client. IMO, we don't really have other options here because the > >experimental API will be deprecated in the future. > > > >For OpenAPI based Airflow REST clients, the current plan is to maintain > >all > >the code gen automation within the main source tree [1], then use it to > >populate each individual language specific client repo like the go > >client > >mentioned earlier. So far, we have the go client completed and > >validated to > >make sure this development flow will meet our needs. The next step I > >think > >the community should focus on is getting API auth implemented [2] > >before we > >move on to generate the python client. How we do API auth could have a > >big > >impact on client code gen automation, so it is worth waiting for. > > > >Once we have authentication implemented in both Airflow core and > >clients, > >we should be all good to start doing version releases for our API > >clients. > > > >That said, adopting open api based clients in the CLI alone won't > >address > >the issue of CLI depending on full airflow installation. Some of the > >cli > >commands like `dags test` depend on a full airflow installation by > >design. > >We will have to either develop a separate CLI intended for remote only > >use > >or add a flag in the existing cli so it can run in a pure remote mode > >where > >it would disable loading of code that requires airflow installation > >entirely. > > > >[1]: https://github.com/apache/airflow/tree/master/clients, > >[2]: https://github.com/apache/airflow/issues/8112 > > > >On Tue, Aug 11, 2020 at 11:05 AM Kamil Breguła > ><kamil.breg...@polidea.com> > >wrote: > > > >> Hello, > >> > >> I think we should remove remote mode in CLI and in-core API Client > >> (airflow.api.client package). > >> Here is docs about remote mode: > >> > >> > > > https://airflow.readthedocs.io/en/latest/usage-cli.html#set-up-connection-to-a-remote-airflow-instance > >> > >> Since these features were introduced, it has never been actively > >developed > >> and I don't think it's widely used. At the same time, Apache Airflow > >is > >> evolving, and this code stands out more and more from the rest. > >> > >> My main reservations about these features: > >> - Remote mode/in-core API Client is rarely used. I asked a few people > >and > >> none of them used it in production. Does anyone use it? > >> - A very small number of commands are available (7 pools command and > >2 dags > >> command only) > >> - Remote mode/API Client depends on experimental REST API. > >> - Remote mode/API Client is a handwritten code that is difficult to > >> maintain. > >> - No documentation for API client > >> - Remote mode/API Client has low test coverage. > >> - Remote mode does not provide a good level of security, because it > >depends > >> on experimental API. There is the only authentication, but the > >> authenticated user can perform any operation. > >> - Requires full Airflow to be installed along with a large number of > >> unnecessary dependencies. Some of them are difficult to install in > >some > >> environments, e.g. setproctitle on Windows > >> - Using this client API changes the logger configuration because it > >> requires importing the airflow package. > >> > >> I think this remote mode in CLI is something valuable, but I think we > >can > >> do it in a different way in the future, e.g. generate a CLI/API > >Client > >> based on the OpenAPI specification. > >> > >> Generated API clients can be installed independently of airflow and > >will be > >> easier to maintain. We already have one API client for golang > >implemented > >> in this way, so new languages will only be developing this idea. > >> - https://github.com/apache/airflow-client-go > >> > >> I will be happy to discuss the vision of the development of these two > >> things. How do we want to develop these two things? > >> > >> Best regards, > >> Kamil Bregula > >> >
