Hi Jarek. Thanks for the discussion. The issue with Connections management in the web server that you described is indeed affected Cloud Composer in the released preview image versions of Airflow 2.0.1 (link to public issue https://issuetracker.google.com/issues/190189297). And as you stated, we do not install pypi packages in web server image mostly because of security concerns.
As a temporary workaround we baked all connections (list of them with their widgets pickled and stored inside) into a web server image, so that customers can add/edit them (even though not all providers packages are pre-installed). This is a temporary workaround that we came up with for now and we are looking for a long-term solution. Our thoughts/ideas for alternative solutions: 1. We do not want to pre-install all providers packages as to not generate unnecessary python dependencies. Or maybe we could do this only for web server images (not scheduler/worker) but then it is not clear if this is a good idea to have such occured discrepancy between pypi dependencies in web server vs scheduler/worker images. 2. Downloading and backing in providers packages (wheel files) into docker image and installing customer specific/required version on demand looks infeasible, taking into account number of providers, their versions and their dependencies. - Eugene On Sun, Jun 13, 2021 at 6:46 PM Jarek Potiuk <[email protected]> wrote: > Dear Airflow community, > > Here is another result of discussions. I would like to raise an attention > to potential Connection management problems that might affect managed > services for Airflow 2.0 and some providers. > > With Airflow 2.0, connection UI "customisations" are baked into the > provider package and in order to see - for example Postgres connection in > the UI, you need to have the "postgres" provider installed in the Webserver. > > As far as I know some of the Managed Airflow services (MWAA, Composer, > possibly other) do not currently allow their users installation of > additional packages in the webserver (the webserver container is different > than the scheduler/worker). This makes it impossible to configure/edit > provider connections via UI (unless those providers are pre-installed in > the webserver image). > > While this is understandable from security point of view to forbid "any'' > package installation, I think the official > "apache-airlfow-providers-*" should be allowlisted for those images and > installed or otherwise made available (for example via pre-installing all > providers in the webserver image if this is not possible from security > point of view to rebuild the image dynamically) > > I wonder what people (and especially the people from MWAA, Composer team) > think about it - do I get it right about the security concerns? Any other > comments? > > > J. > > -- > +48 660 796 129 > -- Eugene
