Can you elaborate (privately if you have to) on what the security concerns are? Since as I understand it the web server is powery deployment, so anything should be limited to one customer/user/deployment.
There is also the new "test connection" feature that will need the provider code installed to work. Then there's the issue of third party connections - of which there is only going to be more of over time. -ash On 14 June 2021 16:35:42 BST, Eugen Kosteev <[email protected]> wrote: >Hi Jarek. > >Thanks for the discussion. >The issue with Connections management in the web server that you described >is indeed affected Cloud Composer in the released preview image versions of >Airflow 2.0.1 (link to public issue >https://issuetracker.google.com/issues/190189297). And as you stated, we do >not install pypi packages in web server image mostly because of security >concerns. > >As a temporary workaround we baked all connections (list of them with their >widgets pickled and stored inside) into a web server image, so that >customers can add/edit them (even though not all providers packages are >pre-installed). This is a temporary workaround that we came up with for now >and we are looking for a long-term solution. > >Our thoughts/ideas for alternative solutions: >1. We do not want to pre-install all providers packages as to not generate >unnecessary python dependencies. Or maybe we could do this only for web >server images (not scheduler/worker) but then it is not clear if this is a >good idea to have such occured discrepancy between pypi dependencies in web >server vs scheduler/worker images. >2. Downloading and backing in providers packages (wheel files) into docker >image and installing customer specific/required version on demand looks >infeasible, taking into account number of providers, their versions and >their dependencies. > >- Eugene > >On Sun, Jun 13, 2021 at 6:46 PM Jarek Potiuk <[email protected]> wrote: > >> Dear Airflow community, >> >> Here is another result of discussions. I would like to raise an attention >> to potential Connection management problems that might affect managed >> services for Airflow 2.0 and some providers. >> >> With Airflow 2.0, connection UI "customisations" are baked into the >> provider package and in order to see - for example Postgres connection in >> the UI, you need to have the "postgres" provider installed in the Webserver. >> >> As far as I know some of the Managed Airflow services (MWAA, Composer, >> possibly other) do not currently allow their users installation of >> additional packages in the webserver (the webserver container is different >> than the scheduler/worker). This makes it impossible to configure/edit >> provider connections via UI (unless those providers are pre-installed in >> the webserver image). >> >> While this is understandable from security point of view to forbid "any'' >> package installation, I think the official >> "apache-airlfow-providers-*" should be allowlisted for those images and >> installed or otherwise made available (for example via pre-installing all >> providers in the webserver image if this is not possible from security >> point of view to rebuild the image dynamically) >> >> I wonder what people (and especially the people from MWAA, Composer team) >> think about it - do I get it right about the security concerns? Any other >> comments? >> >> >> J. >> >> -- >> +48 660 796 129 >> > > >-- >Eugene
