Can you elaborate (privately if you have to) on what the security
concerns are? Since as I understand it the web server is powery
deployment, so anything should be limited to one
customer/user/deployment.
There is also the new "test connection" feature that will need the
provider code installed to work.
Then there's the issue of third party connections - of which there
is only going to be more of over time.
-ash
On 14 June 2021 16:35:42 BST, Eugen Kosteev <[email protected]
<mailto:[email protected]>> wrote:
>Hi Jarek.
>
>Thanks for the discussion.
>The issue with Connections management in the web server that you
described
>is indeed affected Cloud Composer in the released preview image
versions of
>Airflow 2.0.1 (link to public issue
><https://issuetracker.google.com/issues/190189297>). And as you
stated, we do
>not install pypi packages in web server image mostly because of
security
>concerns.
>
>As a temporary workaround we baked all connections (list of them
with their
>widgets pickled and stored inside) into a web server image, so that
>customers can add/edit them (even though not all providers
packages are
>pre-installed). This is a temporary workaround that we came up
with for now
>and we are looking for a long-term solution.
>
>Our thoughts/ideas for alternative solutions:
>1. We do not want to pre-install all providers packages as to not
generate
>unnecessary python dependencies. Or maybe we could do this only
for web
>server images (not scheduler/worker) but then it is not clear if
this is a
>good idea to have such occured discrepancy between pypi
dependencies in web
>server vs scheduler/worker images.
>2. Downloading and backing in providers packages (wheel files)
into docker
>image and installing customer specific/required version on demand
looks
>infeasible, taking into account number of providers, their
versions and
>their dependencies.
>
>- Eugene
>
>On Sun, Jun 13, 2021 at 6:46 PM Jarek Potiuk <[email protected]
<mailto:[email protected]>> wrote:
>
>> Dear Airflow community,
>>
>> Here is another result of discussions. I would like to raise an
attention
>> to potential Connection management problems that might affect
managed
>> services for Airflow 2.0 and some providers.
>>
>> With Airflow 2.0, connection UI "customisations" are baked into
the
>> provider package and in order to see - for example Postgres
connection in
>> the UI, you need to have the "postgres" provider installed in
the Webserver.
>>
>> As far as I know some of the Managed Airflow services (MWAA,
Composer,
>> possibly other) do not currently allow their users installation
of
>> additional packages in the webserver (the webserver container is
different
>> than the scheduler/worker). This makes it impossible to
configure/edit
>> provider connections via UI (unless those providers are
pre-installed in
>> the webserver image).
>>
>> While this is understandable from security point of view to
forbid "any''
>> package installation, I think the official
>> "apache-airlfow-providers-*" should be allowlisted for those
images and
>> installed or otherwise made available (for example via
pre-installing all
>> providers in the webserver image if this is not possible from
security
>> point of view to rebuild the image dynamically)
>>
>> I wonder what people (and especially the people from MWAA,
Composer team)
>> think about it - do I get it right about the security concerns?
Any other
>> comments?
>>
>>
>> J.
>>
>> --
>> +48 660 796 129
>>
>
>
>--
>Eugene
