If the connection templates were available as a shim, for sure we will bake that into the webserver container. For now, we are choosing to bake any provider with a preferred license to alleviate pain for most customers – the current list that we know is ok to go ahead (in addition to the amazon provider package) is the below ones. apache-airflow-providers-databricks apache-airflow-providers-ssh apache-airflow-providers-postgres apache-airflow-providers-docker apache-airflow-providers-facebook apache-airflow-providers-oracle apache-airflow-providers-presto apache-airflow-providers-salesforce apache-airflow-providers-sftp apache-airflow-providers-tableau
I will start a separate discussion on the private@ list. I might not have subscription on that. I should run that by AWS OSS group before I send it out. Thanks Subash From: Jarek Potiuk <[email protected]> Reply-To: "[email protected]" <[email protected]> Date: Tuesday, June 15, 2021 at 11:45 AM To: "[email protected]" <[email protected]> Subject: RE: [EXTERNAL] [DISCUSS] Managing provider Connections via UI in managed Airflow services CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. On Tue, Jun 15, 2021 at 8:22 PM Canapathy, Subash <[email protected]> wrote: @Ash Berlin-Taylor<mailto:[email protected]> – I don’t think that is entirely true. In 1.10 the connection templates code was part of the flask application and not bundled with the provider. Managed services took the webserver baseline as is and let the customers take decision on additives like FB-business, oracle etc.. without bundling them into the managed service software per AWS compliance guidelines. In 2.0 if we bake in all the providers, it will mean that we are baking in their dependencies along with. Just to clarify - in 1.10 there were no providers (there were backports for them) but the dependencies you are talking about were coming through extras (and I think you had very limited set of extras which indeed made you not depend on the dependencies). So you are right in 1.10 you did not have this coupling (all the connection types were baked-in airflow as you explained). Regarding security constraints on why we disallow plugins and requirements on the webserver, I will have to discuss this in person on PMC but on a high level this comes down to remote code execution prevention on managed instances, opening possibilities of exploiting vulnerabilities on the flask-app-builder and the underlying python runtime. There is 2 levels of isolation – one on the single tenancy of environments in MWAA under separate VPCs, and secondly on Fargate that prevents exploits to break out of the container boundaries into the hypervisor. Even with those, our security team had other possibilities of exploits unearthed in penetration testing that led to this decision. Yep. I was involved in a vetting process of similar UI-facing apps and I understand where it comes from, however we would love to hear what your concerns are Subash. Maybe you can send a message to [email protected]<mailto:[email protected]> and explain ? Coming back to my question. Would that help if you could generate such a "shim" package and you install it at the webserver? I can imagine a tool/command where you specify which built-in providers you installed, and run a command that could generate (out of Airflow sources corresponding to your version) a "whl" package containing: * entrypoint returning set of connections that you miss * extracted classes containing the meta-data and Field definitions for the UI for the relevant connections * no additional dependencies There is a small potential caveat with versioning for future versions of providers (but those connections change rarely and mostly in backwards compatible ways and likely you could regenerate the "shim" periodically). J.
