Side comment~ I am super glad we have now also Composer and MWAA people commenting and answering on the devlist. I hope this is just a beginning and you will get more involved over time :D
@Daniel: > Is at all feasible to deprecate connection UI customization? Then everything > can just use `extra` json where the other params fall short. Seems like an > area where the benefit does not outweigh the complexity. We could also take > the opportunity to deprecate the long `extra` key names like > `extra__google_cloud_platform__keyfile_dict` in favor of simpler ones e.g. > `keyfile_dict`. I think this is a very useful feature. As Ash mentioned, there is also the test_connection feature added recently, and I think I took part in quite a number of discussions that people actually find it useful, We had a bug in 2.0.0 and there were complaints :) > Thank you for surfacing this issue on a discussion. The major hurdle for > managed services apart from the security constraints is on the licensing > side. Previously when the code needed for connection templates was part of > Airflow, we were able to bundle them as a solution as the code was under the > Apache v2 license. Now that we have them separated out as provider packages, > those come with dependencies that do not have "blessed" licenses that allow > bundling them into managed service. I am sure GCP folks have similar > restrictions on why they cannot add all 60+ providers as is into the base > image. > We recently did the manual exercise to assess each of those provider package > and their dependencies, and only 20 of them made the cut for not having to > use additional licenses like Facebook license, LGPL etc. Valid point. Perfectly understandable. I can sympathise with that - we had very similar to the discussions we had in ASF about releasing source packages). While for Airflow those provider packages are optional so according to ASF policies we can do, it, I understand for managed services it might be a bit different. > > >As a temporary workaround we baked all connections (list of them with their > > >widgets pickled and stored inside) into a web server image, so that > > >customers can add/edit them (even though not all providers packages are > > >pre-installed). This is a temporary workaround that we came up with for now > > >and we are looking for a long-term solution. Yeah, This is my thought exactly on how to fix it but we can do it in a manageable way. I thought that we could simply provide a tool/script/airflow command (might be community managed) that could produce provider's "shims" to handle the case. Provider manager already extracts all the relevant information during CI for all the providers released, and it could easily bundle this information into a separate set of packages that will be a super-slimmed down version of providers with JUST UI customization. You would then be able to build + install such "shim" packages automatically instead of the main provider whenever new providers are released. They will contain just ASF-licensed code (So likely you should be ok Subash) and no dependencies (Eugene) and then you could choose which "real" and which "shim" packages you would like to install on the webserver. Would that work? Maybe that's a nice contribution to the community. Happy to help on that one :) J.
