Licensing wise there is no issue from me: The astronomerinc images are just re-packaging of the upstream images to apply security fixes so are licensed under whatever the original image is (MIT or Apache2 usually, else we wouldn't have put them in the helm chart PR)
For background, the reason that we at Astronomer created ap-pgbouncer-exporter in the first place is that the upstream package does not patch/rebuild to address security vulnerabilities. By taking this in to airflow-ext it means we as a project become responsible for monitoring and testing that. (And don't be fooled in to thinking the free scanners can detect all vulns here, we've found them to be very of variable, and questionable accuracy.) That is a non-trivial amount of work for an open source project. Has this ever caused us any problems outside of Pip/python dependencies? (I'm not aware of any.) For runtime this maybe makes sense (again, I'm not yet convinced), but for test-only/dev-only deps this seems like a lot of work that we could better spend on working on Airflow. If we pin versions of docker image used then the only real risk is a left-pad scenario of "I'm deleting all my images" which is a minor risk. Do any other project do anything like this? I haven't seen it before. I'd vote for doing nothing and addressing this in specific cases when it becomes a problem. Because I do not see using thidy party docker images as a risk. I see it as a time saving measure. -ash On Jun 22 2020, at 1:42 pm, Jarek Potiuk <[email protected]> wrote: > Hello everyone, > > TL;DR; I noticed that we are accumulating some dependencies to external > binaries (downloads and Docker images) which make the Apache Airflow > Community a bit vulnerable to external dependencies. I would love your > comments/opinions on the proposal I made around this. > > *More explanation/status:* > > While dependence is fine for officially "released" and "managed" by the > owning organizations, I think it is a bit risky to depend on those long > term and I think we should aim to bring all those "vulnerable" dependencies > into community control. > > I reviewed all our code (or I think all !) looking for such dependencies > and prepared an "umbrella" issue where I proposed the approach we can take > for all such dependencies. > > I could have missed some - so if you find others feel free to comment/add > the new ones. > All the details are captured here: > https://github.com/apache/airflow/issues/9401 - I discussed the > context/motivation/current status and approach we can take for those > dependencies. > > A lot of those dependencies just need review and maybe some updates to > latest versions. And I do not think there is a lot to discuss for those. > > There is one point, however, that requires more deliberate action and some > decisions I think. > > We have some dependencies on Docker images that we are using from various > sources: > 1) officially maintained images > 2) images released by organizations that released them for their own > purpose, but they are not "officially maintained" by those organizations > 3) images released by private individuals > > While 1) is perfectly OK, I think for 2) and 3) we should bring the images > to Airflow community management. Here is the list of those images I found > that need to be moved to Airflow: > > - aneeshkj/helm-unittest > - ashb/apache-rat:0.13-1 > - godatadriven/krb5-kdc-server > - polinux/stress (?) > - osixia/openldap:1.2.0 > - astronomerinc/ap-statsd-exporter:0.11.0 > - astronomerinc/ap-pgbouncer:1.8.1 > - astronomerinc/ap-pgbouncer-exporter:0.5.0-1 > > > *Proposal*: > > My proposal is to make a folder in our repository on Github (continue with > the mono-repo approach we follow) to keep corresponding Dockerfiles and > scripts that build and release images from there. Now the only > question is > where to keep those images. We currently have apache/airflow but I > think we > should reserve it for airflow images only and we should keep those images > elsewhere. Unfortunately, we cannot have "sub-images" of any sort in > DockerHub. We are already abusing a bit the "apache/airflow" namespace as > we are keeping both CI and production images there (but that's quite > OK as > the images are similar). > > My proposal will be to create an* "apache/airflow-ext"* DockerHub > repository and keep the images there. They will also be a little > abused because we will have to name them with tags - for example: > > - apache/airflow-ext:helm-unittest-[version] > - apache/airflow-ext:apache-rat-[version] > > I am also open to other names for the repo and proposals other ways > how to > handle that. > > I believe there is no issue with Licences for either of those images (Ash, > Kaxil, Fokko - some of the images are Astronomer's/GoDataDriven's ones - > can you comment on that ?) but I believe licensing on all those > images are > ok for us to copy with attribution (I will double-check that for other > images). > > WDYT? > > J. > > > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> >
