> Yes, it should be easy to build a docker image «from source», or at least as a gradle build task. That could piggy-back on the distro tgz file which should make it not too different - we just pull the release from local disk instead of from the mirrors.
We do this at Salesforce in our local Lucene_Solr fork to also produce a docker image. It's not a big deal but I could share it if we want to consider going this direction. It's kinda necessary if we want to release this all at once instead of requiring a 'tgz' be released first, which in turn somewhat requires some signatures of that binary that then become irrelevant to check when producing the Docker image. It's also super nice for those who fork Solr to also produce a Docker image easily (like us). ~ David Smiley Apache Lucene/Solr Search Developer http://www.linkedin.com/in/davidwsmiley On Sat, Jan 11, 2020 at 5:45 PM Jan Høydahl <[email protected]> wrote: > 1. Are we allowed to maintain ASF code in a non-ASF repo? If not, how do > we transition to > an ASF git repo? > * Can it be a sub folder in our main repo or does it need to be a > separate repo? > > > The way it works (from the official library’s point of view), is that we > maintain > https://github.com/docker-library/official-images/blob/master/library/solr > which > contains a link to a repo (in our case > https://github.com/docker-solr/docker-solr.git) and particular git > commit, and a particular directory for different versions. That is consumed > by their build infrastructure. The library team reviews changes we make to > that file, and the corresponding changes we made to the Dockerfiles and > bash scripts in the docker-solr repo, so it needs to be readily available > and it needs to be easy to see what has changed. > > I think one could theoretically move this into the main Solr repo and > point to its GitHub address, but that would make things slower and much > harder to review. So I think it’s much better to keep the separate repo. I > briefly looked for some official guidance on this, but couldn’t find it > spelled out explicitly. I did see > https://github.com/docker-library/official-images#maintainership which > talks about maintaining git history. > Note also that I already use a “docker-solr” GitHub org for the repo, > rather than my own account, to make it easier to vary ownership. > > If you are dead-set to put it into the main repo, I’d run that discussion > past the library team first before sinking engineering time. > > > I just discovered https://hub.docker.com/u/apache - which is Apache’s own > docker org. I see some images there are hosted in separate apache git > repos, example CouchDB: https://github.com/apache/couchdb-docker pushed > to https://hub.docker.com/r/apache/couchdb - and > https://hub.docker.com/_/couchdb (official). The source of both hub > locations seems to be the same apache/couchdb-docker git repo. I see that > the person who files PRs aginst the official image repo is Joan Touzet ( > http://people.apache.org/~wohali/) who is a CouchDB committer. Perhpas > this is a model for us to follow. > > We may also want to consult LEGAL-503 > <https://issues.apache.org/jira/browse/LEGAL-503?focusedCommentId=17003438&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17003438> > where > the Beam project asked a similar question a few weeks ago, and the reply is: > > *if you would like to continue linking to the Docker release artifact from > the https://beam.apache.org <https://beam.apache.org> you will have:* > *1. Transition to the official ASF dockerhub org: > https://hub.docker.com/u/apache <https://hub.docker.com/u/apache>* > *2. Start including that binary convenience artifact into your VOTE > threads on Beam releases* > *3. Make sure that all Cat-X licenses are ONLY brought into your container > via FROM statements* > > > So bullet point #1 there answers this question. Regarding point #2 and #3 > see below. > > 2. How will the current build/test/publish process need to change? > * Can we continue using travis for CI? > > > In the short term, sure. > > Travis has been great for us — it is free, it builds fast enough, the UI > is nice, the config is simple, the integration is good, and support was > helpful. > Last year Travis CI got acquired, followed by layoffs of senior > engineering staff, so there are concerns about its future, but nothing has > really changed to affect us. > > I imagine it would be nicer to have it in the normal Apache Jenkins world, > but I’m not volunteering for that migration. :-) > > If we want to stay on Travis, there may be some configuration changes > required (roles/permissions/credentials and such that are tied to my > account). > > Oh and just to make it clear: the CI does 2 things: > - it sets build status on GitHub commits (although there is currently no > enforcement to allow only passing PRs to be merged or things like that, or > have review/automerge workflows which would be nice to have) > - and it pushes builds to the > https://hub.docker.com/repository/docker/dockersolr/docker-solr repo — > but those are only used for testing, they are not the docker images that > provide the official images. I've found that occasionally useful, but we > could decide to not do that, or do it differently within the Apache > infrastructure. > > > So I see other ASF projects using travis as well, perhaps ASF has an > account/license? If we continue to use it or if we migrate to Jenkins, we > either way need to run the build and test and then push builds to the > Apache Docker Hub repository space (making the image pull’able with docker > pull apache/solr:tag > The actual producing of official image will be yet another PR to the > docker owned official-images repo. > > * Should publishing of new Docker be a RM responsibility, or something > that happens right > > after each release like the ref-guide? > > > I don’t have a strong opinion. I typically tried to do it as soon as I > became aware of a new version via the solr-user mailing list or twitter. > Sometimes same day, sometimes it would take a week because of changes I > need to make or extra things I wanted to do. > But if I’m more than a few days late someone would be asking about it :-) > The official library team review is usually very fast, same day or 24h. > > > See point #2 from LEGAL-503 above. If we want to officially document / > endorse / link to the image on hub we may want to include the docker image > in the VOTE. I see that the Beam project includes this in their > release-guide (publishing SDK images): > https://beam.apache.org/contribute/release-guide/. What they do is that > push a RC tagged version to their docker-hub as part of the release and > include it in the VOTE. > > 3. Legal stuff - when we as a project file a PR to update the official > solr docker images, > are we then legally releasing a binary version of Solr? > Technically it is Docker CI that build and publish the images, we just > initiate it… > > > I don’t know about that (or how that matters?) > > > Oh, legal stuff matters a lot for Apache :) Again, I think LEGAL-503 > answers this. Bullet #3 there requres the project to make sure that our > Dockerfile does not bring in Cat-X licensed software into the Docker layers > built by us. Since we base our image on the ‘openjdk’ base image, which > contains GNU/Linux binaries and the JDK, the only things we'd need to > verify is what we bring into our Docker layers through apt-get, wget etc. > Below is a list of what I found: > > acl - GPL - provides tool setfacl, used only in tests, can be removed? > dirmngr, gpg - GPL - used only during docker build phase, may be apt > install and uninstalled in the same RUN command > lsof - BSD license > procps - GPL - provides the ‘ps’ command needed by bin/solr. This is part > of openjdk:11 but not openjdk:11-slim... > wget - GPL - used during build only, can be uninstalled after use > netcat - PublicDomain > gosu - GPL - can be removed or replaced with su-exec (MIT) > tini - MIT > > Do we know any other ASF project that maintain their own official > docker image? > > > I've looked at > https://github.com/docker-library/official-images/tree/master/library and > spotted https://github.com/carlossg/docker-maven which is maintained by > an Apache committer. > > > So couchDB is another example. And there are so many other projects in > Apache’s docker-hub org that I suppose there may be others. > > Marcus wrote: > > I think that regardless of what the community decides to do with the > docker-solr repo, a good first step would be to add a Docker folder to the > Apache repository that contains a base Dockerfile and a README. In that > README, users can be directed to the location of the docker-solr repo, > wherever that may be, or leverage the Dockerfile in the Apache repo as a > starting point for building their own image. > > > I think that could be useful; but it then does start to become messy > almost immediately: Users will expect these self-built images and the > official images to work the same, and given that docker-solr has various > extra scripts (eg to create collections at startup), you’d then have to > copy them into the repo (and now have duplicate maintenance, need to test > them). Or you could explicitly decide not to do that, but then your users > will be asking how to achieve the same functionality with their images. > > I would address this as a separate issue. Let’s get the existing image > flow taken care of first. > > > Yes, it should be easy to build a docker image «from source», or at least > as a gradle build task. That could piggy-back on the distro tgz file which > should make it not too different - we just pull the release from local disk > instead of from the mirrors. > > I also saw some projects that have Jenkins routinely publish SNAPSHOT > releases to docker-hub, see e.g. > https://hub.docker.com/r/apache/syncope/tags which is also nice if we > want to have people test out things with unreleased versions or master > branch, then it is always only a docker run command away :) > > Well, I hope other committers also join this discussion and bring perhaps > other points of view here before we start fleshing out actual JIRA tasks to > add to https://issues.apache.org/jira/browse/SOLR-14168. > > If we end up releasing official Solr Docker images together with the > normal release, it would be cool to add documentation to the RefGuide and > perhaps tutorial, on how to run Solr with Docker. > > Jan > > >
