Hi Bruno, I have created several branches for all supported distributions (ubuntu and debian). Both (yade and yadedaily) are installed, but the doc-packages were dropped, I do not think they are needed for production.
Also I tried to reduce the size of the images, so they are just a little more than 1gb. Pipeline are scheduled for every day (after newly built yadedaily-packages are placed into the repository). For yadedaily one needs to install also the python-mpi4py package. It should be fixed with this MR [1]. [1] https://gitlab.com/yade-dev/trunk/-/merge_requests/636 Please test the setup. And when it is OK, I will update the documentation. Basically you need to run the following: docker run --rm -it registry.gitlab.com/yade-dev/docker-prod:ubuntu20.04 or docker run --rm -it registry.gitlab.com/yade-dev/docker-prod:debian-buster Best regards Anton Am Fr., 5. März 2021 um 10:38 Uhr schrieb Bruno Chareyre < bruno.chare...@3sr-grenoble.fr>: > Hi there, > > I'm planning to build new docker images in yade's gitlab for production, > and possibly for development (see second part of this message, some > background comes first). This is open to suggestions. > > * Background: > > I recently started playing with "Singularity" images since I found our HPC > department made it available on the clusters. There was also a user > mentioning that on launchpad recently. From end-user POV, singularity > images work like docker images, but a very practical difference is that it > is allowed on our (and others') HPC. Docker isn't, for security reason. > > It made running yade so easy. The above command worked immediately, and > should work just the same on every system with singularity installed: > *ssh myHPC* > *singularity exec > docker://registry.gitlab.com/bchareyre/docker-yade:ubuntu20.04-daily > <http://registry.gitlab.com/bchareyre/docker-yade:ubuntu20.04-daily> > yadedaily --check* > > or equivalently: > > > > > *export YADE='singularity exec > docker://registry.gitlab.com/bchareyre/docker-yade:ubuntu20.04-daily > <http://registry.gitlab.com/bchareyre/docker-yade:ubuntu20.04-daily> > yadedaily' $YADE --check $YADE myScript.py $ etc. * > > Key points: > 1- singularity accepts docker images in input. > 2- the above command is using some custom docker with yadedaily > pre-installed (which then needs to be downloadable from somewhere where > docker is permitted) > 3- it is compatible with MPI(!). The host system's MPI is able to > communicate with the image system's MPI in a scenario like this, as if it > was just yade running natively on the host: > > *mpirun -np 500 $YADE someParrallelStuff.py *4- a condition for this MPI > magic to work is that the mpi library is in the same version for the host > and for the executed image > 5- performance: no measurable difference compared to a yade compiled on > the host (be it running -j1, -jN or mpiexec). > > For the moment the custom dockers are built in [1] > <https://gitlab.com/bchareyre/docker-yade>. > I'm also building a Singularity images with [2] > <https://gitlab.com/bchareyre/yade-singularity/-/blob/master/.gitlab-ci.yml> > but I didn't really use it since I can build it from docker directly on the > cluster (building the singularity image is implicit in *singularity exec > docker://...*). Building on-site may not be allowed everywhere, though, > and in that case [2] could be useful. > > * What can be done: > > I will move [1,2] or something similar to gitlab/yade-dev and advertise it > in the install page. Also build more versions for people to use them. More > versions because of the MPI point above (4): depending on the host system > someone may want OMPI v1 (unbuntu16), or v2 (ubuntu18), etc. > > For production images it would make sense to just use canonical > debian/ubuntu with yade and/or yadedaily preinstalled. But, it is not > exactly what I did for the moment. Instead I used docker images from our > registry. Which implies the images have yade, and also what it needs to > compile yade (I didn't test compilation yet but it should work). > > I was thinking of splitting that into two types of images; minimal images > for production and "dev" images with all compilation pre-requisites. Then I > realized that the best "dev" image would be - by far - one reflecting the > state of the system at the end of our current pipeline, i.e. one with a > full /build folder and possibly ccache info (if not too large). > > If such dev images were pushed to yade registry then anyone could grab > latest build and recompile incrementally. It could save a lot of > (compilation) time for us when trying to debug something on multiple > distros. > > And what about that?: compiling with a ubuntu20 docker image on a ubuntu20 > host should make it possible to use the pipeline's ccache while still > running yade on the native system (provided that the install path is in the > host filesystem). > > Maybe pushing to registry could be done directly as part of current > pipeline, not sure yet. I am still thinking about some aspects but I think > you get the general idea. Suggestions and advices are welcome. :) > > Chers > > Bruno > > [1] https://gitlab.com/bchareyre/docker-yade > > [2] > https://gitlab.com/bchareyre/yade-singularity/-/blob/master/.gitlab-ci.yml > > > > > > > > > _______________________________________________ > Mailing list: https://launchpad.net/~yade-dev > Post to : yade-dev@lists.launchpad.net > Unsubscribe : https://launchpad.net/~yade-dev > More help : https://help.launchpad.net/ListHelp >
_______________________________________________ Mailing list: https://launchpad.net/~yade-dev Post to : yade-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~yade-dev More help : https://help.launchpad.net/ListHelp