On 06/03/2021 17:06, Janek Kozicki (yade) wrote:
I am not exactly sure what you want to discuss,

I don't know either LOL. That's more an announcement in advance so someone can raise issues, ask features, etc.

Do you want to create some sort of packages with yade installed inside?

You can call it package, but that's more like some docker images in a different format (from a very macroscopic point of view). Main thing is that it is allowed on HPC (where compiling yade can be big pain) and it seems to become more popular.

Hence people will look for a yade-docker target (one with yade inside) in order to build their singularity images, and it is fairly easy to offer some.

Mind that before using Singularity I have never been able to get all checkTests to pass on our HPC cluster. I was able to run what I needed most of the time, but never to pass all tests. There was always an issue with something.

I am not sure if yade-dev registry will be able to hold big
docker images.

Good point, though the images have no reason to be much bigger than our current docker images. The problem would be more images, not larger images. I will check registry limit. If it is a problem I can keep pushing to gitlab.com/bchareyre registry, not an issue.   See: https://gitlab.com/bchareyre/docker-yade/container_registry/1672064

As you see the images are from 1.1GB to 1.7GB, not a big increment.

We may run out of space if we don't start paying
gitlab for hosting.

Not an issue. What I described is what I'm already doing under my account (without paying). If migrating one thing from gitlab/bchareyre to gitlab/yade-dev is the cause of running out of space, then I'll just not migrate. It is not an problem to provide the images to the users under my registry.

Perhaps these singularity_docker packages should also be on yade-dem.org ?

Excessively complex. We would have to setup a registry on our local server while gitlab does that very well.

The interesting stuff for me would be if we could use these HPC
singularity servers in our gitlab continuous integration pipeline :)

If you mean accessing more hardware ressources, no, it will not work in Grenoble. The HPC clusters are dedicated to scientific computing. They have special job submission systems, it will absolutely not integrate in a CI framework.

The yade-runner-01 quickly runs out of space whenever try to I enable
it ;-)

Yeah, but this is a completely different type of ressources, even if they are provided by the same people overall. Maybe it is a good time to check again how I could get gitlab runners for yade. They improved a number of things and offered new services in the recent years. There might be docker farms more easily accessible than when Rémi configured yade-runner-01, now. Rémi was basically ahead of things.

Maybe it is only a matter of single line in
file /etc/gitlab-runner/config.toml , change:

   executor = "docker"


   executor = "singularity"

I think this is quite likely.

Very likely but there is no point doing that, I think.
Why would you generate a singularity image from a docker image to achieve something the docker image does just as well? In the context of using gitlabCI/docker we have root privileges, hence no issue with docker.

We already have incremental recompilation in our gitlab CI pipeline.
The ccache is used for that. The trick was to mount inside docker
(for you: inside singularity) a local directory from the host
filesystem, where the ccache files are stored.

That the gitlab compilation is incremental doesn't make my own local compilation incremental. However if I can download a snapshot of the gitlab pipeline as a virtual machine I can compile incrementally, locally, even though the initial compilation wasn't local.

Note that the docker images are re-downloaded from gitlab only when
they have been rebuilt on https://gitlab.com/yade-dev/docker-yade/-/pipelines
And this download is pretty slow. Fortunately it happens only every
few weeks. Otherwise docker uses the cached linux distro image.

I see where I lost you. Singularity images (at least in my project) are not in any way related to CI. They are related to, primarily: how actual users get actual results (production).
And optionally, to how devs actually compile locally.

Well, download once (wait for download to finish) then start working.
Not much difference to waiting for local compilation (for me that's
inside chroot, sometimes inside docker) then start working :)

With my university connection speed, downloading a docker image and recompiling just one *.cpp is way faster than downloading trunk and compiling everything from scratch. Like incredibly faster. I'm not speaking of what happens on gitlab, I'm speaking of what happens on my own computer.

pushing to registry is part of the pipeline on docker-yade:


Yes, that's my fault <https://gitlab.com/yade-dev/docker-yade/-/commit/c0674c4aacdd3207bb156d2f385704ac5bf5d763>. :)

But I was speaking of pushing from trunk pipeline. Anyway, the incremental compilation part is a secondary point.

Main point is we can easily provide some docker/singularity images that people could directly use to run yade (on HPC especially). Currently we don't. Depending on storage limits they can be under yade-dev or bchareyre, I don't mind. Either way we can point to them in the doc.



best regards

Bruno Chareyre said:     (by the date of Fri, 5 Mar 2021 10:37:59 +0100)

Hi there,

I'm planning to build new docker images in yade's gitlab for production,
and possibly for development (see second part of this message, some
background comes first).  This is open to suggestions.

* Background:

I recently started playing with "Singularity" images since I found our
HPC department made it available on the clusters. There was also a user
mentioning that on launchpad recently. From end-user POV, singularity
images work like docker images, but a very practical difference is that
it is allowed on our (and others') HPC. Docker isn't, for security reason.

It made running yade so easy. The above command worked immediately, and
should work just the same on every system with singularity installed:
/ssh myHPC//
//singularity exec
yadedaily --check/

or equivalently: /
///export YADE=//'singularity exec
$YADE --check
$YADE myScript.py
$ etc.

Key points:
1- singularity accepts docker images in input.
2- the above command is using some custom docker with yadedaily
pre-installed (which then needs to be downloadable from somewhere where
docker is permitted)
3- it is compatible with MPI(!). The host system's MPI is able to
communicate with the image system's MPI in a scenario like this, as if
it was just yade running natively on the host:
/mpirun -np 500 $YADE someParrallelStuff.py
/4- a condition for this MPI magic to work is that the mpi library is in
the same version for the host and for the executed image
5- performance: no measurable difference compared to a yade compiled on
the host (be it running -j1, -jN or mpiexec).

For the moment the custom dockers are built in [1]
I'm also building a Singularity images with [2]
but I didn't really use it since I can build it from docker directly on
the cluster (building the singularity image is implicit in /singularity
exec docker://.../). Building on-site may not be allowed everywhere,
though, and in that case [2] could be useful.

* What can be done:

I will move [1,2] or something similar to gitlab/yade-dev and advertise
it in the install page. Also build more versions for people to use them.
More versions because of the MPI point above (4): depending on the host
system someone may want OMPI v1 (unbuntu16), or v2 (ubuntu18), etc.

For production images it would make sense to just use canonical
debian/ubuntu with yade and/or yadedaily preinstalled. But, it is not
exactly what I did for the moment. Instead I used docker images from our
registry. Which implies the images have yade, and also what it needs to
compile yade (I didn't test compilation yet but it should work).

I was thinking of splitting that into two types of images; minimal
images for production and "dev" images with all compilation
pre-requisites. Then I realized that the best "dev" image would be - by
far - one reflecting the state of the system at the end of our current
pipeline, i.e. one with a full /build folder and possibly ccache info
(if not too large).

If such dev images were pushed to yade registry then anyone could grab
latest build and recompile incrementally. It could save a lot of
(compilation) time for us when trying to debug something on multiple

And what about that?: compiling with a ubuntu20 docker image on a
ubuntu20 host should make it possible to use the pipeline's ccache while
still running yade on the native system (provided that the install path
is in the host filesystem).

Maybe pushing to registry could be done directly as part of current
pipeline, not sure yet. I am still thinking about some aspects but I
think you get the general idea. Suggestions and advices are welcome. :)



[1] https://gitlab.com/bchareyre/docker-yade


Mailing list: https://launchpad.net/~yade-dev
Post to     : yade-dev@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yade-dev
More help   : https://help.launchpad.net/ListHelp

Reply via email to