vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add singularity to airflow URL: https://github.com/apache/airflow/pull/7191#issuecomment-577373099 okay, I've taken a look at the docker-compose approach. Specifically, I've created a `integration-singularity.yml` file that uses a container base that provides Singularity (v3.5.1). ```yaml version: "2.2" services: singularity: image: quay.io/singularity/singularity:v3.5.1 command: -F /etc/alpine-release entrypoint: /usr/bin/tail privileged: true volumes: - /dev/urandom:/dev/random # Required to get non-blocking entropy source ``` Note that I'm using privileged, which likely requires the sudo:true parameter for the testing setup. The traditional entrypoint is the Singularity executable, so I have a basic entrypoint/command to just keep the container running (and allow me to shell inside). Even outside of testing (which is fairly complicated and I'm hoping to not need to reproduce it locally) we can test if the basic interaction with Singularity would work inside the container. ```bash docker-compose --log-level INFO -f "scripts/ci/docker-compose/integration-singularity.yml" up -d ``` Check that everything went up ok... ```bash docker-compose --log-level INFO -f "scripts/ci/docker-compose/integration-singularity.yml" logs Attaching to docker-compose_singularity_1_7d898ecbdbb0 singularity_1_7d898ecbdbb0 | 3.10.3 ``` And that we continued running... ```bash $ docker-compose --log-level INFO -f "scripts/ci/docker-compose/integration-singularity.yml" ps Name Command State Ports -------------------------------------------------------------------------------------- docker-compose_singularity_1_7d898ecbdbb0 /usr/bin/tail -F /etc/alpi ... Up ``` Great! And then we can shell inside. We confirm the version of singularity. ```bash $ docker exec -it 0081d687eeb1 bash bash-5.0# which singularity /usr/local/bin/singularity bash-5.0# singularity --version singularity version 3.5.1 ``` The basic thing we would need to do is start instances. We can actually build the SIF binary successfully, and let's do that first, pulling the container: ```bash $ docker exec -it 3e73d2d5b41e bash bash-5.0# which singularity /usr/local/bin/singularity bash-5.0# singularity pull docker://busybox INFO: Converting OCI blobs to SIF format INFO: Starting build... Getting image source signatures Copying blob bdbbaa22dec6 done Copying config b075e856be done Writing manifest to image destination Storing signatures 2020/01/22 20:24:24 info unpack layer: sha256:bdbbaa22dec6b7fe23106d2c1b1f43d9598cd8fc33706cc27c1d938ecd5bffc7 INFO: Creating SIF file... INFO: Build complete: busybox_latest.sif bash-5.0# ``` Now let's start the instance. ```bash bash-5.0# singularity instance start busybox_latest.sif busybox ERROR: container cleanup failed: no instance found with name busybox FATAL: container creation failed: mount /etc/localtime->/etc/localtime error: while mounting /etc/localtime: mount source /etc/localtime doesn't exist FATAL: failed to start instance: while running /usr/local/libexec/singularity/bin/starter: exit status 255 ``` okay, let's trace back and add this to our docker-compose yml file, we will mount as a volume from the host and redo the above. ```yaml version: "2.2" services: singularity: image: quay.io/singularity/singularity:v3.5.1 command: -F /etc/alpine-release entrypoint: /usr/bin/tail privileged: true volumes: - /etc/localtime:/etc/localtime ``` Zooming ahead, let's recreate and then start the instance: ```bash bash-5.0# singularity instance start busybox_latest.sif busybox INFO: instance started successfully ``` Success! ```bash bash-5.0# singularity instance list INSTANCE NAME PID IP IMAGE busybox 78 /go/busybox_latest.sif ``` And we can exec to it: ```bash bash-5.0# singularity exec instance://busybox echo "HELLO" HELLO ``` So, although I'm hesitant that something about travis won't provide us with all the permissions that we need, I'm going to give it a try to add a docker compose file to reproduce this. My question for you is how I should go about providing this integration - the examples given previously expose services, and for this case we would want an environment with airflow and singularity in the same space. Do you have thoughts or examples for how to approach this? And then given that we can add a custom container, it would need to use one of the Singularity bases (note they are alpine), and then also include dependencies for airflow (or is the install already handled given the container, and happens before the testing?) Either way, if customization is needed on top of the Singularity base (that isn't handled by the testing suite) what would be the best way to do this - building the image elsewhere and using here, or building on demand? Thanks!
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
