potiuk commented on issue #7191: [AIRFLOW-4030] second attempt to add singularity to airflow URL: https://github.com/apache/airflow/pull/7191#issuecomment-579315631 Thanks @vsoch ! Really sorry for not getting back before - I've been travelling and missed that last comment. Fantastic work! I really liked how you explained the steps you did to make it works! Kudos! > So, although I'm hesitant that something about travis won't provide us with all the permissions that we need, I'm going to give it a try to add a docker compose file to reproduce this. My question for you is how I should go about providing this integration - the examples given previously expose services, and for this case we would want an environment with airflow and singularity in the same space. Do you have thoughts or examples for how to approach this? Travis and any other environment will provide all that we need. We already run in privileged mode (and pretty much all CI systems allow that in order to make all kinds of docker-in-docker approach). In fact what we already do is we are mounting docker socket to within the airflow-testing so that you can even run docker commands to reach out the host docker engine. This is needed to run kind (Kubernetes-In-Docker). > And then given that we can add a custom container, it would need to use one of the Singularity bases (note they are alpine), and then also include dependencies for airflow (or is the install already handled given the container, and happens before the testing?) Either way, if customization is needed on top of the Singularity base (that isn't handled by the testing suite) what would be the best way to do this - building the image elsewhere and using here, or building on demand? Hmm, I'd assume that having a base Singularity should be enough and that we can reach it via "singularity" host name via docker compose. But I understand you need to build in some Airflow dependencies in? Am I right? What kind of dependencies you need to build in that container? Is this like local airflow sources or some airflow dependencies? Excuse my ignorance as I do not know much about singularity. We have few approaches we can get: 1) In case of Kubernetes (Kind) we are using RUNTIME="kubernetes" env and we are checking that in the initial CI scripts - if the RUNTIME="kubernetes" we will install kubernetes-in-docker cluster and provision it with the latest Airflow locally built CI image and then we run our tests with this. This is most complex thing we do and I hope we do not have to do the same. 2) If those are about some small packages to install at running singularity after it is started already, we could simply do it in the CI scripts when we start up and see "INTEGRATION_SINGULARITY" is set to "true" (this will be properly set to true if you add it similarly to other integrations of ours and when --integration singularity switch is used). 3) If you need something pre-backed in the Docker image itself then I think we need a separate Dockerfile with FROM: singularity and some steps to build the dockerfile before docker-compose is run (and refer to the newly built image in Docker compose. My best bet if that is the case, that would be that you build and test such images locally for local testing and once it works I can help to integrate it in the CI scripts properly (You can try yourself - this will likely be just appropriate 'docker build' command executed only when --integration singularity is used) I hope this is helpful!
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
