vsoch commented on issue #4846: [AIRFLOW-4030] adding start to singularity for airflow URL: https://github.com/apache/airflow/pull/4846#issuecomment-470208501 hey @mik-laj ! Thanks for your quick feedback, I created and linked the Jira issue, and I'll get the fixes pushed up in a bit (doing more testing first). However, I'd like to quickly answer your questions! > What is the difference between Docker, Docker Swarm, Kubernetes and Signularity? Docker and Singularity are both container technologies - the unit of operation for something like Docker Swarm, or Kubernetes. Singularity recently became OCI compliant with version 3.1.0, so we will (soon) see an ability to plug Singularity containers into a cluster akin to Docker (but more development is needed first). From an abstraction, From a "what are they made of" perspective, Docker and Singularity are hugely similar, the main difference being that Docker containers are assembled from .tar.gz layers, while Singularity containers are single, read only filesystems. From a software / security perspective, they are very different. Singularity came along to solve the problem of Docker having a root daemon (and thus not being installable on a shared HPC resource) and as a result, Singularity is the container of choice for Scientific compute. So logically, if we want to use containers with AIrflow in this environment, we need Singularity. This is why I want to explore this operator - I would like to test and use AIrflow in this context. If Singularity containers work well with Airflow, I'd next look into adding (what I think are considered executors?) akin to Kubernetes, we could use a SLURM or SGE job manager. But for now - Singularity on the level of operator is the right thing - because it's just like Docker. From a high level, you can think of a Singularity container as a binary with (some other software) inside. Does that answer your questions?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
