Re: [OMPI devel] Running on Kubernetes

[Rong Ou]

Thanks Ralph! I created an issue to track this:
https://github.com/kubeflow/mpi-operator/issues/12.

On Mon, May 28, 2018 at 5:25 AM r...@open-mpi.org  wrote:

> One suggestion: this approach requires that the job be executed using
> “mpirun”. Another approach would be to integrate PMIx into Kubernetes, thus
> allowing any job to call MPI_Init regardless of how it was started. The
> advantage would be that it enables the use of MPI by workflow-based
> applications that really aren’t supported by mpirun and require their own
> application manager.
>
> See https://pmix.org for more info
>
> Ralph
>
>
> On May 24, 2018, at 9:02 PM, Rong Ou  wrote:
>
> Hi guys,
>
> Thanks for all the suggestions! It's been a while but we finally got it
> approved for open sourcing. I've submitted a proposal to kubeflow:
> https://github.com/kubeflow/community/blob/master/proposals/mpi-operator-proposal.md.
> In this version we've managed to not use ssh, relying on `kubectl exec`
> instead. It's still pretty "ghetto", but at least we've managed to train
> some tensorflow models with it. :) Please take a look and let me know what
> you think.
>
> Thanks,
>
> Rong
>
> On Fri, Mar 16, 2018 at 11:38 AM r...@open-mpi.org 
> wrote:
>
>> I haven’t really spent any time with Kubernetes, but it seems to me you
>> could just write a Kubernetes plm (and maybe an odls) component and bypass
>> the ssh stuff completely given that you say there is a launcher API.
>>
>> > On Mar 16, 2018, at 11:02 AM, Jeff Squyres (jsquyres) <
>> jsquy...@cisco.com> wrote:
>> >
>> > On Mar 16, 2018, at 10:01 AM, Gilles Gouaillardet <
>> gilles.gouaillar...@gmail.com> wrote:
>> >>
>> >> By default, Open MPI uses the rsh PLM in order to start a job.
>> >
>> > To clarify one thing here: the name of our plugin is "rsh" for
>> historical reasons, but it defaults to looking to looking for "ssh" first.
>> If it finds ssh, it uses it.  Otherwise, it tries to find rsh and use that.
>> >
>> > --
>> > Jeff Squyres
>> > jsquy...@cisco.com
>> >
>> > ___
>> > devel mailing list
>> > devel@lists.open-mpi.org
>> > https://lists.open-mpi.org/mailman/listinfo/devel
>>
>> ___
>> devel mailing list
>> devel@lists.open-mpi.org
>> https://lists.open-mpi.org/mailman/listinfo/devel
>
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/devel
>
>
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/devel
___
devel mailing list
devel@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/devel

Re: [OMPI devel] Running on Kubernetes

[r...@open-mpi.org]

One suggestion: this approach requires that the job be executed using “mpirun”. 
Another approach would be to integrate PMIx into Kubernetes, thus allowing any 
job to call MPI_Init regardless of how it was started. The advantage would be 
that it enables the use of MPI by workflow-based applications that really 
aren’t supported by mpirun and require their own application manager.

See https://pmix.org  for more info

Ralph


> On May 24, 2018, at 9:02 PM, Rong Ou  wrote:
> 
> Hi guys,
> 
> Thanks for all the suggestions! It's been a while but we finally got it 
> approved for open sourcing. I've submitted a proposal to kubeflow: 
> https://github.com/kubeflow/community/blob/master/proposals/mpi-operator-proposal.md
>  
> .
>  In this version we've managed to not use ssh, relying on `kubectl exec` 
> instead. It's still pretty "ghetto", but at least we've managed to train some 
> tensorflow models with it. :) Please take a look and let me know what you 
> think.
> 
> Thanks,
> 
> Rong
> 
> On Fri, Mar 16, 2018 at 11:38 AM r...@open-mpi.org  
> > wrote:
> I haven’t really spent any time with Kubernetes, but it seems to me you could 
> just write a Kubernetes plm (and maybe an odls) component and bypass the ssh 
> stuff completely given that you say there is a launcher API.
> 
> > On Mar 16, 2018, at 11:02 AM, Jeff Squyres (jsquyres)  > > wrote:
> > 
> > On Mar 16, 2018, at 10:01 AM, Gilles Gouaillardet 
> > > 
> > wrote:
> >> 
> >> By default, Open MPI uses the rsh PLM in order to start a job.
> > 
> > To clarify one thing here: the name of our plugin is "rsh" for historical 
> > reasons, but it defaults to looking to looking for "ssh" first.  If it 
> > finds ssh, it uses it.  Otherwise, it tries to find rsh and use that.
> > 
> > -- 
> > Jeff Squyres
> > jsquy...@cisco.com 
> > 
> > ___
> > devel mailing list
> > devel@lists.open-mpi.org 
> > https://lists.open-mpi.org/mailman/listinfo/devel 
> > 
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org 
> https://lists.open-mpi.org/mailman/listinfo/devel 
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/devel

Re: [OMPI devel] Running on Kubernetes

[Rong Ou]

Hi guys,

Thanks for all the suggestions! It's been a while but we finally got it
approved for open sourcing. I've submitted a proposal to kubeflow:
https://github.com/kubeflow/community/blob/master/proposals/mpi-operator-proposal.md.
In this version we've managed to not use ssh, relying on `kubectl exec`
instead. It's still pretty "ghetto", but at least we've managed to train
some tensorflow models with it. :) Please take a look and let me know what
you think.

Thanks,

Rong

On Fri, Mar 16, 2018 at 11:38 AM r...@open-mpi.org  wrote:

> I haven’t really spent any time with Kubernetes, but it seems to me you
> could just write a Kubernetes plm (and maybe an odls) component and bypass
> the ssh stuff completely given that you say there is a launcher API.
>
> > On Mar 16, 2018, at 11:02 AM, Jeff Squyres (jsquyres) <
> jsquy...@cisco.com> wrote:
> >
> > On Mar 16, 2018, at 10:01 AM, Gilles Gouaillardet <
> gilles.gouaillar...@gmail.com> wrote:
> >>
> >> By default, Open MPI uses the rsh PLM in order to start a job.
> >
> > To clarify one thing here: the name of our plugin is "rsh" for
> historical reasons, but it defaults to looking to looking for "ssh" first.
> If it finds ssh, it uses it.  Otherwise, it tries to find rsh and use that.
> >
> > --
> > Jeff Squyres
> > jsquy...@cisco.com
> >
> > ___
> > devel mailing list
> > devel@lists.open-mpi.org
> > https://lists.open-mpi.org/mailman/listinfo/devel
>
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/devel
___
devel mailing list
devel@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/devel

Re: [OMPI devel] Running on Kubernetes

[r...@open-mpi.org]

I haven’t really spent any time with Kubernetes, but it seems to me you could 
just write a Kubernetes plm (and maybe an odls) component and bypass the ssh 
stuff completely given that you say there is a launcher API.

> On Mar 16, 2018, at 11:02 AM, Jeff Squyres (jsquyres)  
> wrote:
> 
> On Mar 16, 2018, at 10:01 AM, Gilles Gouaillardet 
>  wrote:
>> 
>> By default, Open MPI uses the rsh PLM in order to start a job.
> 
> To clarify one thing here: the name of our plugin is "rsh" for historical 
> reasons, but it defaults to looking to looking for "ssh" first.  If it finds 
> ssh, it uses it.  Otherwise, it tries to find rsh and use that.
> 
> -- 
> Jeff Squyres
> jsquy...@cisco.com
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/devel

Re: [OMPI devel] Running on Kubernetes

[Jeff Squyres (jsquyres)]

On Mar 16, 2018, at 10:01 AM, Gilles Gouaillardet 
 wrote:
> 
> By default, Open MPI uses the rsh PLM in order to start a job.

To clarify one thing here: the name of our plugin is "rsh" for historical 
reasons, but it defaults to looking to looking for "ssh" first.  If it finds 
ssh, it uses it.  Otherwise, it tries to find rsh and use that.

-- 
Jeff Squyres
jsquy...@cisco.com

___
devel mailing list
devel@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/devel

Re: [OMPI devel] Running on Kubernetes

[Gilles Gouaillardet]

Hi Rong,

SSH is safe when properly implemented. That being said, some sites
does not allow endusers to directly SSH into compute nodes because
they do not want them to do anything without the resource manager
knowing about it. What is your concern with SSH ?

You can run a resource manager (such as SLURM) in each container, so
SSH is no more needed. Not sure that will address your concerns
though.
An other option is to run a PMIx server in each container.

By default, Open MPI uses the rsh PLM in order to start a job.
Basically, mpirun ssh orted on each  compute nodes (this is a tree
spawn also by default), and then orted fork the MPI task.
Yet an other option is to use three  agents to alter this behavior
mpirun --mca plm_rsh_agent /.../rsh --mca orte_launch_agent
/.../launcher --mca orte_fork_agent /.../forker ...
mpirun will /.../rsh /.../launcher instead of ssh orted,
and then orted will fork /.../forker a.out instead of fork a.out

with a bit of creativity, you should be able to plug docker via some
adhoc agents.

Cheers,

Gilles

On Fri, Mar 16, 2018 at 10:30 PM, Rong Ou  wrote:
> Hi,
>
> I've implemented a Kubernetes operator for running OpenMPI jobs there. An
> operator is just a Custom Resource Definition (CRD) for a new type of
> mpi-aware batch job, and a custom controller that manages the states of
> those jobs. Unfortunately the code can't be open sourced just yet, but I've
> put together a demo that roughly matches the current design
> (https://github.com/rongou/k8s-openmpi).
>
> The basic steps are:
>
> Create a pair of one-time-use ssh keys and store them as a Kubernetes
> secret.
> Launch a set of worker pods (in this context, a pod is just a container),
> and start sshd on them.
> Find the IP addresses of the worker pods (OpenMPI doesn't seem to like
> Kubernetes' DNS names "foo.bar.my-namespace.svc.cluster.local").
> Pass the list of worker IPs to a launcher job that invokes `mpirun`.
>
> This seems to work pretty well. I was able to run the tensorflow benchmark
> on 32 GPUs across 4 nodes, on both AWS and an on-prem cluster.
>
> Although I don't think it's an issue, some people are nervous about running
> sshd in a container in a shared cluster environment. So I have two
> questions:
>
> From your operational experience, how do you deal with security concerns
> with regard to allowing ssh traffic?
> If we want to do this without using ssh, what's the best option? Kubernetes
> has a remote execution api similar to `docker exec`, would it be possible to
> launch all the worker pods first, and exec some command on all of them?
> Similar to what SLURM does?
>
> I'm pretty new to OpenMPI and haven't dug around the codebase much yet, so
> I'm at a bit of a loss on how to proceed. Any pointer would be greatly
> appreciated!
>
> Thanks,
>
> Rong
>
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/devel
___
devel mailing list
devel@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/devel

[OMPI devel] Running on Kubernetes

[Rong Ou]

Hi,

I've implemented a Kubernetes operator
 for running OpenMPI
jobs there. An operator is just a Custom Resource Definition (CRD) for a
new type of mpi-aware batch job, and a custom controller that manages the
states of those jobs. Unfortunately the code can't be open sourced just
yet, but I've put together a demo that roughly matches the current design (
https://github.com/rongou/k8s-openmpi).

The basic steps are:

   - Create a pair of one-time-use ssh keys and store them as a Kubernetes
   secret.
   - Launch a set of worker pods (in this context, a pod is just a
   container), and start sshd on them.
   - Find the IP addresses of the worker pods (OpenMPI doesn't seem to like
   Kubernetes' DNS names "foo.bar.my-namespace.svc.cluster.local").
   - Pass the list of worker IPs to a launcher job that invokes `mpirun`.

This seems to work pretty well. I was able to run the tensorflow benchmark
on 32 GPUs across 4 nodes, on both AWS and an on-prem cluster.

Although I don't think it's an issue, some people are nervous about running
sshd in a container in a shared cluster environment. So I have two
questions:

   - From your operational experience, how do you deal with security
   concerns with regard to allowing ssh traffic?
   - If we want to do this without using ssh, what's the best option?
   Kubernetes has a remote execution api similar to `docker exec`, would it be
   possible to launch all the worker pods first, and exec some command on all
   of them? Similar to what SLURM does?

I'm pretty new to OpenMPI and haven't dug around the codebase much yet, so
I'm at a bit of a loss on how to proceed. Any pointer would be greatly
appreciated!

Thanks,

Rong
___
devel mailing list
devel@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/devel