Re: [OMPI users] Problem with 'orted: command not found'

[Maciek Lewiński]

Thank you! I have set up my env paths at the end of the script and thanks
to you I just noticed that at the beginning of the bashrc script there's a
simple IF that returns when it's opened as non-interactive. I moved my
exports above it and it finally works. Again, thank you very much.

2016-05-03 3:19 GMT+02:00 Jeff Squyres (jsquyres) :

> Make sure you check that these paths are set for *non-interactive* logins.
>
>
> > On May 2, 2016, at 6:14 PM, Maciek Lewiński 
> wrote:
> >
> > I already had correct paths in .bashrc:
> >
> > export
> PATH=/usr/lib/lightdm/lightdm:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/bin:/usr/local/bin:/home/$USER/.openmpi/bin
> >
> > export
> LD_LIBRARY_PATH=:/usr/local/lib:/usr/local/lib:/home/$USER/.openmpi/lib
> >
> > I can run mpi normally from slave1 so I'm sure they work. I also
> exported these paths in .profile just to be sure a moment before but it
> didn't work. Still the same error.
> >
> >
> >
> > 2016-05-02 23:40 GMT+02:00 Jeff Squyres (jsquyres) :
> > The key is this error:
> >
> > bash: orted: command not found
> >
> > Meaning: you need to set your PATH and LD_LIBRARY_PATH properly for
> non-interactive logins.  See
> https://www.open-mpi.org/faq/?category=running#adding-ompi-to-path.
> >
> >
> > > On May 2, 2016, at 5:36 PM, Maciek Lewiński 
> wrote:
> > >
> > > Hi, I'm having problem with Open MPI version 1.10.2.
> > > I've installed two virtual machines on VirtualBox, both are the same
> images of Ubuntu 12.04 64bit.
> > > Both have the same accounts, both have everything configured almost
> exactly the same.
> > > I have configured OMPI only with the --prefix to specify my location
> of install folder which is /home/$USER/.openmpi.
> > > Users on both machines are identical.
> > >
> > > On both machines running mpirun on one of the example programs work
> perfectly, on both machines I've added bin and lib to corresponding PATHs,
> env command ran on master and through ssh on slave1 gives these results:
> > > osboxes@osboxes:~/cloud$ env | grep PATH
> > >
> LD_LIBRARY_PATH=:/usr/local/lib:/usr/local/lib:/home/osboxes/.openmpi/lib
> > >
> PATH=/usr/lib/lightdm/lightdm:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/bin:/usr/local/bin:/home/osboxes/.openmpi/bin
> > >
> > > On both hosts I have installed ssh which works, I've enabled
> passwordless connection, which again, works,
> > >
> > > I have created an NFS folder in which I hold the hello_c program from
> examples and hosts file that looks like this:
> > > 192.168.0.191master
> > > 192.168.0.190slave1
> > >
> > > The same two lines are placed in /etc/hosts for ssh to work.
> > >
> > > On both machines, running:
> > > osboxes@osboxes:~/cloud$ mpirun -np 1 ./hello_c
> > > Hello, world, I am 0 of 1, (Open MPI v1.10.2, package: Open MPI
> osboxes@osboxes Distribution, ident: 1.10.2, repo rev:
> v1.10.1-145-g799148f, Jan 21, 2016, 126)
> > >
> > > Works, even running this command on slave1 through ssh from master
> works as expected.
> > >
> > > Yet when I try to execute the following command I get the error:
> > > osboxes@osboxes:~/cloud$ mpirun -np 2 --hostfile hosts ./hello_c
> > > bash: orted: command not found
> > >
> --
> > > ORTE was unable to reliably start one or more daemons.
> > > This usually is caused by:
> > >
> > > * not finding the required libraries and/or binaries on
> > >   one or more nodes. Please check your PATH and LD_LIBRARY_PATH
> > >   settings, or configure OMPI with --enable-orterun-prefix-by-default
> > >
> > > * lack of authority to execute on one or more specified nodes.
> > >   Please verify your allocation and authorities.
> > >
> > > * the inability to write startup files into /tmp
> (--tmpdir/orte_tmpdir_base).
> > >   Please check with your sys admin to determine the correct location
> to use.
> > >
> > > *  compilation of the orted with dynamic libraries when static are
> required
> > >   (e.g., on Cray). Please check your configure cmd line and consider
> using
> > >   one of the contrib/platform definitions for your system type.
> > >
> > > * an inability to create a connection back to mpirun due to a
> > >   lack of common network interfaces and/or no route found between
> > >   them. Please check network connectivity (including firewalls
> > >   and network routing requirements).
> > >
> --
> > >
> > > The same happens when I use -host master,slave1 instead of --hostfile
> hosts. I'm slowly running out if ideas, I've tried anything I could find on
> the internet or OMPI FAQ, nothing seems to work. What am I doing wrong?
> > > ___
> > > users mailing list
> > > us...@open-mpi.org
> > > Subscription: 

Re: [OMPI users] Problem with 'orted: command not found'

[Gilles Gouaillardet]

If OpenMPI is installed at the same path on every node, the easiest optin
is to re-configure with
--enable-mpirun-prefix-by-default
an other option is to use
`which mpirun` instead of mpirun
and yet an other option is to
mpirun --prefix=$USER/.openmpi

Cheers,

Gilles

On Tuesday, May 3, 2016, Maciek Lewiński  wrote:

> I already had correct paths in .bashrc:
>
> export
> PATH=/usr/lib/lightdm/lightdm:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/bin:/usr/local/bin:/home/$USER/.openmpi/bin
>
> export
> LD_LIBRARY_PATH=:/usr/local/lib:/usr/local/lib:/home/$USER/.openmpi/lib
>
> I can run mpi normally from slave1 so I'm sure they work. I also exported
> these paths in .profile just to be sure a moment before but it didn't work.
> Still the same error.
>
>
>
> 2016-05-02 23:40 GMT+02:00 Jeff Squyres (jsquyres)  >:
>
>> The key is this error:
>>
>> bash: orted: command not found
>>
>> Meaning: you need to set your PATH and LD_LIBRARY_PATH properly for
>> non-interactive logins.  See
>> https://www.open-mpi.org/faq/?category=running#adding-ompi-to-path.
>>
>>
>> > On May 2, 2016, at 5:36 PM, Maciek Lewiński > > wrote:
>> >
>> > Hi, I'm having problem with Open MPI version 1.10.2.
>> > I've installed two virtual machines on VirtualBox, both are the same
>> images of Ubuntu 12.04 64bit.
>> > Both have the same accounts, both have everything configured almost
>> exactly the same.
>> > I have configured OMPI only with the --prefix to specify my location of
>> install folder which is /home/$USER/.openmpi.
>> > Users on both machines are identical.
>> >
>> > On both machines running mpirun on one of the example programs work
>> perfectly, on both machines I've added bin and lib to corresponding PATHs,
>> env command ran on master and through ssh on slave1 gives these results:
>> > osboxes@osboxes:~/cloud$ env | grep PATH
>> >
>> LD_LIBRARY_PATH=:/usr/local/lib:/usr/local/lib:/home/osboxes/.openmpi/lib
>> >
>> PATH=/usr/lib/lightdm/lightdm:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/bin:/usr/local/bin:/home/osboxes/.openmpi/bin
>> >
>> > On both hosts I have installed ssh which works, I've enabled
>> passwordless connection, which again, works,
>> >
>> > I have created an NFS folder in which I hold the hello_c program from
>> examples and hosts file that looks like this:
>> > 192.168.0.191master
>> > 192.168.0.190slave1
>> >
>> > The same two lines are placed in /etc/hosts for ssh to work.
>> >
>> > On both machines, running:
>> > osboxes@osboxes:~/cloud$ mpirun -np 1 ./hello_c
>> > Hello, world, I am 0 of 1, (Open MPI v1.10.2, package: Open MPI
>> osboxes@osboxes Distribution, ident: 1.10.2, repo rev:
>> v1.10.1-145-g799148f, Jan 21, 2016, 126)
>> >
>> > Works, even running this command on slave1 through ssh from master
>> works as expected.
>> >
>> > Yet when I try to execute the following command I get the error:
>> > osboxes@osboxes:~/cloud$ mpirun -np 2 --hostfile hosts ./hello_c
>> > bash: orted: command not found
>> >
>> --
>> > ORTE was unable to reliably start one or more daemons.
>> > This usually is caused by:
>> >
>> > * not finding the required libraries and/or binaries on
>> >   one or more nodes. Please check your PATH and LD_LIBRARY_PATH
>> >   settings, or configure OMPI with --enable-orterun-prefix-by-default
>> >
>> > * lack of authority to execute on one or more specified nodes.
>> >   Please verify your allocation and authorities.
>> >
>> > * the inability to write startup files into /tmp
>> (--tmpdir/orte_tmpdir_base).
>> >   Please check with your sys admin to determine the correct location to
>> use.
>> >
>> > *  compilation of the orted with dynamic libraries when static are
>> required
>> >   (e.g., on Cray). Please check your configure cmd line and consider
>> using
>> >   one of the contrib/platform definitions for your system type.
>> >
>> > * an inability to create a connection back to mpirun due to a
>> >   lack of common network interfaces and/or no route found between
>> >   them. Please check network connectivity (including firewalls
>> >   and network routing requirements).
>> >
>> --
>> >
>> > The same happens when I use -host master,slave1 instead of --hostfile
>> hosts. I'm slowly running out if ideas, I've tried anything I could find on
>> the internet or OMPI FAQ, nothing seems to work. What am I doing wrong?
>> > ___
>> > users mailing list
>> > us...@open-mpi.org 
>> > Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users
>> > Link to this post:
>> 

Re: [OMPI users] Problem with 'orted: command not found'

[Jeff Squyres (jsquyres)]

Make sure you check that these paths are set for *non-interactive* logins.


> On May 2, 2016, at 6:14 PM, Maciek Lewiński  wrote:
> 
> I already had correct paths in .bashrc:
> 
> export 
> PATH=/usr/lib/lightdm/lightdm:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/bin:/usr/local/bin:/home/$USER/.openmpi/bin
> 
> export LD_LIBRARY_PATH=:/usr/local/lib:/usr/local/lib:/home/$USER/.openmpi/lib
> 
> I can run mpi normally from slave1 so I'm sure they work. I also exported 
> these paths in .profile just to be sure a moment before but it didn't work. 
> Still the same error.
> 
> 
> 
> 2016-05-02 23:40 GMT+02:00 Jeff Squyres (jsquyres) :
> The key is this error:
> 
> bash: orted: command not found
> 
> Meaning: you need to set your PATH and LD_LIBRARY_PATH properly for 
> non-interactive logins.  See 
> https://www.open-mpi.org/faq/?category=running#adding-ompi-to-path.
> 
> 
> > On May 2, 2016, at 5:36 PM, Maciek Lewiński  
> > wrote:
> >
> > Hi, I'm having problem with Open MPI version 1.10.2.
> > I've installed two virtual machines on VirtualBox, both are the same images 
> > of Ubuntu 12.04 64bit.
> > Both have the same accounts, both have everything configured almost exactly 
> > the same.
> > I have configured OMPI only with the --prefix to specify my location of 
> > install folder which is /home/$USER/.openmpi.
> > Users on both machines are identical.
> >
> > On both machines running mpirun on one of the example programs work 
> > perfectly, on both machines I've added bin and lib to corresponding PATHs, 
> > env command ran on master and through ssh on slave1 gives these results:
> > osboxes@osboxes:~/cloud$ env | grep PATH
> > LD_LIBRARY_PATH=:/usr/local/lib:/usr/local/lib:/home/osboxes/.openmpi/lib
> > PATH=/usr/lib/lightdm/lightdm:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/bin:/usr/local/bin:/home/osboxes/.openmpi/bin
> >
> > On both hosts I have installed ssh which works, I've enabled passwordless 
> > connection, which again, works,
> >
> > I have created an NFS folder in which I hold the hello_c program from 
> > examples and hosts file that looks like this:
> > 192.168.0.191master
> > 192.168.0.190slave1
> >
> > The same two lines are placed in /etc/hosts for ssh to work.
> >
> > On both machines, running:
> > osboxes@osboxes:~/cloud$ mpirun -np 1 ./hello_c
> > Hello, world, I am 0 of 1, (Open MPI v1.10.2, package: Open MPI 
> > osboxes@osboxes Distribution, ident: 1.10.2, repo rev: 
> > v1.10.1-145-g799148f, Jan 21, 2016, 126)
> >
> > Works, even running this command on slave1 through ssh from master works as 
> > expected.
> >
> > Yet when I try to execute the following command I get the error:
> > osboxes@osboxes:~/cloud$ mpirun -np 2 --hostfile hosts ./hello_c
> > bash: orted: command not found
> > --
> > ORTE was unable to reliably start one or more daemons.
> > This usually is caused by:
> >
> > * not finding the required libraries and/or binaries on
> >   one or more nodes. Please check your PATH and LD_LIBRARY_PATH
> >   settings, or configure OMPI with --enable-orterun-prefix-by-default
> >
> > * lack of authority to execute on one or more specified nodes.
> >   Please verify your allocation and authorities.
> >
> > * the inability to write startup files into /tmp 
> > (--tmpdir/orte_tmpdir_base).
> >   Please check with your sys admin to determine the correct location to use.
> >
> > *  compilation of the orted with dynamic libraries when static are required
> >   (e.g., on Cray). Please check your configure cmd line and consider using
> >   one of the contrib/platform definitions for your system type.
> >
> > * an inability to create a connection back to mpirun due to a
> >   lack of common network interfaces and/or no route found between
> >   them. Please check network connectivity (including firewalls
> >   and network routing requirements).
> > --
> >
> > The same happens when I use -host master,slave1 instead of --hostfile 
> > hosts. I'm slowly running out if ideas, I've tried anything I could find on 
> > the internet or OMPI FAQ, nothing seems to work. What am I doing wrong?
> > ___
> > users mailing list
> > us...@open-mpi.org
> > Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users
> > Link to this post: 
> > http://www.open-mpi.org/community/lists/users/2016/05/29068.php
> 
> 
> --
> Jeff Squyres
> jsquy...@cisco.com
> For corporate legal information go to: 
> http://www.cisco.com/web/about/doing_business/legal/cri/
> 
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> 

Re: [OMPI users] Problem with 'orted: command not found'

[Maciek Lewiński]

I already had correct paths in .bashrc:

export
PATH=/usr/lib/lightdm/lightdm:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/bin:/usr/local/bin:/home/$USER/.openmpi/bin

export
LD_LIBRARY_PATH=:/usr/local/lib:/usr/local/lib:/home/$USER/.openmpi/lib

I can run mpi normally from slave1 so I'm sure they work. I also exported
these paths in .profile just to be sure a moment before but it didn't work.
Still the same error.



2016-05-02 23:40 GMT+02:00 Jeff Squyres (jsquyres) :

> The key is this error:
>
> bash: orted: command not found
>
> Meaning: you need to set your PATH and LD_LIBRARY_PATH properly for
> non-interactive logins.  See
> https://www.open-mpi.org/faq/?category=running#adding-ompi-to-path.
>
>
> > On May 2, 2016, at 5:36 PM, Maciek Lewiński 
> wrote:
> >
> > Hi, I'm having problem with Open MPI version 1.10.2.
> > I've installed two virtual machines on VirtualBox, both are the same
> images of Ubuntu 12.04 64bit.
> > Both have the same accounts, both have everything configured almost
> exactly the same.
> > I have configured OMPI only with the --prefix to specify my location of
> install folder which is /home/$USER/.openmpi.
> > Users on both machines are identical.
> >
> > On both machines running mpirun on one of the example programs work
> perfectly, on both machines I've added bin and lib to corresponding PATHs,
> env command ran on master and through ssh on slave1 gives these results:
> > osboxes@osboxes:~/cloud$ env | grep PATH
> > LD_LIBRARY_PATH=:/usr/local/lib:/usr/local/lib:/home/osboxes/.openmpi/lib
> >
> PATH=/usr/lib/lightdm/lightdm:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/bin:/usr/local/bin:/home/osboxes/.openmpi/bin
> >
> > On both hosts I have installed ssh which works, I've enabled
> passwordless connection, which again, works,
> >
> > I have created an NFS folder in which I hold the hello_c program from
> examples and hosts file that looks like this:
> > 192.168.0.191master
> > 192.168.0.190slave1
> >
> > The same two lines are placed in /etc/hosts for ssh to work.
> >
> > On both machines, running:
> > osboxes@osboxes:~/cloud$ mpirun -np 1 ./hello_c
> > Hello, world, I am 0 of 1, (Open MPI v1.10.2, package: Open MPI
> osboxes@osboxes Distribution, ident: 1.10.2, repo rev:
> v1.10.1-145-g799148f, Jan 21, 2016, 126)
> >
> > Works, even running this command on slave1 through ssh from master works
> as expected.
> >
> > Yet when I try to execute the following command I get the error:
> > osboxes@osboxes:~/cloud$ mpirun -np 2 --hostfile hosts ./hello_c
> > bash: orted: command not found
> >
> --
> > ORTE was unable to reliably start one or more daemons.
> > This usually is caused by:
> >
> > * not finding the required libraries and/or binaries on
> >   one or more nodes. Please check your PATH and LD_LIBRARY_PATH
> >   settings, or configure OMPI with --enable-orterun-prefix-by-default
> >
> > * lack of authority to execute on one or more specified nodes.
> >   Please verify your allocation and authorities.
> >
> > * the inability to write startup files into /tmp
> (--tmpdir/orte_tmpdir_base).
> >   Please check with your sys admin to determine the correct location to
> use.
> >
> > *  compilation of the orted with dynamic libraries when static are
> required
> >   (e.g., on Cray). Please check your configure cmd line and consider
> using
> >   one of the contrib/platform definitions for your system type.
> >
> > * an inability to create a connection back to mpirun due to a
> >   lack of common network interfaces and/or no route found between
> >   them. Please check network connectivity (including firewalls
> >   and network routing requirements).
> >
> --
> >
> > The same happens when I use -host master,slave1 instead of --hostfile
> hosts. I'm slowly running out if ideas, I've tried anything I could find on
> the internet or OMPI FAQ, nothing seems to work. What am I doing wrong?
> > ___
> > users mailing list
> > us...@open-mpi.org
> > Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users
> > Link to this post:
> http://www.open-mpi.org/community/lists/users/2016/05/29068.php
>
>
> --
> Jeff Squyres
> jsquy...@cisco.com
> For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/
>
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2016/05/29069.php

Re: [OMPI users] Problem with 'orted: command not found'

[Jeff Squyres (jsquyres)]

The key is this error:

bash: orted: command not found

Meaning: you need to set your PATH and LD_LIBRARY_PATH properly for 
non-interactive logins.  See 
https://www.open-mpi.org/faq/?category=running#adding-ompi-to-path.


> On May 2, 2016, at 5:36 PM, Maciek Lewiński  wrote:
> 
> Hi, I'm having problem with Open MPI version 1.10.2.
> I've installed two virtual machines on VirtualBox, both are the same images 
> of Ubuntu 12.04 64bit.
> Both have the same accounts, both have everything configured almost exactly 
> the same.
> I have configured OMPI only with the --prefix to specify my location of 
> install folder which is /home/$USER/.openmpi.
> Users on both machines are identical.
> 
> On both machines running mpirun on one of the example programs work 
> perfectly, on both machines I've added bin and lib to corresponding PATHs, 
> env command ran on master and through ssh on slave1 gives these results:
> osboxes@osboxes:~/cloud$ env | grep PATH
> LD_LIBRARY_PATH=:/usr/local/lib:/usr/local/lib:/home/osboxes/.openmpi/lib
> PATH=/usr/lib/lightdm/lightdm:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/bin:/usr/local/bin:/home/osboxes/.openmpi/bin
> 
> On both hosts I have installed ssh which works, I've enabled passwordless 
> connection, which again, works,
> 
> I have created an NFS folder in which I hold the hello_c program from 
> examples and hosts file that looks like this:
> 192.168.0.191master
> 192.168.0.190slave1
> 
> The same two lines are placed in /etc/hosts for ssh to work.
> 
> On both machines, running:
> osboxes@osboxes:~/cloud$ mpirun -np 1 ./hello_c
> Hello, world, I am 0 of 1, (Open MPI v1.10.2, package: Open MPI 
> osboxes@osboxes Distribution, ident: 1.10.2, repo rev: v1.10.1-145-g799148f, 
> Jan 21, 2016, 126)
> 
> Works, even running this command on slave1 through ssh from master works as 
> expected.
> 
> Yet when I try to execute the following command I get the error: 
> osboxes@osboxes:~/cloud$ mpirun -np 2 --hostfile hosts ./hello_c
> bash: orted: command not found
> --
> ORTE was unable to reliably start one or more daemons.
> This usually is caused by:
> 
> * not finding the required libraries and/or binaries on
>   one or more nodes. Please check your PATH and LD_LIBRARY_PATH
>   settings, or configure OMPI with --enable-orterun-prefix-by-default
> 
> * lack of authority to execute on one or more specified nodes.
>   Please verify your allocation and authorities.
> 
> * the inability to write startup files into /tmp (--tmpdir/orte_tmpdir_base).
>   Please check with your sys admin to determine the correct location to use.
> 
> *  compilation of the orted with dynamic libraries when static are required
>   (e.g., on Cray). Please check your configure cmd line and consider using
>   one of the contrib/platform definitions for your system type.
> 
> * an inability to create a connection back to mpirun due to a
>   lack of common network interfaces and/or no route found between
>   them. Please check network connectivity (including firewalls
>   and network routing requirements).
> --
> 
> The same happens when I use -host master,slave1 instead of --hostfile hosts. 
> I'm slowly running out if ideas, I've tried anything I could find on the 
> internet or OMPI FAQ, nothing seems to work. What am I doing wrong?
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2016/05/29068.php


-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to: 
http://www.cisco.com/web/about/doing_business/legal/cri/

[OMPI users] Problem with 'orted: command not found'

[Maciek Lewiński]

Hi, I'm having problem with Open MPI version 1.10.2.
I've installed two virtual machines on VirtualBox, both are the same images
of Ubuntu 12.04 64bit.
Both have the same accounts, both have everything configured almost exactly
the same.
I have configured OMPI only with the --prefix to specify my location of
install folder which is /home/$USER/.openmpi.
Users on both machines are identical.

On both machines running mpirun on one of the example programs work
perfectly, on both machines I've added bin and lib to corresponding PATHs,
env command ran on master and through ssh on slave1 gives these results:
osboxes@osboxes:~/cloud$ env | grep PATH
LD_LIBRARY_PATH=:/usr/local/lib:/usr/local/lib:/home/osboxes/.openmpi/lib
PATH=/usr/lib/lightdm/lightdm:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/bin:/usr/local/bin:/home/osboxes/.openmpi/bin

On both hosts I have installed ssh which works, I've enabled passwordless
connection, which again, works,

I have created an NFS folder in which I hold the hello_c program from
examples and hosts file that looks like this:
192.168.0.191master
192.168.0.190slave1

The same two lines are placed in /etc/hosts for ssh to work.

On both machines, running:
osboxes@osboxes:~/cloud$ mpirun -np 1 ./hello_c
Hello, world, I am 0 of 1, (Open MPI v1.10.2, package: Open MPI
osboxes@osboxes Distribution, ident: 1.10.2, repo rev:
v1.10.1-145-g799148f, Jan 21, 2016, 126)

Works, even running this command on slave1 through ssh from master works as
expected.

Yet when I try to execute the following command I get the error:
osboxes@osboxes:~/cloud$ mpirun -np 2 --hostfile hosts ./hello_c
bash: orted: command not found
--
ORTE was unable to reliably start one or more daemons.
This usually is caused by:

* not finding the required libraries and/or binaries on
  one or more nodes. Please check your PATH and LD_LIBRARY_PATH
  settings, or configure OMPI with --enable-orterun-prefix-by-default

* lack of authority to execute on one or more specified nodes.
  Please verify your allocation and authorities.

* the inability to write startup files into /tmp
(--tmpdir/orte_tmpdir_base).
  Please check with your sys admin to determine the correct location to use.

*  compilation of the orted with dynamic libraries when static are required
  (e.g., on Cray). Please check your configure cmd line and consider using
  one of the contrib/platform definitions for your system type.

* an inability to create a connection back to mpirun due to a
  lack of common network interfaces and/or no route found between
  them. Please check network connectivity (including firewalls
  and network routing requirements).
--

The same happens when I use -host master,slave1 instead of --hostfile
hosts. I'm slowly running out if ideas, I've tried anything I could find on
the internet or OMPI FAQ, nothing seems to work. What am I doing wrong?