Re: [OMPI users] a question about [MPI]IO on systems without network filesystem

[Terry Frankcombe]

Hi Paul

I think you should clarify whether you mean you want you application to
send all it's data back to a particular rank, which then does all IO (in
which case the answer is any MPI implementation can do this... it's a
matter of how you code the app), or if you want the application to know
nothing about it, but have the system somehow intercept all IO and make
it magically appear at a particular node (much harder).


On Wed, 2010-09-29 at 11:34 +0200, Paul Kapinos wrote:
> Dear OpenMPI developer,
> 
> We have a question about the possibility to use MPI IO (and possible 
> regular I/O) on clusters which does *not* have a common filesystem 
> (network filesystem) on all nodes.
> 
> A common filesystem is mainly NOT a hard precondition to use OpenMPI:
> http://www.open-mpi.org/faq/?category=running#do-i-need-a-common-filesystem
> 
> 
> Say, we have a (diskless? equipped with very small disks?) cluster, on 
> which only one node have access to a filesystem.
> 
> Is it possible to configure/run OpenMPI in a such way, that only _one_ 
> process (e.g. master) performs real disk I/O, and other processes sends 
> the data to the master which works as an agent?
> 
> Of course this would impacts the performance, because all data must be 
> send over network, and the master may became a bottleneck. But is such 
> scenario - IO of all processes bundled to one  process - practicable at all?
> 
> 
> Best wishes
> Paul
> 
> 
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

[OMPI users] how to tell if opempi is using rsh or ssh

[Borenstein, Bernard S]

We are eliminating the use of rsh at our company and I'm trying to test out 
openmpi with the Nasa Overflow program using ssh.
I've been testing other MPI's (MPICH1 and LAM/MPI) and if I tried to use rsh 
the programs would just die when running
using PBS.  I submitted my Overflow job using  --mca plm_rsh_agent rsh and was 
surprised to see the job run.
Does this setting fail over to ssh if rsh is not available or should it just 
use rsh only???  Also is there any command
(this is a linux cluster) to see if ssh is being used.  I'm trying to 
definitely make sure that it is using ssh and not rsh.
rsh has been turned off on these nodes, so I'm pretty sure it's using ssh, but 
want to make sure.

Thanx in advance for any help that can be provided.

Bernie Borenstein
The Boeing Company

Re: [OMPI users] [openib] segfault when using openib btl

[Terry Dontje]

In some of the testing Eloi did earlier he did disabled eager rdma and 
still saw the issue.


--td

Shamis, Pavel wrote:

Terry,
Ishai Rabinovitz is HPC team manager (I added him to CC)

Eloi,

Back to issue. I have seen very similar issue long time ago on some hardware 
platforms that support relaxed ordering memory operations. If I remember 
correct it was some IBM platform.
Do you know if relaxed memory ordering is enabled on your platform ? If it is 
enabled you have to disable eager rdma.

Regards,
Pasha

On Sep 29, 2010, at 1:04 PM, Terry Dontje wrote:

Pasha, do you by any chance know who at Mellanox might be responsible for OMPI 
working?

--td

Eloi Gaudry wrote:
 Hi Nysal, Terry,
Thanks for your input on this issue.
I'll follow your advice. Do you know any Mellanox developer I may discuss with, 
preferably someone who has spent some time inside the openib btl ?

Regards,
Eloi

On 29/09/2010 06:01, Nysal Jan wrote:
Hi Eloi,
We discussed this issue during the weekly developer meeting & there were no 
further suggestions, apart from checking the driver and firmware levels. The 
consensus was that it would be better if you could take this up directly with your 
IB vendor.

Regards
--Nysal
___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users


--

Terry D. Dontje | Principal Software Engineer
Developer Tools Engineering | +1.781.442.2631
Oracle - Performance Technologies
95 Network Drive, Burlington, MA 01803
Email terry.don...@oracle.com




___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
  



--
Oracle
Terry D. Dontje | Principal Software Engineer
Developer Tools Engineering | +1.781.442.2631
Oracle * - Performance Technologies*
95 Network Drive, Burlington, MA 01803
Email terry.don...@oracle.com 

Re: [OMPI users] [openib] segfault when using openib btl

[Eloi Gaudry]

 Pasha,
Thanks for your help.

I'm not aware of such memory configuration on the new cluster of our 
customer (each computing node is running the Red-Hat 5.x operating 
system on Intel X5570 processors).
Anyway, I've already tried to deactivate eager_rdma, but this wouldn't 
solve the hdr->tag=0 issue (in 
share/openmpi/mca-btl-openib-device-params.ini, eager_rdma is on 
[vendor_part_id=26428]).


Ishai,
If you need anymore information, please feel free to ask.

Regards,
Eloi

On 29/09/2010 19:49, Shamis, Pavel wrote:

Terry,
Ishai Rabinovitz is HPC team manager (I added him to CC)

Eloi,

Back to issue. I have seen very similar issue long time ago on some hardware 
platforms that support relaxed ordering memory operations. If I remember 
correct it was some IBM platform.
Do you know if relaxed memory ordering is enabled on your platform ? If it is 
enabled you have to disable eager rdma.

Regards,
Pasha

On Sep 29, 2010, at 1:04 PM, Terry Dontje wrote:

Pasha, do you by any chance know who at Mellanox might be responsible for OMPI 
working?

--td

Eloi Gaudry wrote:
  Hi Nysal, Terry,
Thanks for your input on this issue.
I'll follow your advice. Do you know any Mellanox developer I may discuss with, 
preferably someone who has spent some time inside the openib btl ?

Regards,
Eloi

On 29/09/2010 06:01, Nysal Jan wrote:
Hi Eloi,
We discussed this issue during the weekly developer meeting&  there were no 
further suggestions, apart from checking the driver and firmware levels. The 
consensus was that it would be better if you could take this up directly with your 
IB vendor.

Regards
--Nysal
___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users


--

Terry D. Dontje | Principal Software Engineer
Developer Tools Engineering | +1.781.442.2631
Oracle - Performance Technologies
95 Network Drive, Burlington, MA 01803
Email terry.don...@oracle.com




___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] a question about [MPI]IO on systems without network filesystem

[jody]

Hi Paul

> Is it possible to configure/run OpenMPI in a such way, that only _one_
> process (e.g. master) performs real disk I/O, and other processes sends the
> data to the master which works as an agent?

It is possible to run OpenMPI this way, but it is not a matter of configuration,
but of implementation alone.

> Of course this would impacts the performance, because all data must be send
> over network, and the master may became a bottleneck. But is such scenario -
> IO of all processes bundled to one  process - practicable at all?

I think this question can only be answered by trying, because it
depends strongly
on the volume of your messages and the quality of your hardware
(network and disk speed)

Jody

Re: [OMPI users] [openib] segfault when using openib btl

[Shamis, Pavel]

Terry,
Ishai Rabinovitz is HPC team manager (I added him to CC)

Eloi,

Back to issue. I have seen very similar issue long time ago on some hardware 
platforms that support relaxed ordering memory operations. If I remember 
correct it was some IBM platform.
Do you know if relaxed memory ordering is enabled on your platform ? If it is 
enabled you have to disable eager rdma.

Regards,
Pasha

On Sep 29, 2010, at 1:04 PM, Terry Dontje wrote:

Pasha, do you by any chance know who at Mellanox might be responsible for OMPI 
working?

--td

Eloi Gaudry wrote:
 Hi Nysal, Terry,
Thanks for your input on this issue.
I'll follow your advice. Do you know any Mellanox developer I may discuss with, 
preferably someone who has spent some time inside the openib btl ?

Regards,
Eloi

On 29/09/2010 06:01, Nysal Jan wrote:
Hi Eloi,
We discussed this issue during the weekly developer meeting & there were no 
further suggestions, apart from checking the driver and firmware levels. The 
consensus was that it would be better if you could take this up directly with 
your IB vendor.

Regards
--Nysal
___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users


--

Terry D. Dontje | Principal Software Engineer
Developer Tools Engineering | +1.781.442.2631
Oracle - Performance Technologies
95 Network Drive, Burlington, MA 01803
Email terry.don...@oracle.com

Re: [OMPI users] [openib] segfault when using openib btl

[Terry Dontje]

Pasha, do you by any chance know who at Mellanox might be responsible 
for OMPI working?


--td

Eloi Gaudry wrote:

 Hi Nysal, Terry,
Thanks for your input on this issue.
I'll follow your advice. Do you know any Mellanox developer I may 
discuss with, preferably someone who has spent some time inside the 
openib btl ?


Regards,
Eloi

On 29/09/2010 06:01, Nysal Jan wrote:

Hi Eloi,
We discussed this issue during the weekly developer meeting & there 
were no further suggestions, apart from checking the driver and 
firmware levels. The consensus was that it would be better if you 
could take this up directly with your IB vendor.


Regards
--Nysal

___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users



--
Oracle
Terry D. Dontje | Principal Software Engineer
Developer Tools Engineering | +1.781.442.2631
Oracle * - Performance Technologies*
95 Network Drive, Burlington, MA 01803
Email terry.don...@oracle.com 

Re: [OMPI users] [openib] segfault when using openib btl

[Eloi Gaudry]

 Hi Nysal, Terry,
Thanks for your input on this issue.
I'll follow your advice. Do you know any Mellanox developer I may 
discuss with, preferably someone who has spent some time inside the 
openib btl ?


Regards,
Eloi

On 29/09/2010 06:01, Nysal Jan wrote:

Hi Eloi,
We discussed this issue during the weekly developer meeting & there 
were no further suggestions, apart from checking the driver and 
firmware levels. The consensus was that it would be better if you 
could take this up directly with your IB vendor.


Regards
--Nysal

[OMPI users] a question about [MPI]IO on systems without network filesystem

[Paul Kapinos]

Dear OpenMPI developer,

We have a question about the possibility to use MPI IO (and possible 
regular I/O) on clusters which does *not* have a common filesystem 
(network filesystem) on all nodes.


A common filesystem is mainly NOT a hard precondition to use OpenMPI:
http://www.open-mpi.org/faq/?category=running#do-i-need-a-common-filesystem


Say, we have a (diskless? equipped with very small disks?) cluster, on 
which only one node have access to a filesystem.


Is it possible to configure/run OpenMPI in a such way, that only _one_ 
process (e.g. master) performs real disk I/O, and other processes sends 
the data to the master which works as an agent?


Of course this would impacts the performance, because all data must be 
send over network, and the master may became a bottleneck. But is such 
scenario - IO of all processes bundled to one  process - practicable at all?



Best wishes
Paul



--
Dipl.-Inform. Paul Kapinos   -   High Performance Computing,
RWTH Aachen University, Center for Computing and Communication
Seffenter Weg 23,  D 52074  Aachen (Germany)
Tel: +49 241/80-24915


smime.p7s
Description: S/MIME Cryptographic Signature

Re: [OMPI users] [openib] segfault when using openib btl

[Nysal Jan]

Hi Eloi,
We discussed this issue during the weekly developer meeting & there were no
further suggestions, apart from checking the driver and firmware levels. The
consensus was that it would be better if you could take this up directly
with your IB vendor.

Regards
--Nysal

On Mon, Sep 27, 2010 at 8:14 PM, Eloi Gaudry  wrote:

> Terry,
>
> Please find enclosed the requested check outputs (using -output-filename
> stdout.tag.null option).
> I'm displaying frag->hdr->tag here.
>
> Eloi
>
> On Monday 27 September 2010 16:29:12 Terry Dontje wrote:
> > Eloi, sorry can you print out frag->hdr->tag?
> >
> > Unfortunately from your last email I think it will still all have
> > non-zero values.
> > If that ends up being the case then there must be something odd with the
> > descriptor pointer to the fragment.
> >
> > --td
> >
> > Eloi Gaudry wrote:
> > > Terry,
> > >
> > > Please find enclosed the requested check outputs (using
> -output-filename
> > > stdout.tag.null option).
> > >
> > > For information, Nysal In his first message referred to
> > > ompi/mca/pml/ob1/pml_ob1_hdr.h and said that hdr->tg value was wrnong
> on
> > > receiving side. #define MCA_PML_OB1_HDR_TYPE_MATCH (MCA_BTL_TAG_PML
> > > + 1)
> > > #define MCA_PML_OB1_HDR_TYPE_RNDV  (MCA_BTL_TAG_PML + 2)
> > > #define MCA_PML_OB1_HDR_TYPE_RGET  (MCA_BTL_TAG_PML + 3)
> > >
> > >  #define MCA_PML_OB1_HDR_TYPE_ACK   (MCA_BTL_TAG_PML + 4)
> > >
> > > #define MCA_PML_OB1_HDR_TYPE_NACK  (MCA_BTL_TAG_PML + 5)
> > > #define MCA_PML_OB1_HDR_TYPE_FRAG  (MCA_BTL_TAG_PML + 6)
> > > #define MCA_PML_OB1_HDR_TYPE_GET   (MCA_BTL_TAG_PML + 7)
> > >
> > >  #define MCA_PML_OB1_HDR_TYPE_PUT   (MCA_BTL_TAG_PML + 8)
> > >
> > > #define MCA_PML_OB1_HDR_TYPE_FIN   (MCA_BTL_TAG_PML + 9)
> > > and in ompi/mca/btl/btl.h
> > > #define MCA_BTL_TAG_PML 0x40
> > >
> > > Eloi
> > >
> > > On Monday 27 September 2010 14:36:59 Terry Dontje wrote:
> > >> I am thinking checking the value of *frag->hdr right before the return
> > >> in the post_send function in
> ompi/mca/btl/openib/btl_openib_endpoint.h.
> > >> It is line 548 in the trunk
> > >>
> https://svn.open-mpi.org/source/xref/ompi-trunk/ompi/mca/btl/openib/btl_
> > >> ope nib_endpoint.h#548
> > >>
> > >> --td
> > >>
> > >> Eloi Gaudry wrote:
> > >>> Hi Terry,
> > >>>
> > >>> Do you have any patch that I could apply to be able to do so ? I'm
> > >>> remotely working on a cluster (with a terminal) and I cannot use any
> > >>> parallel debugger or sequential debugger (with a call to xterm...). I
> > >>> can track frag->hdr->tag value in
> > >>> ompi/mca/btl/openib/btl_openib_component.c::handle_wc in the
> > >>> SEND/RDMA_WRITE case, but this is all I can think of alone.
> > >>>
> > >>> You'll find a stacktrace (receive side) in this thread (10th or 11th
> > >>> message) but it might be pointless.
> > >>>
> > >>> Regards,
> > >>> Eloi
> > >>>
> > >>> On Monday 27 September 2010 11:43:55 Terry Dontje wrote:
> >  So it sounds like coalescing is not your issue and that the problem
> >  has something to do with the queue sizes.  It would be helpful if we
> >  could detect the hdr->tag == 0 issue on the sending side and get at
> >  least a stack trace.  There is something really odd going on here.
> > 
> >  --td
> > 
> >  Eloi Gaudry wrote:
> > > Hi Terry,
> > >
> > > I'm sorry to say that I might have missed a point here.
> > >
> > > I've lately been relaunching all previously failing computations
> with
> > > the message coalescing feature being switched off, and I saw the
> same
> > > hdr->tag=0 error several times, always during a collective call
> > > (MPI_Comm_create, MPI_Allreduce and MPI_Broadcast, so far). And as
> > > soon as I switched to the peer queue option I was previously using
> > > (--mca btl_openib_receive_queues P,65536,256,192,128 instead of
> using
> > > --mca btl_openib_use_message_coalescing 0), all computations ran
> > > flawlessly.
> > >
> > > As for the reproducer, I've already tried to write something but I
> > > haven't succeeded so far at reproducing the hdr->tag=0 issue with
> it.
> > >
> > > Eloi
> > >
> > > On 24/09/2010 18:37, Terry Dontje wrote:
> > >> Eloi Gaudry wrote:
> > >>> Terry,
> > >>>
> > >>> You were right, the error indeed seems to come from the message
> > >>> coalescing feature. If I turn it off using the "--mca
> > >>> btl_openib_use_message_coalescing 0", I'm not able to observe the
> > >>> "hdr->tag=0" error.
> > >>>
> > >>> There are some trac requests associated to very similar error
> > >>> (https://svn.open-mpi.org/trac/ompi/search?q=coalescing) but
> they
> > >>> are all closed (except
> > >>> https://svn.open-mpi.org/trac/ompi/ticket/2352 that might be
> > >>> related), aren't they ? What would you suggest Terry ?
> > >>
> > >> Interesting, though it looks