Re: [OMPI users] latest Intel CPU bug

[Matthieu Brucher]

Hi,

I think, on the contrary, that he did notice the AMD/ARM issue. I suppose
you haven't read the text (and I like the fact that there are different
opinions on this issue).

Matthieu

2018-01-05 8:23 GMT+01:00 Gilles Gouaillardet :

> John,
>
>
> The technical assessment so to speak is linked in the article and is
> available at https://googleprojectzero.blogspot.jp/2018/01/reading-privil
> eged-memory-with-side.html.
>
> The long rant against Intel PR blinded you and you did not notice AMD and
> ARM (and though not mentionned here, Power and Sparc too) are vulnerable to
> some bugs.
>
>
> Full disclosure, i have no affiliation with Intel, but i am getting pissed
> with the hysteria around this issue.
>
> Gilles
>
>
> On 1/5/2018 3:54 PM, John Chludzinski wrote:
>
>> That article gives the best technical assessment I've seen of Intel's
>> architecture bug. I noted the discussion's subject and thought I'd add some
>> clarity. Nothing more.
>>
>> For the TL;DR crowd: get an AMD chip in your computer.
>>
>> On Thursday, January 4, 2018, r...@open-mpi.org 
>> > wrote:
>>
>> Yes, please - that was totally inappropriate for this mailing list.
>> Ralph
>>
>>
>> On Jan 4, 2018, at 4:33 PM, Jeff Hammond >> > wrote:
>>>
>>> Can we restrain ourselves to talk about Open-MPI or at least
>>> technical aspects of HPC communication on this list and leave the
>>> stock market tips for Hacker News and Twitter?
>>>
>>> Thanks,
>>>
>>> Jeff
>>>
>>> On Thu, Jan 4, 2018 at 3:53 PM, John
>>> Chludzinski>> >wrote:
>>>
>>> Fromhttps://semiaccurate.com/2018/01/04/kaiser-security-hole
>>> s-will-devastate-intels-marketshare/
>>> >> will-devastate-intels-marketshare/>
>>>
>>>
>>>
>>>   Kaiser security holes will devastate Intel’s marketshare
>>>
>>>
>>>   Analysis: This one tips the balance toward AMD in a big way
>>>
>>>
>>>   Jan 4, 2018 by Charlie Demerjian
>>>   
>>>
>>>
>>>
>>> This latest decade-long critical security hole in Intel CPUs
>>> is going to cost the company significant market share.
>>> SemiAccurate thinks it is not only consequential but will
>>> shift the balance of power away from Intel CPUs for at least
>>> the next several years.
>>>
>>> Today’s latest crop of gaping security flaws have three sets
>>> of holes across Intel, AMD, and ARM processors along with a
>>> slew of official statements and detailed analyses. On top of
>>> that the statements from vendors range from detailed and
>>> direct to intentionally misleading and slimy. Lets take a
>>> look at what the problems are, who they effect and what the
>>> outcome will be. Those outcomes range from trivial patching
>>> to destroying the market share of Intel servers, and no we
>>> are not joking.
>>>
>>> (*Authors Note 1:* For the technical readers we are
>>> simplifying a lot, sorry we know this hurts. The full
>>> disclosure docs are linked, read them for the details.)
>>>
>>> (*Authors Note 2:* For the financial oriented subscribers out
>>> there, the parts relevant to you are at the very end, the
>>> section is titled *Rubber Meet Road*.)
>>>
>>> *The Problem(s):*
>>>
>>> As we said earlier there are three distinct security flaws
>>> that all fall somewhat under the same umbrella. All are ‘new’
>>> in the sense that the class of attacks hasn’t been publicly
>>> described before, and all are very obscure CPU speculative
>>> execution and timing related problems. The extent the fixes
>>> affect differing architectures also ranges from minor to
>>> near-crippling slowdowns. Worse yet is that all three flaws
>>> aren’t bugs or errors, they exploit correct CPU behavior to
>>> allow the systems to be hacked.
>>>
>>> The three problems are cleverly labeled Variant One, Variant
>>> Two, and Variant Three. Google Project Zero was the original
>>> discoverer of them and has labeled the classes as Bounds
>>> Bypass Check, Branch Target Injection, and Rogue Data Cache
>>> Load respectively. You can read up on the extensive and gory
>>> details here
>>> >> ileged-memory-with-side.html> if
>>>
>>> you wish.
>>>
>>> If you are the TLDR type the very simplified summary is that
>>> modern CPUs will speculatively execute operations ahead of
>>> the one they are currently 

Re: [OMPI users] OMPI users] Still "illegal instruction"

[Matthieu Brucher]

I don't think there is anything OpenMPI can do for you here. The issue is
clearly on how you are compiling your application.
To start, you can try to compile without the --march=generic and use
something as generic as possible (i.e. only SSE2). Then if this doesn't
work for your app, do the same for any 3rd party library.

Cheers,

2016-09-15 19:01 GMT+01:00 Mahmood Naderan :

> Excuse me, which is most suitable for me to find the name of the illegal
> instruction?
>
> --verbose
> --debug-level
> --debug-daemons
> --debug-daemons-file
>
>
> Regards,
> Mahmood
>
>
> ___
> users mailing list
> users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users
>



-- 
Information System Engineer, Ph.D.
Blog: http://blog.audio-tk.com/
LinkedIn: http://www.linkedin.com/in/matthieubrucher
___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] Fwd: Can I just use non-blocking send/receive without calling MPI_Wait ever

[Matthieu Brucher]

If you don't need to know if the data was transferred or not, then why do
you transfer it in the first place? The schema seems kind of strange, as
you don't have any clue that the data was actually transferred. Actually
without Wait and Test, you can pretty much assume you don't transfer
anything.

Cheers,

Matthieu

2015-04-03 22:51 GMT+01:00 Lei Shi :

> Hi Jeff,
>
> Thanks for your reminder. I don't need to make sure the data is correct or
> not. I know it sounds crazy at first time, but there are some numerical
> schemes designed for this situation. I just want to call
> MPI_ISend/MPI_IRecv without calling waiting or testing but can still run my
> program smoothly.
>
> Sincerely Yours,
>
> Lei Shi
> -
>
> On Fri, Apr 3, 2015 at 11:52 AM, Jeff Squyres (jsquyres) <
> jsquy...@cisco.com> wrote:
>
>> On Apr 3, 2015, at 12:50 PM, Lei Shi  wrote:
>> >
>> > P.S.  Pavan suggests me to use MPI_Request_free. I will give it a try.
>>
>> Keep in mind that you have zero indication of when a send or receive
>> completes if you MPI_Request_free (Pavan implied this, too).  You could be
>> reading half a message from a prior receive, for example.
>>
>> --
>> Jeff Squyres
>> jsquy...@cisco.com
>> For corporate legal information go to:
>> http://www.cisco.com/web/about/doing_business/legal/cri/
>>
>>
>
>
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2015/04/26605.php
>



-- 
Information System Engineer, Ph.D.
Blog: http://matt.eifelle.com
LinkedIn: http://www.linkedin.com/in/matthieubrucher
Music band: http://liliejay.com/

Re: [OMPI users] MPI_Isend with no recieve

[Matthieu Brucher]

The problem is that for the moment, the implementation uses
Isend/Irecv, but you don't know what will happen in the future
(hopefully, it will use something else).
If your program bypasses the required call to MPI_Iscatterv, then you
only have one option: implement MPI_Iscatterv yourself, with only the
required parts. It's not that complicated, it can be done with an
empty array of requests, and then for each required Isend, increment
the request counter, same for the Irecv, and wait for all at the end.

2014-07-16 15:53 GMT+02:00 Ziv Aginsky <zivagin...@gmail.com>:
> Thanks a lot.
> You are right I am using MPI_Iscatterv, in a domain decomposition code, but
> the problem is that for the domain which I have no data to send fro, the
> program will jump the routine. I can not redesign the whole program.
> Do you know what will happen to send call with zero size buffer? Can I just
> set the request to MPI_SUCCESS for ranks which I will send zero buffer to
> and have no receive call?
> Is there any other MPI routine that can do MPI_Scatterv on selected ranks?
> without creating a new communicator?
>
>
>
>
> On Wed, Jul 16, 2014 at 3:42 PM, Matthieu Brucher
> <matthieu.bruc...@gmail.com> wrote:
>>
>> If you are using Iscatterv (I guess it is that one), it handles the
>> pairs itself. You shouldn't bypass it because you think it is better.
>> You don't know how it is implemented, so just call Iscatterv for all
>> ranks.
>>
>> 2014-07-16 14:33 GMT+01:00 Ziv Aginsky <zivagin...@gmail.com>:
>> > I know the standard, but what if I can not bypass the send message. For
>> > example if I have MPI_Iscatter and for some ranks the send buffer has
>> > zero
>> > size. At those ranks it will jump the MPI_Iscatter routine, which means
>> > I
>> > have some zero size send and no receive.
>> >
>> >
>> >
>> >
>> > On Wed, Jul 16, 2014 at 3:28 PM, Matthieu Brucher
>> > <matthieu.bruc...@gmail.com> wrote:
>> >>
>> >> Hi,
>> >>
>> >> The easiest would also to bypass the Isend as well! The standard is
>> >> clear, you need a pair of Isend/Irecv.
>> >>
>> >> Cheers,
>> >>
>> >> 2014-07-16 14:27 GMT+01:00 Ziv Aginsky <zivagin...@gmail.com>:
>> >> > I have a loop in which I will do some MPI_Isend. According to the MPI
>> >> > standard, for every send you need a recv
>> >> >
>> >> > If one or several of my MPI_Isend have zero size buffer, should I
>> >> > still
>> >> > have
>> >> > the mpi_recv or I can do it without recv? I mean on the processor
>> >> > which
>> >> > I
>> >> > should recv the data I know priory that my buffer is with zero size.
>> >> > Can
>> >> > I
>> >> > jump from MPI_Recv.
>> >> >
>> >> > The question is because of the format of the program I am using if it
>> >> > knows
>> >> > that the receiving buffer is zero it will not call the routine which
>> >> > contains mpi_Recv.
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > ___
>> >> > users mailing list
>> >> > us...@open-mpi.org
>> >> > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> >> > Link to this post:
>> >> > http://www.open-mpi.org/community/lists/users/2014/07/24781.php
>> >>
>> >>
>> >>
>> >> --
>> >> Information System Engineer, Ph.D.
>> >> Blog: http://matt.eifelle.com
>> >> LinkedIn: http://www.linkedin.com/in/matthieubrucher
>> >> Music band: http://liliejay.com/
>> >> ___
>> >> users mailing list
>> >> us...@open-mpi.org
>> >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> >> Link to this post:
>> >> http://www.open-mpi.org/community/lists/users/2014/07/24782.php
>> >
>> >
>> >
>> > ___
>> > users mailing list
>> > us...@open-mpi.org
>> > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> > Link to this post:
>> > http://www.open-mpi.org/community/lists/users/2014/07/24783.php
>>
>>
>>
>> --
>> Information System Engineer, Ph.D.
>> Blog: http://matt.eifelle.com
>> LinkedIn: http://www.linkedin.com/in/matthieubrucher
>> Music band: http://liliejay.com/
>> ___
>> users mailing list
>> us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post:
>> http://www.open-mpi.org/community/lists/users/2014/07/24784.php
>
>
>
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2014/07/24785.php



-- 
Information System Engineer, Ph.D.
Blog: http://matt.eifelle.com
LinkedIn: http://www.linkedin.com/in/matthieubrucher
Music band: http://liliejay.com/

Re: [OMPI users] MPI_Isend with no recieve

[Matthieu Brucher]

If you are using Iscatterv (I guess it is that one), it handles the
pairs itself. You shouldn't bypass it because you think it is better.
You don't know how it is implemented, so just call Iscatterv for all
ranks.

2014-07-16 14:33 GMT+01:00 Ziv Aginsky <zivagin...@gmail.com>:
> I know the standard, but what if I can not bypass the send message. For
> example if I have MPI_Iscatter and for some ranks the send buffer has zero
> size. At those ranks it will jump the MPI_Iscatter routine, which means I
> have some zero size send and no receive.
>
>
>
>
> On Wed, Jul 16, 2014 at 3:28 PM, Matthieu Brucher
> <matthieu.bruc...@gmail.com> wrote:
>>
>> Hi,
>>
>> The easiest would also to bypass the Isend as well! The standard is
>> clear, you need a pair of Isend/Irecv.
>>
>> Cheers,
>>
>> 2014-07-16 14:27 GMT+01:00 Ziv Aginsky <zivagin...@gmail.com>:
>> > I have a loop in which I will do some MPI_Isend. According to the MPI
>> > standard, for every send you need a recv
>> >
>> > If one or several of my MPI_Isend have zero size buffer, should I still
>> > have
>> > the mpi_recv or I can do it without recv? I mean on the processor which
>> > I
>> > should recv the data I know priory that my buffer is with zero size. Can
>> > I
>> > jump from MPI_Recv.
>> >
>> > The question is because of the format of the program I am using if it
>> > knows
>> > that the receiving buffer is zero it will not call the routine which
>> > contains mpi_Recv.
>> >
>> >
>> >
>> >
>> > ___
>> > users mailing list
>> > us...@open-mpi.org
>> > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> > Link to this post:
>> > http://www.open-mpi.org/community/lists/users/2014/07/24781.php
>>
>>
>>
>> --
>> Information System Engineer, Ph.D.
>> Blog: http://matt.eifelle.com
>> LinkedIn: http://www.linkedin.com/in/matthieubrucher
>> Music band: http://liliejay.com/
>> ___
>> users mailing list
>> us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post:
>> http://www.open-mpi.org/community/lists/users/2014/07/24782.php
>
>
>
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2014/07/24783.php



-- 
Information System Engineer, Ph.D.
Blog: http://matt.eifelle.com
LinkedIn: http://www.linkedin.com/in/matthieubrucher
Music band: http://liliejay.com/

Re: [OMPI users] MPI_Isend with no recieve

[Matthieu Brucher]

Hi,

The easiest would also to bypass the Isend as well! The standard is
clear, you need a pair of Isend/Irecv.

Cheers,

2014-07-16 14:27 GMT+01:00 Ziv Aginsky :
> I have a loop in which I will do some MPI_Isend. According to the MPI
> standard, for every send you need a recv
>
> If one or several of my MPI_Isend have zero size buffer, should I still have
> the mpi_recv or I can do it without recv? I mean on the processor which I
> should recv the data I know priory that my buffer is with zero size. Can I
> jump from MPI_Recv.
>
> The question is because of the format of the program I am using if it knows
> that the receiving buffer is zero it will not call the routine which
> contains mpi_Recv.
>
>
>
>
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2014/07/24781.php



-- 
Information System Engineer, Ph.D.
Blog: http://matt.eifelle.com
LinkedIn: http://www.linkedin.com/in/matthieubrucher
Music band: http://liliejay.com/

Re: [OMPI users] MPI_Alltoall with Vector Datatype

[Matthieu Brucher]

A simple test would be to run it with valgrind, so that out of bound
reads and writes will be obvious.

Cheers,

Matthieu

2014-05-08 21:16 GMT+02:00 Spenser Gilliland :
> George & Mattheiu,
>
>> The Alltoall should only return when all data is sent and received on
>> the current rank, so there shouldn't be any race condition.
>
> Your right this is MPI not pthreads.  That should never happen. Duh!
>
>> I think the issue is with the way you define the send and receive
>> buffer in the MPI_Alltoall. You have to keep in mind that the
>> all-to-all pattern will overwrite the entire data in the receive
>> buffer. Thus, starting from a relative displacement in the data (in
>> this case matrix[wrank*wrows]), begs for troubles, as you will write
>> outside the receive buffer.
>
> The submatrix corresponding to matrix[wrank*wrows][0] to
> matrix[(wrank+1)*wrows-1][:] is valid only on the wrank process.  This
> is a block distribution of the rows like what MPI_Scatter would
> produce.  As wrows is equal to N (matrix width/height) divided by
> wsize, the number of mpi_all_t blocks in each message is equal to
> wsize.  Therefore, there should be no writing outside the bounds of
> the submatrix.
>
> On another note,
> I just ported the example to use dynamic memory and now I'm getting
> segfaults when I call MPI_Finalize().  Any idea what in the code could
> have caused this?
>
> It's paste binned here: https://gist.github.com/anonymous/a80e0679c3cbffb82e39
>
> The result is
>
> [sgillila@jarvis src]$ mpirun -npernode 2 transpose2 8
> N = 8
> Matrix =
>  0: 0 1 2 3 4 5 6 7
>  0: 8 9101112131415
>  0:1617181920212223
>  0:2425262728293031
>  1:3233343536373839
>  1:4041424344454647
>  1:4849505152535455
>  1:5657585960616263
> Matrix =
>  0: 0 8162432404856
>  0: 1 9172533414957
>  0: 210182634425058
>  0: 311192735435159
>  1: 412202836445260
>  1: 513212937455361
>  1: 614223038465462
>  1: 715233139475563
> [jarvis:09314] *** Process received signal ***
> [jarvis:09314] Signal: Segmentation fault (11)
> [jarvis:09314] Signal code: Address not mapped (1)
> [jarvis:09314] Failing at address: 0x21da228
> [jarvis:09314] [ 0] /lib64/libpthread.so.0() [0x371480f500]
> [jarvis:09314] [ 1]
> /opt/openmpi/lib/libmpi.so.1(opal_memory_ptmalloc2_int_free+0x75)
> [0x7f2e85452575]
> [jarvis:09314] [ 2]
> /opt/openmpi/lib/libmpi.so.1(opal_memory_ptmalloc2_free+0xd3)
> [0x7f2e85452bc3]
> [jarvis:09314] [ 3] transpose2(main+0x160) [0x4012a0]
> [jarvis:09314] [ 4] /lib64/libc.so.6(__libc_start_main+0xfd) [0x3713c1ecdd]
> [jarvis:09314] [ 5] transpose2() [0x400d49]
> [jarvis:09314] *** End of error message ***
> --
> mpirun noticed that process rank 1 with PID 9314 on node
> jarvis.cs.iit.edu exited on signal 11 (Segmentation fault).
> --
>
> --
> Spenser Gilliland
> Computer Engineer
> Doctoral Candidate
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users



-- 
Information System Engineer, Ph.D.
Blog: http://matt.eifelle.com
LinkedIn: http://www.linkedin.com/in/matthieubrucher
Music band: http://liliejay.com/

Re: [OMPI users] MPI_Alltoall with Vector Datatype

[Matthieu Brucher]

The Alltoall should only return when all data is sent and received on
the current rank, so there shouldn't be any race condition.

Cheers,

Matthieu

2014-05-08 15:53 GMT+02:00 Spenser Gilliland :
> George & other list members,
>
> I think I may have a race condition in this example that is masked by
> the print_matrix statement.
>
> For example, lets say rank one has a large sleep before reaching the
> local transpose, will the other ranks have completed the Alltoall and
> when rank one reaches the local transpose it is altering the data that
> the other processors sent it?
>
> Thanks,
> Spenser
>
>
> --
> Spenser Gilliland
> Computer Engineer
> Doctoral Candidate
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users



-- 
Information System Engineer, Ph.D.
Blog: http://matt.eifelle.com
LinkedIn: http://www.linkedin.com/in/matthieubrucher
Music band: http://liliejay.com/

Re: [OMPI users] performance of MPI_Iallgatherv

[Matthieu Brucher]

Yes, usually the MPI libraries don't allow that. You can launch
another thread for the computation, make calls to MPI_Test during that
time and join at the end.

Cheers,

2014-04-07 4:12 GMT+01:00 Zehan Cui <zehan@gmail.com>:
> Hi Matthieu,
>
> Thanks for your suggestion. I tried MPI_Waitall(), but the results are
> the same. It seems the communication didn't overlap with computation.
>
> Regards,
> Zehan
>
> On 4/5/14, Matthieu Brucher <matthieu.bruc...@gmail.com> wrote:
>> Hi,
>>
>> Try waiting on all gathers at the same time, not one by one (this is
>> what non blocking collectives are made for!)
>>
>> Cheers,
>>
>> Matthieu
>>
>> 2014-04-05 10:35 GMT+01:00 Zehan Cui <zehan@gmail.com>:
>>> Hi,
>>>
>>> I'm testing the non-blocking collective of OpenMPI-1.8.
>>>
>>> I have two nodes with Infiniband to perform allgather on totally 128MB
>>> data.
>>>
>>> I split the 128MB data into eight pieces, and perform computation and
>>> MPI_Iallgatherv() on one piece of data each iteration, hoping that the
>>> MPI_Iallgatherv() of last iteration can be overlapped with computation of
>>> current iteration. A MPI_Wait() is called at the end of last iteration.
>>>
>>> However, the total communication time (including the final wait time) is
>>> similar with that of the traditional blocking MPI_Allgatherv, even
>>> slightly
>>> higher.
>>>
>>>
>>> Following is the test pseudo-code, the source code are attached.
>>>
>>> ===
>>>
>>> Using MPI_Allgatherv:
>>>
>>> for( i=0; i<8; i++ )
>>> {
>>>   // computation
>>> mytime( t_begin );
>>> computation;
>>> mytime( t_end );
>>> comp_time += (t_end - t_begin);
>>>
>>>   // communication
>>> t_begin = t_end;
>>> MPI_Allgatherv();
>>> mytime( t_end );
>>> comm_time += (t_end - t_begin);
>>> }
>>> 
>>>
>>> Using MPI_Iallgatherv:
>>>
>>> for( i=0; i<8; i++ )
>>> {
>>>   // computation
>>> mytime( t_begin );
>>> computation;
>>> mytime( t_end );
>>> comp_time += (t_end - t_begin);
>>>
>>>   // communication
>>> t_begin = t_end;
>>> MPI_Iallgatherv();
>>> mytime( t_end );
>>> comm_time += (t_end - t_begin);
>>> }
>>>
>>> // wait for non-blocking allgather to complete
>>> mytime( t_begin );
>>> for( i=0; i<8; i++ )
>>> MPI_Wait;
>>> mytime( t_end );
>>> wait_time = t_end - t_begin;
>>>
>>> ==
>>>
>>> The results of Allgatherv is:
>>> [cmy@gnode102 test_nbc]$ /home3/cmy/czh/opt/ompi-1.8/bin/mpirun -n 2
>>> --host
>>> gnode102,gnode103 ./Allgatherv 128 2 | grep time
>>> Computation time  : 8481279 us
>>> Communication time: 319803 us
>>>
>>> The results of Iallgatherv is:
>>> [cmy@gnode102 test_nbc]$ /home3/cmy/czh/opt/ompi-1.8/bin/mpirun -n 2
>>> --host
>>> gnode102,gnode103 ./Iallgatherv 128 2 | grep time
>>> Computation time  : 8479177 us
>>> Communication time: 199046 us
>>> Wait time:  139841 us
>>>
>>>
>>> So, does this mean that current OpenMPI implementation of MPI_Iallgatherv
>>> doesn't support offloading of collective communication to dedicated cores
>>> or
>>> network interface?
>>>
>>> Best regards,
>>> Zehan
>>>
>>>
>>>
>>>
>>>
>>> ___
>>> users mailing list
>>> us...@open-mpi.org
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>>
>>
>> --
>> Information System Engineer, Ph.D.
>> Blog: http://matt.eifelle.com
>> LinkedIn: http://www.linkedin.com/in/matthieubrucher
>> Music band: http://liliejay.com/
>> ___
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>
>
> --
> Best Regards
> Zehan Cui(崔泽汉)
> ---
> Institute of Computing Technology, Chinese Academy of Sciences.
> No.6 Kexueyuan South Road Zhongguancun,Haidian District Beijing,China
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users



-- 
Information System Engineer, Ph.D.
Blog: http://matt.eifelle.com
LinkedIn: http://www.linkedin.com/in/matthieubrucher
Music band: http://liliejay.com/

Re: [OMPI users] performance of MPI_Iallgatherv

[Matthieu Brucher]

Hi,

Try waiting on all gathers at the same time, not one by one (this is
what non blocking collectives are made for!)

Cheers,

Matthieu

2014-04-05 10:35 GMT+01:00 Zehan Cui :
> Hi,
>
> I'm testing the non-blocking collective of OpenMPI-1.8.
>
> I have two nodes with Infiniband to perform allgather on totally 128MB data.
>
> I split the 128MB data into eight pieces, and perform computation and
> MPI_Iallgatherv() on one piece of data each iteration, hoping that the
> MPI_Iallgatherv() of last iteration can be overlapped with computation of
> current iteration. A MPI_Wait() is called at the end of last iteration.
>
> However, the total communication time (including the final wait time) is
> similar with that of the traditional blocking MPI_Allgatherv, even slightly
> higher.
>
>
> Following is the test pseudo-code, the source code are attached.
>
> ===
>
> Using MPI_Allgatherv:
>
> for( i=0; i<8; i++ )
> {
>   // computation
> mytime( t_begin );
> computation;
> mytime( t_end );
> comp_time += (t_end - t_begin);
>
>   // communication
> t_begin = t_end;
> MPI_Allgatherv();
> mytime( t_end );
> comm_time += (t_end - t_begin);
> }
> 
>
> Using MPI_Iallgatherv:
>
> for( i=0; i<8; i++ )
> {
>   // computation
> mytime( t_begin );
> computation;
> mytime( t_end );
> comp_time += (t_end - t_begin);
>
>   // communication
> t_begin = t_end;
> MPI_Iallgatherv();
> mytime( t_end );
> comm_time += (t_end - t_begin);
> }
>
> // wait for non-blocking allgather to complete
> mytime( t_begin );
> for( i=0; i<8; i++ )
> MPI_Wait;
> mytime( t_end );
> wait_time = t_end - t_begin;
>
> ==
>
> The results of Allgatherv is:
> [cmy@gnode102 test_nbc]$ /home3/cmy/czh/opt/ompi-1.8/bin/mpirun -n 2 --host
> gnode102,gnode103 ./Allgatherv 128 2 | grep time
> Computation time  : 8481279 us
> Communication time: 319803 us
>
> The results of Iallgatherv is:
> [cmy@gnode102 test_nbc]$ /home3/cmy/czh/opt/ompi-1.8/bin/mpirun -n 2 --host
> gnode102,gnode103 ./Iallgatherv 128 2 | grep time
> Computation time  : 8479177 us
> Communication time: 199046 us
> Wait time:  139841 us
>
>
> So, does this mean that current OpenMPI implementation of MPI_Iallgatherv
> doesn't support offloading of collective communication to dedicated cores or
> network interface?
>
> Best regards,
> Zehan
>
>
>
>
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users



-- 
Information System Engineer, Ph.D.
Blog: http://matt.eifelle.com
LinkedIn: http://www.linkedin.com/in/matthieubrucher
Music band: http://liliejay.com/

Re: [OMPI users] Segmentation fault in MPI_Init when passing pointers allocated in main()

[Matthieu Brucher]

It seems that argv[argc] should always be NULL according to the
standard. So OMPI failure is not actually a bug!

Cheers,

2013/11/12 Matthieu Brucher <matthieu.bruc...@gmail.com>:
> Interestingly enough, in ompi_mpi_init, opal_argv_join is called
> without then array length, so I suppose that in the usual argc/argv
> couple, you have an additional value to argv which may be NULL. So try
> allocating 3 additional values, the last being NULL, and it may work.
>
> Cheers,
>
> Matthieu
>
> 2013/11/12 Tang, Yu-Hang <yuhang_t...@brown.edu>:
>> I tried the following code without CUDA, the error is still there:
>>
>> #include "mpi.h"
>>
>> #include 
>> #include 
>> #include 
>>
>> int main(int argc, char **argv)
>> {
>> // override command line arguments to make sure cudaengine get the
>> correct one
>> char **argv_new = new char*[ argc + 2 ];
>> for( int i = 0 ; i < argc ; i++ )
>> {
>> argv_new[i] = new char[ strlen( argv[i] ) + 1 ];
>> strcpy( argv_new[i], argv[i] );
>> }
>> argv_new[ argc   ] = new char[ 32 ];
>> argv_new[ argc+1 ] = new char[ 32 ];
>> strcpy( argv_new[argc],   "-device" );
>> sprintf( argv_new[argc+1], "%d", 0 );
>>
>> argc += 2;
>> argv = argv_new;
>>
>> MPI_Init(,);
>>
>> // do something...
>>
>> MPI_Finalize();
>>
>> for( int i = 0 ; i < argc ; i++ ) delete [] argv[i];
>> delete [] argv;
>> }
>>
>> At the end of the program the pointer stored in argv is exactly that of
>> argv_new so this should not be a problem. Manually inserting printf tells me
>> that the fault occured at MPI_Init. The code works fine if I use
>> MPI_Init(NULL,NULL) instead. The same code also compiles and runs without a
>> problem on my laptop with mpich2-1.4.
>>
>> Best,
>> Yu-Hang
>>
>>
>>
>> On Tue, Nov 12, 2013 at 11:18 AM, Matthieu Brucher
>> <matthieu.bruc...@gmail.com> wrote:
>>>
>>> Hi,
>>>
>>> Are you sure this is the correct code? This seems strange and not a good
>>> idea:
>>>
>>>MPI_Init(,);
>>>
>>> // do something...
>>>
>>> for( int i = 0 ; i < argc ; i++ ) delete [] argv[i];
>>> delete [] argv;
>>>
>>> Did you mean argc_new and argv_new instead?
>>> Do you have the same error without CUDA?
>>>
>>> Cheers,
>>>
>>> Matthieu
>>>
>>>
>>> 2013/11/12 Tang, Yu-Hang <yuhang_t...@brown.edu>:
>>> > Hi,
>>> >
>>> > I tried to augment the command line argument list by allocating my own
>>> > list
>>> > of strings and passing them to MPI_Init, yet I got a segmentation fault
>>> > for
>>> > both OpenMPI 1.6.3 and 1.7.2, while the code works fine with MPICH2. The
>>> > code is:
>>> >
>>> > #include "mpi.h"
>>> > #include "cuda_runtime.h"
>>> > #include 
>>> > #include 
>>> > #include 
>>> >
>>> > int main(int argc, char **argv)
>>> > {
>>> > int device = 0;
>>> > int skip = 0;
>>> > bool skipmode = false;
>>> > bool specified = false;
>>> > for( int i = 0 ; i < argc ; i++ )
>>> > {
>>> > if ( strcmp( argv[i], "-device" ) == 0 )
>>> > {
>>> > i++;
>>> > if ( argv[i][0] == '-' )
>>> > {
>>> > skipmode = true;
>>> > skip = fabs( atoi( argv[i] ) );
>>> > }
>>> > else
>>> > {
>>> > skipmode = false;
>>> > device = atoi( argv[i] );
>>> > }
>>> > specified = true;
>>> > }
>>> > }
>>> >
>>> > if ( !specified || skipmode )
>>> > {
>>> > char* var;
>>> > int dev_count, local_rank = 0;
>>> > if ( (var = getenv("SLURM_LOCALID")) != NULL) local_rank =
>>> > atoi(var);
>>> > else if( (var = getenv("MV2_COMM_WORLD_LOCAL_RANK"))  != NULL)
>>> > local_rank = atoi(var);
>>> > 

Re: [OMPI users] Segmentation fault in MPI_Init when passing pointers allocated in main()

[Matthieu Brucher]

Interestingly enough, in ompi_mpi_init, opal_argv_join is called
without then array length, so I suppose that in the usual argc/argv
couple, you have an additional value to argv which may be NULL. So try
allocating 3 additional values, the last being NULL, and it may work.

Cheers,

Matthieu

2013/11/12 Tang, Yu-Hang <yuhang_t...@brown.edu>:
> I tried the following code without CUDA, the error is still there:
>
> #include "mpi.h"
>
> #include 
> #include 
> #include 
>
> int main(int argc, char **argv)
> {
> // override command line arguments to make sure cudaengine get the
> correct one
> char **argv_new = new char*[ argc + 2 ];
> for( int i = 0 ; i < argc ; i++ )
> {
> argv_new[i] = new char[ strlen( argv[i] ) + 1 ];
> strcpy( argv_new[i], argv[i] );
> }
> argv_new[ argc   ] = new char[ 32 ];
> argv_new[ argc+1 ] = new char[ 32 ];
> strcpy( argv_new[argc],   "-device" );
> sprintf( argv_new[argc+1], "%d", 0 );
>
> argc += 2;
> argv = argv_new;
>
> MPI_Init(,);
>
> // do something...
>
> MPI_Finalize();
>
> for( int i = 0 ; i < argc ; i++ ) delete [] argv[i];
> delete [] argv;
> }
>
> At the end of the program the pointer stored in argv is exactly that of
> argv_new so this should not be a problem. Manually inserting printf tells me
> that the fault occured at MPI_Init. The code works fine if I use
> MPI_Init(NULL,NULL) instead. The same code also compiles and runs without a
> problem on my laptop with mpich2-1.4.
>
> Best,
> Yu-Hang
>
>
>
> On Tue, Nov 12, 2013 at 11:18 AM, Matthieu Brucher
> <matthieu.bruc...@gmail.com> wrote:
>>
>> Hi,
>>
>> Are you sure this is the correct code? This seems strange and not a good
>> idea:
>>
>>MPI_Init(,);
>>
>> // do something...
>>
>> for( int i = 0 ; i < argc ; i++ ) delete [] argv[i];
>> delete [] argv;
>>
>> Did you mean argc_new and argv_new instead?
>> Do you have the same error without CUDA?
>>
>> Cheers,
>>
>> Matthieu
>>
>>
>> 2013/11/12 Tang, Yu-Hang <yuhang_t...@brown.edu>:
>> > Hi,
>> >
>> > I tried to augment the command line argument list by allocating my own
>> > list
>> > of strings and passing them to MPI_Init, yet I got a segmentation fault
>> > for
>> > both OpenMPI 1.6.3 and 1.7.2, while the code works fine with MPICH2. The
>> > code is:
>> >
>> > #include "mpi.h"
>> > #include "cuda_runtime.h"
>> > #include 
>> > #include 
>> > #include 
>> >
>> > int main(int argc, char **argv)
>> > {
>> > int device = 0;
>> > int skip = 0;
>> > bool skipmode = false;
>> > bool specified = false;
>> > for( int i = 0 ; i < argc ; i++ )
>> > {
>> > if ( strcmp( argv[i], "-device" ) == 0 )
>> > {
>> > i++;
>> > if ( argv[i][0] == '-' )
>> > {
>> > skipmode = true;
>> > skip = fabs( atoi( argv[i] ) );
>> > }
>> > else
>> > {
>> > skipmode = false;
>> > device = atoi( argv[i] );
>> > }
>> > specified = true;
>> > }
>> > }
>> >
>> > if ( !specified || skipmode )
>> > {
>> > char* var;
>> > int dev_count, local_rank = 0;
>> > if ( (var = getenv("SLURM_LOCALID")) != NULL) local_rank =
>> > atoi(var);
>> > else if( (var = getenv("MV2_COMM_WORLD_LOCAL_RANK"))  != NULL)
>> > local_rank = atoi(var);
>> > else if( (var = getenv("OMPI_COMM_WORLD_LOCAL_RANK")) != NULL)
>> > local_rank = atoi(var);
>> > cudaGetDeviceCount( _count );
>> > if ( skipmode )
>> > {
>> > device = 0;
>> > if ( device == skip ) local_rank++;
>> > while( local_rank-- > 0 )
>> > {
>> > device = (++device) % dev_count;
>> > if ( device == skip ) local_rank++;
>> > }
>> > }
>> > else device = local_rank % dev_count;
>> > }
>> >
>> > // override co

Re: [OMPI users] Segmentation fault in MPI_Init when passing pointers allocated in main()

[Matthieu Brucher]

Hi,

Are you sure this is the correct code? This seems strange and not a good idea:

   MPI_Init(,);

// do something...

for( int i = 0 ; i < argc ; i++ ) delete [] argv[i];
delete [] argv;

Did you mean argc_new and argv_new instead?
Do you have the same error without CUDA?

Cheers,

Matthieu


2013/11/12 Tang, Yu-Hang :
> Hi,
>
> I tried to augment the command line argument list by allocating my own list
> of strings and passing them to MPI_Init, yet I got a segmentation fault for
> both OpenMPI 1.6.3 and 1.7.2, while the code works fine with MPICH2. The
> code is:
>
> #include "mpi.h"
> #include "cuda_runtime.h"
> #include 
> #include 
> #include 
>
> int main(int argc, char **argv)
> {
> int device = 0;
> int skip = 0;
> bool skipmode = false;
> bool specified = false;
> for( int i = 0 ; i < argc ; i++ )
> {
> if ( strcmp( argv[i], "-device" ) == 0 )
> {
> i++;
> if ( argv[i][0] == '-' )
> {
> skipmode = true;
> skip = fabs( atoi( argv[i] ) );
> }
> else
> {
> skipmode = false;
> device = atoi( argv[i] );
> }
> specified = true;
> }
> }
>
> if ( !specified || skipmode )
> {
> char* var;
> int dev_count, local_rank = 0;
> if ( (var = getenv("SLURM_LOCALID")) != NULL) local_rank =
> atoi(var);
> else if( (var = getenv("MV2_COMM_WORLD_LOCAL_RANK"))  != NULL)
> local_rank = atoi(var);
> else if( (var = getenv("OMPI_COMM_WORLD_LOCAL_RANK")) != NULL)
> local_rank = atoi(var);
> cudaGetDeviceCount( _count );
> if ( skipmode )
> {
> device = 0;
> if ( device == skip ) local_rank++;
> while( local_rank-- > 0 )
> {
> device = (++device) % dev_count;
> if ( device == skip ) local_rank++;
> }
> }
> else device = local_rank % dev_count;
> }
>
> // override command line arguments to make sure cudaengine get the
> correct one
> char **argv_new = new char*[ argc + 2 ];
> for( int i = 0 ; i < argc ; i++ )
> {
> argv_new[i] = new char[ strlen( argv[i] ) + 1 ];
> strcpy( argv_new[i], argv[i] );
> }
> argv_new[ argc   ] = new char[ 32 ];
> argv_new[ argc+1 ] = new char[ 32 ];
> strcpy( argv_new[argc],   "-device" );
> sprintf( argv_new[argc+1], "%d", device );
> argc += 2;
> argv = argv_new;
>
> cudaSetDevice( device );
>
> MPI_Init(,);
>
> // do something...
>
> MPI_Finalize();
>
> cudaDeviceReset();
> for( int i = 0 ; i < argc ; i++ ) delete [] argv[i];
> delete [] argv;
> }
>
> When compiled using nvcc -ccbin mpic++, The error I got was:
>
> [jueying:16317] *** Process received signal ***
> [jueying:16317] Signal: Segmentation fault (11)
> [jueying:16317] Signal code: Address not mapped (1)
> [jueying:16317] Failing at address: 0x21
> [jueying:16317] [ 0] /usr/lib64/libpthread.so.0() [0x39e5e0f000]
> [jueying:16317] [ 1] /usr/lib64/libc.so.6() [0x39e5760551]
> [jueying:16317] [ 2]
> /opt/openmpi/1.7.2/lib/libopen-pal.so.5(opal_argv_join+0x39)
> [0x7f460b993079]
> [jueying:16317] [ 3] /opt/openmpi/1.7.2/lib/libmpi.so.1(ompi_mpi_init+0x347)
> [0x7f460c106a57]
> [jueying:16317] [ 4] /opt/openmpi/1.7.2/lib/libmpi.so.1(MPI_Init+0x16b)
> [0x7f460c12523b]
> [jueying:16317] [ 5] ./lmp_jueying() [0x40c035]
> [jueying:16317] [ 6] /usr/lib64/libc.so.6(__libc_start_main+0xf5)
> [0x39e5621a05]
> [jueying:16317] [ 7] ./lmp_jueying() [0x40dd21]
> [jueying:16317] *** End of error message ***
>
> Thanks for the help.
>
> Best regards,
> Yu-Hang Tang
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users



-- 
Information System Engineer, Ph.D.
Blog: http://matt.eifelle.com
LinkedIn: http://www.linkedin.com/in/matthieubrucher
Music band: http://liliejay.com/

Re: [OMPI users] Segmentation fault with fresh compilation of 1.7.2

[Matthieu Brucher]

Hi,

I tried with the latest nightly (well now it may not be the latest
anymore), and orte-info didn't crash. So I'll try again later with my
app.

thanks,

Matthieu

2013/9/15 Matthieu Brucher <matthieu.bruc...@gmail.com>:
> I can try later this week, yes.
> Thanks
>
> Le 15 sept. 2013 19:09, "Ralph Castain" <r...@open-mpi.org> a écrit :
>
>> Could you try the current 1.7.3 nightly tarball instead? I don't see a
>> problem there, and I'm wondering if this is something we already fixed. We
>> will be releasing 1.7.3 shortly and it is mostly complete at this time.
>>
>>
>> On Sep 15, 2013, at 10:43 AM, Matthieu Brucher
>> <matthieu.bruc...@gmail.com> wrote:
>>
>> Yes, ompi_info does not crash.
>>
>> Le 15 sept. 2013 18:05, "Ralph Castain" <r...@open-mpi.org> a écrit :
>>>
>>> No - out of curiosity, does ompi_info work? I'm wondering if this is
>>> strictly an orte-info problem.
>>>
>>> On Sep 15, 2013, at 10:03 AM, Matthieu Brucher
>>> <matthieu.bruc...@gmail.com> wrote:
>>>
>>> Just --with-lsf. Perhaps because then it must be launched through lsf?
>>>
>>> Le 15 sept. 2013 18:02, "Ralph Castain" <r...@open-mpi.org> a écrit :
>>>>
>>>> I'm not entirely sure - I don't see anything that would cause that
>>>> problem in that location. How did you configure this?
>>>>
>>>>
>>>> On Sep 12, 2013, at 3:17 AM, Matthieu Brucher
>>>> <matthieu.bruc...@gmail.com> wrote:
>>>>
>>>> > Hi,
>>>> >
>>>> > I compiled OpenMPI on a RHEL6 box with LSF support, but when I run
>>>> > sonthing, it crashes. Also orte-info crashes:
>>>> >
>>>> > Package: Open MPI mbruc...@xxx.com Distribution
>>>> >Open RTE: 1.7.2
>>>> >  Open RTE repo revision: r28673
>>>> >   Open RTE release date: Jun 26, 2013
>>>> >OPAL: 1.7.2
>>>> >  OPAL repo revision: r28673
>>>> >   OPAL release date: Jun 26, 2013
>>>> >Ident string: 1.7.2
>>>> >  Prefix: /xxx/mbrucher/openmpi
>>>> > Configured architecture: x86_64-unknown-linux-gnu
>>>> >  Configure host: xxx.xxx.com
>>>> >   Configured by: mbrucher
>>>> >   Configured on: Thu Sep 12 10:22:06 BST 2013
>>>> >  Configure host: xxx.xxx.com
>>>> >Built by: mbrucher
>>>> >Built on: Thu Sep 12 10:24:59 BST 2013
>>>> >  Built host: xxx.xxx.com
>>>> >  C compiler: gcc
>>>> > C compiler absolute: /usr/bin/gcc
>>>> >  C compiler family name: GNU
>>>> >  C compiler version: 4.4.6
>>>> >  Internal debug support: no
>>>> > Memory profiling support: no
>>>> > Memory debugging support: no
>>>> > libltdl support: yes
>>>> >   Heterogeneous support: no
>>>> > orterun default --prefix: no
>>>> >   MPI_WTIME support: gettimeofday
>>>> > Symbol vis. support: yes
>>>> >   FT Checkpoint support: no (checkpoint thread: no)
>>>> > [abgengcluster:45509] *** Process received signal ***
>>>> > [abgengcluster:45509] Signal: Segmentation fault (11)
>>>> > [abgengcluster:45509] Signal code: Address not mapped (1)
>>>> > [abgengcluster:45509] Failing at address: 0xf8
>>>> > [abgengcluster:45509] [ 0] /lib64/libpthread.so.0() [0x3ffc00f4a0]
>>>> > [abgengcluster:45509] [ 1]
>>>> >
>>>> > /xxx/mbrucher/openmpi/lib/libopen-pal.so.5(opal_libevent2019_event_priority_set+0x6f)
>>>> > [0x2aae84a736ef]
>>>> > [abgengcluster:45509] [ 2]
>>>> > /xxx/mbrucher/openmpi/lib/libopen-rte.so.5(orte_iof_base_open+0x31c)
>>>> > [0x2aae847edfbc]
>>>> > [abgengcluster:45509] [ 3] orte-info(orte_info_open_components+0x71f)
>>>> > [0x406b8f]
>>>> > [abgengcluster:45509] [ 4] orte-info(main+0x93d) [0x40450d]
>>>> > [abgengcluster:45509] [ 5] /lib64/libc.so.6(__libc_start_main+0xfd)
>>>> > [0x3ffb81ecdd]
>>>> > [abgengcluster:45509] [ 6] orte-info() [0x403b09]
>>>> > [abgengcluster:45509] **

Re: [OMPI users] Segmentation fault with fresh compilation of 1.7.2

[Matthieu Brucher]

I can try later this week, yes.
Thanks
Le 15 sept. 2013 19:09, "Ralph Castain" <r...@open-mpi.org> a écrit :

> Could you try the current 1.7.3 nightly tarball instead? I don't see a
> problem there, and I'm wondering if this is something we already fixed. We
> will be releasing 1.7.3 shortly and it is mostly complete at this time.
>
>
> On Sep 15, 2013, at 10:43 AM, Matthieu Brucher <matthieu.bruc...@gmail.com>
> wrote:
>
> Yes, ompi_info does not crash.
> Le 15 sept. 2013 18:05, "Ralph Castain" <r...@open-mpi.org> a écrit :
>
>> No - out of curiosity, does ompi_info work? I'm wondering if this is
>> strictly an orte-info problem.
>>
>> On Sep 15, 2013, at 10:03 AM, Matthieu Brucher <
>> matthieu.bruc...@gmail.com> wrote:
>>
>> Just --with-lsf. Perhaps because then it must be launched through lsf?
>> Le 15 sept. 2013 18:02, "Ralph Castain" <r...@open-mpi.org> a écrit :
>>
>>> I'm not entirely sure - I don't see anything that would cause that
>>> problem in that location. How did you configure this?
>>>
>>>
>>> On Sep 12, 2013, at 3:17 AM, Matthieu Brucher <
>>> matthieu.bruc...@gmail.com> wrote:
>>>
>>> > Hi,
>>> >
>>> > I compiled OpenMPI on a RHEL6 box with LSF support, but when I run
>>> > sonthing, it crashes. Also orte-info crashes:
>>> >
>>> > Package: Open MPI mbruc...@xxx.com Distribution
>>> >Open RTE: 1.7.2
>>> >  Open RTE repo revision: r28673
>>> >   Open RTE release date: Jun 26, 2013
>>> >OPAL: 1.7.2
>>> >  OPAL repo revision: r28673
>>> >   OPAL release date: Jun 26, 2013
>>> >Ident string: 1.7.2
>>> >  Prefix: /xxx/mbrucher/openmpi
>>> > Configured architecture: x86_64-unknown-linux-gnu
>>> >  Configure host: xxx.xxx.com
>>> >   Configured by: mbrucher
>>> >   Configured on: Thu Sep 12 10:22:06 BST 2013
>>> >  Configure host: xxx.xxx.com
>>> >Built by: mbrucher
>>> >Built on: Thu Sep 12 10:24:59 BST 2013
>>> >  Built host: xxx.xxx.com
>>> >  C compiler: gcc
>>> > C compiler absolute: /usr/bin/gcc
>>> >  C compiler family name: GNU
>>> >  C compiler version: 4.4.6
>>> >  Internal debug support: no
>>> > Memory profiling support: no
>>> > Memory debugging support: no
>>> > libltdl support: yes
>>> >   Heterogeneous support: no
>>> > orterun default --prefix: no
>>> >   MPI_WTIME support: gettimeofday
>>> > Symbol vis. support: yes
>>> >   FT Checkpoint support: no (checkpoint thread: no)
>>> > [abgengcluster:45509] *** Process received signal ***
>>> > [abgengcluster:45509] Signal: Segmentation fault (11)
>>> > [abgengcluster:45509] Signal code: Address not mapped (1)
>>> > [abgengcluster:45509] Failing at address: 0xf8
>>> > [abgengcluster:45509] [ 0] /lib64/libpthread.so.0() [0x3ffc00f4a0]
>>> > [abgengcluster:45509] [ 1]
>>> >
>>> /xxx/mbrucher/openmpi/lib/libopen-pal.so.5(opal_libevent2019_event_priority_set+0x6f)
>>> > [0x2aae84a736ef]
>>> > [abgengcluster:45509] [ 2]
>>> > /xxx/mbrucher/openmpi/lib/libopen-rte.so.5(orte_iof_base_open+0x31c)
>>> > [0x2aae847edfbc]
>>> > [abgengcluster:45509] [ 3] orte-info(orte_info_open_components+0x71f)
>>> [0x406b8f]
>>> > [abgengcluster:45509] [ 4] orte-info(main+0x93d) [0x40450d]
>>> > [abgengcluster:45509] [ 5] /lib64/libc.so.6(__libc_start_main+0xfd)
>>> > [0x3ffb81ecdd]
>>> > [abgengcluster:45509] [ 6] orte-info() [0x403b09]
>>> > [abgengcluster:45509] *** End of error message ***
>>> > Segmentation fault (core dumped)
>>> >
>>> > Is there something that I missed?
>>> >
>>> > Cheers,
>>> >
>>> > Matthieu
>>> > --
>>> > Information System Engineer, Ph.D.
>>> > Blog: http://matt.eifelle.com
>>> > LinkedIn: http://www.linkedin.com/in/matthieubrucher
>>> > Music band: http://liliejay.com/
>>> > ___
>>> > users mailing list
>>> > us...@open-mpi.org
>>> > http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>
>>> ___
>>> users mailing list
>>> us...@open-mpi.org
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>
>> ___
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>>
>>
>> ___
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>

Re: [OMPI users] Segmentation fault with fresh compilation of 1.7.2

[Matthieu Brucher]

Yes, ompi_info does not crash.
Le 15 sept. 2013 18:05, "Ralph Castain" <r...@open-mpi.org> a écrit :

> No - out of curiosity, does ompi_info work? I'm wondering if this is
> strictly an orte-info problem.
>
> On Sep 15, 2013, at 10:03 AM, Matthieu Brucher <matthieu.bruc...@gmail.com>
> wrote:
>
> Just --with-lsf. Perhaps because then it must be launched through lsf?
> Le 15 sept. 2013 18:02, "Ralph Castain" <r...@open-mpi.org> a écrit :
>
>> I'm not entirely sure - I don't see anything that would cause that
>> problem in that location. How did you configure this?
>>
>>
>> On Sep 12, 2013, at 3:17 AM, Matthieu Brucher <matthieu.bruc...@gmail.com>
>> wrote:
>>
>> > Hi,
>> >
>> > I compiled OpenMPI on a RHEL6 box with LSF support, but when I run
>> > sonthing, it crashes. Also orte-info crashes:
>> >
>> > Package: Open MPI mbruc...@xxx.com Distribution
>> >Open RTE: 1.7.2
>> >  Open RTE repo revision: r28673
>> >   Open RTE release date: Jun 26, 2013
>> >OPAL: 1.7.2
>> >  OPAL repo revision: r28673
>> >   OPAL release date: Jun 26, 2013
>> >Ident string: 1.7.2
>> >  Prefix: /xxx/mbrucher/openmpi
>> > Configured architecture: x86_64-unknown-linux-gnu
>> >  Configure host: xxx.xxx.com
>> >   Configured by: mbrucher
>> >   Configured on: Thu Sep 12 10:22:06 BST 2013
>> >  Configure host: xxx.xxx.com
>> >Built by: mbrucher
>> >Built on: Thu Sep 12 10:24:59 BST 2013
>> >  Built host: xxx.xxx.com
>> >  C compiler: gcc
>> > C compiler absolute: /usr/bin/gcc
>> >  C compiler family name: GNU
>> >  C compiler version: 4.4.6
>> >  Internal debug support: no
>> > Memory profiling support: no
>> > Memory debugging support: no
>> > libltdl support: yes
>> >   Heterogeneous support: no
>> > orterun default --prefix: no
>> >   MPI_WTIME support: gettimeofday
>> > Symbol vis. support: yes
>> >   FT Checkpoint support: no (checkpoint thread: no)
>> > [abgengcluster:45509] *** Process received signal ***
>> > [abgengcluster:45509] Signal: Segmentation fault (11)
>> > [abgengcluster:45509] Signal code: Address not mapped (1)
>> > [abgengcluster:45509] Failing at address: 0xf8
>> > [abgengcluster:45509] [ 0] /lib64/libpthread.so.0() [0x3ffc00f4a0]
>> > [abgengcluster:45509] [ 1]
>> >
>> /xxx/mbrucher/openmpi/lib/libopen-pal.so.5(opal_libevent2019_event_priority_set+0x6f)
>> > [0x2aae84a736ef]
>> > [abgengcluster:45509] [ 2]
>> > /xxx/mbrucher/openmpi/lib/libopen-rte.so.5(orte_iof_base_open+0x31c)
>> > [0x2aae847edfbc]
>> > [abgengcluster:45509] [ 3] orte-info(orte_info_open_components+0x71f)
>> [0x406b8f]
>> > [abgengcluster:45509] [ 4] orte-info(main+0x93d) [0x40450d]
>> > [abgengcluster:45509] [ 5] /lib64/libc.so.6(__libc_start_main+0xfd)
>> > [0x3ffb81ecdd]
>> > [abgengcluster:45509] [ 6] orte-info() [0x403b09]
>> > [abgengcluster:45509] *** End of error message ***
>> > Segmentation fault (core dumped)
>> >
>> > Is there something that I missed?
>> >
>> > Cheers,
>> >
>> > Matthieu
>> > --
>> > Information System Engineer, Ph.D.
>> > Blog: http://matt.eifelle.com
>> > LinkedIn: http://www.linkedin.com/in/matthieubrucher
>> > Music band: http://liliejay.com/
>> > ___
>> > users mailing list
>> > us...@open-mpi.org
>> > http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>> ___
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>

Re: [OMPI users] Segmentation fault with fresh compilation of 1.7.2

[Matthieu Brucher]

Just --with-lsf. Perhaps because then it must be launched through lsf?
Le 15 sept. 2013 18:02, "Ralph Castain" <r...@open-mpi.org> a écrit :

> I'm not entirely sure - I don't see anything that would cause that problem
> in that location. How did you configure this?
>
>
> On Sep 12, 2013, at 3:17 AM, Matthieu Brucher <matthieu.bruc...@gmail.com>
> wrote:
>
> > Hi,
> >
> > I compiled OpenMPI on a RHEL6 box with LSF support, but when I run
> > sonthing, it crashes. Also orte-info crashes:
> >
> > Package: Open MPI mbruc...@xxx.com Distribution
> >Open RTE: 1.7.2
> >  Open RTE repo revision: r28673
> >   Open RTE release date: Jun 26, 2013
> >OPAL: 1.7.2
> >  OPAL repo revision: r28673
> >   OPAL release date: Jun 26, 2013
> >Ident string: 1.7.2
> >  Prefix: /xxx/mbrucher/openmpi
> > Configured architecture: x86_64-unknown-linux-gnu
> >  Configure host: xxx.xxx.com
> >   Configured by: mbrucher
> >   Configured on: Thu Sep 12 10:22:06 BST 2013
> >  Configure host: xxx.xxx.com
> >Built by: mbrucher
> >Built on: Thu Sep 12 10:24:59 BST 2013
> >  Built host: xxx.xxx.com
> >  C compiler: gcc
> > C compiler absolute: /usr/bin/gcc
> >  C compiler family name: GNU
> >  C compiler version: 4.4.6
> >  Internal debug support: no
> > Memory profiling support: no
> > Memory debugging support: no
> > libltdl support: yes
> >   Heterogeneous support: no
> > orterun default --prefix: no
> >   MPI_WTIME support: gettimeofday
> > Symbol vis. support: yes
> >   FT Checkpoint support: no (checkpoint thread: no)
> > [abgengcluster:45509] *** Process received signal ***
> > [abgengcluster:45509] Signal: Segmentation fault (11)
> > [abgengcluster:45509] Signal code: Address not mapped (1)
> > [abgengcluster:45509] Failing at address: 0xf8
> > [abgengcluster:45509] [ 0] /lib64/libpthread.so.0() [0x3ffc00f4a0]
> > [abgengcluster:45509] [ 1]
> >
> /xxx/mbrucher/openmpi/lib/libopen-pal.so.5(opal_libevent2019_event_priority_set+0x6f)
> > [0x2aae84a736ef]
> > [abgengcluster:45509] [ 2]
> > /xxx/mbrucher/openmpi/lib/libopen-rte.so.5(orte_iof_base_open+0x31c)
> > [0x2aae847edfbc]
> > [abgengcluster:45509] [ 3] orte-info(orte_info_open_components+0x71f)
> [0x406b8f]
> > [abgengcluster:45509] [ 4] orte-info(main+0x93d) [0x40450d]
> > [abgengcluster:45509] [ 5] /lib64/libc.so.6(__libc_start_main+0xfd)
> > [0x3ffb81ecdd]
> > [abgengcluster:45509] [ 6] orte-info() [0x403b09]
> > [abgengcluster:45509] *** End of error message ***
> > Segmentation fault (core dumped)
> >
> > Is there something that I missed?
> >
> > Cheers,
> >
> > Matthieu
> > --
> > Information System Engineer, Ph.D.
> > Blog: http://matt.eifelle.com
> > LinkedIn: http://www.linkedin.com/in/matthieubrucher
> > Music band: http://liliejay.com/
> > ___
> > users mailing list
> > us...@open-mpi.org
> > http://www.open-mpi.org/mailman/listinfo.cgi/users
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>

[OMPI users] Segmentation fault with fresh compilation of 1.7.2

[Matthieu Brucher]

Hi,

I compiled OpenMPI on a RHEL6 box with LSF support, but when I run
sonthing, it crashes. Also orte-info crashes:

 Package: Open MPI mbruc...@xxx.com Distribution
Open RTE: 1.7.2
  Open RTE repo revision: r28673
   Open RTE release date: Jun 26, 2013
OPAL: 1.7.2
  OPAL repo revision: r28673
   OPAL release date: Jun 26, 2013
Ident string: 1.7.2
  Prefix: /xxx/mbrucher/openmpi
 Configured architecture: x86_64-unknown-linux-gnu
  Configure host: xxx.xxx.com
   Configured by: mbrucher
   Configured on: Thu Sep 12 10:22:06 BST 2013
  Configure host: xxx.xxx.com
Built by: mbrucher
Built on: Thu Sep 12 10:24:59 BST 2013
  Built host: xxx.xxx.com
  C compiler: gcc
 C compiler absolute: /usr/bin/gcc
  C compiler family name: GNU
  C compiler version: 4.4.6
  Internal debug support: no
Memory profiling support: no
Memory debugging support: no
 libltdl support: yes
   Heterogeneous support: no
orterun default --prefix: no
   MPI_WTIME support: gettimeofday
 Symbol vis. support: yes
   FT Checkpoint support: no (checkpoint thread: no)
[abgengcluster:45509] *** Process received signal ***
[abgengcluster:45509] Signal: Segmentation fault (11)
[abgengcluster:45509] Signal code: Address not mapped (1)
[abgengcluster:45509] Failing at address: 0xf8
[abgengcluster:45509] [ 0] /lib64/libpthread.so.0() [0x3ffc00f4a0]
[abgengcluster:45509] [ 1]
/xxx/mbrucher/openmpi/lib/libopen-pal.so.5(opal_libevent2019_event_priority_set+0x6f)
[0x2aae84a736ef]
[abgengcluster:45509] [ 2]
/xxx/mbrucher/openmpi/lib/libopen-rte.so.5(orte_iof_base_open+0x31c)
[0x2aae847edfbc]
[abgengcluster:45509] [ 3] orte-info(orte_info_open_components+0x71f) [0x406b8f]
[abgengcluster:45509] [ 4] orte-info(main+0x93d) [0x40450d]
[abgengcluster:45509] [ 5] /lib64/libc.so.6(__libc_start_main+0xfd)
[0x3ffb81ecdd]
[abgengcluster:45509] [ 6] orte-info() [0x403b09]
[abgengcluster:45509] *** End of error message ***
Segmentation fault (core dumped)

Is there something that I missed?

Cheers,

Matthieu
-- 
Information System Engineer, Ph.D.
Blog: http://matt.eifelle.com
LinkedIn: http://www.linkedin.com/in/matthieubrucher
Music band: http://liliejay.com/

[OMPI users] Typo on the FAQ page

[Matthieu Brucher]

Hi,

I saw a typo on the FAQ page
http://www.open-mpi.org/faq/?category=mpi-apps. It says that the
variable to change the CXX compiler is OMPI_MPIXX, but it is
OMPI_MPICXX (a C is missing).

Cheers,
-- 
Information System Engineer, Ph.D.
Blog: http://matt.eifelle.com
LinkedIn: http://www.linkedin.com/in/matthieubrucher
Music band: http://liliejay.com/

Re: [OMPI users] mpirun (Aborted) error

[Matthieu Brucher]

Hi,
This may be because you have an error in the parallel communication
pattern. Without other information, it is difficult to say something
else. Try degugging your application.

Matthieu

2013/2/24, Mohammad Mohsenie :
> Dear All,
> Greetings,
>
> I have installed openmpi to make my siesta package run in parallel in ubuntu
> 12. After all lateral packages (BLAS, LAPACK, BLACS, SCALAPACK ) are build I
> run an example with this command:
>  mpirun -np 4 siesta h20.out
>
> but this error showed up :
>
> mpirun noticed that process rank 2 with PID 6371 on node
> mahdi-System-Product-Name exited on signal 6 (Aborted).
>
> but when I changed command to serial mode ( mpirun -np 1 siesta >h20.out) there was no problem.
>
> Any help regard to my problem would be highly appreciated,
>
> Regards,
> SMM1370
>
>
>
>


-- 
Information System Engineer, Ph.D.
Blog: http://matt.eifelle.com
LinkedIn: http://www.linkedin.com/in/matthieubrucher
Music band: http://liliejay.com/

Re: [OMPI users] starting open-mpi

[Matthieu Brucher]

Hi,

You need to use the command prompt provided by Visual Studio and it will
work.

Matthieu

2012/5/18 Ghobad Zarrinchian 

> Hi. I've installed Visual Studio 2008 on my machine. But i have still the
> same problem. How can i solve it? thx
>
>
> On Fri, May 11, 2012 at 10:50 PM, Ghobad Zarrinchian <
> gzarrin@gmail.com> wrote:
>
>> Thanks Dmitry for your reply. :)
>>
>>
>> On Fri, May 11, 2012 at 4:47 PM, Dmitry N. Mikushin 
>> wrote:
>>
>>> Hi Ghobad,
>>>
>>> The error message means the OpenMPI wants to use cl.exe - the compiler
>>> from Microsoft Visual Studio.
>>>
>>> Here http://www.open-mpi.org/software/ompi/v1.5/ms-windows.php is it
>>> stated:
>>>
>>> This is the first binary release for Windows, with basic MPI libraries
>>> and executables. The supported platforms are Windows XP, Windows
>>> Vista, Windows Server 2003/2008, and Windows 7 (including both 32 and
>>> 64 bit versions). The installers were configured with CMake 2.8.1 and
>>> compiled under Visual Studio 2010, and they support for C/C++
>>> compilers of Visual Studio 2005, 2008 and 2010.
>>>
>>> So, to compile MPI programs you probably need one of this compilers to
>>> be installed.
>>>
>>> Best regards.
>>> - Dima.
>>>
>>> 2012/5/10 Ghobad Zarrinchian :
>>>  > Hi all. I'm a new open-mpi user. I've downloaded the
>>> > OpenMPI_v1.5.5-1_win32.exe file to install open-mpi on my dual-core
>>> windows
>>> > 7 machine. I installed the file but now i can't compile my mpi
>>> programs. I
>>> > use command below (in command prompt window) to compile my 'test.cpp'
>>> > program:
>>> >
>>> >>> mpic++ -o test test.cpp
>>> >
>>> > but i get error as follows:
>>> >
>>> >>> The open mpi wrapper compiler was unable to find the specified
>>> compiler
>>> >>> cl.exe in your path.
>>> >  Note that this compiler was either specified at configure time or
>>> in
>>> > one several possible environment variables.
>>> >
>>> > What is the problem? Is my compilation command right? Is there any
>>> remained
>>> > necessary steps to complete my open-mpi installation?
>>> > Is it necessary to specify some environment variables?
>>> >
>>> > Thanks in advanced.
>>> >
>>> > ___
>>> > users mailing list
>>> > us...@open-mpi.org
>>> > http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>
>>> ___
>>> users mailing list
>>> us...@open-mpi.org
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>
>>
>>
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>



-- 
Information System Engineer, Ph.D.
Blog: http://matt.eifelle.com
LinkedIn: http://www.linkedin.com/in/matthieubrucher

Re: [OMPI users] over-subscription of cores

[Matthieu Brucher]

Hi,

If your problem is memory bound and if you don't use the whole memory
capacity of one node, it means that you are limited by your memory
bandwidth. In this case oversubscribing the number of processes will lead
to worse behavior, as all processes will fight for the same memory
bandwidth.

Just my opinion.

Matthieu Brucher

2011/12/23 Santosh Ansumali <ansum...@gmail.com>

>  Dear All,
>We are running a PDE solver which is memory bound. Due to
> cache related issue,   smaller  number of grid point per core leads to
> better performance for this code.  Thus, though available memory per
> core is more than 2 GB, we are able to good  performance   by using
> less than 1 GB per core.
>
>  I want to know whether oversubscribing the cores can potentially
> improve performance of such a code.  My thinking is that if I
> oversubscribe the cores,  each thread will be using less than 1 GB so
> cache related problems will be less severe.  Is this logic correct or
> due to cache conflict performance will deteriorate further?
>  In case, over-subscription can help, how shall I modify
> submission file (using sun grid engine) to enable over-subscription of
> cores?
> my current submission file is written as follows
> #!/bin/bash
> #$ -N first
> #$ -S /bin/bash
> #$ -cwd
> #$ -e $JOB_ID.$JOB_NAME.ERROR
> #$ -o $JOB_ID.$JOB_NAME.OUTPUT
> #$ -P faculty_prj
> #$ -p 0
> #$ -pe orte 8
> /opt/mpi/openmpi/1.3.3/gnu/bin/mpirun -np $NSLOTS ./test_vel.out
>
> Is it possible to allow over-subscription by modifying submission file
> itself?  Or do I need to change hostfiles somehow?
> Thanks for your help!
> Best Regards
> Santosh Ansumali,
> Faculty Fellow,
> Engineering Mechanics Unit
> Jawaharlal Nehru Centre for Advanced Scientific Research (JNCASR)
>  Jakkur, Bangalore-560 064, India
> Tel: + 91 80 22082938
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>



-- 
Information System Engineer, Ph.D.
Blog: http://matt.eifelle.com
LinkedIn: http://www.linkedin.com/in/matthieubrucher

Re: [OMPI users] Running OpenMPI on SGI Altix with 4096 cores: very poor performance

[Matthieu Brucher]

Don't forget that MPT has some optimizations OpenMPI may not have, as
"overriding" free(). This way, MPT can have a huge performance boost
if you're allocating and freeing memory, and the same happens if you
communicate often.

Matthieu

2010/12/21 Gilbert Grosdidier :
> Hi George,
>  Thanks for your help. The bottom line is that the processes are neatly
> placed on the nodes/cores,
> as far as I can tell from the map :
> [...]
>         Process OMPI jobid: [33285,1] Process rank: 4
>         Process OMPI jobid: [33285,1] Process rank: 5
>         Process OMPI jobid: [33285,1] Process rank: 6
>         Process OMPI jobid: [33285,1] Process rank: 7
>  Data for node: Name: r34i0n1   Num procs: 8
>         Process OMPI jobid: [33285,1] Process rank: 8
>         Process OMPI jobid: [33285,1] Process rank: 9
>         Process OMPI jobid: [33285,1] Process rank: 10
>         Process OMPI jobid: [33285,1] Process rank: 11
>         Process OMPI jobid: [33285,1] Process rank: 12
>         Process OMPI jobid: [33285,1] Process rank: 13
>         Process OMPI jobid: [33285,1] Process rank: 14
>         Process OMPI jobid: [33285,1] Process rank: 15
>  Data for node: Name: r34i0n2   Num procs: 8
>         Process OMPI jobid: [33285,1] Process rank: 16
>         Process OMPI jobid: [33285,1] Process rank: 17
>         Process OMPI jobid: [33285,1] Process rank: 18
>         Process OMPI jobid: [33285,1] Process rank: 19
>         Process OMPI jobid: [33285,1] Process rank: 20
> [...]
>  But the perfs are still very low ;-(
>  Best,    G.
> Le 20 déc. 10 à 22:27, George Bosilca a écrit :
>
> That's a first step. My question was more related to the process overlay on
> the cores. If the MPI implementation place one process per node, then rank k
> and rank k+1 will always be on separate node, and the communications will
> have to go over IB. In the opposite if the MPI implementation places the
> processes per core, then rank k and k+1 will [mostly] be on the same node
> and the communications will be over shared memory. Depending on how the
> processes are placed and how you create the neighborhoods the performance
> can be drastically impacted.
>
> There is a pretty good description of the problem at:
> http://www.hpccommunity.org/f55/behind-scenes-mpi-process-placement-640/
>
> Some hints at
> http://www.open-mpi.org/faq/?category=running#mpirun-scheduling. I suggest
> you play with the --byslot --bynode options to see how this affect the
> performance of your application.
>
> For the hardcore cases we provide a rankfile feature. More info at:
> http://www.open-mpi.org/faq/?category=tuning#using-paffinity
>
> Enjoy,
>  george.
>
>
>
> On Dec 20, 2010, at 15:45 , Gilbert Grosdidier wrote:
>
> Yes, there is definitely only 1 process per core with both MPI
> implementations.
>
> Thanks,   G.
>
>
> Le 20/12/2010 20:39, George Bosilca a écrit :
>
> Are your processes places the same way with the two MPI implementations?
> Per-node vs. per-core ?
>
>  george.
>
> On Dec 20, 2010, at 11:14 , Gilbert Grosdidier wrote:
>
> Bonjour,
>
> I am now at a loss with my running of OpenMPI (namely 1.4.3)
>
> on a SGI Altix cluster with 2048 or 4096 cores, running over Infiniband.
>
> After fixing several rather obvious failures with Ralph, Jeff and John help,
>
> I am now facing the bottom of this story since :
>
> - there are no more obvious failures with messages
>
> - compared to the running of the application with SGI-MPT, the CPU
> performances I get
>
> are very low, decreasing when the number of cores increases (cf below)
>
> - these performances are highly reproducible
>
> - I tried a very high number of -mca parameters, to no avail
>
> If I take as a reference the MPT CPU speed performance,
>
> it is of about 900 (in some arbitrary unit), whatever the
>
> number of cores I used (up to 8192).
>
> But, when running with OMPI, I get:
>
> - 700 with 1024 cores (which is already rather low)
>
> - 300 with 2048 cores
>
> - 60   with 4096 cores.
>
> The computing loop, over which the above CPU performance is evaluated,
> includes
>
> a stack of MPI exchanges [per core : 8 x (MPI_Isend + MPI_Irecv) +
> MPI_Waitall]
>
> The application is of the 'domain partition' type,
>
> and the performances, together with the memory footprint,
>
> are very identical on all  cores. The memory footprint is twice higher in
>
> the OMPI case (1.5GB/core) than in the MPT case (0.7GB/core).
>
> What could be wrong with all these, please ?
>
> I provided (in attachment) the 'ompi_info -all ' output.
>
> The config.log is in attachment as well.
>
> I compiled OMPI with icc. I checked numa and affinity are OK.
>
> I use the following command to run my OMPI app:
>
> "mpiexec -mca btl_openib_rdma_pipeline_send_length 65536\
>
> -mca btl_openib_rdma_pipeline_frag_size 65536\
>
> -mca btl_openib_min_rdma_pipeline_size 65536\
>
> -mca btl_self_rdma_pipeline_send_length 262144\
>
> -mca btl_self_rdma_pipeline_frag_size 262144\
>
> -mca 

Re: [OMPI users] Open MPI task scheduler

[Matthieu Brucher]

2010/6/21 Jack Bryan :
> Hi,
> thank you very much for your help.
> What is the meaning of " must find a system so that every task can be
> serialized in the same form." What is the meaning of "serize " ?

Serialize is the process of converting an object instance into a
text/binary stream, and to create a new object instance from this
stream. It allows communication of data between processes. With MPI,
you send one data type after another, with serialization, you send
everything in one big chunk.

> I have no experience of programming with python and XML.

Python is not mandatory at all. I use it to automate the wrappers/SOAP
files generation, and to talk to the daemon. You can do this is any
language you are comfortable with.

> I have studied your blog.
> Where can I find a simple example to use the techniques you have said ?

If you are looking for RPC samples, you can ask google with just SOAP
as key, it will find several tutorials on how this works. As Jody
said, you may want something simplier if you can have all tasks in one
MPI process, but once you go on a big cluster with variable resources,
you will be stuck.

> For exmple, I have 5 task (print "hello world !").
> I want to use 6 processors to do it in parallel.
> One processr is the manager node who distributes tasks and other 5
> processors
> do the printing jobs and when they are done, they tell this to the manager
> noitde.

In this case, you have your daemon working in parallel from the batch
scheduler, and then each process asks the daemon for a new ticket. You
may add tickets by talking to the dameon directly, without having to
launch a new job.


> Boost.Asio is a cross-platform C++ library for network and low-level I/O
> programming. I have no experiences of using it. Will it take a long time to
> learn
> how to use it ?

The longest time will not be to master Boost, but more to understand
how to create your TCP server and to serialize your parameters.

> If the messages are transferred by SOAP+TCP, how the manager node calls it
> and push task into it ?

You have to think of SOAP + TCP as just a simple function call that
hides everything. From the client node point of view, it's a simple
function call server.get_ticket(). The manager node will be talked to
by different kind of programs: task programs or by a program that will
push tickets. The latter one will just be another function call like
this in C++:

std::vector tickets;
daemon.connect(somewhere, port);
daemon.add_tickets(tickets);

> Do I need to install SOAP+TCP on my cluster so that I can use it ?

As Jody said, you can do things with MPI directly. I would not
recommand it, but this will help you with a fast solution. You will
have to use some MPI2 calls to create a socket on the daemon to talk
to it, and in fact, you will have to create exactly what I proposed,
but less portable.

Matthieu
-- 
Information System Engineer, Ph.D.
Blog: http://matt.eifelle.com
LinkedIn: http://www.linkedin.com/in/matthieubrucher

Re: [OMPI users] Open MPI task scheduler

[Matthieu Brucher]

2010/6/20 Jack Bryan :
> Hi, Matthieu:
> Thanks for your help.
> Most of your ideas show that what I want to do.
> My scheduler should be able to be called from any C++ program, which can
> put
> a list of tasks to the scheduler and then the scheduler distributes the
> tasks to other client nodes.
> It may work like in this way:
> while(still tasks available) {
> myScheduler.push(tasks);
> myScheduler.get(tasks results from client nodes);
> }

Exactly. In your case, you want only one server, so you must find a
system so that every task can be serialized in the same form. The
easiest way to do so is to serialize your parameter set as an XML
fragment and add the type of task as another field.

> My cluster has 400 nodes with Open MPI. The tasks should be transferred b y
> MPI protocol.

No, they should not ;) MPI can be used, but it is not the easiest way
to do so. You still have to serialize your ticket, and you have to use
some functions that are from MPI2 (so perhaps not as portable as MPI1
functions). Besides, it cannot be used from programs that do not know
of using MPI protocols.

> I am not familiar with  RPC Protocol.

RPC is not a protocol per se. SOAP is. RPC stands for Remote Procedure
Call. It is basically your scheduler that has several functions
clients can call:
- add tickets
- retrieve ticket
- ticket is done

> If I use Boost.ASIO and some Python/GCCXML script to generate the code, it
> can be
> called from C++ program on Open MPI cluster ?

Yes, SOAP is just an XML way of representing the fact that you call a
function on the server. You can use it with C++, Java, ... I use it
with Python to monitor how many tasks are remaining, for instance.

> I cannot find the skeletton on your blog.
> Would you please tell me where to find it ?

It's not complete as some of the work is property of my employer. This
is how I use GCCXML to generate the calling code:
http://matt.eifelle.com/2009/07/21/using-gccxml-to-automate-c-wrappers-creation/
You have some additional code to write, but this is the main idea.

> I really appreciate your help.

No sweat, I hope I can give you correct hints!

Matthieu
-- 
Information System Engineer, Ph.D.
Blog: http://matt.eifelle.com
LinkedIn: http://www.linkedin.com/in/matthieubrucher

Re: [OMPI users] Open MPI task scheduler

[Matthieu Brucher]

Hi Jack,

What you are seeking is the client/server pattern. Have one node act
as a server. It will create a list of tasks or even a graph of tasks
if you have dependencies, and then create clients that will connect to
the server with an RPC protocol (I've done this with a SOAP+TCP
protocol, the severance of the TCP connection meaning that the client
is dead and that its task should be recycled, ités easy to do with
Boost.ASIO and some Python/GCCXML scripts to automatically generate
your code, I've written a skeletton on my blog). You may even have
clients with different sizes or capabilities and tell the server what
each client can do, and then the server may dispatch appropriate
tickets to the clients.

Each client and server can be a MPI process, you don't have to create
all clients inside one MPI process (you may use several if the
smallest resource your batch scheduler allocates is bigger that one of
your tasks). With a batch scheduler, it's better to allocate your
tasks as small as possible so that you can balance the resources you
need.

Matthieu

2010/6/20 Jack Bryan :
> Hi, all:
> I need to design a task scheduler (not PBS job scheduler) on Open MPI
> cluster.
> I need to parallelize an algorithm so that a big problem is decomposed into
> small tasks, which can be distributed
> to other worker nodes by the Scheduler and after being solved, the results
> of these tasks are returned to the manager node with the Scheduler, which
> will distribute more tasks on the base of the collected results.
> I need to use C++ to design the scheduler.
> I have searched online and I cannot find any scheduler available
> for this purpose.
> Any help is appreciated.
> thanks
> Jack
> June 19  2010
> 
> Hotmail has tools for the New Busy. Search, chat and e-mail from your inbox.
> Learn more.
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>



-- 
Information System Engineer, Ph.D.
Blog: http://matt.eifelle.com
LinkedIn: http://www.linkedin.com/in/matthieubrucher

Re: [OMPI users] bin/orted: Command not found.

[Matthieu Brucher]

Strange that it indicates the whole path. I had the same issue, but it
only said that orted couldn't be found. In my .bashrc, I put what it
needed to get orted in my PATH, and it worked.

Matthieu

2009/8/8 Ralph Castain :
> Not that I know of - I don't think we currently have any way for you to
> specify a location for individual nodes.
>
> Is there some reason why you installed it this way?
>
>
> On Fri, Aug 7, 2009 at 11:27 AM, Kenneth Yoshimoto  wrote:
>>
>> Hello,
>>
>>  I have three sets of nodes, each with openmpi installed in
>> a different location.  I am getting an error related to orted:
>>
>> /users/kenneth/info/openmpi/install/bin/orted: Command not found.
>>
>>  I think it's looking for orted in the wrong place on some of
>> the nodes.  Is there an easy way to have mpirun look for
>> orted in the right place on the different sets of nodes?
>>
>> Thanks,
>> Kenneth
>> ___
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>



-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher

Re: [OMPI users] MPI and C++ (Boost)

[Matthieu Brucher]

> IF boost is attached to MPI 3 (or whatever), AND it becomes part of the
> mainstream MPI implementations, THEN you can have the discussion again.

Hi,

At the moment, I think that Boost.MPI only supports MPI1.1, and even
then, some additional work may be done, at least regarding the complex
datatypes.

Matthieu
-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher

Re: [OMPI users] New warning messages in 1.3.2 (not present in1.2.8)

[Matthieu Brucher]

Thank you a lot for this.

I've just checked everything again, recompiled my code as well (I'm
using SCons so it detects that the headers and the libraries changed)
and it works without a warning.

Matthieu

2009/5/12 Jeff Squyres <jsquy...@cisco.com>:
> On May 12, 2009, at 8:17 AM, Matthieu Brucher wrote:
>
>> OK, this is indeed the case. I'll try to clean the tree (I have
>> several other package and deleted the original 1.2.8 package) and test
>> again.
>>
>
>
> This misleading libltdl error message continues to bite us over and over
> again (users and developers alike), so I just put in a workaround with some
> heuristics to try to print a better error message in cases like yours.  Now
> you'll see something like this:
>
> [foo.example.com:24273] mca: base: component_find: unable to open
> /home/jsquyres/bogus/lib/openmpi/mca_btl_openib: perhaps a missing symbol,
> or compiled for a different version of Open MPI? (ignored)
>
> Changeset here (Shiqing tells me it'll work on Windows, too -- MTT will tell
> us for sure :-) ):
>
>    https://svn.open-mpi.org/trac/ompi/changeset/21214
>
> And I filed to move it to v1.3.3 here:
>
>    https://svn.open-mpi.org/trac/ompi/ticket/1917
>
> --
> Jeff Squyres
> Cisco Systems
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>



-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher

Re: [OMPI users] New warning messages in 1.3.2 (not present in1.2.8)

[Matthieu Brucher]

2009/5/12 Jeff Squyres :
> Or it could be that you installed 1.3.2 over 1.2.8 -- some of the 1.2.8
> components that no longer exist in the 1.3 series are still in the
> installation tree, but failed to open properly (unfortunately, libltdl gives
> an incorrect "file not found" error message if it is unable to load a plugin
> for any reason, such as if a symbol is unable to be resolved from that
> plugin).
>
> The best thing to do is to install 1.3 in a clean, fresh tree, or uninstall
> your 1.2.8 before installing 1.3.

OK, this is indeed the case. I'll try to clean the tree (I have
several other package and deleted the original 1.2.8 package) and test
again.

Thanks for the answers!

Matthieu
-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher

[OMPI users] New warning messages in 1.3.2 (not present in 1.2.8)

[Matthieu Brucher]

Hi,

I've managed to use 1.3.2 (still not with LSF and InfiniPath, I start
one step after another), but I have additional warnings that didn't
show up in 1.2.8:

[host-b:09180] mca: base: component_find: unable to open
/home/brucher/lib/openmpi/mca_ras_dash_host: file not found (ignored)
[host-b:09180] mca: base: component_find: unable to open
/home/brucher/lib/openmpi/mca_ras_gridengine: file not found (ignored)
[host-b:09180] mca: base: component_find: unable to open
/home/brucher/lib/openmpi/mca_ras_localhost: file not found (ignored)
[host-b:09180] mca: base: component_find: unable to open
/home/brucher/lib/openmpi/mca_errmgr_hnp: file not found (ignored)
[host-b:09180] mca: base: component_find: unable to open
/home/brucher/lib/openmpi/mca_errmgr_orted: file not found (ignored)
[host-b:09180] mca: base: component_find: unable to open
/home/brucher/lib/openmpi/mca_errmgr_proxy: file not found (ignored)
[host-b:09180] mca: base: component_find: unable to open
/home/brucher/lib/openmpi/mca_iof_proxy: file not found (ignored)
[host-b:09180] mca: base: component_find: unable to open
/home/brucher//lib/openmpi/mca_iof_svc: file not found (ignored)

Can this be fixed in some way?

Matthieu
-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher

Re: [OMPI users] LSF launch with OpenMPI

[Matthieu Brucher]

Hi,

Thank you for the tip, this seems to be what I was looking for.

Matthieu

2009/5/7 Mehdi Bozzo-Rey <mbozz...@platform.com>:
> Hello Jeroen,
>
>
>
> There are 2 ways of launching OpenMPI jobs (using a recent version of LSF):
>
> 1.   The one you have just described; it uses the generic PJL (Parallel
> Job Launcher) framework. You can easily recognise it because of the use of
> the –a openmpi flag and mpirun.lsf
>
> 2.   In recent versions of LSF, another framework is also available, and
> it permits a tight (native) integration with the MPIs (this is why there is
> the OpenMPI integration)
>
>
>
> So, for 1., a typical command line would be, as you mentioned, something
> like:
>
>
>
> bsub -o %J.out -e %J.err -n 4 -R "span[ptile=1]" -a openmpi mpirun.lsf
> ./test
>
>
>
> And for 2., you would use something like:
>
>
>
> bsub -o %J.out -e %J.err -n 4 -R "span[ptile=1]" mpirun ./test
>
>
>
> Cheers,
>
>
>
> Mehdi
>
>
>
>
>
>
>
> From: users-boun...@open-mpi.org [mailto:users-boun...@open-mpi.org] On
> Behalf Of Jeroen Kleijer
> Sent: May-05-09 9:26 AM
> To: Open MPI Users
> Subject: Re: [OMPI users] LSF launch with OpenMPI
>
>
>
> If you wish to submit to lsf using its native commands (bsub) you can do the
> following:
>
>
>
> bsub -q ${QUEUE} -a openmpi -n ${CPUS} "mpirun.lsf  -x PATH -x
> LD_LIBRARY_PATH -x MPI_BUFFER_SIZE ${COMMAND} ${OPTIONS}"
>
>
>
> It should be noted that in this case you don't call OpenMPI's mpirun
> directly but use the mpirun.lsf, a wrapper script provided by LSF. This
> wrapper script takes care of setting the necessary environment variables and
> eventually calls the correct mpirun. (the option "-a openmpi" tells LSF that
> we're using OpenMPI so don't try to autodetect)
>
>
>
> Regards,
>
>
>
> Jeroen Kleijer
>
> On Tue, May 5, 2009 at 2:23 PM, Jeff Squyres <jsquy...@cisco.com> wrote:
>
> On May 5, 2009, at 6:10 AM, Matthieu Brucher wrote:
>
> The first is what the support of LSF by OpenMPI means. When mpirun is
> executed, it is an LSF job that is actually ran? Or what does it
> imply? I've tried to search on the openmpi website as well as on the
> internet, but I couldn't find a clear answer/use case.
>
>
>
> What Terry said is correct.  It means that "mpirun" will use, under the
> covers, the "native" launching mechanism of LSF to launch jobs (vs., say,
> rsh or ssh).  It'll also discover the hosts to use for this job without the
> use of a hostfile -- it'll query LSF directly to see what hosts it should
> use.
>
> My second question is about the LSF detection. lsf.h is detected, but
> when lsb_launch is searched for ion libbat.so, it fails because
> parse_time and parse_time_ex are not found. Is there a way to add
> additional lsf libraries so that the search can be done?
>
> Can you send all the data shown here:
>
>    http://www.open-mpi.org/community/help/
>
> --
> Jeff Squyres
> Cisco Systems
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>



-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher

Re: [OMPI users] LSF launch with OpenMPI

[Matthieu Brucher]

2009/5/6 Jeff Squyres <jsquy...@cisco.com>:
> On May 5, 2009, at 10:01 AM, Matthieu Brucher wrote:
>
>> > What Terry said is correct.  It means that "mpirun" will use, under the
>> > covers, the "native" launching mechanism of LSF to launch jobs (vs.,
>> > say,
>> > rsh or ssh).  It'll also discover the hosts to use for this job without
>> > the
>> > use of a hostfile -- it'll query LSF directly to see what hosts it
>> > should
>> > use.
>>
>> OK, so I have to do something like:
>> bsub -n ${CPUS} mpirun myapplication
>>
>> Is it what I think?
>>
>
> I don't know what you think.  ;-)  But I think that your above command might
> be correct.  You want *1* copy of mpirun to execute.  Hence, if
>
> bsub -n ${CPUS} uptime
>
> launches ${CPUS} copies of uptime, then the above command is not correct.
>  You want to submit an ${CPUS} processor job to LSF and have *one* copy of
> "mpirun myapplication" run -- mpirun will then invoke the underlying stuff
> to launch ${CPUS} copies of myapplication and join them together into a
> single MPI job.
>
>> I've enclosed the configure output as well as the config.log. The
>> problem is that my LSF (I didn't install it) 7.0.3 need libbat to be
>> linked against llsbstream (I modified the configure script to add
>> -llsbstream, and it compiled).
>>
>
> Huh!  Odd -- we didn't need that before.  Let me check with Platform...
>
> FWIW, you should be able to run like this without modifying configure:
>
>    ./configure LIBS=-llsbstream etc
>
> That should add -llsbstream in the Right places.

Thanks, I'll try this (provided I'm able to run OpenMPI 1.3.2, I have
some strange errors I didn't get with 1.2.8)

Matthieu
-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher

Re: [OMPI users] LSF launch with OpenMPI

[Matthieu Brucher]

2009/5/5 Jeff Squyres <jsquy...@cisco.com>:
> On May 5, 2009, at 6:10 AM, Matthieu Brucher wrote:
>
>> The first is what the support of LSF by OpenMPI means. When mpirun is
>> executed, it is an LSF job that is actually ran? Or what does it
>> imply? I've tried to search on the openmpi website as well as on the
>> internet, but I couldn't find a clear answer/use case.
>>
>
> What Terry said is correct.  It means that "mpirun" will use, under the
> covers, the "native" launching mechanism of LSF to launch jobs (vs., say,
> rsh or ssh).  It'll also discover the hosts to use for this job without the
> use of a hostfile -- it'll query LSF directly to see what hosts it should
> use.

OK, so I have to do something like:
bsub -n ${CPUS} mpirun myapplication

Is it what I think?

>> My second question is about the LSF detection. lsf.h is detected, but
>> when lsb_launch is searched for ion libbat.so, it fails because
>> parse_time and parse_time_ex are not found. Is there a way to add
>> additional lsf libraries so that the search can be done?
>>
>
>
> Can you send all the data shown here:
>
>    http://www.open-mpi.org/community/help/

I've enclosed the configure output as well as the config.log. The
problem is that my LSF (I didn't install it) 7.0.3 need libbat to be
linked against llsbstream (I modified the configure script to add
-llsbstream, and it compiled).

I can't use the official way of launching a batch job, LSF doesn't
pickup the correct LSF script wrapper (due to a bogus installation).

Thank you for all the answers! (I will have others, as I'm trying to
use the InfiniPath support as well)

Matthieu
-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher


output.tar.bz
Description: Binary data

[OMPI users] LSF launch with OpenMPI

[Matthieu Brucher]

Hello,

I have two questions, in fact.

The first is what the support of LSF by OpenMPI means. When mpirun is
executed, it is an LSF job that is actually ran? Or what does it
imply? I've tried to search on the openmpi website as well as on the
internet, but I couldn't find a clear answer/use case.

My second question is about the LSF detection. lsf.h is detected, but
when lsb_launch is searched for ion libbat.so, it fails because
parse_time and parse_time_ex are not found. Is there a way to add
additional lsf libraries so that the search can be done?

Matthieu Brucher
-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher