Re: [OMPI users] bin/orted: Command not found.
Not that I know of - I don't think we currently have any way for you to specify a location for individual nodes. Is there some reason why you installed it this way? On Fri, Aug 7, 2009 at 11:27 AM, Kenneth Yoshimoto wrote: > > Hello, > > I have three sets of nodes, each with openmpi installed in > a different location. I am getting an error related to orted: > > /users/kenneth/info/openmpi/install/bin/orted: Command not found. > > I think it's looking for orted in the wrong place on some of > the nodes. Is there an easy way to have mpirun look for > orted in the right place on the different sets of nodes? > > Thanks, > Kenneth > ___ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users >
Re: [OMPI users] Automated tuning tool
Gus Correa wrote: Terry Frankcombe wrote: There's been quite some discussion here lately about the effect of OMPI tuning parameters, particularly w.r.t. collectives. Is there some tool to probe performance on any given network/collection of nodes to aid optimisation of these parameters? (I'm thinking something along the philosophy of ATLAS.) ___ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users Hi Terry We are also looking for this holy grail. So far I found this 2008 reference to a certain "Open Tool for Parameter Optimization (OTPO)": http://www.springerlink.com/content/h5162153l184r7p0/ OTPO defines itself as this: "OTPO systematically tests large numbers of combinations of Open MPI’s run-time tunable parameters for common communication patterns and performance metrics to determine the “best” set for a given platform." you can checkout the OTPO code at http://svn.open-mpi.org/svn/otpo/trunk/ It supports as of now netpipe and skampi collectives for tuning. It is far from perfect, but it is a starting point. If there are any issues, please let us know... Thanks Edgar However, I couldn't find any reference to the actual code or scripts, and whether it is available, tested, free, downloadable, etc. At this point I am doing these performance tests in a laborious and inefficient manual way, when I have the time to do it. As some of the aforementioned article authors are list subscribers (and OpenMPI developers), maybe they can shed some light about OTPO, tuned collective optimization, OpenMPI runtime parameter optimization, etc. IMHO, this topic deserves at least a FAQ. Developers, Jeff: Any suggestions? :) Many thanks, Gus Correa - Gustavo Correa Lamont-Doherty Earth Observatory - Columbia University Palisades, NY, 10964-8000 - USA - ___ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users -- Edgar Gabriel Assistant Professor Parallel Software Technologies Lab http://pstl.cs.uh.edu Department of Computer Science University of Houston Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA Tel: +1 (713) 743-3857 Fax: +1 (713) 743-3335
[OMPI users] x4100 with IB
I have several Sun x4100 with Infiniband which appear to be running at 400MB/sec instead of 800MB/sec. It a freshly reformatted cluster converting from solaris to linux. We also reset the bios settings with "load optimal defaults". Does anyone know which bios setting i changed to dump the BW? x4100 mellanox ib ofed-1.4.1-rc6 w/ openmpi
Re: [OMPI users] Automated tuning tool
I'm not sure that there is a general "best set" of parameters, given the dependence of that set on comms patterns, etc... Still, this *is* a classic parameter sweep and optimization problem (unlike ATLAS), with a small number of parameters, and is the sort of thing one should be able to hook up fairly easily in a python script connected to a batch scheduler. Especially since you'd be likely to submit and run either a single job, or a number of equal sized jobs in parallel. In fact, here is a python script that works with SGE http://www.cs.umass.edu/~swarm/index.php?n=Sge.Py Now, you'd just have to choose the app, or apps that are important to you Gus Correa wrote: Terry Frankcombe wrote: There's been quite some discussion here lately about the effect of OMPI tuning parameters, particularly w.r.t. collectives. Is there some tool to probe performance on any given network/collection of nodes to aid optimisation of these parameters? (I'm thinking something along the philosophy of ATLAS.) ___ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users Hi Terry We are also looking for this holy grail. So far I found this 2008 reference to a certain "Open Tool for Parameter Optimization (OTPO)": http://www.springerlink.com/content/h5162153l184r7p0/ OTPO defines itself as this: "OTPO systematically tests large numbers of combinations of Open MPI’s run-time tunable parameters for common communication patterns and performance metrics to determine the “best” set for a given platform." However, I couldn't find any reference to the actual code or scripts, and whether it is available, tested, free, downloadable, etc. At this point I am doing these performance tests in a laborious and inefficient manual way, when I have the time to do it. As some of the aforementioned article authors are list subscribers (and OpenMPI developers), maybe they can shed some light about OTPO, tuned collective optimization, OpenMPI runtime parameter optimization, etc. IMHO, this topic deserves at least a FAQ. Developers, Jeff: Any suggestions? :) Many thanks, Gus Correa - Gustavo Correa Lamont-Doherty Earth Observatory - Columbia University Palisades, NY, 10964-8000 - USA - ___ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users -- John Casu, Chiral Dynamics, Inc 15525 Upper Ellen Road, Los Gatos, CA 95033. Phone: (831) 840 0142
Re: [OMPI users] Automated tuning tool
Terry Frankcombe wrote: There's been quite some discussion here lately about the effect of OMPI tuning parameters, particularly w.r.t. collectives. Is there some tool to probe performance on any given network/collection of nodes to aid optimisation of these parameters? (I'm thinking something along the philosophy of ATLAS.) ___ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users Hi Terry We are also looking for this holy grail. So far I found this 2008 reference to a certain "Open Tool for Parameter Optimization (OTPO)": http://www.springerlink.com/content/h5162153l184r7p0/ OTPO defines itself as this: "OTPO systematically tests large numbers of combinations of Open MPI’s run-time tunable parameters for common communication patterns and performance metrics to determine the “best” set for a given platform." However, I couldn't find any reference to the actual code or scripts, and whether it is available, tested, free, downloadable, etc. At this point I am doing these performance tests in a laborious and inefficient manual way, when I have the time to do it. As some of the aforementioned article authors are list subscribers (and OpenMPI developers), maybe they can shed some light about OTPO, tuned collective optimization, OpenMPI runtime parameter optimization, etc. IMHO, this topic deserves at least a FAQ. Developers, Jeff: Any suggestions? :) Many thanks, Gus Correa - Gustavo Correa Lamont-Doherty Earth Observatory - Columbia University Palisades, NY, 10964-8000 - USA -
[OMPI users] bin/orted: Command not found.
Hello, I have three sets of nodes, each with openmpi installed in a different location. I am getting an error related to orted: /users/kenneth/info/openmpi/install/bin/orted: Command not found. I think it's looking for orted in the wrong place on some of the nodes. Is there an easy way to have mpirun look for orted in the right place on the different sets of nodes? Thanks, Kenneth
Re: [OMPI users] Performance question about OpenMPI and MVAPICH2 on IB
Hi Craig, Terry, Neeraj, list Craig: A fellow here runs WRF. I grep the code and there is plenty of collectives there: MPI_[All]Gather[v], MPI_[All]Reduce, etc. Domain decomposition code like WRF, MITgcm, and other atmosphere and ocean codes has point-to-point communication to exchange subdomain boundaries, but also collective operations to calculate sums, etc, in various types of PDE (matrix) solvers that require global information. Terry: On the MITgcm, the apparent culprit is MPI_Allreduce, which seems to be bad on **small** messages (rather than big ones). This is the same behavior pattern that was reported here on May, regarding MPI_Alltoall, by Roman Martonak, a list subscriber using a computational chemistry package on an IB cluster: http://www.open-mpi.org/community/lists/users/2009/07/10045.php http://www.open-mpi.org/community/lists/users/2009/05/9419.php At that point Pavel Shamis, Peter Kjellstrom, and others gave very good suggestions, but they were only focused on MPI_Alltoall. No other collectives were considered. All: Any insights on how to tune MPI_Allreduce? Maybe a hint on the other collectives also? Any benchmark tool available that one can use to find the sweet spot of each collective? Many thanks, Gus Correa - Gustavo Correa Lamont-Doherty Earth Observatory - Columbia University Palisades, NY, 10964-8000 - USA - nee...@crlindia.com wrote: Hi Terry, I had tested mostly MPI_Bcast, MPI_Reduce, MPI_Gather kind of MPI with openmpi-1.3 and hierarchical option enabled.In all these, i found results slower than regular tuned collectives. We have HP Blade with intel clovertown processor(two quad core) connected with DDR infiniband clos network. Results were tested on 12-16 nodes with 8 mpi process each node. Regards Neeraj Chourasia (MTS) Computational Research Laboratories Ltd. (A wholly Owned Subsidiary of TATA SONS Ltd) B-101, ICC Trade Towers, Senapati Bapat Road Pune 411016 (Mah) INDIA (O) +91-20-6620 9863 (Fax) +91-20-6620 9862 M: +91.9225520634 *Terry Dontje * Sent by: users-boun...@open-mpi.org 08/07/2009 05:15 PM Please respond to Open MPI Users To us...@open-mpi.org cc Subject Re: [OMPI users] Performance question about OpenMPI and MVAPICH2 onIB Hi Neeraj, Were there specific collectives that were slower? Also what kind of cluster were you running on? How many nodes and cores per node? thanks, --td > Message: 3 > Date: Fri, 7 Aug 2009 16:51:05 +0530 > From: nee...@crlindia.com > Subject: Re: [OMPI users] Performance question about OpenMPI and > MVAPICH2 on IB > To: Open MPI Users > Cc: us...@open-mpi.org, users-boun...@open-mpi.org > Message-ID: > > > Content-Type: text/plain; charset="us-ascii" > > Hi Terry, > > I feel hierarchical collectives are slower compare to tuned one. I > had done some benchmark in the past specific to collectives, and this is > what i feel based on my observation. > > Regards > > Neeraj Chourasia (MTS) > Computational Research Laboratories Ltd. > (A wholly Owned Subsidiary of TATA SONS Ltd) > B-101, ICC Trade Towers, Senapati Bapat Road > Pune 411016 (Mah) INDIA > (O) +91-20-6620 9863 (Fax) +91-20-6620 9862 > M: +91.9225520634 ___ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users =-=-= Notice: The information contained in this e-mail message and/or attachments to it may contain confidential or privileged information. If you are not the intended recipient, any dissemination, use, review, distribution, printing or copying of the information contained in this e-mail message and/or attachments to it are strictly prohibited. If you have received this communication in error, please notify us by reply e-mail or telephone and immediately and permanently delete the message and any attachments. Internet communications cannot be guaranteed to be timely, secure, error or virus-free. The sender does not accept liability for any errors or omissions.Thank you =-=-= ___ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users
Re: [OMPI users] pipes system limit
Rolf Vandevaart wrote: This message is telling you that you have run out of file descriptors. I am surprised that the -mca parameter setting did not fix the problem. Can you run limit or ulimit on your shell and send the information? I typically set my limit to 65536 assuming the system allows it. burl-16 58 =>limit descriptors descriptors 65536 burl-16 59 => bash-3.00$ ulimit -n 65536 bash-3.00$ Rolf Thanks for the fast answer ! I've done : limit descriptors 1024 (csh style) And that's working fine. I took the linux descriptors by default. Thanks again, Yann On 08/07/09 11:21, Yann JOBIC wrote: Hello all, I'm using hpc8.2 : Lidia-jobic% ompi_info Displaying Open MPI information for 32-bit ... Package: ClusterTools 8.2 Open MPI: 1.3.3r21324-ct8.2-b09j-r40 [...] And i've got a X4600 machine (8*4 cores). When i'm trying to run a 32 processor jobs, i've got : Lidia-jobic% mpiexec --mca opal_set_max_sys_limits 1 -n 32 ./exe [Lidia:29384] [[61597,0],0] ORTE_ERROR_LOG: The system limit on number of pipes a process can open was reached in file base/iof_base_setup.c at line 112 [Lidia:29384] [[61597,0],0] ORTE_ERROR_LOG: The system limit on number of pipes a process can open was reached in file odls_default_module.c at line 203 [Lidia:29384] [[61597,0],0] ORTE_ERROR_LOG: The system limit on number of network connections a process can open was reached in file oob_tcp.c at line 446 -- Error: system limit exceeded on number of network connections that can be open This can be resolved by setting the mca parameter opal_set_max_sys_limits to 1, increasing your limit descriptor setting (using limit or ulimit commands), or asking the system administrator to increase the system limit. -- I tried the ulimit, the mca parameter, i've got no idea of where to look at. I've got the same computer under linux, and it's working fine... Have you seen it ? Do you know a way to bypass it ? Many thanks, Yann -- ___ Yann JOBIC HPC engineer Polytech Marseille DME IUSTI-CNRS UMR 6595 Technopôle de Château Gombert 5 rue Enrico Fermi 13453 Marseille cedex 13 Tel : (33) 4 91 10 69 39 ou (33) 4 91 10 69 43 Fax : (33) 4 91 10 69 69
Re: [OMPI users] pipes system limit
This message is telling you that you have run out of file descriptors. I am surprised that the -mca parameter setting did not fix the problem. Can you run limit or ulimit on your shell and send the information? I typically set my limit to 65536 assuming the system allows it. burl-16 58 =>limit descriptors descriptors 65536 burl-16 59 => bash-3.00$ ulimit -n 65536 bash-3.00$ Rolf On 08/07/09 11:21, Yann JOBIC wrote: Hello all, I'm using hpc8.2 : Lidia-jobic% ompi_info Displaying Open MPI information for 32-bit ... Package: ClusterTools 8.2 Open MPI: 1.3.3r21324-ct8.2-b09j-r40 [...] And i've got a X4600 machine (8*4 cores). When i'm trying to run a 32 processor jobs, i've got : Lidia-jobic% mpiexec --mca opal_set_max_sys_limits 1 -n 32 ./exe [Lidia:29384] [[61597,0],0] ORTE_ERROR_LOG: The system limit on number of pipes a process can open was reached in file base/iof_base_setup.c at line 112 [Lidia:29384] [[61597,0],0] ORTE_ERROR_LOG: The system limit on number of pipes a process can open was reached in file odls_default_module.c at line 203 [Lidia:29384] [[61597,0],0] ORTE_ERROR_LOG: The system limit on number of network connections a process can open was reached in file oob_tcp.c at line 446 -- Error: system limit exceeded on number of network connections that can be open This can be resolved by setting the mca parameter opal_set_max_sys_limits to 1, increasing your limit descriptor setting (using limit or ulimit commands), or asking the system administrator to increase the system limit. -- I tried the ulimit, the mca parameter, i've got no idea of where to look at. I've got the same computer under linux, and it's working fine... Have you seen it ? Do you know a way to bypass it ? Many thanks, Yann -- = rolf.vandeva...@sun.com 781-442-3043 =
[OMPI users] pipes system limit
Hello all, I'm using hpc8.2 : Lidia-jobic% ompi_info Displaying Open MPI information for 32-bit ... Package: ClusterTools 8.2 Open MPI: 1.3.3r21324-ct8.2-b09j-r40 [...] And i've got a X4600 machine (8*4 cores). When i'm trying to run a 32 processor jobs, i've got : Lidia-jobic% mpiexec --mca opal_set_max_sys_limits 1 -n 32 ./exe [Lidia:29384] [[61597,0],0] ORTE_ERROR_LOG: The system limit on number of pipes a process can open was reached in file base/iof_base_setup.c at line 112 [Lidia:29384] [[61597,0],0] ORTE_ERROR_LOG: The system limit on number of pipes a process can open was reached in file odls_default_module.c at line 203 [Lidia:29384] [[61597,0],0] ORTE_ERROR_LOG: The system limit on number of network connections a process can open was reached in file oob_tcp.c at line 446 -- Error: system limit exceeded on number of network connections that can be open This can be resolved by setting the mca parameter opal_set_max_sys_limits to 1, increasing your limit descriptor setting (using limit or ulimit commands), or asking the system administrator to increase the system limit. -- I tried the ulimit, the mca parameter, i've got no idea of where to look at. I've got the same computer under linux, and it's working fine... Have you seen it ? Do you know a way to bypass it ? Many thanks, Yann -- ___ Yann JOBIC HPC engineer Polytech Marseille DME IUSTI-CNRS UMR 6595 Technopôle de Château Gombert 5 rue Enrico Fermi 13453 Marseille cedex 13 Tel : (33) 4 91 10 69 39 ou (33) 4 91 10 69 43 Fax : (33) 4 91 10 69 69
[OMPI users] Failure trying to use tuned collectives
To use tuned collectives, do all I have to do is add --mca coll tuned? I am trying to run with: # mpirun -np 8 --mca coll tuned --mca orte_base_help_aggregate 0 ./wrf.exe But all the processes fail with the folling message: -- It looks like MPI_INIT failed for some reason; your parallel process is likely to abort. There are many reasons that a parallel process can fail during MPI_INIT; some of which are due to configuration or environment problems. This failure appears to be an internal failure; here's some additional information (which may only be relevant to an Open MPI developer): mca_coll_base_comm_select(MPI_COMM_WORLD) failed --> Returned "Not found" (-13) instead of "Success" (0) -- Thanks, Craig
Re: [OMPI users] Performance question about OpenMPI and MVAPICH2 on IB
Terry Dontje wrote: Craig, Did your affinity script bind the processes per socket or linearly to cores. If the former you'll want to look at using rankfiles and place the ranks based on sockets. TWe have found this especially useful if you are not running fully subscribed on your machines. Also, if you think the main issue is collectives performance you may want to try using the hierarchical and SM collectives. However, be forewarned we are right now trying to pound out some errors with these modules. To enable them you add the following parameters "--mca coll_hierarch_priority 100 --mca coll_sm_priority 100". We would be very interested in any results you get (failures, improvements, non-improvements). Adding these two options causes the code to segfault at startup. Craig thanks, --td Message: 4 Date: Thu, 06 Aug 2009 17:03:08 -0600 From: Craig Tierney Subject: Re: [OMPI users] Performance question about OpenMPI and MVAPICH2 onIB To: Open MPI Users Message-ID: <4a7b612c.8070...@noaa.gov> Content-Type: text/plain; charset=ISO-8859-1 A followup Part of problem was affinity. I had written a script to do processor and memory affinity (which works fine with MVAPICH2). It is an idea that I got from TACC. However, the script didn't seem to work correctly with OpenMPI (or I still have bugs). Setting --mca mpi_paffinity_alone 1 made things better. However, the performance is still not as good: Cores Mvapich2Openmpi --- 8 17.317.3 16 31.731.5 32 62.962.8 64 110.8 108.0 128 219.2 201.4 256 384.5 342.7 512 687.2 537.6 The performance number is GFlops (so larger is better). The first few numbers show that the executable is the right speed. I verified that IB is being used by using OMB and checking latency and bandwidth. Those numbers are what I expect (3GB/s, 1.5mu/s for QDR). However, the Openmpi version is not scaling as well. Any ideas on why that might be the case? Thanks, Craig ___ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users
Re: [OMPI users] Performance question about OpenMPI and MVAPICH2 on IB
Date: Fri, 07 Aug 2009 07:12:45 -0600 From: Craig Tierney Subject: Re: [OMPI users] Performance question about OpenMPI and MVAPICH2 on IB To: Open MPI Users Message-ID: <4a7c284d.3040...@noaa.gov> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Terry Dontje wrote: > Craig, > > Did your affinity script bind the processes per socket or linearly to > cores. If the former you'll want to look at using rankfiles and place > the ranks based on sockets. TWe have found this especially useful if > you are not running fully subscribed on your machines. > The script binds them to sockets and also binds memory per node. It is smart enough that if the machine_file does not use all the cores (because the user reordered them) then the script will lay out the tasks evenly between the two sockets. Ok, so you'll probably want to look at using rankfile (described in the mpirun manpage) because mpi_paffinity_alone just does a linear binding (rank 0 to cpu0, rank 1 to cpu 1...). > Also, if you think the main issue is collectives performance you may > want to try using the hierarchical and SM collectives. However, be > forewarned we are right now trying to pound out some errors with these > modules. To enable them you add the following parameters "--mca > coll_hierarch_priority 100 --mca coll_sm_priority 100". We would be > very interested in any results you get (failures, improvements, > non-improvements). > I don't know what it is slow. OpenMPI is so flexible in how the stack can be tuned. But I also have 100s of users runing dozens of major codes, and what I need is a set of options that 'just work' in most cases. I will try the above options and get back to you. Ok, thanks. --td
Re: [OMPI users] init failing
I'm way behind on my mail; apologies for the delay in replying. Did you figure this out? As a pure guess, it sounds like you have a heterogeneous setup -- nodes have different distros and/or versions. As such, your glibc's may be different, etc. In such situations, it is definitely recommended to have a separate installation of Open MPI *on each node* (i.e., compiled/built for that distro/version/platform). If you're copying the files from system A to system B and A and B are different distros/versions, it could be a good reason why it fails to work. Hope that helps. On Jul 28, 2009, at 4:07 AM, Dominik Táborský wrote: Hi everyone, I am trying to build my own system for my nodes - minimalistic. I tried to make things easy so I didn't even recompile openMPI for it, I just copied everything from my Ubuntu installation (I know, it's very dirty, but I stick to KISS :) ). Before, things just worked perfectly with the libraries. I only recompile executable binaries, not Glibc (not openMPI, I also didn't succeed compiling openMPI but that's a different story). So, as I keep trying to run Hello world! program, I keep getting the same error message every time. Everything in the system is fine from my point of view. The error message is this: [user:24307] mca: base: components_open: component timer / linux open function failed -- It looks like opal_init failed for some reason; your parallel process is likely to abort. There are many reasons that a parallel process can fail during opal_init; some of which are due to configuration or environment problems. This failure appears to be an internal failure; here's some additional information (which may only be relevant to an Open MPI developer): opal_carto_base_select failed --> Returned value -13 instead of OPAL_SUCCESS -- [user:24307] [[INVALID],INVALID] ORTE_ERROR_LOG: Not found in file ../../../orte/runtime/orte_init.c at line 77 [user:24307] [[INVALID],INVALID] ORTE_ERROR_LOG: Not found in file ../../../orte/orted/orted_main.c at line 315 I tried googling and searching the archives, nothing gave me a hint. What might be missing? Should I really try to recompile openMPI? What needs to be on/off in the kernel? Any ideas? Thanks in advance, bremby ___ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users -- Jeff Squyres jsquy...@cisco.com
Re: [OMPI users] Performance question about OpenMPI and MVAPICH2 on IB
nee...@crlindia.com wrote: Hi Craig, How was the nodefile selected for execution? Whether it was provided by scheduler say LSF/SGE/PBS or you manually gave it? With WRF, we observed giving sequential nodes (Blades which are in the same order as in enclosure) gave us some performance benefit. Regards I figured this might be the case. Right now the batch system is giving the nodes to the applciation. They are not sorted, and I have considered doing that. I have also launched numerous cases of one problems size, and I don't get that much variation in run time, not to explain the differences in MPI stack. Craig Neeraj Chourasia (MTS) Computational Research Laboratories Ltd. (A wholly Owned Subsidiary of TATA SONS Ltd) B-101, ICC Trade Towers, Senapati Bapat Road Pune 411016 (Mah) INDIA (O) +91-20-6620 9863 (Fax) +91-20-6620 9862 M: +91.9225520634 *Craig Tierney * Sent by: users-boun...@open-mpi.org 08/07/2009 04:43 AM Please respond to Open MPI Users To Open MPI Users cc Subject Re: [OMPI users] Performance question about OpenMPI and MVAPICH2 on IB Gus Correa wrote: > Hi Craig, list > > I suppose WRF uses MPI collective calls (MPI_Reduce, > MPI_Bcast, MPI_Alltoall etc), > just like the climate models we run here do. > A recursive grep on the source code will tell. > I will check this out. I am not the WRF expert, but I was under the impression that most weather models are nearest neighbor communications, not collectives. > If that is the case, you may need to tune the collectives dynamically. > We are experimenting with tuned collectives here also. > > Specifically, we had a scaling problem with the MITgcm > (also running on an IB cluster) > that is probably due to collectives. > Similar problems were reported on this list before, > with computational chemistry software. > See these threads: > http://www.open-mpi.org/community/lists/users/2009/07/10045.php > http://www.open-mpi.org/community/lists/users/2009/05/9419.php > > If WRF outputs timing information, particularly the time spent on MPI > routines, you may also want to compare how the OpenMPI and > MVAPICH versions fare w.r.t. MPI collectives. > > I hope this helps. > I will look into this. Thanks for the ideas. Craig > Gus Correa > - > Gustavo Correa > Lamont-Doherty Earth Observatory - Columbia University > Palisades, NY, 10964-8000 - USA > - > > > > Craig Tierney wrote: >> I am running openmpi-1.3.3 on my cluster which is using >> OFED-1.4.1 for Infiniband support. I am comparing performance >> between this version of OpenMPI and Mvapich2, and seeing a >> very large difference in performance. >> >> The code I am testing is WRF v3.0.1. I am running the >> 12km benchmark. >> >> The two builds are the exact same codes and configuration >> files. All I did different was use modules to switch versions >> of MPI, and recompiled the code. >> >> Performance: >> >> Cores Mvapich2Openmpi >> --- >>8 17.313.9 >> 16 31.725.9 >> 32 62.951.6 >> 64 110.892.8 >> 128 219.2 189.4 >> 256 384.5 317.8 >> 512 687.2 516.7 >> >> The performance number is GFlops (so larger is better). >> >> I am calling openmpi as: >> >> /opt/openmpi/1.3.3-intel/bin/mpirun --mca plm_rsh_disable_qrsh 1 >> --mca btl openib,sm,self \ >> -machinefile /tmp/6026489.1.qntest.q/machines -x LD_LIBRARY_PATH -np >> $NSLOTS /home/ctierney/bin/noaa_affinity ./wrf.exe >> >> So, >> >> Is this expected? Are some common sense optimizations to use? >> Is there a way to verify that I am really using the IB? When >> I try: >> >> -mca bta ^tcp,openib,sm,self >> >> I get the errors: >> -- >> >> No available btl components were found! >> >> This means that there are no components of this type installed on your >> system or all the components reported that they could not be used. >> >> This is a fatal error; your MPI process is likely to abort. Check the >> output of the "ompi_info" command and ensure that components of this >> type are available on your system. You may also wish to check the >> value of the "component_path" MCA parameter and ensure that it has at >> least one directory that contains valid MCA components. >> -- >> >> >> But ompi_info is telling me that I have openib support: >> >>MCA btl: openib (MCA v2.0, API v2.0, Component v1.3.3) >> >> Note, I did rebuild OFED and put it in a different directory >> and did not rebuild OpenMPI. However, since ompi_info isn't >> complaining and the libraries
Re: [OMPI users] Performance question about OpenMPI and MVAPICH2 on IB
Terry Dontje wrote: Craig, Did your affinity script bind the processes per socket or linearly to cores. If the former you'll want to look at using rankfiles and place the ranks based on sockets. TWe have found this especially useful if you are not running fully subscribed on your machines. The script binds them to sockets and also binds memory per node. It is smart enough that if the machine_file does not use all the cores (because the user reordered them) then the script will lay out the tasks evenly between the two sockets. Also, if you think the main issue is collectives performance you may want to try using the hierarchical and SM collectives. However, be forewarned we are right now trying to pound out some errors with these modules. To enable them you add the following parameters "--mca coll_hierarch_priority 100 --mca coll_sm_priority 100". We would be very interested in any results you get (failures, improvements, non-improvements). I don't know what it is slow. OpenMPI is so flexible in how the stack can be tuned. But I also have 100s of users runing dozens of major codes, and what I need is a set of options that 'just work' in most cases. I will try the above options and get back to you. Craig thanks, --td Message: 4 Date: Thu, 06 Aug 2009 17:03:08 -0600 From: Craig Tierney Subject: Re: [OMPI users] Performance question about OpenMPI and MVAPICH2 onIB To: Open MPI Users Message-ID: <4a7b612c.8070...@noaa.gov> Content-Type: text/plain; charset=ISO-8859-1 A followup Part of problem was affinity. I had written a script to do processor and memory affinity (which works fine with MVAPICH2). It is an idea that I got from TACC. However, the script didn't seem to work correctly with OpenMPI (or I still have bugs). Setting --mca mpi_paffinity_alone 1 made things better. However, the performance is still not as good: Cores Mvapich2Openmpi --- 8 17.317.3 16 31.731.5 32 62.962.8 64 110.8 108.0 128 219.2 201.4 256 384.5 342.7 512 687.2 537.6 The performance number is GFlops (so larger is better). The first few numbers show that the executable is the right speed. I verified that IB is being used by using OMB and checking latency and bandwidth. Those numbers are what I expect (3GB/s, 1.5mu/s for QDR). However, the Openmpi version is not scaling as well. Any ideas on why that might be the case? Thanks, Craig ___ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users
Re: [OMPI users] Performance question about OpenMPI and MVAPICH2 on IB
Hi Terry, I had tested mostly MPI_Bcast, MPI_Reduce, MPI_Gather kind of MPI with openmpi-1.3 and hierarchical option enabled.In all these, i found results slower than regular tuned collectives. We have HP Blade with intel clovertown processor(two quad core) connected with DDR infiniband clos network. Results were tested on 12-16 nodes with 8 mpi process each node. Regards Neeraj Chourasia (MTS) Computational Research Laboratories Ltd. (A wholly Owned Subsidiary of TATA SONS Ltd) B-101, ICC Trade Towers, Senapati Bapat Road Pune 411016 (Mah) INDIA (O) +91-20-6620 9863 (Fax) +91-20-6620 9862 M: +91.9225520634 Terry Dontje Sent by: users-boun...@open-mpi.org 08/07/2009 05:15 PM Please respond to Open MPI Users To us...@open-mpi.org cc Subject Re: [OMPI users] Performance question about OpenMPI and MVAPICH2on IB Hi Neeraj, Were there specific collectives that were slower? Also what kind of cluster were you running on? How many nodes and cores per node? thanks, --td > Message: 3 > Date: Fri, 7 Aug 2009 16:51:05 +0530 > From: nee...@crlindia.com > Subject: Re: [OMPI users] Performance question about OpenMPI and >MVAPICH2on IB > To: Open MPI Users > Cc: us...@open-mpi.org, users-boun...@open-mpi.org > Message-ID: > > > Content-Type: text/plain; charset="us-ascii" > > Hi Terry, > > I feel hierarchical collectives are slower compare to tuned one. I > had done some benchmark in the past specific to collectives, and this is > what i feel based on my observation. > > Regards > > Neeraj Chourasia (MTS) > Computational Research Laboratories Ltd. > (A wholly Owned Subsidiary of TATA SONS Ltd) > B-101, ICC Trade Towers, Senapati Bapat Road > Pune 411016 (Mah) INDIA > (O) +91-20-6620 9863 (Fax) +91-20-6620 9862 > M: +91.9225520634 ___ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users =-=-= Notice: The information contained in this e-mail message and/or attachments to it may contain confidential or privileged information. If you are not the intended recipient, any dissemination, use, review, distribution, printing or copying of the information contained in this e-mail message and/or attachments to it are strictly prohibited. If you have received this communication in error, please notify us by reply e-mail or telephone and immediately and permanently delete the message and any attachments. Internet communications cannot be guaranteed to be timely, secure, error or virus-free. The sender does not accept liability for any errors or omissions.Thank you =-=-=
Re: [OMPI users] Performance question about OpenMPI and MVAPICH2 on IB
Hi Neeraj, Were there specific collectives that were slower? Also what kind of cluster were you running on? How many nodes and cores per node? thanks, --td Message: 3 Date: Fri, 7 Aug 2009 16:51:05 +0530 From: nee...@crlindia.com Subject: Re: [OMPI users] Performance question about OpenMPI and MVAPICH2on IB To: Open MPI Users Cc: us...@open-mpi.org, users-boun...@open-mpi.org Message-ID: Content-Type: text/plain; charset="us-ascii" Hi Terry, I feel hierarchical collectives are slower compare to tuned one. I had done some benchmark in the past specific to collectives, and this is what i feel based on my observation. Regards Neeraj Chourasia (MTS) Computational Research Laboratories Ltd. (A wholly Owned Subsidiary of TATA SONS Ltd) B-101, ICC Trade Towers, Senapati Bapat Road Pune 411016 (Mah) INDIA (O) +91-20-6620 9863 (Fax) +91-20-6620 9862 M: +91.9225520634
Re: [OMPI users] Performance question about OpenMPI and MVAPICH2 on IB
Hi Craig, How was the nodefile selected for execution? Whether it was provided by scheduler say LSF/SGE/PBS or you manually gave it? With WRF, we observed giving sequential nodes (Blades which are in the same order as in enclosure) gave us some performance benefit. Regards Neeraj Chourasia (MTS) Computational Research Laboratories Ltd. (A wholly Owned Subsidiary of TATA SONS Ltd) B-101, ICC Trade Towers, Senapati Bapat Road Pune 411016 (Mah) INDIA (O) +91-20-6620 9863 (Fax) +91-20-6620 9862 M: +91.9225520634 Craig Tierney Sent by: users-boun...@open-mpi.org 08/07/2009 04:43 AM Please respond to Open MPI Users To Open MPI Users cc Subject Re: [OMPI users] Performance question about OpenMPI and MVAPICH2 on IB Gus Correa wrote: > Hi Craig, list > > I suppose WRF uses MPI collective calls (MPI_Reduce, > MPI_Bcast, MPI_Alltoall etc), > just like the climate models we run here do. > A recursive grep on the source code will tell. > I will check this out. I am not the WRF expert, but I was under the impression that most weather models are nearest neighbor communications, not collectives. > If that is the case, you may need to tune the collectives dynamically. > We are experimenting with tuned collectives here also. > > Specifically, we had a scaling problem with the MITgcm > (also running on an IB cluster) > that is probably due to collectives. > Similar problems were reported on this list before, > with computational chemistry software. > See these threads: > http://www.open-mpi.org/community/lists/users/2009/07/10045.php > http://www.open-mpi.org/community/lists/users/2009/05/9419.php > > If WRF outputs timing information, particularly the time spent on MPI > routines, you may also want to compare how the OpenMPI and > MVAPICH versions fare w.r.t. MPI collectives. > > I hope this helps. > I will look into this. Thanks for the ideas. Craig > Gus Correa > - > Gustavo Correa > Lamont-Doherty Earth Observatory - Columbia University > Palisades, NY, 10964-8000 - USA > - > > > > Craig Tierney wrote: >> I am running openmpi-1.3.3 on my cluster which is using >> OFED-1.4.1 for Infiniband support. I am comparing performance >> between this version of OpenMPI and Mvapich2, and seeing a >> very large difference in performance. >> >> The code I am testing is WRF v3.0.1. I am running the >> 12km benchmark. >> >> The two builds are the exact same codes and configuration >> files. All I did different was use modules to switch versions >> of MPI, and recompiled the code. >> >> Performance: >> >> Cores Mvapich2Openmpi >> --- >>8 17.313.9 >> 16 31.725.9 >> 32 62.951.6 >> 64 110.892.8 >> 128 219.2 189.4 >> 256 384.5 317.8 >> 512 687.2 516.7 >> >> The performance number is GFlops (so larger is better). >> >> I am calling openmpi as: >> >> /opt/openmpi/1.3.3-intel/bin/mpirun --mca plm_rsh_disable_qrsh 1 >> --mca btl openib,sm,self \ >> -machinefile /tmp/6026489.1.qntest.q/machines -x LD_LIBRARY_PATH -np >> $NSLOTS /home/ctierney/bin/noaa_affinity ./wrf.exe >> >> So, >> >> Is this expected? Are some common sense optimizations to use? >> Is there a way to verify that I am really using the IB? When >> I try: >> >> -mca bta ^tcp,openib,sm,self >> >> I get the errors: >> -- >> >> No available btl components were found! >> >> This means that there are no components of this type installed on your >> system or all the components reported that they could not be used. >> >> This is a fatal error; your MPI process is likely to abort. Check the >> output of the "ompi_info" command and ensure that components of this >> type are available on your system. You may also wish to check the >> value of the "component_path" MCA parameter and ensure that it has at >> least one directory that contains valid MCA components. >> -- >> >> >> But ompi_info is telling me that I have openib support: >> >>MCA btl: openib (MCA v2.0, API v2.0, Component v1.3.3) >> >> Note, I did rebuild OFED and put it in a different directory >> and did not rebuild OpenMPI. However, since ompi_info isn't >> complaining and the libraries are available, I am thinking that >> is isn't a problem. I could be wrong. >> >> Thanks, >> Craig > > ___ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users > -- Craig Tierney (craig.tier...@noaa.gov) ___ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users =-=-= Notice: The information co
Re: [OMPI users] Performance question about OpenMPI and MVAPICH2 on IB
Hi Terry, I feel hierarchical collectives are slower compare to tuned one. I had done some benchmark in the past specific to collectives, and this is what i feel based on my observation. Regards Neeraj Chourasia (MTS) Computational Research Laboratories Ltd. (A wholly Owned Subsidiary of TATA SONS Ltd) B-101, ICC Trade Towers, Senapati Bapat Road Pune 411016 (Mah) INDIA (O) +91-20-6620 9863 (Fax) +91-20-6620 9862 M: +91.9225520634 Terry Dontje Sent by: users-boun...@open-mpi.org 08/07/2009 04:35 PM Please respond to Open MPI Users To us...@open-mpi.org cc Subject Re: [OMPI users] Performance question about OpenMPI and MVAPICH2on IB Craig, Did your affinity script bind the processes per socket or linearly to cores. If the former you'll want to look at using rankfiles and place the ranks based on sockets. TWe have found this especially useful if you are not running fully subscribed on your machines. Also, if you think the main issue is collectives performance you may want to try using the hierarchical and SM collectives. However, be forewarned we are right now trying to pound out some errors with these modules. To enable them you add the following parameters "--mca coll_hierarch_priority 100 --mca coll_sm_priority 100". We would be very interested in any results you get (failures, improvements, non-improvements). thanks, --td > Message: 4 > Date: Thu, 06 Aug 2009 17:03:08 -0600 > From: Craig Tierney > Subject: Re: [OMPI users] Performance question about OpenMPI and >MVAPICH2 on IB > To: Open MPI Users > Message-ID: <4a7b612c.8070...@noaa.gov> > Content-Type: text/plain; charset=ISO-8859-1 > > A followup > > Part of problem was affinity. I had written a script to do processor > and memory affinity (which works fine with MVAPICH2). It is an > idea that I got from TACC. However, the script didn't seem to > work correctly with OpenMPI (or I still have bugs). > > Setting --mca mpi_paffinity_alone 1 made things better. However, > the performance is still not as good: > > Cores Mvapich2Openmpi > --- >8 17.317.3 > 16 31.731.5 > 32 62.962.8 > 64 110.8 108.0 > 128 219.2 201.4 > 256 384.5 342.7 > 512 687.2 537.6 > > The performance number is GFlops (so larger is better). > > The first few numbers show that the executable is the right > speed. I verified that IB is being used by using OMB and > checking latency and bandwidth. Those numbers are what I > expect (3GB/s, 1.5mu/s for QDR). > > However, the Openmpi version is not scaling as well. Any > ideas on why that might be the case? > > Thanks, > Craig ___ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users =-=-= Notice: The information contained in this e-mail message and/or attachments to it may contain confidential or privileged information. If you are not the intended recipient, any dissemination, use, review, distribution, printing or copying of the information contained in this e-mail message and/or attachments to it are strictly prohibited. If you have received this communication in error, please notify us by reply e-mail or telephone and immediately and permanently delete the message and any attachments. Internet communications cannot be guaranteed to be timely, secure, error or virus-free. The sender does not accept liability for any errors or omissions.Thank you =-=-=
Re: [OMPI users] Performance question about OpenMPI and MVAPICH2 on IB
Craig, Did your affinity script bind the processes per socket or linearly to cores. If the former you'll want to look at using rankfiles and place the ranks based on sockets. TWe have found this especially useful if you are not running fully subscribed on your machines. Also, if you think the main issue is collectives performance you may want to try using the hierarchical and SM collectives. However, be forewarned we are right now trying to pound out some errors with these modules. To enable them you add the following parameters "--mca coll_hierarch_priority 100 --mca coll_sm_priority 100". We would be very interested in any results you get (failures, improvements, non-improvements). thanks, --td Message: 4 Date: Thu, 06 Aug 2009 17:03:08 -0600 From: Craig Tierney Subject: Re: [OMPI users] Performance question about OpenMPI and MVAPICH2 on IB To: Open MPI Users Message-ID: <4a7b612c.8070...@noaa.gov> Content-Type: text/plain; charset=ISO-8859-1 A followup Part of problem was affinity. I had written a script to do processor and memory affinity (which works fine with MVAPICH2). It is an idea that I got from TACC. However, the script didn't seem to work correctly with OpenMPI (or I still have bugs). Setting --mca mpi_paffinity_alone 1 made things better. However, the performance is still not as good: Cores Mvapich2Openmpi --- 8 17.317.3 16 31.731.5 32 62.962.8 64 110.8 108.0 128 219.2 201.4 256 384.5 342.7 512 687.2 537.6 The performance number is GFlops (so larger is better). The first few numbers show that the executable is the right speed. I verified that IB is being used by using OMB and checking latency and bandwidth. Those numbers are what I expect (3GB/s, 1.5mu/s for QDR). However, the Openmpi version is not scaling as well. Any ideas on why that might be the case? Thanks, Craig
[OMPI users] Automated tuning tool
There's been quite some discussion here lately about the effect of OMPI tuning parameters, particularly w.r.t. collectives. Is there some tool to probe performance on any given network/collection of nodes to aid optimisation of these parameters? (I'm thinking something along the philosophy of ATLAS.)
Re: [OMPI users] Performance question about OpenMPI and MVAPICH2 on IB
Hi Craig, WRF has pattern of talking to nearest neighbours like p+1, p-1, p+3 and p-3, where p is the particular process. But in addition to that, it also uses collective calls like MPI_Bcast, MPI_AlltoAllv, MPI_Allgather, MPI_Gather, MPI_Gatherv, MPI_Scatterv. Apparently openmpi-1.3 series are not better in terms of collectives as compare to its 1.2 series. But there are lot of parameters which has been added to tune collectives like giving dynamic file option which would override openmpi default selection of algorithm for particular collective operation. Since collectives depend heavily on your network architecture and message size, i would like you to first fine tune your collectives on your network fabric before running any scientific application. Regards Neeraj Chourasia (MTS) Computational Research Laboratories Ltd. (A wholly Owned Subsidiary of TATA SONS Ltd) B-101, ICC Trade Towers, Senapati Bapat Road Pune 411016 (Mah) INDIA (O) +91-20-6620 9863 (Fax) +91-20-6620 9862 M: +91.9225520634 Craig Tierney Sent by: users-boun...@open-mpi.org 08/07/2009 04:43 AM Please respond to Open MPI Users To Open MPI Users cc Subject Re: [OMPI users] Performance question about OpenMPI and MVAPICH2 on IB Gus Correa wrote: > Hi Craig, list > > I suppose WRF uses MPI collective calls (MPI_Reduce, > MPI_Bcast, MPI_Alltoall etc), > just like the climate models we run here do. > A recursive grep on the source code will tell. > I will check this out. I am not the WRF expert, but I was under the impression that most weather models are nearest neighbor communications, not collectives. > If that is the case, you may need to tune the collectives dynamically. > We are experimenting with tuned collectives here also. > > Specifically, we had a scaling problem with the MITgcm > (also running on an IB cluster) > that is probably due to collectives. > Similar problems were reported on this list before, > with computational chemistry software. > See these threads: > http://www.open-mpi.org/community/lists/users/2009/07/10045.php > http://www.open-mpi.org/community/lists/users/2009/05/9419.php > > If WRF outputs timing information, particularly the time spent on MPI > routines, you may also want to compare how the OpenMPI and > MVAPICH versions fare w.r.t. MPI collectives. > > I hope this helps. > I will look into this. Thanks for the ideas. Craig > Gus Correa > - > Gustavo Correa > Lamont-Doherty Earth Observatory - Columbia University > Palisades, NY, 10964-8000 - USA > - > > > > Craig Tierney wrote: >> I am running openmpi-1.3.3 on my cluster which is using >> OFED-1.4.1 for Infiniband support. I am comparing performance >> between this version of OpenMPI and Mvapich2, and seeing a >> very large difference in performance. >> >> The code I am testing is WRF v3.0.1. I am running the >> 12km benchmark. >> >> The two builds are the exact same codes and configuration >> files. All I did different was use modules to switch versions >> of MPI, and recompiled the code. >> >> Performance: >> >> Cores Mvapich2Openmpi >> --- >>8 17.313.9 >> 16 31.725.9 >> 32 62.951.6 >> 64 110.892.8 >> 128 219.2 189.4 >> 256 384.5 317.8 >> 512 687.2 516.7 >> >> The performance number is GFlops (so larger is better). >> >> I am calling openmpi as: >> >> /opt/openmpi/1.3.3-intel/bin/mpirun --mca plm_rsh_disable_qrsh 1 >> --mca btl openib,sm,self \ >> -machinefile /tmp/6026489.1.qntest.q/machines -x LD_LIBRARY_PATH -np >> $NSLOTS /home/ctierney/bin/noaa_affinity ./wrf.exe >> >> So, >> >> Is this expected? Are some common sense optimizations to use? >> Is there a way to verify that I am really using the IB? When >> I try: >> >> -mca bta ^tcp,openib,sm,self >> >> I get the errors: >> -- >> >> No available btl components were found! >> >> This means that there are no components of this type installed on your >> system or all the components reported that they could not be used. >> >> This is a fatal error; your MPI process is likely to abort. Check the >> output of the "ompi_info" command and ensure that components of this >> type are available on your system. You may also wish to check the >> value of the "component_path" MCA parameter and ensure that it has at >> least one directory that contains valid MCA components. >> -- >> >> >> But ompi_info is telling me that I have openib support: >> >>MCA btl: openib (MCA v2.0, API v2.0, Component v1.3.3) >> >> Note, I did rebuild OFED and put it in a different directory >> and did not rebuild OpenMPI. However, since ompi_info isn't >
Re: [OMPI users] Bus Error (7) on PS3 running HPL (OpenMPI 1.2.8)
may be because of insufficient storage space...???( I mean hard disk space.) On Thu, Aug 6, 2009 at 11:23 PM, Jeff Squyres wrote: > Any chance you could re-try the experiment with Open MPI 1.3.3? > > > On Aug 4, 2009, at 11:10 AM, Hoelzlwimmer Andreas - S0810595005 wrote: > > Hello, >> >> I’ve wanted to run MPI on a couple of PS3 here. According to a colleague >> who set it up, I had to set several HugePages. As the PS3 RAM is limited I >> had to allocate 2 HugePages. I ran HPL at first with the following command >> (out of a tutorial): >> mpirun --mca btl_openib_want_fork_support 0 -np 1 numactl --physcpubind=0 >> ./xhpl : -np 1 numactl --physcpubind=1 ./xhpl >> >> Now as I had very little memory I had to disable some services. I did so >> (Wifi Service, Bluetooth, printing, unneeded). After running the same >> command again, I got the an error message (see below). Can anyone help me >> here, I have no idea what the error message actually means, and I can’t find >> anything useful about it. It’s running on Yellow Dog Linux, using OpenMPI >> 1.2.8 >> >> Cheers, >> Andreas Hoelzlwimmer >> >> Error Message: >> [PS02:04815] *** Process received signal *** >> [PS02:04815] Signal: Bus error (7) >> [PS02:04815] Signal code: (2) >> [PS02:04815] Failing at address: 0x4000ca78008 >> [PS02:04816] *** Process received signal *** >> [PS02:04816] Signal: Bus error (7) >> [PS02:04816] Signal code: (2) >> [PS02:04816] Failing at address: 0x4000ca78008 >> [PS02:04816] [ 0] [0x1003e8] >> [PS02:04816] [ 1] ./xhpl(HPL_hpalloc-0x17cc8c) [0x1001103c] >> [PS02:04816] [ 2] ./xhpl(HPL_pdtest-0x17da40) [0x100101f8] >> [PS02:04816] [ 3] ./xhpl(main-0x182f2c) [0x1000acdc] >> [PS02:04816] [ 4] /lib64/libc.so.6 [0x80ca0e966c] >> [PS02:04816] [ 5] /lib64/libc.so.6(__libc_start_main-0x1473e0) >> [0x80ca0e98e8] >> [PS02:04816] *** End of error message *** >> [PS02:04815] [ 0] [0x1003e8] >> [PS02:04815] [ 1] ./xhpl(HPL_hpalloc-0x17cc8c) [0x1001103c] >> [PS02:04815] [ 2] ./xhpl(HPL_pdtest-0x17da40) [0x100101f8] >> [PS02:04815] [ 3] ./xhpl(main-0x182f2c) [0x1000acdc] >> [PS02:04815] [ 4] /lib64/libc.so.6 [0x80ca0e966c] >> [PS02:04815] [ 5] /lib64/libc.so.6(__libc_start_main-0x1473e0) >> [0x80ca0e98e8] >> [PS02:04815] *** End of error message *** >> mpirun noticed that job rank 0 with PID 4815 on node PS02 exited on signal >> 7 (Bus error). >> 1 additional process aborted (not shown) >> ___ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users >> > > > -- > Jeff Squyres > jsquy...@cisco.com > > > ___ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users > -- Vipin K. Research Engineer, C-DOTB, India
