Re: [OMPI users] slowdown with infiniband and latest CentOS kernel
On Wed, 2013-12-18 at 11:47 -0500, Noam Bernstein wrote: > Yes - I never characterized it fully, but we attached with gdb to every > single vasp running process, and all were stuck in the same > call to MPI_allreduce() every time. It's only happening on a rather large > jobs, so it's not the easiest setup to debug. That sounds like one of the bugs i found i VASP. Could you send me the input data that triggers this (with info on how it was run, i.e. #mpi-tasks etc) and i can check if our heavily fixed version hits it. /Åke S.
[OMPI users] typo in opal/memoryhooks/memory.h (1.6.5)
Hi! Not sure if this has been caught already or not, but there is a typo in opal/memoryhooks/memory.h in 1.6.5. #ifndef OPAL_MEMORY_MEMORY_H #define OPAl_MEMORY_MEMORY_H Note the lower case "l" in the define. /Åke S.
Re: [OMPI users] Calling MPI_send MPI_recv from a fortran subroutine
On Fri, 2013-03-01 at 01:24 +0900, Pradeep Jha wrote: > Sorry for those mistakes. I addressed all the three problems > - I put "implicit none" at the top of main program > - I initialized tag. > - changed MPI_INT to MPI_INTEGER > - "send_length" should be just "send", it was a typo. > > > But the code is still hanging in sendrecv. The present form is below: > "tag" isn't iniitalized to anything so it may very well be totally different in all the processes. ALWAYS initialize variables before using them. > main.f > > > program main > > implicit none > > include 'mpif.h' > > integer me, np, ierror > > call MPI_init( ierror ) > call MPI_comm_rank( mpi_comm_world, me, ierror ) > call MPI_comm_size( mpi_comm_world, np, ierror ) > > call sendrecv(me, np) > > call mpi_finalize( ierror ) > > stop > end > > sendrecv.f > > > subroutine sendrecv(me, np) > > include 'mpif.h' > > integer np, me, sender, tag > integer, dimension(mpi_status_size) :: status > > integer, dimension(1) :: recv, send > > if (me.eq.0) then > > do sender = 1, np-1 > call mpi_recv(recv, 1, mpi_integer, sender, tag, > & mpi_comm_world, status, ierror) > > end do > end if > > if ((me.ge.1).and.(me.lt.np)) then > send(1) = me*12 > > call mpi_send(send, 1, mpi_integer, 0, tag, > &mpi_comm_world, ierror) > end if > > return > end
[OMPI users] libmpi_f90 shared lib version number change in 1.6.3
Hi! Was the change for libmpi_f90 in VERSION intentional or a typo? This is from openmpi 1.6.3 libmpi_f90_so_version=4:0:1 1.6.1 had libmpi_f90_so_version=2:0:1 -- Ake Sandgren, HPC2N, Umea University, S-90187 Umea, Sweden Internet: a...@hpc2n.umu.se Phone: +46 90 7866134 Fax: +46 90 7866126 Mobile: +46 70 7716134 WWW: http://www.hpc2n.umu.se
Re: [OMPI users] grpcomm component hier gone...
On Thu, 2013-01-03 at 07:14 -0800, Ralph Castain wrote: > > Well, it isn't :-) > > configure says: > > --- MCA component grpcomm:pmi (m4 configuration macro) > > checking for MCA component grpcomm:pmi compile mode... dso > > checking if user requested PMI support... no > > checking if MCA component grpcomm:pmi can compile... no > > Ah - that is the problem. You need to configure > --with-pmi= Ahh thanks. Was assuming i needed something like that. > > Not sure what you mean here. slurm's pmi module is available (and Intel > > MPI can use it if i point it to it). > > Yeah, we need to be pointed to it just like Intel. Doh :-( > > > > Anyway, I think that if there is code that tries to use the hier > > component it shouldn't have been removed. > > Agreed - it looks like something picked up an unintended change. Just trying > to help you work with it as I don't know when a 1.6.4 will occur. I pulled the 1.6.1 hier component and reran autogen so i have it working but it's good to know what's to be expected in later releases.
Re: [OMPI users] grpcomm component hier gone...
On Thu, 2013-01-03 at 07:00 -0800, Ralph Castain wrote: > On Jan 3, 2013, at 6:52 AM, Ake Sandgren <ake.sandg...@hpc2n.umu.se> wrote: > > > On Thu, 2013-01-03 at 06:18 -0800, Ralph Castain wrote: > >> On Jan 3, 2013, at 3:01 AM, Ake Sandgren <ake.sandg...@hpc2n.umu.se> wrote: > >> > >>> On Thu, 2013-01-03 at 11:54 +0100, Ake Sandgren wrote: > >>>> On Thu, 2013-01-03 at 11:15 +0100, Ake Sandgren wrote: > >>>>> Hi! > >>>>> > >>>>> The grpcomm component hier seems to have vanished between 1.6.1 and > >>>>> 1.6.3. > >>>>> Why? > >>>>> It seems that the version of slurm we are using (not the latest at the > >>>>> moment) is using it for startup. > >> > >> It should be using PMI if you are directly launching processes via srun, > >> and should not be using hier any more. > > > > Shouldn't the grpcomm pmi component be turned on by default then, if it > > is needed? > > It should be Well, it isn't :-) configure says: --- MCA component grpcomm:pmi (m4 configuration macro) checking for MCA component grpcomm:pmi compile mode... dso checking if user requested PMI support... no checking if MCA component grpcomm:pmi can compile... no > > So what is the real problem here? > > Do you have PMI installed and running on your system? I think that is the > source of the trouble - if PMI isn't running, then this will fail. Not sure what you mean here. slurm's pmi module is available (and Intel MPI can use it if i point it to it). Anyway, I think that if there is code that tries to use the hier component it shouldn't have been removed.
Re: [OMPI users] grpcomm component hier gone...
On Thu, 2013-01-03 at 06:18 -0800, Ralph Castain wrote: > On Jan 3, 2013, at 3:01 AM, Ake Sandgren <ake.sandg...@hpc2n.umu.se> wrote: > > > On Thu, 2013-01-03 at 11:54 +0100, Ake Sandgren wrote: > >> On Thu, 2013-01-03 at 11:15 +0100, Ake Sandgren wrote: > >>> Hi! > >>> > >>> The grpcomm component hier seems to have vanished between 1.6.1 and > >>> 1.6.3. > >>> Why? > >>> It seems that the version of slurm we are using (not the latest at the > >>> moment) is using it for startup. > > It should be using PMI if you are directly launching processes via srun, and > should not be using hier any more. Shouldn't the grpcomm pmi component be turned on by default then, if it is needed? > >>> > >> > >> Hmm it seems it is the ess_slurmd_module.c that is using grpcomm=hier. > > Yes - that is the *only* scenario (a direct launch of procs via srun) that > should use hier What i have in my submit file is: #SBATCH -n x srun some-mpi-binary This fails since hier is missing. The reason one wants to use srun and not mpirun is getting slurms cgroup containement. > > > > orte/mca/plm/base/plm_base_rsh_support.c also tries to use the hier > > grpcomm > > Something is very wrong if that is true. How was this configured, and how are > you starting this job? Not sure if it actually tries to use hier at runtime, i just noticed that it had a setenv OMPI_MCA_grpcomm=hier in the code. So what is the real problem here? configure line is: ./configure --enable-orterun-prefix-by-default --enable-cxx-exceptions
Re: [OMPI users] grpcomm component hier gone...
On Thu, 2013-01-03 at 11:54 +0100, Ake Sandgren wrote: > On Thu, 2013-01-03 at 11:15 +0100, Ake Sandgren wrote: > > Hi! > > > > The grpcomm component hier seems to have vanished between 1.6.1 and > > 1.6.3. > > Why? > > It seems that the version of slurm we are using (not the latest at the > > moment) is using it for startup. > > > > Hmm it seems it is the ess_slurmd_module.c that is using grpcomm=hier. orte/mca/plm/base/plm_base_rsh_support.c also tries to use the hier grpcomm
Re: [OMPI users] grpcomm component hier gone...
On Thu, 2013-01-03 at 11:15 +0100, Ake Sandgren wrote: > Hi! > > The grpcomm component hier seems to have vanished between 1.6.1 and > 1.6.3. > Why? > It seems that the version of slurm we are using (not the latest at the > moment) is using it for startup. > Hmm it seems it is the ess_slurmd_module.c that is using grpcomm=hier. Please fix :-)
[OMPI users] grpcomm component hier gone...
Hi! The grpcomm component hier seems to have vanished between 1.6.1 and 1.6.3. Why? It seems that the version of slurm we are using (not the latest at the moment) is using it for startup. -- Ake Sandgren, HPC2N, Umea University, S-90187 Umea, Sweden Internet: a...@hpc2n.umu.se Phone: +46 90 7866134 Fax: +46 90 7866126 Mobile: +46 70 7716134 WWW: http://www.hpc2n.umu.se
Re: [OMPI users] fortran bindings for MPI_Op_commutative
On Thu, 2012-09-27 at 16:31 +0200, Ake Sandgren wrote: > Hi! > > Building 1.6.1 and 1.6.2 i seem to be missing the actual fortran > bindings for MPI_Op_commutative and a bunch of other functions. > > My configure is > ./configure --enable-orterun-prefix-by-default --enable-cxx-exceptions > > When looking in libmpi_f77.so there is no mpi_op_commutative_ defined. > mpi_init_ is there (as a weak) as it should. > > All compilers give me the same result. > > Any ideas why? Ahh, pop_commutative_f.c is missing from the profile/Makefile.am
[OMPI users] fortran bindings for MPI_Op_commutative
Hi! Building 1.6.1 and 1.6.2 i seem to be missing the actual fortran bindings for MPI_Op_commutative and a bunch of other functions. My configure is ./configure --enable-orterun-prefix-by-default --enable-cxx-exceptions When looking in libmpi_f77.so there is no mpi_op_commutative_ defined. mpi_init_ is there (as a weak) as it should. All compilers give me the same result. Any ideas why? -- Ake Sandgren, HPC2N, Umea University, S-90187 Umea, Sweden Internet: a...@hpc2n.umu.se Phone: +46 90 7866134 Fax: +46 90 7866126 Mobile: +46 70 7716134 WWW: http://www.hpc2n.umu.se
[OMPI users] Bug in openmpi 1.5.4 in paffinity
Hi! I'm getting a segfault in hwloc_setup_distances_from_os_matrix in the call to hwloc_bitmap_or due to objs or objs[i]->cpuset being freed and containing garbage, objs[i]->cpuset has infinite < 0. I only get this when using slurm with cgroups, asking for 2 nodes with 1 cpu each. The cpuset is then already set when mpiexec starts and something breaks down. valgrind on mpiexec says: ==27540== Invalid read of size 8 ==27540==at 0x7178F79: opal_paffinity_hwloc_finalize_logical_distances (distances.c:412) ==27540==by 0x7172C1E: hwloc_discover (topology.c:1805) ==27540==by 0x71745F2: opal_paffinity_hwloc_topology_load (topology.c:2244) ==27540==by 0x7164FB4: hwloc_open (paffinity_hwloc_component.c:93) ==27540==by 0x4F98D2E: mca_base_components_open (mca_base_components_open.c:214) ==27540==by 0x500084B: opal_paffinity_base_open (paffinity_base_open.c:120) ==27540==by 0x4F525BB: opal_init (opal_init.c:307) ==27540==by 0x4E50CA8: orte_init (orte_init.c:78) ==27540==by 0x403C8F: orterun (orterun.c:615) ==27540==by 0x4032C3: main (main.c:13) ==27540== Address 0x6e38380 is 160 bytes inside a block of size 248 free'd ==27540==at 0x4C270BD: free (vg_replace_malloc.c:366) ==27540==by 0x716B6A1: unlink_and_free_object_and_children (topology.c:1131) ==27540==by 0x716BB35: remove_empty (topology.c:1150) ==27540==by 0x7170CBB: hwloc_discover (topology.c:1768) ==27540==by 0x71745F2: opal_paffinity_hwloc_topology_load (topology.c:2244) ==27540==by 0x7164FB4: hwloc_open (paffinity_hwloc_component.c:93) ==27540==by 0x4F98D2E: mca_base_components_open (mca_base_components_open.c:214) ==27540==by 0x500084B: opal_paffinity_base_open (paffinity_base_open.c:120) ==27540==by 0x4F525BB: opal_init (opal_init.c:307) ==27540==by 0x4E50CA8: orte_init (orte_init.c:78) ==27540==by 0x403C8F: orterun (orterun.c:615) ==27540==by 0x4032C3: main (main.c:13) I hope the above info is enough and that you can fix it :-) /Åke S.
Re: [OMPI users] PathScale problems persist
On Wed, 2010-09-22 at 14:16 +0200, Ake Sandgren wrote: > On Wed, 2010-09-22 at 07:42 -0400, Jeff Squyres wrote: > > This is a problem with the Pathscale compiler and old versions of GCC. See: > > > > > > http://www.open-mpi.org/faq/?category=building#pathscale-broken-with-mpi-c%2B%2B-api > > > > I note that you said you're already using GCC 4.x, but it's not clear from > > your text whether pathscale is using that compiler or a different GCC on > > the back-end. If you can confirm that pathscale *is* using GCC 4.x on the > > back-end, then this is worth reporting to the pathscale support people. > > I have no problem running the code below compiled with openmpi 1.4.2 and > pathscale 3.2. And i should of course have specified that this is with a GCC4.x backend. -- Ake Sandgren, HPC2N, Umea University, S-90187 Umea, Sweden Internet: a...@hpc2n.umu.se Phone: +46 90 7866134 Fax: +46 90 7866126 Mobile: +46 70 7716134 WWW: http://www.hpc2n.umu.se
Re: [OMPI users] PathScale problems persist
On Wed, 2010-09-22 at 07:42 -0400, Jeff Squyres wrote:
> This is a problem with the Pathscale compiler and old versions of GCC. See:
>
>
> http://www.open-mpi.org/faq/?category=building#pathscale-broken-with-mpi-c%2B%2B-api
>
> I note that you said you're already using GCC 4.x, but it's not clear from
> your text whether pathscale is using that compiler or a different GCC on the
> back-end. If you can confirm that pathscale *is* using GCC 4.x on the
> back-end, then this is worth reporting to the pathscale support people.
I have no problem running the code below compiled with openmpi 1.4.2 and
pathscale 3.2.
> > However, now we are having trouble with the 1.4.2, PathScale 3.2, and
> > the C++ bindings. The following code:
> >
> > #include
> > #include
> >
> > int main(int argc, char* argv[]) {
> > int node, size;
> >
> > MPI::Init(argc, argv);
> > MPI::COMM_WORLD.Set_errhandler(MPI::ERRORS_THROW_EXCEPTIONS);
> >
> > try {
> >int rank = MPI::COMM_WORLD.Get_rank();
> >int size = MPI::COMM_WORLD.Get_size();
> >
> >std::cout << "Hello world from process " << rank << " out of "
> > << size << "!" << std::endl;
> > }
> >
> > catch(MPI::Exception e) {
> >std::cerr << "MPI Error: " << e.Get_error_code()
> > << " - " << e.Get_error_string() << std::endl;
> > }
> >
> > MPI::Finalize();
> > return 0;
> > }
> >
> > generates the following output:
> >
> > [host1:29934] *** An error occurred in MPI_Comm_set_errhandler
> > [host1:29934] *** on communicator MPI_COMM_WORLD
> > [host1:29934] *** MPI_ERR_COMM: invalid communicator
> > [host1:29934] *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort)
> > --
> > mpirun has exited due to process rank 2 with PID 29934 on
> > node host1 exiting without calling "finalize". This may
> > have caused other processes in the application to be
> > terminated by signals sent by mpirun (as reported here).
> > ------
> > [host1:29931] 3 more processes have sent help message
> > help-mpi-errors.txt / mpi_errors_are_fatal
> > [host1:29931] Set MCA parameter "orte_base_help_aggregate" to 0 to see
> > all help / error messages
> >
> > There are no problems when Open MPI 1.4.2 is built with GCC (GCC 4.1.2).
> > No problems are found with Open MPI 1.2.6 and PathScale either.
--
Ake Sandgren, HPC2N, Umea University, S-90187 Umea, Sweden
Internet: a...@hpc2n.umu.se Phone: +46 90 7866134 Fax: +46 90 7866126
Mobile: +46 70 7716134 WWW: http://www.hpc2n.umu.se
[OMPI users] opal_mutex_lock(): Resource deadlock avoided
Hi! We have a code that trips on this fairly often. I've seen cases where it works but mostly it gets stuck here. The actual mpi call is call mpi_file_open(...) I'm currently just wondering if there has been other reports on/anyone have seen deadlock in mpi-io parts of the code or if this most likely caused by our setup. openmpi version is 1.4.2 (fails with 1.3.3 too) Filesystem used is GPFS openmpi built with mpi-threads but without progress-threads -- Ake Sandgren, HPC2N, Umea University, S-90187 Umea, Sweden Internet: a...@hpc2n.umu.se Phone: +46 90 7866134 Fax: +46 90 7866126 Mobile: +46 70 7716134 WWW: http://www.hpc2n.umu.se
Re: [OMPI users] Segmentation fault in mca_btl_tcp
On Thu, 2010-04-15 at 15:57 +0900, Werner Van Geit wrote: > Hi, > > We are using openmpi 1.4.1 on our cluster computer (in conjunction with > Torque). One of our users has a problem with his jobs generating a > segmentation fault on one of the slaves, this is the backtrace: > > [cstone-00613:28461] *** Process received signal *** > [cstone-00613:28461] Signal: Segmentation fault (11) > [cstone-00613:28461] Signal code: (128) > [cstone-00613:28461] Failing at address: (nil) > [cstone-00613:28462] *** Process received signal *** > [cstone-00613:28462] Signal: Segmentation fault (11) > [cstone-00613:28462] Signal code: Address not mapped (1) > [cstone-00613:28462] Failing at address: (nil) > [cstone-00613:28461] [ 0] /lib64/libc.so.6 [0x2ba1933dce20] > [cstone-00613:28461] [ 1] /opt/openmpi-1.3/lib/openmpi/mca_btl_tcp.so > [0x2ba19530ec7a] > [cstone-00613:28461] [ 2] /opt/openmpi-1.3/lib/openmpi/mca_btl_tcp.so > [0x2ba19530d860] > [cstone-00613:28461] [ 3] /opt/openmpi/lib/libopen-pal.so.0 [0x2ba1938eb16b] > [cstone-00613:28461] [ 4] > /opt/openmpi/lib/libopen-pal.so.0(opal_progress+0x9e) [0x2ba1938e072e] > [cstone-00613:28461] [ 5] /opt/openmpi/lib/libmpi.so.0 [0x2ba193621b38] > [cstone-00613:28461] [ 6] /opt/openmpi/lib/libmpi.so.0(PMPI_Wait+0x5b) > [0x2ba19364c63b] > [cstone-00613:28461] [ 7] /opt/openmpi/lib/libmpi_f77.so.0(mpi_wait_+0x3a) > [0x2ba192e98b8a] > [cstone-00613:28461] [ 8] ./roms [0x44976c] > [cstone-00613:28461] [ 9] ./roms [0x449d96] > [cstone-00613:28461] [10] ./roms [0x422708] > [cstone-00613:28461] [11] ./roms [0x402908] > [cstone-00613:28461] [12] ./roms [0x402467] > [cstone-00613:28461] [13] ./roms [0x46d20e] > [cstone-00613:28461] [14] /lib64/libc.so.6(__libc_start_main+0xf4) > [0x2ba1933ca164] > [cstone-00613:28461] [15] ./roms [0x401dd9] > [cstone-00613:28461] *** End of error message *** > [cstone-00613:28462] [ 0] /lib64/libc.so.6 [0x2b5d57db6e20] > [cstone-00613:28462] *** End of error message *** > > The other slaves crash with: > [cstone-00612][[21785,1],35][btl_tcp_frag.c:216:mca_btl_tcp_frag_recv] > mca_btl_tcp_frag_recv: readv failed: Connection reset by peer (104) > > Since this problem seems to be happening in the network part of MPI my guess > is that there is, or something wrong with the network, or a bug in OpenMPI. > This same problem also appeared at the time that we were using openmpi 1.3 > > How could this problem be solved ? We sometimes see mysterious crashes like this one. At least some of them are caused by port scanners, i.e. unexpected non-mpi related packets coming in on the sockets will sometimes cause havoc. We've been getting http traffic in the jobs stdout/err sometimes. That really makes the users confused :-) And yes, we are going to block this but we haven't had time... -- Ake Sandgren, HPC2N, Umea University, S-90187 Umea, Sweden Internet: a...@hpc2n.umu.se Phone: +46 90 7866134 Fax: +46 90 7866126 Mobile: +46 70 7716134 WWW: http://www.hpc2n.umu.se
Re: [OMPI users] Problems building Open MPI 1.4.1 with Pathscale
On Wed, 2010-02-10 at 08:42 -0700, Barrett, Brian W wrote:
> Adding the memory and cc will certainly do no harm, and someone tried to
> remove them as an optimization. I wouldn't change the input and output lines
> - the differences are mainly syntactic sugar.
Gcc actually didn't like the example i sent earlier.
Another iteration gave this as a working (gcc/intel/pgi/pathscale works)
code.
static inline int opal_atomic_cmpset_32( volatile int32_t *addr,
int32_t oldval, int32_t newval)
{
unsigned char ret;
__asm__ __volatile__ (
SMPLOCK "cmpxchgl %3,%2 \n\t"
"sete %0 \n\t"
: "=qm" (ret), "+a" (oldval), "+m" (*addr)
: "q"(newval)
: "memory", "cc");
return (int)ret;
}
--
Ake Sandgren, HPC2N, Umea University, S-90187 Umea, Sweden
Internet: a...@hpc2n.umu.se Phone: +46 90 7866134 Fax: +46 90 7866126
Mobile: +46 70 7716134 WWW: http://www.hpc2n.umu.se
Re: [OMPI users] Problems building Open MPI 1.4.1 with Pathscale
On Wed, 2010-02-10 at 08:21 -0500, Jeff Squyres wrote:
> On Feb 10, 2010, at 7:47 AM, Ake Sandgren wrote:
>
> > According to people who knows asm statements fairly well (compiler
> > developers), it should be
>
> > static inline int opal_atomic_cmpset_32( volatile int32_t *addr,
> > int32_t oldval, int32_t newval)
> > {
> > unsigned char ret;
> > __asm__ __volatile__ (
> > SMPLOCK "cmpxchgl %3,%2 \n\t"
> > "sete %0 \n\t"
> > : "=qm" (ret), "=a" (oldval), "=m" (*addr)
> > : "q"(newval), "2"(*addr), "1"(oldval)
> > : "memory", "cc");
> >
> > return (int)ret;
> > }
>
> Disclaimer: I know almost nothing about assembly.
>
> I know that OMPI's asm is a carefully crafted set of assembly that works
> across a broad range of compilers. So what might not be "quite right" for
> one compiler may actually be there because another compiler needs it.
>
> That being said, if the changes above are for correctness, not
> neatness/style/etc., I can't speak for that...
The above should be correct for gcc style unless i misunderstood them.
Quoting from their reply:
'it should be "memory", "cc" since you also have to tell gcc you're
clobbering the EFLAGS'
And i don't know asm either so...
--
Ake Sandgren, HPC2N, Umea University, S-90187 Umea, Sweden
Internet: a...@hpc2n.umu.se Phone: +46 90 7866134 Fax: +46 90 7866126
Mobile: +46 70 7716134 WWW: http://www.hpc2n.umu.se
Re: [OMPI users] Problems building Open MPI 1.4.1 with Pathscale
On Tue, 2010-02-09 at 14:44 -0800, Mostyn Lewis wrote:
> The old opal_atomic_cmpset_32 worked:
>
> static inline int opal_atomic_cmpset_32( volatile int32_t *addr,
> unsigned char ret;
> __asm__ __volatile__ (
> SMPLOCK "cmpxchgl %1,%2 \n\t"
> "sete %0 \n\t"
> : "=qm" (ret)
> : "q"(newval), "m"(*addr), "a"(oldval)
> : "memory");
>
> return (int)ret;
> }
>
> The new opal_atomic_cmpset_32 fails:
>
> static inline int opal_atomic_cmpset_32( volatile int32_t *addr,
> int32_t oldval, int32_t newval)
> {
> unsigned char ret;
> __asm__ __volatile__ (
> SMPLOCK "cmpxchgl %3,%4 \n\t"
> "sete %0 \n\t"
> : "=qm" (ret), "=a" (oldval), "=m" (*addr)
> : "q"(newval), "m"(*addr), "1"(oldval)
> return (int)ret;
> }
>
> **However** if you put back the "clobber" for memory line (3rd :), it works:
>
> static inline int opal_atomic_cmpset_32( volatile int32_t *addr,
> int32_t oldval, int32_t newval)
> {
> unsigned char ret;
> __asm__ __volatile__ (
> SMPLOCK "cmpxchgl %3,%4 \n\t"
> "sete %0 \n\t"
> : "=qm" (ret), "=a" (oldval), "=m" (*addr)
> : "q"(newval), "m"(*addr), "1"(oldval)
> : "memory");
>
> return (int)ret;
> }
>
> This works in a test case for pathcc, gcc, icc, pgcc, SUN studio cc and
> open64 (pathscale
> lineage - which also fails with 1.4.1).
> Also the SMPLOCK above is defined as "lock; " - the ";" is a GNU as statement
> delimter - is
> that right? Seems to work with/without the ";".
>
>
> Also, a question - I see you generate via perl another "lock" asm file which
> you put into
> opal/asm/generated/<whatever, e.g. atomic-amd64-linux.s> and stick into
> libasm - what you
> generate there for whatever usage hasn't changed 1.4->1.4.1->svn trunk?
According to people who knows asm statements fairly well (compiler
developers), it should be
static inline int opal_atomic_cmpset_32( volatile int32_t *addr,
int32_t oldval, int32_t newval)
{
unsigned char ret;
__asm__ __volatile__ (
SMPLOCK "cmpxchgl %3,%2 \n\t"
"sete %0 \n\t"
: "=qm" (ret), "=a" (oldval), "=m" (*addr)
: "q"(newval), "2"(*addr), "1"(oldval)
: "memory", "cc");
return (int)ret;
}
--
Ake Sandgren, HPC2N, Umea University, S-90187 Umea, Sweden
Internet: a...@hpc2n.umu.se Phone: +46 90 7866134 Fax: +46 90 7866126
Mobile: +46 70 7716134 WWW: http://www.hpc2n.umu.se
Re: [OMPI users] Problems building Open MPI 1.4.1 with Pathscale
On Tue, 2010-02-09 at 08:49 -0500, Jeff Squyres wrote: > FWIW, I have had terrible luck with the patschale compiler over the years. > Repeated attempts to get support from them -- even when I was a paying > customer -- resulted in no help (e.g., a pathCC bug with the OMPI C++ > bindings that I filed years ago was never resolved). > > Is this compiler even supported anymore? I.e., is there a support department > somewhere that you have a hope of getting any help from? > > I can't say for sure, of course, but if MPI hello world hangs, it smells like > a compiler bug. You might want to attach to "hello world" in a debugger and > see where it's hung. You might need to compile OMPI with debugging symbols > to get any meaningful information. > > ** NOTE: My personal feelings about the pathscale compiler suite do not > reflect anyone else's feelings in the Open MPI community. Perhaps someone > could change my mind someday, but *I* have personally given up on this > compiler. :-( Pathscale is not dead, in fact I'm talking to them more or less daily at the moment. They have been restructuring since the demise of SciCortex last year. I hope they will be able to release a new version fairly soon. In my opinion (working mostly with Fortran codes, shudder) it is the best compiler around. Although they have had problems over the years in coming out with fixes for bugs in a timely fashion. -- Ake Sandgren, HPC2N, Umea University, S-90187 Umea, Sweden Internet: a...@hpc2n.umu.se Phone: +46 90 7866134 Fax: +46 90 7866126 Mobile: +46 70 7716134 WWW: http://www.hpc2n.umu.se
Re: [OMPI users] Problems compiling OpenMPI 1.4 with PGI 9.0-3
On Thu, 2010-01-07 at 11:57 +0100, Peter Kjellstrom wrote: > On Wednesday 06 January 2010, Tim Miller wrote: > > Hi All, > > > > I am trying to compile OpenMPI 1.4 with PGI 9.0-3 and am getting the > > following error in configure: > > > > checking for functional offsetof macro... no > > configure: WARNING: Your compiler does not support offsetof macro > > configure: error: Configure: Cannot continue > > > > I have searched around and found that this error occurs because of a > > problem in the configure scripts when PGI 10 is used, but I'm using 9.0-3 > > which should not have the configure script issue. Here is the output of > > pgcc -V: > > > > pgcc 9.0-3 64-bit target on x86-64 Linux -tp k8-64e > > Copyright 1989-2000, The Portland Group, Inc. All Rights Reserved. > > Copyright 2000-2009, STMicroelectronics, Inc. All Rights Reserved. > > > > I'm not sure what's wrong here as other people have reported being able to > > build OpenMPI with PGI 9. Does anyone have any ideas? > > Maybe a late enough PGI-9 behaves like PGI-10. You could try the 1.4.1-rc1 > which should work with PGI-10 and see if it fixes your problems too. Our PGI 9.0-3 doesn't have any problems building openmpi 1.3.3 or 1.4 -- Ake Sandgren, HPC2N, Umea University, S-90187 Umea, Sweden Internet: a...@hpc2n.umu.se Phone: +46 90 7866134 Fax: +46 90 7866126 Mobile: +46 70 7716134 WWW: http://www.hpc2n.umu.se
Re: [OMPI users] MPI_Irecv segmentation fault
On Mon, 2009-09-21 at 19:26 -0400, Everette Clemmer wrote:
> Hey all,
>
> I'm getting a segmentation fault when I attempt to receive a single
> character via MPI_Irecv. Code follows:
>
> void recv_func() {
> if( !MASTER ) {
> charbuffer[ 1 ];
> int flag;
> MPI_Request request;
> MPI_Status status;
>
> MPI_Irecv( , 1, MPI_CHAR, 0, MPI_ANY_TAG,
> MPI_COMM_WORLD, );
It should be MPI_Irecv(buffer, 1, ...)
> The segfault disappears if I comment out the MPI_Irecv call in
> recv_func so I'm assuming that there's something wrong with the
> parameters that I'm passing to it. Thoughts?
--
Ake Sandgren, HPC2N, Umea University, S-90187 Umea, Sweden
Internet: a...@hpc2n.umu.se Phone: +46 90 7866134 Fax: +46 90 7866126
Mobile: +46 70 7716134 WWW: http://www.hpc2n.umu.se
Re: [OMPI users] Bad MPI_Bcast behaviour when running over openib
On Fri, 2009-09-11 at 13:18 +0200, Ake Sandgren wrote: > Hi! > > The following code shows a bad behaviour when running over openib. Oops. Red Face big time. I happened to run the IB test between two systems that don't have IB connectivity. Goes and hide in a dark corner... -- Ake Sandgren, HPC2N, Umea University, S-90187 Umea, Sweden Internet: a...@hpc2n.umu.se Phone: +46 90 7866134 Fax: +46 90 7866126 Mobile: +46 70 7716134 WWW: http://www.hpc2n.umu.se
[OMPI users] Bad MPI_Bcast behaviour when running over openib
Hi!
The following code shows a bad behaviour when running over openib.
Openmpi: 1.3.3
With openib it dies with "error polling HP CQ with status WORK REQUEST
FLUSHED ERROR status number 5 ", with tcp or shmem it works as expected.
#include
#include
#include
#include "mpi.h"
int main(int argc, char *argv[])
{
int rank;
int n;
MPI_Init( , );
MPI_Comm_rank( MPI_COMM_WORLD, );
fprintf(stderr, "I am %d at %d\n", rank, time(NULL));
fflush(stderr);
n = 4;
MPI_Bcast(, 1, MPI_INTEGER, 0, MPI_COMM_WORLD);
fprintf(stderr, "I am %d at %d\n", rank, time(NULL));
fflush(stderr);
if (rank == 0) {
sleep(60);
}
MPI_Barrier(MPI_COMM_WORLD);
MPI_Finalize( );
exit(0);
}
I know about the internal openmpi reason for it do behave as it does.
But i think that it should be allowed to behave as it does.
This example is a bit engineered but there are codes where a similar
situation can occur, i.e. the Bcast sender doing lots of other work
after the Bcast before the next MPI call. VASP is a candidate for this.
--
Ake Sandgren, HPC2N, Umea University, S-90187 Umea, Sweden
Internet: a...@hpc2n.umu.se Phone: +46 90 7866134 Fax: +46 90 7866126
Mobile: +46 70 7716134 WWW: http://www.hpc2n.umu.se
[OMPI users] Need help with tuning of IB for OpenMPI 1.3.3
Hi! We have one user code that is having lots of problems with RNRs or sometimes hangs. (The same code runs ok on another IB based system which has full connectivity and on our Myrinet system) The IB network has a 7:3 overload, i.e. 7 nodes per 3 IB links up to the main Cisco switch. In other words, we have 48 bladecenters with 14 blades (8 cores) in each with a IB switch per bladecenter and 2x3 IB lines per bladecenter to the main Cisco switch. Now to the question, do you have any good suggestions on parameters that will help us get around this problem. I tried changing the queue-pair settings and it does affect the problem but so far i haven't been able to fix it completely. The code usually works when running with nodes=8:ppn=8, but always fails sooner or later with nodes=16:ppn=8. Also turning off leave_pinned helps a bit. The best settings i have so far are: -mca mpi_leave_pinned 0 -mca btl_openib_receive_queues "P,128,512:S,2048, 512,128,2:S,12288,512,128,2:S,65536,512,128,2" I have tried almost anything i can think of and desperately need help here. Building everything in debug mode helps somewhat due to the code getting so slow that the network can keep up a lot better but not completely. OS: CentOS5.3 (OFED 1.3.2 and 1.4.2 tested) HW: Mellanox MT25208 InfiniHost III Ex (128MB) -- Ake Sandgren, HPC2N, Umea University, S-90187 Umea, Sweden Internet: a...@hpc2n.umu.se Phone: +46 90 7866134 Fax: +46 90 7866126 Mobile: +46 70 7716134 WWW: http://www.hpc2n.umu.se
Re: [OMPI users] BLACS vs. OpenMPI 1.1.1 & 1.3
On Sat, 2006-10-28 at 08:45 -0400, Jeff Squyres wrote: > Sorry for the delay on this -- is this still the case with the OMPI > trunk? > > We think we finally have all the issues solved with MPI_ABORT on the > trunk. > Nah, it was a problem with overutilization, i.e. 4tasks on 2 cpus in one node. Turning on yield_when_idle made the problem go away. As to why it didn't figure this out by itself is another problem. Summary: 1.1.2 works correctly, no problems. (Except when building with PGI, i have patches...) > >> Lets add some more data to this... > >> BLACS 1.1p3 > >> Ubuntu Dapper 6.06 > >> dual opteron > >> gcc 4.0 > >> gfortran 4.0 (for both f77 and f90) > >> standard tests with 4 tasks on one node (i.e. 2 tasks per cpu) > >> > >> OpenMPI 1.1.2rc3 > >> The tests comes to a complete standstill at the integer bsbr tests > >> It consumes cpu all the time but nothing happens. > > > > Actually if i'm not too inpatient i will progress but VERY slowly. > > A complete run of the blacstest takes +30min cpu-time... > >> From the bsbr tests and onwards everything takes "forever". -- Ake Sandgren, HPC2N, Umea University, S-90187 Umea, Sweden Internet: a...@hpc2n.umu.se Phone: +46 90 7866134 Fax: +46 90 7866126 Mobile: +46 70 7716134 WWW: http://www.hpc2n.umu.se
