[OMPI users] Problem with MPI_FILE_WRITE_AT

2017-09-15 Thread McGrattan, Kevin B. Dr. (Fed)
I am using MPI_FILE_WRITE_AT to print out the timings of subroutines in a big Fortran code. I have noticed since upgrading to Open MPI 2.1.1 that sometimes the file to be written is corrupted. Each MPI process is supposed to write out a character string that is 159 characters in length, plus a

[OMPI users] Memory leak in Open MPI 2.1.1

2017-07-28 Thread McGrattan, Kevin B. Dr. (Fed)
I am using Open MPI 2.1.1 along with Intel Fortran 17 update 4 and I am experiencing what I think is a memory leak with a job that uses 184 MPI processes. The memory used per process appears to be increasing by about 1 to 2 percent per hour. My code uses mostly persistent sends and receives to

Re: [OMPI users] MPI_WAIT hangs after a call to MPI_CANCEL

2017-04-05 Thread McGrattan, Kevin B. Dr. (Fed)
just ignore a dropped packet now and again, or try to figure out what’s wrong with our IB. Thanks Kevin From: George Bosilca [mailto:bosi...@icl.utk.edu] Sent: Monday, April 03, 2017 5:59 PM To: McGrattan, Kevin B. Dr. (Fed) <kevin.mcgrat...@nist.gov> Cc: Open MPI Users <users@lists.ope

[OMPI users] MPI_WAIT hangs after a call to MPI_CANCEL

2017-04-01 Thread McGrattan, Kevin B. Dr. (Fed)
I am running a large computational fluid dynamics code on a linux cluster (Centos 6.8, Open MPI 1.8.4). The code is written in Fortran and compiled with Intel Fortran 16.0.3. The cluster has 36 nodes, each node has two sockets, each socket has six cores. I have noticed that the code hangs when

[OMPI users] Seg fault in MPI_FINALIZE

2015-10-16 Thread McGrattan, Kevin B. Dr.
My group is running a fairly large CFD code compiled with Intel Fortran 16.0.0 and OpenMPI 1.8.4. Each night we run hundreds of simple test cases, using a range of MPI processes from 1 to 16. I have noticed that if we submit these jobs on our linux cluster and assign each job exclusive rights

Re: [OMPI users] Hybrid OpenMPI/OpenMP leading to deadlocks?

2014-10-16 Thread McGrattan, Kevin B. Dr.
ponse, etc), and code with poorly designed IO. My two cents, Gus Correa On 10/16/2014 10:16 AM, McGrattan, Kevin B. Dr. wrote: > The individual MPI processes appear to be using a few percent of the > system memory. > > I have created a loop containing repeated calls to MPI_TESTAL

Re: [OMPI users] Hybrid OpenMPI/OpenMP leading to deadlocks?

2014-10-16 Thread McGrattan, Kevin B. Dr.
output? If you are using IB and run out of QP, you should at least get something saying that. On Oct 15, 2014, at 8:22 AM, McGrattan, Kevin B. Dr. <kevin.mcgrat...@nist.gov<mailto:kevin.mcgrat...@nist.gov>> wrote: I am using OpenMPI 1.8.3 on a linux cluster to run fairly long CFD (c

[OMPI users] Hybrid OpenMPI/OpenMP leading to deadlocks?

2014-10-15 Thread McGrattan, Kevin B. Dr.
I am using OpenMPI 1.8.3 on a linux cluster to run fairly long CFD (computational fluid dynamics) simulations using 16 MPI processes. The calculations last several days and typically involve millions of MPI exchanges. I use the Intel Fortran compiler, and when I compile with the -openmp option

Re: [OMPI users] How does binding option affect network traffic?

2014-09-18 Thread McGrattan, Kevin B. Dr.
Yes and no. When I ran a single job that uses 16 MPI processes, and I mapped by socket and used 8 nodes, 2 ppn, the job ran 30% faster than the same job mapped by core on 2 nodes. Each process was fairly CPU intensive compared to the communication, so I suspect that the speed up was due to the

Re: [OMPI users] How does binding option affect network traffic?

2014-09-05 Thread McGrattan, Kevin B. Dr.
small corner case (because no one runs that way) that we've honestly never bothered to fix the issue (it's actually a really complicated resource exhaustion issue -- it's kinda hard to know what the Right Thing is to do when you've run out of memory...). On Sep 2, 2014, at 9:37 AM, McGrattan,

Re: [OMPI users] How does binding option affect network traffic?

2014-09-03 Thread McGrattan, Kevin B. Dr.
bothered to fix the issue (it's actually a really complicated resource exhaustion issue -- it's kinda hard to know what the Right Thing is to do when you've run out of memory...). On Sep 2, 2014, at 9:37 AM, McGrattan, Kevin B. Dr. <kevin.mcgrat...@nist.gov> wrote: > Thanks for the advice.

Re: [OMPI users] How does binding option affect network traffic?

2014-09-02 Thread McGrattan, Kevin B. Dr.
Thanks for the advice. Our jobs vary in size, from just a few MPI processes to about 64. Jobs are submitted at random, which is why I want to map by socket. If the cluster is empty, and someone submits a job with 16 MPI processes, I would think it would run most efficiently if it used 8 nodes,

Re: [OMPI users] How does binding option affect network traffic?

2014-08-29 Thread McGrattan, Kevin B. Dr.
29, 2014 3:26 PM To: Open MPI Users Subject: Re: [OMPI users] How does binding option affect network traffic? On Aug 29, 2014, at 10:51 AM, McGrattan, Kevin B. Dr. <kevin.mcgrat...@nist.gov<mailto:kevin.mcgrat...@nist.gov>> wrote: Thanks for the tip. I understand how using the --c

Re: [OMPI users] How does binding option affect network traffic?

2014-08-29 Thread McGrattan, Kevin B. Dr.
2" On Aug 28, 2014, at 11:50 AM, McGrattan, Kevin B. Dr. <kevin.mcgrat...@nist.gov> wrote: > My institute recently purchased a linux cluster with 20 nodes; 2 sockets per > node; 6 cores per socket. OpenMPI v 1.8.1 is installed. I want to run 15 > jobs. Each job requir

[OMPI users] How does binding option affect network traffic?

2014-08-28 Thread McGrattan, Kevin B. Dr.
My institute recently purchased a linux cluster with 20 nodes; 2 sockets per node; 6 cores per socket. OpenMPI v 1.8.1 is installed. I want to run 15 jobs. Each job requires 16 MPI processes. For each job, I want to use two cores on each node, mapping by socket. If I use these options: #PBS