Re: [OMPI users] memory per core/process

2013-04-03 Thread Ralph Castain
Here is a v1.6 port of what was committed to the trunk. Let me know if/how it works for you. The option you will want to use is: mpirun -mca opal_set_max_sys_limits stacksize:unlimited or whatever number you want to give (see ulimit for the units). Note that you won't see any impact if you run

Re: [OMPI users] memory per core/process

2013-04-02 Thread Duke Nguyen
On 4/2/13 11:03 PM, Gus Correa wrote: On 04/02/2013 11:40 AM, Duke Nguyen wrote: On 3/30/13 8:46 PM, Patrick Bégou wrote: Ok, so your problem is identified as a stack size problem. I went into these limitations using Intel fortran compilers on large data problems. First, it seems you can

Re: [OMPI users] memory per core/process

2013-04-02 Thread Duke Nguyen
On 4/2/13 10:45 PM, Ralph Castain wrote: Hmmm...tell you what. I'll add the ability for OMPI to set the limit to a user-specified level upon launch of each process. This will give you some protection and flexibility. That would be excellent ;) I forget, so please forgive the old man's

Re: [OMPI users] memory per core/process

2013-04-02 Thread Gus Correa
On 04/02/2013 11:40 AM, Duke Nguyen wrote: On 3/30/13 8:46 PM, Patrick Bégou wrote: Ok, so your problem is identified as a stack size problem. I went into these limitations using Intel fortran compilers on large data problems. First, it seems you can increase your stack size as "ulimit -s

Re: [OMPI users] memory per core/process

2013-04-02 Thread Ralph Castain
Hmmm...tell you what. I'll add the ability for OMPI to set the limit to a user-specified level upon launch of each process. This will give you some protection and flexibility. I forget, so please forgive the old man's fading memory - what version of OMPI are you using? I'll backport a patch

Re: [OMPI users] memory per core/process

2013-04-02 Thread Duke Nguyen
On 3/30/13 8:46 PM, Patrick Bégou wrote: Ok, so your problem is identified as a stack size problem. I went into these limitations using Intel fortran compilers on large data problems. First, it seems you can increase your stack size as "ulimit -s unlimited" works (you didn't enforce the

Re: [OMPI users] memory per core/process

2013-04-02 Thread Duke Nguyen
On 4/2/13 6:50 PM, Reuti wrote: Hi, Am 30.03.2013 um 14:46 schrieb Patrick Bégou: Ok, so your problem is identified as a stack size problem. I went into these limitations using Intel fortran compilers on large data problems. First, it seems you can increase your stack size as "ulimit -s

Re: [OMPI users] memory per core/process

2013-04-02 Thread Duke Nguyen
On 4/2/13 6:42 PM, Reuti wrote: /usr/local/bin/mpirun -npernode 1 -tag-output sh -c "ulimit -a" You are right :) $ /usr/local/bin/mpirun -npernode 1 -tag-output sh -c "ulimit -a" [1,0]:core file size (blocks, -c) 0 [1,0]:data seg size (kbytes, -d) unlimited

Re: [OMPI users] memory per core/process

2013-04-02 Thread Reuti
Hi, Am 30.03.2013 um 15:35 schrieb Gustavo Correa: > On Mar 30, 2013, at 10:02 AM, Duke Nguyen wrote: > >> On 3/30/13 8:20 PM, Reuti wrote: >>> Am 30.03.2013 um 13:26 schrieb Tim Prince: >>> On 03/30/2013 06:36 AM, Duke Nguyen wrote: > On 3/30/13 5:22 PM, Duke Nguyen wrote: >> On

Re: [OMPI users] memory per core/process

2013-04-02 Thread Reuti
Hi, Am 30.03.2013 um 14:46 schrieb Patrick Bégou: > Ok, so your problem is identified as a stack size problem. I went into these > limitations using Intel fortran compilers on large data problems. > > First, it seems you can increase your stack size as "ulimit -s unlimited" > works (you

Re: [OMPI users] memory per core/process

2013-04-02 Thread Reuti
Hi, Am 02.04.2013 um 13:22 schrieb Duke Nguyen: > On 4/1/13 9:20 PM, Ralph Castain wrote: >> It's probably the same problem - try running 'mpirun -npernode 1 -tag-output >> ulimit -a" on the remote nodes and see what it says. I suspect you'll find >> that they aren't correct. > > Somehow I

Re: [OMPI users] memory per core/process

2013-04-02 Thread Duke Nguyen
On 4/1/13 9:20 PM, Ralph Castain wrote: It's probably the same problem - try running 'mpirun -npernode 1 -tag-output ulimit -a" on the remote nodes and see what it says. I suspect you'll find that they aren't correct. Somehow I could not run your advised CMD: $ qsub -l nodes=4:ppn=8 -I

Re: [OMPI users] memory per core/process

2013-04-01 Thread Ralph Castain
It's probably the same problem - try running 'mpirun -npernode 1 -tag-output ulimit -a" on the remote nodes and see what it says. I suspect you'll find that they aren't correct. BTW: the "-tag-output'" option marks each line of output with the rank of the process. Since all the outputs will

Re: [OMPI users] memory per core/process

2013-04-01 Thread Duke Nguyen
On 3/31/13 12:20 AM, Duke Nguyen wrote: I should really have asked earlier. Thanks for all the helps. I think I was excited too soon :). Increasing stacksize does help if I run a job in a dedicated server. Today I tried to modify the cluster (/etc/security/limits.conf, /etc/init.d/pbs_mom)

Re: [OMPI users] memory per core/process

2013-03-30 Thread Duke Nguyen
I should really have asked earlier. Thanks for all the helps. D. On 3/30/13 10:28 PM, Ralph Castain wrote: FWIW: there is an MCA param that helps with such problems: opal_set_max_sys_limits "Set to non-zero to automatically set any system-imposed limits to the

Re: [OMPI users] memory per core/process

2013-03-30 Thread Ralph Castain
FWIW: there is an MCA param that helps with such problems: opal_set_max_sys_limits "Set to non-zero to automatically set any system-imposed limits to the maximum allowed", At the moment, it only sets the limits on number of files open, and max size of a file we can

Re: [OMPI users] memory per core/process

2013-03-30 Thread Gustavo Correa
On Mar 30, 2013, at 10:02 AM, Duke Nguyen wrote: > On 3/30/13 8:20 PM, Reuti wrote: >> Am 30.03.2013 um 13:26 schrieb Tim Prince: >> >>> On 03/30/2013 06:36 AM, Duke Nguyen wrote: On 3/30/13 5:22 PM, Duke Nguyen wrote: > On 3/30/13 3:13 PM, Patrick Bégou wrote: >> I do not know

Re: [OMPI users] memory per core/process

2013-03-30 Thread Duke Nguyen
On 3/30/13 8:20 PM, Reuti wrote: Am 30.03.2013 um 13:26 schrieb Tim Prince: On 03/30/2013 06:36 AM, Duke Nguyen wrote: On 3/30/13 5:22 PM, Duke Nguyen wrote: On 3/30/13 3:13 PM, Patrick Bégou wrote: I do not know about your code but: 1) did you check stack limitations ? Typically intel

Re: [OMPI users] memory per core/process

2013-03-30 Thread Patrick Bégou
Ok, so your problem is identified as a stack size problem. I went into these limitations using Intel fortran compilers on large data problems. First, it seems you can increase your stack size as "ulimit -s unlimited" works (you didn't enforce the system hard limit). The best way is to set

Re: [OMPI users] memory per core/process

2013-03-30 Thread Reuti
Am 30.03.2013 um 13:26 schrieb Tim Prince: > On 03/30/2013 06:36 AM, Duke Nguyen wrote: >> On 3/30/13 5:22 PM, Duke Nguyen wrote: >>> On 3/30/13 3:13 PM, Patrick Bégou wrote: I do not know about your code but: 1) did you check stack limitations ? Typically intel fortran codes

Re: [OMPI users] memory per core/process

2013-03-30 Thread Tim Prince
On 03/30/2013 06:36 AM, Duke Nguyen wrote: On 3/30/13 5:22 PM, Duke Nguyen wrote: On 3/30/13 3:13 PM, Patrick Bégou wrote: I do not know about your code but: 1) did you check stack limitations ? Typically intel fortran codes needs large amount of stack when the problem size increase. Check

Re: [OMPI users] memory per core/process

2013-03-30 Thread Duke Nguyen
On 3/30/13 5:22 PM, Duke Nguyen wrote: On 3/30/13 3:13 PM, Patrick Bégou wrote: I do not know about your code but: 1) did you check stack limitations ? Typically intel fortran codes needs large amount of stack when the problem size increase. Check ulimit -a First time I heard of stack

Re: [OMPI users] memory per core/process

2013-03-30 Thread Reuti
Am 30.03.2013 um 05:21 schrieb Duke Nguyen: > Hi folks, > > I am sorry if this question had been asked before, but after ten days of > searching/working on the system, I surrender :(. We try to use mpirun to run > abinit (abinit.org) which in turns will call an input file to run some >

Re: [OMPI users] memory per core/process

2013-03-30 Thread Duke Nguyen
On 3/30/13 3:13 PM, Patrick Bégou wrote: I do not know about your code but: 1) did you check stack limitations ? Typically intel fortran codes needs large amount of stack when the problem size increase. Check ulimit -a First time I heard of stack limitations. Anyway, ulimit -a gives $

Re: [OMPI users] memory per core/process

2013-03-30 Thread Patrick Bégou
I do not know about your code but: 1) did you check stack limitations ? Typically intel fortran codes needs large amount of stack when the problem size increase. Check ulimit -a 2) did your node uses cpuset and memory limitation like fake numa to set the maximum amount of memory available

[OMPI users] memory per core/process

2013-03-30 Thread Duke Nguyen
Hi folks, I am sorry if this question had been asked before, but after ten days of searching/working on the system, I surrender :(. We try to use mpirun to run abinit (abinit.org) which in turns will call an input file to run some simulation. The command to run is pretty simple $ mpirun -np