Re: [OMPI users] low CPU utilization with OpenMPI

[Vinson Leung]

Later I change another machine and set the TMPDIR to default /tmp, but the
problem (low CPU utilization under 20%) still occur :<

Vincent

On Thu, Oct 23, 2014 at 10:38 PM, Jeff Squyres (jsquyres) <
jsquy...@cisco.com> wrote:

> If normal users can't write to /tmp (or if /tmp is an NFS-mounted
> filesystem), that's the underlying problem.
>
> @Vinson -- you should probably try to get that fixed.
>
>
>
> On Oct 23, 2014, at 10:35 AM, Joshua Ladd  wrote:
>
> > It's not coming from OSHMEM but from the OPAL "shmem" framework. You are
> going to get terrible performance - possibly slowing to a crawl having all
> processes open their backing files for mmap on NSF. I think that's the
> error that he's getting.
> >
> >
> > Josh
> >
> > On Thu, Oct 23, 2014 at 6:06 AM, Vinson Leung 
> wrote:
> > HI, Thanks for your reply:)
> > I really run an MPI program (compile with OpenMPI and run with "mpirun
> -n 8 .."). My OpenMPI version is 1.8.3 and my program is Gromacs. BTW,
> what is OSHMEM ?
> >
> > Best
> > Vincent
> >
> > On Thu, Oct 23, 2014 at 12:21 PM, Ralph Castain 
> wrote:
> > From your error message, I gather you are not running an MPI program,
> but rather an OSHMEM one? Otherwise, I find the message strange as it only
> would be emitted from an OSHMEM program.
> >
> > What version of OMPI are you trying to use?
> >
> >> On Oct 22, 2014, at 7:12 PM, Vinson Leung 
> wrote:
> >>
> >> Thanks for your reply:)
> >> Follow your advice I tried to set the TMPDIR to /var/tmp and /dev/shm
> and even reset to /tmp (I get the system permission), the problem still
> occur (CPU utilization still lower than 20%). I have no idea why and ready
> to give up OpenMPI instead of using other MPI library.
> >>
> >> Old Message-
> >>
> >> Date: Tue, 21 Oct 2014 22:21:31 -0400
> >> From: Brock Palen 
> >> To: Open MPI Users 
> >> Subject: Re: [OMPI users] low CPU utilization with OpenMPI
> >> Message-ID: 
> >> Content-Type: text/plain; charset=us-ascii
> >>
> >> Doing special files on NFS can be weird,  try the other /tmp/ locations:
> >>
> >> /var/tmp/
> >> /dev/shm  (ram disk careful!)
> >>
> >> Brock Palen
> >> www.umich.edu/~brockp
> >> CAEN Advanced Computing
> >> XSEDE Campus Champion
> >> bro...@umich.edu
> >> (734)936-1985
> >>
> >>
> >>
> >> > On Oct 21, 2014, at 10:18 PM, Vinson Leung 
> wrote:
> >> >
> >> > Because of permission reason (OpenMPI can not write temporary file to
> the default /tmp directory), I change the TMPDIR to my local directory
> (export TMPDIR=/home/user/tmp ) and then the MPI program can run. But the
> CPU utilization is very low under 20% (8 MPI rank running in Intel Xeon
> 8-core CPU).
> >> >
> >> > And I also got some message when I run with OpenMPI:
> >> > [cn3:28072] 9 more processes have sent help message
> help-opal-shmem-mmap.txt / mmap on nfs
> >> > [cn3:28072] Set MCA parameter "orte_base_help_aggregate" to 0 to see
> all help / error messages
> >> >
> >> > Any idea?
> >> > Thanks
> >> >
> >> > VIncent
> >> > ___
> >> > users mailing list
> >> > us...@open-mpi.org
> >> > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> >> > Link to this post:
> http://www.open-mpi.org/community/lists/users/2014/10/25548.php
> >> ___
> >> users mailing list
> >> us...@open-mpi.org
> >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> >> Link to this post:
> http://www.open-mpi.org/community/lists/users/2014/10/2.php
> >
> >
> > ___
> > users mailing list
> > us...@open-mpi.org
> > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> > Link to this post:
> http://www.open-mpi.org/community/lists/users/2014/10/25556.php
> >
> >
> > ___
> > users mailing list
> > us...@open-mpi.org
> > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> > Link to this post:
> http://www.open-mpi.org/community/lists/users/2014/10/25558.php
> >
> > ___
> > users mailing list
> > us...@open-mpi.org
> > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> > Link to this post:
> http://www.open-mpi.org/community/lists/users/2014/10/25560.php
>
>
> --
> Jeff Squyres
> jsquy...@cisco.com
> For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/
>
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2014/10/25561.php
>

Re: [OMPI users] low CPU utilization with OpenMPI

[Vinson Leung]

How can I fix the error if all processes open their backing files for mmap
on NSF like you said?

Vincent

On Thu, Oct 23, 2014 at 10:35 PM, Joshua Ladd  wrote:

> It's not coming from OSHMEM but from the OPAL "shmem" framework. You are
> going to get terrible performance - possibly slowing to a crawl having all
> processes open their backing files for mmap on NSF. I think that's the
> error that he's getting.
>
>
> Josh
>
> On Thu, Oct 23, 2014 at 6:06 AM, Vinson Leung 
> wrote:
>
>> HI, Thanks for your reply:)
>> I really run an MPI program (compile with OpenMPI and run with "mpirun -n
>> 8 .."). My OpenMPI version is 1.8.3 and my program is Gromacs. BTW,
>> what is OSHMEM ?
>>
>> Best
>> Vincent
>>
>> On Thu, Oct 23, 2014 at 12:21 PM, Ralph Castain  wrote:
>>
>>> From your error message, I gather you are not running an MPI program,
>>> but rather an OSHMEM one? Otherwise, I find the message strange as it only
>>> would be emitted from an OSHMEM program.
>>>
>>> What version of OMPI are you trying to use?
>>>
>>> On Oct 22, 2014, at 7:12 PM, Vinson Leung 
>>> wrote:
>>>
>>> Thanks for your reply:)
>>> Follow your advice I tried to set the TMPDIR to /var/tmp and /dev/shm
>>> and even reset to /tmp (I get the system permission), the problem still
>>> occur (CPU utilization still lower than 20%). I have no idea why and ready
>>> to give up OpenMPI instead of using other MPI library.
>>>
>>> Old Message-
>>>
>>> Date: Tue, 21 Oct 2014 22:21:31 -0400
>>> From: Brock Palen 
>>> To: Open MPI Users 
>>> Subject: Re: [OMPI users] low CPU utilization with OpenMPI
>>> Message-ID: 
>>> Content-Type: text/plain; charset=us-ascii
>>>
>>> Doing special files on NFS can be weird,  try the other /tmp/ locations:
>>>
>>> /var/tmp/
>>> /dev/shm  (ram disk careful!)
>>>
>>> Brock Palen
>>> www.umich.edu/~brockp
>>> CAEN Advanced Computing
>>> XSEDE Campus Champion
>>> bro...@umich.edu
>>> (734)936-1985
>>>
>>>
>>>
>>> > On Oct 21, 2014, at 10:18 PM, Vinson Leung 
>>> wrote:
>>> >
>>> > Because of permission reason (OpenMPI can not write temporary file to
>>> the default /tmp directory), I change the TMPDIR to my local directory
>>> (export TMPDIR=/home/user/tmp ) and then the MPI program can run. But the
>>> CPU utilization is very low under 20% (8 MPI rank running in Intel Xeon
>>> 8-core CPU).
>>> >
>>> > And I also got some message when I run with OpenMPI:
>>> > [cn3:28072] 9 more processes have sent help message
>>> help-opal-shmem-mmap.txt / mmap on nfs
>>> > [cn3:28072] Set MCA parameter "orte_base_help_aggregate" to 0 to see
>>> all help / error messages
>>> >
>>> > Any idea?
>>> > Thanks
>>> >
>>> > VIncent
>>> > ___
>>> > users mailing list
>>> > us...@open-mpi.org
>>> > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>>> > Link to this post:
>>> http://www.open-mpi.org/community/lists/users/2014/10/25548.php
>>> ___
>>> users mailing list
>>> us...@open-mpi.org
>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>>> Link to this post:
>>> http://www.open-mpi.org/community/lists/users/2014/10/2.php
>>>
>>>
>>>
>>> ___
>>> users mailing list
>>> us...@open-mpi.org
>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>>> Link to this post:
>>> http://www.open-mpi.org/community/lists/users/2014/10/25556.php
>>>
>>
>>
>> ___
>> users mailing list
>> us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post:
>> http://www.open-mpi.org/community/lists/users/2014/10/25558.php
>>
>
>
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2014/10/25560.php
>

Re: [OMPI users] which info is needed for SIGSEGV in Java foropenmpi-dev-124-g91e9686 on Solaris

[Kawashima, Takahiro]

Hi Siegmar,

The attached JRE log shows very important information.

When JRE loads the MPI class, JNI_OnLoad function in
libmpi_java.so (Open MPI library; written in C) is called.
And probably mca_base_var_cache_files function passes NULL
to asprintf function. I don't know how this situation occurs.
You may be able to track this by inserting debug printf in
Open MPI code shown in the stack trace, or by using gdb or
something.

hs_err_pid13080.log:

siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR), 
si_addr=0x

Stack: [0x7b40,0x7b50],  sp=0x7b4fc730,  free 
space=1009k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libc.so.1+0x3c7f0]  strlen+0x50
C  [libc.so.1+0xaf640]  vsnprintf+0x84
C  [libc.so.1+0xaadb4]  vasprintf+0x20
C  [libc.so.1+0xaaf04]  asprintf+0x28
C  [libopen-pal.so.0.0.0+0xaf3cc]  mca_base_var_cache_files+0x160
C  [libopen-pal.so.0.0.0+0xaed90]  mca_base_var_init+0x4e8
C  [libopen-pal.so.0.0.0+0xb260c]  register_variable+0x214
C  [libopen-pal.so.0.0.0+0xb36a0]  mca_base_var_register+0x104
C  [libmpi_java.so.0.0.0+0x221e8]  JNI_OnLoad+0x128
C  [libjava.so+0x10860]  Java_java_lang_ClassLoader_00024NativeLibrary_load+0xb8
j  java.lang.ClassLoader$NativeLibrary.load(Ljava/lang/String;Z)V+-665819
j  java.lang.ClassLoader$NativeLibrary.load(Ljava/lang/String;Z)V+0
j  java.lang.ClassLoader.loadLibrary0(Ljava/lang/Class;Ljava/io/File;)Z+328
j  java.lang.ClassLoader.loadLibrary(Ljava/lang/Class;Ljava/lang/String;Z)V+290
j  java.lang.Runtime.loadLibrary0(Ljava/lang/Class;Ljava/lang/String;)V+54
j  java.lang.System.loadLibrary(Ljava/lang/String;)V+7
j  mpi.MPI.()V+28


Regards,
Takahiro

> Hello Oscar,
> 
> do you have time to look into my problem? Probably Takahiro has a
> point and gdb behaves differently on Solaris and Linux, so that
> the differing outputs have no meaning. I tried to debug my Java
> program, but without success so far, because I wasn't able to get
> into the Java program to set a breakpoint or to see the code. Have
> you succeeded to debug a mpiJava program? If so, how must I call
> gdb (I normally use "gdb mipexec" and then "run -np 1 java ...")?
> What can I do to get helpful information to track the error down?
> I have attached the error log file. Perhaps you can see if something
> is going wrong with the Java interface. Thank you very much for your
> help and any hints for the usage of gdb with mpiJava in advance.
> Please let me know if I can provide anything else.
> 
> 
> Kind regards
> 
> Siegmar
> 
> 
> > > I think that it must have to do with MPI, because everything
> > > works fine on Linux and my Java program works fine with an older
> > > MPI version (openmpi-1.8.2a1r31804) as well.
> > 
> > Yes. I also think it must have to do with MPI.
> > But java process side, not mpiexec process side.
> > 
> > When you run Java MPI program via mpiexec, a mpiexec process
> > process launch a java process. When the java process (your
> > Java program) calls a MPI method, native part (written in C/C++)
> > of the MPI library is called. It runs in java process, not in
> > mpiexec process. I suspect that part.
> > 
> > > On Solaris things are different.
> > 
> > Are you saying the following difference?
> > After this line,
> > > 881 ORTE_ACTIVATE_JOB_STATE(jdata, ORTE_JOB_STATE_INIT);
> > Linux shows
> > > orte_job_state_to_str (state=1)
> > > at ../../openmpi-dev-124-g91e9686/orte/util/error_strings.c:217
> > > 217 switch(state) {
> > but Solaris shows
> > > orte_util_print_name_args (name=0x100118380 )
> > > at ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:122
> > > 122 if (NULL == name) {
> > 
> > Each macro is defined as:
> > 
> > #define ORTE_ACTIVATE_JOB_STATE(j, s)   \
> > do {\
> > orte_job_t *shadow=(j); \
> > opal_output_verbose(1, orte_state_base_framework.framework_output, \
> > "%s ACTIVATE JOB %s STATE %s AT %s:%d", \
> > ORTE_NAME_PRINT(ORTE_PROC_MY_NAME), \
> > (NULL == shadow) ? "NULL" : \
> > ORTE_JOBID_PRINT(shadow->jobid),
> > \
> > orte_job_state_to_str((s)), \
> > __FILE__, __LINE__);\
> > orte_state.activate_job_state(shadow, (s)); \
> > } while(0);
> > 
> > #define ORTE_NAME_PRINT(n) \
> > orte_util_print_name_args(n)
> > 
> > #define ORTE_JOBID_PRINT(n) \
> > orte_util_print_jobids(n)
> > 
> > I'm not sure, but I think the gdb on Solaris steps 

Re: [OMPI users] which info is needed for SIGSEGV in Java foropenmpi-dev-124-g91e9686 on Solaris

[Oscar Vega-Gisbert]

Hello Siegmar,

If your Java program only calls to MPI.Init and MPI.Finalize you don't 
need debug Java. The JNI layer is very thin, so I think the problem is 
not in Java. Also, if the process crash is in the JNI side, debugging 
won't provides you useful information.


But if you want debug 2 processes, you can do the following.

You must launch 2 instances of the Java debugger (jdb) or NetBeans, 
Eclipse,... listening on port 8000.


The 2 processes must be launched with the necessary parameters to attach 
to the listening debuggers:


mpirun -np 2 java -agentlib:jdwp=transport=dt_socket,\
server=n,address=localhost:8000 Hello

I checked it on NetBeans and it works.
Here you have more details about debugging:

http://docs.oracle.com/javase/8/docs/technotes/guides/jpda/conninv.html

Regards,
Oscar

El 23/10/14 17:03, Siegmar Gross escribió:

Hello Oscar,

do you have time to look into my problem? Probably Takahiro has a
point and gdb behaves differently on Solaris and Linux, so that
the differing outputs have no meaning. I tried to debug my Java
program, but without success so far, because I wasn't able to get
into the Java program to set a breakpoint or to see the code. Have
you succeeded to debug a mpiJava program? If so, how must I call
gdb (I normally use "gdb mipexec" and then "run -np 1 java ...")?
What can I do to get helpful information to track the error down?
I have attached the error log file. Perhaps you can see if something
is going wrong with the Java interface. Thank you very much for your
help and any hints for the usage of gdb with mpiJava in advance.
Please let me know if I can provide anything else.


Kind regards

Siegmar



I think that it must have to do with MPI, because everything
works fine on Linux and my Java program works fine with an older
MPI version (openmpi-1.8.2a1r31804) as well.

Yes. I also think it must have to do with MPI.
But java process side, not mpiexec process side.

When you run Java MPI program via mpiexec, a mpiexec process
process launch a java process. When the java process (your
Java program) calls a MPI method, native part (written in C/C++)
of the MPI library is called. It runs in java process, not in
mpiexec process. I suspect that part.


On Solaris things are different.

Are you saying the following difference?
After this line,

881 ORTE_ACTIVATE_JOB_STATE(jdata, ORTE_JOB_STATE_INIT);

Linux shows

orte_job_state_to_str (state=1)
 at ../../openmpi-dev-124-g91e9686/orte/util/error_strings.c:217
217 switch(state) {

but Solaris shows

orte_util_print_name_args (name=0x100118380 )
 at ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:122
122 if (NULL == name) {

Each macro is defined as:

#define ORTE_ACTIVATE_JOB_STATE(j, s)   \
 do {\
 orte_job_t *shadow=(j); \
 opal_output_verbose(1, orte_state_base_framework.framework_output, \
 "%s ACTIVATE JOB %s STATE %s AT %s:%d",  \
 ORTE_NAME_PRINT(ORTE_PROC_MY_NAME), \
 (NULL == shadow) ? "NULL" : \
 ORTE_JOBID_PRINT(shadow->jobid),\
 orte_job_state_to_str((s)), \
 __FILE__, __LINE__);   \
 orte_state.activate_job_state(shadow, (s)); \
 } while(0);

#define ORTE_NAME_PRINT(n) \
 orte_util_print_name_args(n)

#define ORTE_JOBID_PRINT(n) \
 orte_util_print_jobids(n)

I'm not sure, but I think the gdb on Solaris steps into
orte_util_print_name_args, but gdb on Linux doesn't step into
orte_util_print_name_args and orte_util_print_jobids for some
reason, or orte_job_state_to_str is evaluated before them.

So I think it's not an important difference.

You showed the following lines.

orterun (argc=5, argv=0x7fffe0d8)
 at

../../../../openmpi-dev-124-g91e9686/orte/tools/orterun/orterun.c:1084

1084while (orte_event_base_active) {
(gdb)
1085opal_event_loop(orte_event_base, OPAL_EVLOOP_ONCE);
(gdb)

I'm not familiar with this code but I think this part (in mpiexec
process) is only waiting the java process to terminate (normally
or abnormally). So I think the problem is not in a mpiexec process
but in a java process.

Regards,
Takahiro


Hi Takahiro,


mpiexec and java run as distinct processes. Your JRE message
says java process raises SEGV. So you should trace the java
process, not the mpiexec process. And more, your JRE message
says the crash happened outside the Java Virtual Machine in
native code. So usual Java program debugger is useless.
You should trace native code part of the java process.
Unfortunately I don't know how to debug such one.

I think that it must have to do with MPI, 

Re: [OMPI users] which info is needed for SIGSEGV in Java foropenmpi-dev-124-g91e9686 on Solaris

[Siegmar Gross]

Hello Oscar,

do you have time to look into my problem? Probably Takahiro has a
point and gdb behaves differently on Solaris and Linux, so that
the differing outputs have no meaning. I tried to debug my Java
program, but without success so far, because I wasn't able to get
into the Java program to set a breakpoint or to see the code. Have
you succeeded to debug a mpiJava program? If so, how must I call
gdb (I normally use "gdb mipexec" and then "run -np 1 java ...")?
What can I do to get helpful information to track the error down?
I have attached the error log file. Perhaps you can see if something
is going wrong with the Java interface. Thank you very much for your
help and any hints for the usage of gdb with mpiJava in advance.
Please let me know if I can provide anything else.


Kind regards

Siegmar


> > I think that it must have to do with MPI, because everything
> > works fine on Linux and my Java program works fine with an older
> > MPI version (openmpi-1.8.2a1r31804) as well.
> 
> Yes. I also think it must have to do with MPI.
> But java process side, not mpiexec process side.
> 
> When you run Java MPI program via mpiexec, a mpiexec process
> process launch a java process. When the java process (your
> Java program) calls a MPI method, native part (written in C/C++)
> of the MPI library is called. It runs in java process, not in
> mpiexec process. I suspect that part.
> 
> > On Solaris things are different.
> 
> Are you saying the following difference?
> After this line,
> > 881 ORTE_ACTIVATE_JOB_STATE(jdata, ORTE_JOB_STATE_INIT);
> Linux shows
> > orte_job_state_to_str (state=1)
> > at ../../openmpi-dev-124-g91e9686/orte/util/error_strings.c:217
> > 217 switch(state) {
> but Solaris shows
> > orte_util_print_name_args (name=0x100118380 )
> > at ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:122
> > 122 if (NULL == name) {
> 
> Each macro is defined as:
> 
> #define ORTE_ACTIVATE_JOB_STATE(j, s)   \
> do {\
> orte_job_t *shadow=(j); \
> opal_output_verbose(1, orte_state_base_framework.framework_output, \
> "%s ACTIVATE JOB %s STATE %s AT %s:%d",   \
> ORTE_NAME_PRINT(ORTE_PROC_MY_NAME), \
> (NULL == shadow) ? "NULL" : \
> ORTE_JOBID_PRINT(shadow->jobid),  \
> orte_job_state_to_str((s)), \
> __FILE__, __LINE__);  \
> orte_state.activate_job_state(shadow, (s)); \
> } while(0);
> 
> #define ORTE_NAME_PRINT(n) \
> orte_util_print_name_args(n)
> 
> #define ORTE_JOBID_PRINT(n) \
> orte_util_print_jobids(n)
> 
> I'm not sure, but I think the gdb on Solaris steps into
> orte_util_print_name_args, but gdb on Linux doesn't step into
> orte_util_print_name_args and orte_util_print_jobids for some
> reason, or orte_job_state_to_str is evaluated before them.
> 
> So I think it's not an important difference.
> 
> You showed the following lines.
> > > > orterun (argc=5, argv=0x7fffe0d8)
> > > > at 
../../../../openmpi-dev-124-g91e9686/orte/tools/orterun/orterun.c:1084
> > > > 1084while (orte_event_base_active) {
> > > > (gdb) 
> > > > 1085opal_event_loop(orte_event_base, OPAL_EVLOOP_ONCE);
> > > > (gdb) 
> 
> I'm not familiar with this code but I think this part (in mpiexec
> process) is only waiting the java process to terminate (normally
> or abnormally). So I think the problem is not in a mpiexec process
> but in a java process.
> 
> Regards,
> Takahiro
> 
> > Hi Takahiro,
> > 
> > > mpiexec and java run as distinct processes. Your JRE message
> > > says java process raises SEGV. So you should trace the java
> > > process, not the mpiexec process. And more, your JRE message
> > > says the crash happened outside the Java Virtual Machine in
> > > native code. So usual Java program debugger is useless.
> > > You should trace native code part of the java process.
> > > Unfortunately I don't know how to debug such one.
> > 
> > I think that it must have to do with MPI, because everything
> > works fine on Linux and my Java program works fine with an older
> > MPI version (openmpi-1.8.2a1r31804) as well.
> > 
> > linpc1 x 112 mpiexec -np 1 java InitFinalizeMain
> > Hello!
> > linpc1 x 113 
> > 
> > Therefore I single stepped through the program on Linux as well
> > and found a difference launching the process. On Linux I get the
> > following sequence.
> > 
> > Breakpoint 1, rsh_launch (jdata=0x614aa0)
> > at 
../../../../../openmpi-dev-124-g91e9686/orte/mca/plm/rsh/plm_rsh_module.c:876
> > 876 if (ORTE_FLAG_TEST(jdata, ORTE_JOB_FLAG_RESTART)) {
> > (gdb) s
> > 881 

Re: [OMPI users] low CPU utilization with OpenMPI

[Jeff Squyres (jsquyres)]

If normal users can't write to /tmp (or if /tmp is an NFS-mounted filesystem), 
that's the underlying problem.

@Vinson -- you should probably try to get that fixed.



On Oct 23, 2014, at 10:35 AM, Joshua Ladd  wrote:

> It's not coming from OSHMEM but from the OPAL "shmem" framework. You are 
> going to get terrible performance - possibly slowing to a crawl having all 
> processes open their backing files for mmap on NSF. I think that's the error 
> that he's getting.
> 
> 
> Josh
> 
> On Thu, Oct 23, 2014 at 6:06 AM, Vinson Leung  wrote:
> HI, Thanks for your reply:)
> I really run an MPI program (compile with OpenMPI and run with "mpirun -n 8 
> .."). My OpenMPI version is 1.8.3 and my program is Gromacs. BTW, what is 
> OSHMEM ?
> 
> Best
> Vincent
> 
> On Thu, Oct 23, 2014 at 12:21 PM, Ralph Castain  wrote:
> From your error message, I gather you are not running an MPI program, but 
> rather an OSHMEM one? Otherwise, I find the message strange as it only would 
> be emitted from an OSHMEM program.
> 
> What version of OMPI are you trying to use?
> 
>> On Oct 22, 2014, at 7:12 PM, Vinson Leung  wrote:
>> 
>> Thanks for your reply:)
>> Follow your advice I tried to set the TMPDIR to /var/tmp and /dev/shm and 
>> even reset to /tmp (I get the system permission), the problem still occur 
>> (CPU utilization still lower than 20%). I have no idea why and ready to give 
>> up OpenMPI instead of using other MPI library.
>> 
>> Old Message-
>> 
>> Date: Tue, 21 Oct 2014 22:21:31 -0400
>> From: Brock Palen 
>> To: Open MPI Users 
>> Subject: Re: [OMPI users] low CPU utilization with OpenMPI
>> Message-ID: 
>> Content-Type: text/plain; charset=us-ascii
>> 
>> Doing special files on NFS can be weird,  try the other /tmp/ locations:
>> 
>> /var/tmp/
>> /dev/shm  (ram disk careful!)
>> 
>> Brock Palen
>> www.umich.edu/~brockp
>> CAEN Advanced Computing
>> XSEDE Campus Champion
>> bro...@umich.edu
>> (734)936-1985
>> 
>> 
>> 
>> > On Oct 21, 2014, at 10:18 PM, Vinson Leung  wrote:
>> >
>> > Because of permission reason (OpenMPI can not write temporary file to the 
>> > default /tmp directory), I change the TMPDIR to my local directory (export 
>> > TMPDIR=/home/user/tmp ) and then the MPI program can run. But the CPU 
>> > utilization is very low under 20% (8 MPI rank running in Intel Xeon 8-core 
>> > CPU).
>> >
>> > And I also got some message when I run with OpenMPI:
>> > [cn3:28072] 9 more processes have sent help message 
>> > help-opal-shmem-mmap.txt / mmap on nfs
>> > [cn3:28072] Set MCA parameter "orte_base_help_aggregate" to 0 to see all 
>> > help / error messages
>> >
>> > Any idea?
>> > Thanks
>> >
>> > VIncent
>> > ___
>> > users mailing list
>> > us...@open-mpi.org
>> > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> > Link to this post: 
>> > http://www.open-mpi.org/community/lists/users/2014/10/25548.php
>> ___
>> users mailing list
>> us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post: 
>> http://www.open-mpi.org/community/lists/users/2014/10/2.php
> 
> 
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2014/10/25556.php
> 
> 
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2014/10/25558.php
> 
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2014/10/25560.php


-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to: 
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI users] low CPU utilization with OpenMPI

[Joshua Ladd]

It's not coming from OSHMEM but from the OPAL "shmem" framework. You are
going to get terrible performance - possibly slowing to a crawl having all
processes open their backing files for mmap on NSF. I think that's the
error that he's getting.


Josh

On Thu, Oct 23, 2014 at 6:06 AM, Vinson Leung 
wrote:

> HI, Thanks for your reply:)
> I really run an MPI program (compile with OpenMPI and run with "mpirun -n
> 8 .."). My OpenMPI version is 1.8.3 and my program is Gromacs. BTW,
> what is OSHMEM ?
>
> Best
> Vincent
>
> On Thu, Oct 23, 2014 at 12:21 PM, Ralph Castain  wrote:
>
>> From your error message, I gather you are not running an MPI program, but
>> rather an OSHMEM one? Otherwise, I find the message strange as it only
>> would be emitted from an OSHMEM program.
>>
>> What version of OMPI are you trying to use?
>>
>> On Oct 22, 2014, at 7:12 PM, Vinson Leung 
>> wrote:
>>
>> Thanks for your reply:)
>> Follow your advice I tried to set the TMPDIR to /var/tmp and /dev/shm and
>> even reset to /tmp (I get the system permission), the problem still occur
>> (CPU utilization still lower than 20%). I have no idea why and ready to
>> give up OpenMPI instead of using other MPI library.
>>
>> Old Message-
>>
>> Date: Tue, 21 Oct 2014 22:21:31 -0400
>> From: Brock Palen 
>> To: Open MPI Users 
>> Subject: Re: [OMPI users] low CPU utilization with OpenMPI
>> Message-ID: 
>> Content-Type: text/plain; charset=us-ascii
>>
>> Doing special files on NFS can be weird,  try the other /tmp/ locations:
>>
>> /var/tmp/
>> /dev/shm  (ram disk careful!)
>>
>> Brock Palen
>> www.umich.edu/~brockp
>> CAEN Advanced Computing
>> XSEDE Campus Champion
>> bro...@umich.edu
>> (734)936-1985
>>
>>
>>
>> > On Oct 21, 2014, at 10:18 PM, Vinson Leung 
>> wrote:
>> >
>> > Because of permission reason (OpenMPI can not write temporary file to
>> the default /tmp directory), I change the TMPDIR to my local directory
>> (export TMPDIR=/home/user/tmp ) and then the MPI program can run. But the
>> CPU utilization is very low under 20% (8 MPI rank running in Intel Xeon
>> 8-core CPU).
>> >
>> > And I also got some message when I run with OpenMPI:
>> > [cn3:28072] 9 more processes have sent help message
>> help-opal-shmem-mmap.txt / mmap on nfs
>> > [cn3:28072] Set MCA parameter "orte_base_help_aggregate" to 0 to see
>> all help / error messages
>> >
>> > Any idea?
>> > Thanks
>> >
>> > VIncent
>> > ___
>> > users mailing list
>> > us...@open-mpi.org
>> > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> > Link to this post:
>> http://www.open-mpi.org/community/lists/users/2014/10/25548.php
>> ___
>> users mailing list
>> us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post:
>> http://www.open-mpi.org/community/lists/users/2014/10/2.php
>>
>>
>>
>> ___
>> users mailing list
>> us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post:
>> http://www.open-mpi.org/community/lists/users/2014/10/25556.php
>>
>
>
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2014/10/25558.php
>

Re: [OMPI users] which info is needed for SIGSEGV in Java for openmpi-dev-124-g91e9686 on Solaris

[Kawashima, Takahiro]

Hi Siegmar,

> I think that it must have to do with MPI, because everything
> works fine on Linux and my Java program works fine with an older
> MPI version (openmpi-1.8.2a1r31804) as well.

Yes. I also think it must have to do with MPI.
But java process side, not mpiexec process side.

When you run Java MPI program via mpiexec, a mpiexec process
process launch a java process. When the java process (your
Java program) calls a MPI method, native part (written in C/C++)
of the MPI library is called. It runs in java process, not in
mpiexec process. I suspect that part.

> On Solaris things are different.

Are you saying the following difference?
After this line,
> 881 ORTE_ACTIVATE_JOB_STATE(jdata, ORTE_JOB_STATE_INIT);
Linux shows
> orte_job_state_to_str (state=1)
> at ../../openmpi-dev-124-g91e9686/orte/util/error_strings.c:217
> 217 switch(state) {
but Solaris shows
> orte_util_print_name_args (name=0x100118380 )
> at ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:122
> 122 if (NULL == name) {

Each macro is defined as:

#define ORTE_ACTIVATE_JOB_STATE(j, s)   \
do {\
orte_job_t *shadow=(j); \
opal_output_verbose(1, orte_state_base_framework.framework_output, \
"%s ACTIVATE JOB %s STATE %s AT %s:%d", \
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME), \
(NULL == shadow) ? "NULL" : \
ORTE_JOBID_PRINT(shadow->jobid),\
orte_job_state_to_str((s)), \
__FILE__, __LINE__);\
orte_state.activate_job_state(shadow, (s)); \
} while(0);

#define ORTE_NAME_PRINT(n) \
orte_util_print_name_args(n)

#define ORTE_JOBID_PRINT(n) \
orte_util_print_jobids(n)

I'm not sure, but I think the gdb on Solaris steps into
orte_util_print_name_args, but gdb on Linux doesn't step into
orte_util_print_name_args and orte_util_print_jobids for some
reason, or orte_job_state_to_str is evaluated before them.

So I think it's not an important difference.

You showed the following lines.
> > > orterun (argc=5, argv=0x7fffe0d8)
> > > at 
> > > ../../../../openmpi-dev-124-g91e9686/orte/tools/orterun/orterun.c:1084
> > > 1084while (orte_event_base_active) {
> > > (gdb) 
> > > 1085opal_event_loop(orte_event_base, OPAL_EVLOOP_ONCE);
> > > (gdb) 

I'm not familiar with this code but I think this part (in mpiexec
process) is only waiting the java process to terminate (normally
or abnormally). So I think the problem is not in a mpiexec process
but in a java process.

Regards,
Takahiro

> Hi Takahiro,
> 
> > mpiexec and java run as distinct processes. Your JRE message
> > says java process raises SEGV. So you should trace the java
> > process, not the mpiexec process. And more, your JRE message
> > says the crash happened outside the Java Virtual Machine in
> > native code. So usual Java program debugger is useless.
> > You should trace native code part of the java process.
> > Unfortunately I don't know how to debug such one.
> 
> I think that it must have to do with MPI, because everything
> works fine on Linux and my Java program works fine with an older
> MPI version (openmpi-1.8.2a1r31804) as well.
> 
> linpc1 x 112 mpiexec -np 1 java InitFinalizeMain
> Hello!
> linpc1 x 113 
> 
> Therefore I single stepped through the program on Linux as well
> and found a difference launching the process. On Linux I get the
> following sequence.
> 
> Breakpoint 1, rsh_launch (jdata=0x614aa0)
> at 
> ../../../../../openmpi-dev-124-g91e9686/orte/mca/plm/rsh/plm_rsh_module.c:876
> 876 if (ORTE_FLAG_TEST(jdata, ORTE_JOB_FLAG_RESTART)) {
> (gdb) s
> 881 ORTE_ACTIVATE_JOB_STATE(jdata, ORTE_JOB_STATE_INIT);
> (gdb) s
> orte_job_state_to_str (state=1)
> at ../../openmpi-dev-124-g91e9686/orte/util/error_strings.c:217
> 217 switch(state) {
> (gdb) 
> 221 return "PENDING INIT";
> (gdb) 
> 317 }
> (gdb) 
> orte_util_print_jobids (job=4294967295)
> at ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:170
> 170 ptr = get_print_name_buffer();
> (gdb) 
> 
> 
> 
> On Solaris things are different.
> 
> Breakpoint 1, rsh_launch (jdata=0x100125250)
> at 
> ../../../../../openmpi-dev-124-g91e9686/orte/mca/plm/rsh/plm_rsh_module.c:876
> 876 if (ORTE_FLAG_TEST(jdata, ORTE_JOB_FLAG_RESTART)) {
> (gdb) s
> 881 ORTE_ACTIVATE_JOB_STATE(jdata, ORTE_JOB_STATE_INIT);
> (gdb) s
> orte_util_print_name_args (name=0x100118380 )
> at ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:122
> 122 if (NULL == name) {
> (gdb) 
> 142 job = 

Re: [OMPI users] low CPU utilization with OpenMPI

[Vinson Leung]

HI, Thanks for your reply:)
I really run an MPI program (compile with OpenMPI and run with "mpirun -n 8
.."). My OpenMPI version is 1.8.3 and my program is Gromacs. BTW, what
is OSHMEM ?

Best
Vincent

On Thu, Oct 23, 2014 at 12:21 PM, Ralph Castain  wrote:

> From your error message, I gather you are not running an MPI program, but
> rather an OSHMEM one? Otherwise, I find the message strange as it only
> would be emitted from an OSHMEM program.
>
> What version of OMPI are you trying to use?
>
> On Oct 22, 2014, at 7:12 PM, Vinson Leung  wrote:
>
> Thanks for your reply:)
> Follow your advice I tried to set the TMPDIR to /var/tmp and /dev/shm and
> even reset to /tmp (I get the system permission), the problem still occur
> (CPU utilization still lower than 20%). I have no idea why and ready to
> give up OpenMPI instead of using other MPI library.
>
> Old Message-
>
> Date: Tue, 21 Oct 2014 22:21:31 -0400
> From: Brock Palen 
> To: Open MPI Users 
> Subject: Re: [OMPI users] low CPU utilization with OpenMPI
> Message-ID: 
> Content-Type: text/plain; charset=us-ascii
>
> Doing special files on NFS can be weird,  try the other /tmp/ locations:
>
> /var/tmp/
> /dev/shm  (ram disk careful!)
>
> Brock Palen
> www.umich.edu/~brockp
> CAEN Advanced Computing
> XSEDE Campus Champion
> bro...@umich.edu
> (734)936-1985
>
>
>
> > On Oct 21, 2014, at 10:18 PM, Vinson Leung 
> wrote:
> >
> > Because of permission reason (OpenMPI can not write temporary file to
> the default /tmp directory), I change the TMPDIR to my local directory
> (export TMPDIR=/home/user/tmp ) and then the MPI program can run. But the
> CPU utilization is very low under 20% (8 MPI rank running in Intel Xeon
> 8-core CPU).
> >
> > And I also got some message when I run with OpenMPI:
> > [cn3:28072] 9 more processes have sent help message
> help-opal-shmem-mmap.txt / mmap on nfs
> > [cn3:28072] Set MCA parameter "orte_base_help_aggregate" to 0 to see all
> help / error messages
> >
> > Any idea?
> > Thanks
> >
> > VIncent
> > ___
> > users mailing list
> > us...@open-mpi.org
> > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> > Link to this post:
> http://www.open-mpi.org/community/lists/users/2014/10/25548.php
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2014/10/2.php
>
>
>
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2014/10/25556.php
>

Re: [OMPI users] New ib locked pages behavior?

[rf]

> "Bill" == Bill Broadley  writes:

Hi Bill,

>> I've commented in detail on this (non-)issue on 2014-08-20:

>> http://www.open-mpi.org/community/lists/users/2014/08/25090.php

Bill> I read that.  It seems pretty clear what the problem is, but
Bill> not so clear on what a user experiencing this problem should
Bill> do about it.

sorry for not to have made that clear enough.

Bill> So for people who are using ubuntu 14.04 and openmpi-1.6.5 and
Bill> 64 GB nodes.

Bill> Should they:
Bill> * bump log_mtts_per_seg from 3 to 4 (64GB) or 5 (128GB)?

No.

Bill> * ignore the error message because it doesn't apply?

Fortunately yes :)

Bill> * ditch ubuntu's packagedopenmpi 1.6.5 and all the packages
Bill>   that depends on it and install something newer than
Bill>   1.8.2rc4?

Not necessary.

-- 
Roland

---
http://www.q-leap.com / http://qlustar.com
  --- HPC / Storage / Cloud Linux Cluster OS ---

Re: [OMPI users] low CPU utilization with OpenMPI

[Ralph Castain]

From your error message, I gather you are not running an MPI program, but 
rather an OSHMEM one? Otherwise, I find the message strange as it only would be 
emitted from an OSHMEM program.

What version of OMPI are you trying to use?

> On Oct 22, 2014, at 7:12 PM, Vinson Leung  wrote:
> 
> Thanks for your reply:)
> Follow your advice I tried to set the TMPDIR to /var/tmp and /dev/shm and 
> even reset to /tmp (I get the system permission), the problem still occur 
> (CPU utilization still lower than 20%). I have no idea why and ready to give 
> up OpenMPI instead of using other MPI library.
> 
> Old Message-
> 
> Date: Tue, 21 Oct 2014 22:21:31 -0400
> From: Brock Palen >
> To: Open MPI Users >
> Subject: Re: [OMPI users] low CPU utilization with OpenMPI
> Message-ID:  >
> Content-Type: text/plain; charset=us-ascii
> 
> Doing special files on NFS can be weird,  try the other /tmp/ locations:
> 
> /var/tmp/
> /dev/shm  (ram disk careful!)
> 
> Brock Palen
> www.umich.edu/~brockp 
> CAEN Advanced Computing
> XSEDE Campus Champion
> bro...@umich.edu 
> (734)936-1985
> 
> 
> 
> > On Oct 21, 2014, at 10:18 PM, Vinson Leung  > > wrote:
> >
> > Because of permission reason (OpenMPI can not write temporary file to the 
> > default /tmp directory), I change the TMPDIR to my local directory (export 
> > TMPDIR=/home/user/tmp ) and then the MPI program can run. But the CPU 
> > utilization is very low under 20% (8 MPI rank running in Intel Xeon 8-core 
> > CPU).
> >
> > And I also got some message when I run with OpenMPI:
> > [cn3:28072] 9 more processes have sent help message 
> > help-opal-shmem-mmap.txt / mmap on nfs
> > [cn3:28072] Set MCA parameter "orte_base_help_aggregate" to 0 to see all 
> > help / error messages
> >
> > Any idea?
> > Thanks
> >
> > VIncent
> > ___
> > users mailing list
> > us...@open-mpi.org 
> > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users 
> > 
> > Link to this post: 
> > http://www.open-mpi.org/community/lists/users/2014/10/25548.php 
> > ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2014/10/2.php