Re: [OMPI users] MPI Error

Jeff Squyres Sun, 21 Dec 2008 08:18:55 -0500

It's not the valgrind that are telling here -- it's the fact thatsomething is saying "INJECTING SIGNAL..."

It looks like something in your program is calling abort() (signal 6is the abort signal).



On Dec 21, 2008, at 4:08 AM, Asad Ali wrote:

--===============1267833741==
Content-Type: multipart/alternative;
 boundary="----=_Part_48219_1955003.1229850505418"

------=_Part_48219_1955003.1229850505418
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

Hi Jeff,
I have changed the position of malloc.h in header files list. Imoved it up
above mpi.h. Now I am getting a different error message see following,
[asad@stat74 T]$ mpirun --np 4 nice -10 ./lisa09EMRIT-P
+---[ lisa14.c ]---------------------------------------------------
| This is process #0 of 4
| initialising : 21.12.2008 21:59:28
Allocating 134217728 bytes for read buffer...
Allocating 134217728 bytes for read buffer...
Allocating 134217728 bytes for read buffer...
Allocating 134217728 bytes for read buffer...
Allocating 134217728 bytes for read buffer...
Allocating 134217728 bytes for read buffer...
Allocating 134217728 bytes for read buffer...
-+--> INJECTING SIGNAL...
Allocating 134217728 bytes for read buffer...
-+--> INJECTING SIGNAL...
-+--> INJECTING SIGNAL...
-+--> INJECTING SIGNAL...
524765
524765
524765
524765
| initialising `TDI framework' (might take several minutes)...
Signal:6 info.si_errno:0(Success) si_code:-6()
[0] func:/usr/lib/openmpi/libopal.so.0 [0x46efd6]
[1] func:[0x110440]
[2] func:[0x110402]
[3] func:/lib/libc.so.6(gsignal+0x50) [0x2c1fa0]
[4] func:/lib/libc.so.6(abort+0x101) [0x2c38b1]
[5] func:/usr/lib/openmpi/libopal.so.0 [0x473d34]
[6] func:/usr/lib/openmpi/libopal.so.0(free+0x36) [0x473786]
[7] func:./lisa09EMRIT-P(main+0x454) [0x80566f4]
[8] func:/lib/libc.so.6(__libc_start_main+0xe0) [0x2aef70]
[9] func:./lisa09EMRIT-P [0x804af21]
*** End of error message ***
Its good that now no user defined functions in code are involved inerror
(it was giving error messages for some of my defined functions too).
malloc.h has also been disappeared in this error message as was inprevious
message.
I have used valgrind to see where the error is but I am unable tointerpret
the following valgrind output error message.


--16515-- object doesn't have a symbol table
--16515-- Reading syms from /usr/lib/openmpi/openmpi/mca_osc_pt2pt.so
(0x4114000)
--16515-- object doesn't have a symbol table
==16513== Warning: set address range perms: large range 536870912(defined)==16512== Warning: set address range perms: large range 536870912(defined)==16514== Warning: set address range perms: large range 536870912(defined)==16515== Warning: set address range perms: large range 536870912(defined)
+---[ lisa14.c ]---------------------------------------------------
| This is process #0 of 4
| initialising : 21.12.2008 21:15:37
Allocating 134217728 bytes for read buffer...
==16515== Warning: set address range perms: large range 134221824(defined)
Allocating 134217728 bytes for read buffer...
==16514== Warning: set address range perms: large range 134221824(defined)
Allocating 134217728 bytes for read buffer...
==16513== Warning: set address range perms: large range 134221824(defined)
Allocating 134217728 bytes for read buffer...
==16512== Warning: set address range perms: large range 134221824(defined)==16515== Warning: set address range perms: large range 134217728(defined)==16514== Warning: set address range perms: large range 134217728(defined)==16513== Warning: set address range perms: large range 134217728(defined)==16512== Warning: set address range perms: large range 134217728(defined)==16515== Warning: set address range perms: large range 134221824(noaccess)==16514== Warning: set address range perms: large range 134221824(noaccess)==16513== Warning: set address range perms: large range 134221824(noaccess)==16512== Warning: set address range perms: large range 134221824(noaccess)
--16515-- memcheck GC: 1024 nodes, 1024 survivors (100.0%)
--16515-- memcheck GC: increase table size to 2048
--16513-- memcheck GC: 1024 nodes, 1024 survivors (100.0%)
--16513-- memcheck GC: increase table size to 2048
--16514-- memcheck GC: 1024 nodes, 1024 survivors (100.0%)
--16514-- memcheck GC: increase table size to 2048
--16512-- memcheck GC: 1024 nodes, 1024 survivors (100.0%)
--16512-- memcheck GC: increase table size to 2048
--16513-- memcheck GC: 2048 nodes, 2048 survivors (100.0%)
--16513-- memcheck GC: increase table size to 4096
--16514-- memcheck GC: 2048 nodes, 2048 survivors (100.0%)
--16514-- memcheck GC: increase table size to 4096
--16515-- memcheck GC: 2048 nodes, 2048 survivors (100.0%)
--16515-- memcheck GC: increase table size to 4096
--16512-- memcheck GC: 2048 nodes, 2048 survivors (100.0%)
--16512-- memcheck GC: increase table size to 4096
Allocating 134217728 bytes for read buffer...
==16515== Warning: set address range perms: large range 134221824(defined)
Allocating 134217728 bytes for read buffer...
==16513== Warning: set address range perms: large range 134221824(defined)
Allocating 134217728 bytes for read buffer...
==16514== Warning: set address range perms: large range 134221824(defined)==16515== Warning: set address range perms: large range 134217728(defined)==16513== Warning: set address range perms: large range 134217728(defined)==16514== Warning: set address range perms: large range 134217728(defined)==16515== Warning: set address range perms: large range 134221824(noaccess)
Allocating 134217728 bytes for read buffer...
==16512== Warning: set address range perms: large range 134221824(defined)==16513== Warning: set address range perms: large range 134221824(noaccess)==16512== Warning: set address range perms: large range 134217728(defined)==16514== Warning: set address range perms: large range 134221824(noaccess)==16512== Warning: set address range perms: large range 134221824(noaccess)
-+--> INJECTING SIGNAL...
-+--> INJECTING SIGNAL...
-+--> INJECTING SIGNAL...
-+--> INJECTING SIGNAL...
524765
524765
524765
524765
Signal:6 info.si_errno:0(Success) si_code:-6()
--16515-- Reading syms from /lib/libgcc_s-4.1.2-20070925.so.1(0x794000)
--16515-- object doesn't have a symbol table
[0] func:/usr/lib/openmpi/libopal.so.0 [0x46efd6]
[1] func:/lib/libpthread.so.0 [0x443290]
[2] func:/lib/ld-linux.so.2 [0x2767f2]
[3] func:/lib/libc.so.6(gsignal+0x50) [0x2c1fa0]
[4] func:/lib/libc.so.6(abort+0x101) [0x2c38b1]
[5] func:/usr/lib/openmpi/libopal.so.0 [0x473d34]
[6] func:/usr/lib/openmpi/libopal.so.0(free+0x36) [0x473786]
[7] func:./lisa09EMRIT-P(main+0x44c) [0x80566ec]
[8] func:/lib/libc.so.6(__libc_start_main+0xe0) [0x2aef70]
[9] func:./lisa09EMRIT-P [0x804af21]
*** End of error message ***
==16515==
==16515== Process terminating with default action of signal 6(SIGABRT)
==16515== at 0x2C1FA0: raise (in /lib/libc-2.6.so)
==16515== by 0x2C39FD: abort (in /lib/libc-2.6.so)
==16515== by 0x473D33: (within /usr/lib/openmpi/libopal.so.0.0.0)
==16515== by 0x473785: free (in /usr/lib/openmpi/libopal.so.0.0.0)
==16515== by 0x80566EB: main (in /home/phd/asad/MLDCs/T/lisa09EMRIT-P)
==16515==
==16515== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 108from 1)
--16515--
--16515-- supp: 108 dl-hack3
==16515== malloc/free: in use at exit: 0 bytes in 0 blocks.
==16515== malloc/free: 0 allocs, 0 frees, 0 bytes allocated.
==16515==
==16515== All heap blocks were freed -- no leaks are possible.
--16515-- memcheck: sanity checks: 22725 cheap, 910 expensive
--16515-- memcheck: auxmaps: 0 auxmap entries (0k, 0M) in use
--16515-- memcheck: auxmaps: 0 searches, 0 comparisons
--16515-- memcheck: SMs: n_issued = 6014 (96224k, 93M)
--16515-- memcheck: SMs: n_deissued = 4726 (75616k, 73M)
--16515-- memcheck: SMs: max_noaccess = 65535 (1048560k, 1023M)
--16515-- memcheck: SMs: max_undefined = 38 (608k, 0M)
--16515-- memcheck: SMs: max_defined = 10813 (173008k, 168M)
--16515-- memcheck: SMs: max_non_DSM = 3109 (49744k, 48M)
--16515-- memcheck: max sec V bit nodes: 2094 (106k, 0M)
--16515-- memcheck: set_sec_vbits8 calls: 3371606 (new: 2094,updates:
3369512)
--16515-- memcheck: max shadow mem size: 50154k, 48M
--16515-- translate: fast SP updates identified: 23,603( 89.1%)--16515-- translate: generic_known SP updates identified: 2,000( 7.5%)--16515-- translate: generic_unknown SP updates identified: 877( 3.3%)
--16515-- tt/tc: 625,692 tt lookups requiring 672,349 probes
--16515-- tt/tc: 625,692 fast-cache updates, 9 flushes
--16515-- transtab: new 20,764 (798,227 -> 14,916,839; ratio186:10)
[0 scs]
--16515-- transtab: dumped 0 (0 -> ??)
--16515-- transtab: discarded 193 (3,337 -> ??)
--16515-- scheduler: 2,272,550,211 jumps (bb entries).
--16515-- scheduler: 22,725/32,791,636 major/minor sched events.
--16515-- sanity: 22726 cheap, 910 expensive checks.
--16515-- exectx: 30,011 lists, 11 contexts (avg 0 per list)
--16515-- exectx: 108 searches, 97 full compares (898 per 1000)
--16515-- exectx: 0 cmp2, 232 cmp4, 0 cmpAll
[8]+ Killed mpirun -np 4 valgrind -v ./lisa09EMRIT-P

Could please tell me how ti interpret the results of valgrind?

Thanking you in advance.

Asad

------=_Part_48219_1955003.1229850505418
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
Hi Jeff,I have changed the position of malloc.h in header files list. I moved it up above mpi.h. Now I am getting a different errormessage see following,[asad@stat74 T]$ mpirun  --np 4 nice -10 ./lisa09EMRIT-P  +---[ lisa14.c ]---------------------------------------------------  | This is process #0 of 4  |initialising  :  21.12.2008  21:59:28 Allocating134217728 bytes for read buffer... Allocating 134217728 bytes for read buffer... Allocating134217728 bytes for read buffer... Allocating 134217728 bytes forread buffer... Allocating 134217728 bytes for readbuffer... Allocating 134217728 bytes for read buffer... Allocating 134217728 bytes for read buffer... -+--> INJECTINGSIGNAL... Allocating 134217728 bytes for read buffer... -+--> INJECTING SIGNAL... -+--> INJECTING SIGNAL... -+-->INJECTING SIGNAL... 524765 524765 524765 524765  | initialising `TDIframework' (might take several minutes)... Signal:6info.si_errno:0(Success) si_code:-6() [0] func:/usr/lib/openmpi/libopal.so.0 [0x46efd6] [1] func:[0x110440] [2] func:[0x110402] [3] func:/lib/libc.so.6(gsignal+0x50)[0x2c1fa0] [4] func:/lib/libc.so.6(abort+0x101) [0x2c38b1] [5]func:/usr/lib/openmpi/libopal.so.0 [0x473d34] [6] func:/usr/lib/openmpi/libopal.so.0(free+0x36) [0x473786] [7] func:./lisa09EMRIT-P(main+0x454) [0x80566f4] [8] func:/lib/libc.so.6(__libc_start_main+0xe0) [0x2aef70] [9] func:./lisa09EMRIT-P [0x804af21] *** End of error message *** Itsgood that now no user defined functions in code are involved inerror (it was giving error messages for some of my defined functionstoo). malloc.h has also been disappeared in this error message aswas in previous message. I have used valgrind to see where the error is but I am unableto interpret the following valgrind output errormessage. --16515--    object doesn'thave a symbol table --16515-- Reading syms from /usr/lib/openmpi/openmpi/mca_osc_pt2pt.so (0x4114000) --16515--    object doesn't have a symboltable ==16513== Warning: set address range perms: large range536870912 (defined) ==16512== Warning: set address range perms:large range 536870912 (defined) ==16514== Warning: set addressrange perms: large range 536870912 (defined) ==16515== Warning: set address range perms: large range 536870912(defined)  +---[ lisa14.c ]---------------------------------------------------  | This is process #0 of 4  |initialising  :  21.12.2008  21:15:37 Allocating 134217728 bytes for read buffer... ==16515== Warning:set address range perms: large range 134221824(defined) Allocating 134217728 bytes for readbuffer... ==16514== Warning: set address range perms: large range134221824 (defined) Allocating 134217728 bytes for read buffer... ==16513== Warning:set address range perms: large range 134221824(defined) Allocating 134217728 bytes for readbuffer... ==16512== Warning: set address range perms: large range134221824 (defined) ==16515== Warning: set address range perms: large range 134217728(defined) ==16514== Warning: set address range perms: large range134217728 (defined) ==16513== Warning: set address range perms:large range 134217728 (defined) ==16512== Warning: set address range perms: large range 134217728(defined) ==16515== Warning: set address range perms: large range134221824 (noaccess) ==16514== Warning: set address range perms:large range 134221824 (noaccess) ==16513== Warning: set address range perms: large range 134221824(noaccess) ==16512== Warning: set address range perms: largerange 134221824 (noaccess) --16515-- memcheck GC: 1024 nodes,1024 survivors (100.0%) --16515-- memcheck GC: increase table size to 2048 --16513--memcheck GC: 1024 nodes, 1024 survivors (100.0%) --16513--memcheck GC: increase table size to 2048 --16514-- memcheck GC:1024 nodes, 1024 survivors (100.0%) --16514-- memcheck GC: increase table size to 2048 --16512--memcheck GC: 1024 nodes, 1024 survivors (100.0%) --16512--memcheck GC: increase table size to 2048 --16513-- memcheck GC:2048 nodes, 2048 survivors (100.0%) --16513-- memcheck GC: increase table size to 4096 --16514--memcheck GC: 2048 nodes, 2048 survivors (100.0%) --16514--memcheck GC: increase table size to 4096 --16515-- memcheck GC:2048 nodes, 2048 survivors (100.0%) --16515-- memcheck GC: increase table size to 4096 --16512--memcheck GC: 2048 nodes, 2048 survivors (100.0%) --16512--memcheck GC: increase table size to 4096 Allocating 134217728bytes for read buffer... ==16515== Warning: set address rangeperms: large range 134221824 (defined) Allocating 134217728 bytes for read buffer... ==16513== Warning:set address range perms: large range 134221824(defined) Allocating 134217728 bytes for readbuffer... ==16514== Warning: set address range perms: large range134221824 (defined) ==16515== Warning: set address range perms: large range 134217728(defined) ==16513== Warning: set address range perms: large range134217728 (defined) ==16514== Warning: set address range perms:large range 134217728 (defined) ==16515== Warning: set address range perms: large range 134221824(noaccess) Allocating 134217728 bytes for readbuffer... ==16512== Warning: set address range perms: large range134221824 (defined) ==16513== Warning: set address range perms:large range 134221824 (noaccess) ==16512== Warning: set address range perms: large range 134217728(defined) ==16514== Warning: set address range perms: large range134221824 (noaccess) ==16512== Warning: set address range perms:large range 134221824 (noaccess) -+--> INJECTING SIGNAL... -+--> INJECTING SIGNAL... -+--> INJECTING SIGNAL... -+--> INJECTINGSIGNAL... 524765 524765 524765 524765 Signal:6info.si_errno:0(Success) si_code:-6() --16515-- Reading symsfrom /lib/libgcc_s-4.1.2-20070925.so.1 (0x794000) --16515--    object doesn't have a symboltable [0] func:/usr/lib/openmpi/libopal.so.0 [0x46efd6] [1]func:/lib/libpthread.so.0 [0x443290] [2] func:/lib/ld-linux.so.2[0x2767f2] [3] func:/lib/libc.so.6(gsignal+0x50) [0x2c1fa0] [4] func:/lib/libc.so.6(abort+0x101) [0x2c38b1] [5] func:/usr/lib/openmpi/libopal.so.0 [0x473d34] [6] func:/usr/lib/openmpi/libopal.so.0(free+0x36) [0x473786] [7] func:./lisa09EMRIT-P(main+0x44c) [0x80566ec] [8] func:/lib/libc.so.6(__libc_start_main+0xe0) [0x2aef70] [9] func:./lisa09EMRIT-P [0x804af21] *** End of error message*** ==16515== ==16515== Process terminating with defaultaction of signal 6 (SIGABRT) ==16515==    at0x2C1FA0: raise (in /lib/<a href="http://libc-2.6.so";>libc-2.6.so</a>) ==16515==    by 0x2C39FD: abort (in /lib/<a href="http://libc-2.6.so">libc-2.6.so</a>) ==16515==    by 0x473D33:(within /usr/lib/openmpi/libopal.so.0.0.0) ==16515==    by 0x473785: free (in /usr/lib/openmpi/libopal.so.0.0.0) ==16515==    by 0x80566EB: main (in /home/phd/asad/MLDCs/T/lisa09EMRIT-P) ==16515== ==16515== ERROR SUMMARY: 0errors from 0 contexts (suppressed: 108 from 1) --16515-- --16515-- supp:  108 dl-hack3 ==16515== malloc/free: inuse at exit: 0 bytes in 0 blocks. ==16515== malloc/free: 0 allocs, 0 frees, 0 bytesallocated. ==16515== ==16515== All heap blocks were freed --no leaks are possible. --16515--  memcheck: sanity checks:22725 cheap, 910 expensive --16515--  memcheck: auxmaps: 0auxmap entries (0k, 0M) in use --16515--  memcheck: auxmaps: 0 searches, 0comparisons --16515--  memcheck: SMs:n_issued      = 6014 (96224k,93M) --16515--  memcheck: SMs: n_deissued   = 4726 (75616k, 73M) --16515--  memcheck: SMs:max_noaccess  = 65535 (1048560k, 1023M) --16515--  memcheck: SMs: max_undefined = 38 (608k,0M) --16515--  memcheck: SMs: max_defined   =10813 (173008k, 168M) --16515--  memcheck: SMs:max_non_DSM   = 3109 (49744k, 48M) --16515-- memcheck: max sec V bit nodes:    2094 (106k, 0M) --16515--  memcheck: set_sec_vbits8 calls: 3371606 (new: 2094,updates: 3369512) --16515--  memcheck: max shadow memsize:   50154k, 48M --16515--translate:           fast SP updates identified: 23,603 ( 89.1%) --16515-- translate:   generic_known SP updatesidentified: 2,000 (  7.5%) --16515-- translate:generic_unknown SP updates identified: 877 (  3.3%) --16515--     tt/tc: 625,692 tt lookups requiring 672,349probes --16515--     tt/tc: 625,692 fast-cache updates,9 flushes --16515--  transtab:new        20,764 (798,227 ->14,916,839; ratio 186:10) [0 scs] --16515--  transtab:dumped     0 (0 -> ??) --16515-- transtab: discarded  193 (3,337 -> ??) --16515-- scheduler: 2,272,550,211 jumps (bb entries). --16515--scheduler: 22,725/32,791,636 major/minor sched events. --16515--    sanity: 22726 cheap, 910 expensivechecks. --16515--    exectx: 30,011 lists, 11contexts (avg 0 per list) --16515--    exectx: 108 searches, 97 full compares(898 per 1000) --16515--    exectx: 0 cmp2, 232cmp4, 0 cmpAll [8]+ Killed                 mpirun -np 4 valgrind -v ./lisa09EMRIT-P Could please tell mehow ti interpret the results of valgrind? 
 Thanking you in advance. Asad 

------=_Part_48219_1955003.1229850505418--

--===============1267833741==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
--===============1267833741==--



--
Jeff Squyres
Cisco Systems

Re: [OMPI users] MPI Error

Reply via email to