Thanks, Gilles! I was looking at that same code just now and completely missed the lack of a case for '%u' (and '%lu'). I will add one now and see if that resolves the problem....
-Paul On Fri, Dec 12, 2014 at 3:10 PM, Gilles Gouaillardet < gilles.gouaillar...@gmail.com> wrote: > Ralph, > > I cannot find a case for the %u format is guess_strlen > And since the default does not invoke va_arg() > I > it seems strlen is invoked on nnuma instead of arch > > Makes sense ? > > Cheers, > > Gilles > > Ralph Castain <r...@open-mpi.org> wrote: > Afraid I'm drawing a blank, Paul - I can't see how we got to a bad address > down there. This is at the beginning of orte_init, so there are no threads > running nor has anything much happened. > > Do you have any suggestions? > > > On Dec 12, 2014, at 9:02 AM, Paul Hargrove <phhargr...@lbl.gov> wrote: > > Ralph, > > The "arch" variable looks fine: > Current function is opal_hwloc_base_get_topo_signature > 2134 nnuma, nsocket, nl3, nl2, nl1, ncore, nhwt, arch); > (dbx) print arch > arch = 0x1001700a0 "sun4v" > > And so is "fmt": > > Current function is opal_asprintf > 194 length = opal_vasprintf(ptr, fmt, ap); > (dbx) print fmt > fmt = 0xffffffff7eeada98 "%uN:%uS:%uL3:%uL2:%uL1:%uC:%uH:%s" > > However, things have gone bad in guess_strlen(): > > Current function is guess_strlen > 71 len += (int)strlen(sarg); > (dbx) print sarg > sarg = 0x2 "<bad address 0x2>" > > -Paul > > On Fri, Dec 12, 2014 at 2:24 AM, Ralph Castain <r...@open-mpi.org> wrote: > >> Hmmm....this is really odd. I actually do have a protection for that arch >> value being NULL, and you are in the code section when it isn't. >> >> Do you still have the core file around? If so, can you print out the >> value of the "arch" variable? It would be in the >> opal_hwloc_base_get_topo_signature level. >> >> I'm wondering if that value has been hosed, and the problem is memory >> corruption somewhere. >> >> >> On Dec 11, 2014, at 8:56 PM, Ralph Castain <r...@open-mpi.org> wrote: >> >> Thanks Paul - I will post a fix for this tomorrow. Looks like Sparc isn't >> returning an architecture type for some reason, and I didn't protect >> against it. >> >> >> On Dec 11, 2014, at 7:39 PM, Paul Hargrove <phhargr...@lbl.gov> wrote: >> >> Backtrace for the Solaris-10/SPARC SEGV appears below. >> I've changed the subject line to distinguish this from the earlier report. >> >> -Paul >> >> program terminated by signal SEGV (no mapping at the fault address) >> 0xffffffff7d93b634: strlen+0x0014: lduh [%o2], %o1 >> Current function is guess_strlen >> 71 len += (int)strlen(sarg); >> (dbx) where >> [1] strlen(0x2, 0x73000000, 0x2, 0x80808080, 0x2, 0x80808080), at >> 0xffffffff7d93b634 >> =>[2] guess_strlen(fmt = 0xffffffff7eeada98 >> "%uN:%uS:%uL3:%uL2:%uL1:%uC:%uH:%s", ap = 0xffffffff7ffff058), line 71 in >> "printf.c" >> [3] opal_vasprintf(ptr = 0xffffffff7ffff0b8, fmt = 0xffffffff7eeada98 >> "%uN:%uS:%uL3:%uL2:%uL1:%uC:%uH:%s", ap = 0xffffffff7ffff050), line 218 in >> "printf.c" >> [4] opal_asprintf(ptr = 0xffffffff7ffff0b8, fmt = 0xffffffff7eeada98 >> "%uN:%uS:%uL3:%uL2:%uL1:%uC:%uH:%s", ... = 0x807ede0103, ...), line 194 in >> "printf.c" >> [5] opal_hwloc_base_get_topo_signature(topo = 0x100128ea0), line 2134 >> in "hwloc_base_util.c" >> [6] rte_init(), line 205 in "ess_hnp_module.c" >> [7] orte_init(pargc = 0xffffffff7ffff61c, pargv = 0xffffffff7ffff610, >> flags = 4U), line 148 in "orte_init.c" >> [8] orterun(argc = 7, argv = 0xffffffff7ffff7a8), line 856 in >> "orterun.c" >> [9] main(argc = 7, argv = 0xffffffff7ffff7a8), line 13 in "main.c" >> >> On Thu, Dec 11, 2014 at 7:17 PM, Ralph Castain <r...@open-mpi.org> wrote: >> >>> No, that looks different - it's failing in mpirun itself. Can you get a >>> line number on it? >>> >>> Sorry for delay - I'm generating rc3 now >>> >>> >>> On Dec 11, 2014, at 6:59 PM, Paul Hargrove <phhargr...@lbl.gov> wrote: >>> >>> Don't see an rc3 yet. >>> >>> My Solaris-10/SPARC runs fail slightly differently (see below). >>> It looks sufficiently similar that it MIGHT be the same root cause. >>> However, lacking an rc3 to test I figured it would be better to report >>> this than to ignore it. >>> >>> The problem is present with both V8+ and V9 ABIs, and with both Gnu and >>> Sun compilers. >>> >>> -Paul >>> >>> [niagara1:29881] *** Process received signal *** >>> [niagara1:29881] Signal: Segmentation Fault (11) >>> [niagara1:29881] Signal code: Address not mapped (1) >>> [niagara1:29881] Failing at address: 2 >>> >>> /sandbox/hargrove/OMPI/openmpi-1.8.4rc2-solaris10-sparcT2-gcc346-v8plus/INST/lib/libopen-pal.so.6.2.1:opal_bac >>> ktrace_print+0x24 >>> >>> /sandbox/hargrove/OMPI/openmpi-1.8.4rc2-solaris10-sparcT2-gcc346-v8plus/INST/lib/libopen-pal.so.6.2.1:0xaa160 >>> /lib/libc.so.1:0xc5364 >>> /lib/libc.so.1:0xb9e64 >>> /lib/libc.so.1:strlen+0x14 [ Signal 11 (SEGV)] >>> >>> /sandbox/hargrove/OMPI/openmpi-1.8.4rc2-solaris10-sparcT2-gcc346-v8plus/INST/lib/libopen-pal.so.6.2.1:opal_vas >>> printf+0x20 >>> >>> /sandbox/hargrove/OMPI/openmpi-1.8.4rc2-solaris10-sparcT2-gcc346-v8plus/INST/lib/libopen-pal.so.6.2.1:opal_asp >>> rintf+0x30 >>> >>> /sandbox/hargrove/OMPI/openmpi-1.8.4rc2-solaris10-sparcT2-gcc346-v8plus/INST/lib/libopen-pal.so.6.2.1:opal_hwl >>> oc_base_get_topo_signature+0x24c >>> >>> /sandbox/hargrove/OMPI/openmpi-1.8.4rc2-solaris10-sparcT2-gcc346-v8plus/INST/lib/openmpi/mca_ess_hnp.so:0x2d90 >>> >>> /sandbox/hargrove/OMPI/openmpi-1.8.4rc2-solaris10-sparcT2-gcc346-v8plus/INST/lib/libopen-rte.so.7.0.5:orte_ini >>> t+0x2f8 >>> >>> /sandbox/hargrove/OMPI/openmpi-1.8.4rc2-solaris10-sparcT2-gcc346-v8plus/INST/bin/orterun:orterun+0xaa8 >>> >>> /sandbox/hargrove/OMPI/openmpi-1.8.4rc2-solaris10-sparcT2-gcc346-v8plus/INST/bin/orterun:main+0x14 >>> >>> /sandbox/hargrove/OMPI/openmpi-1.8.4rc2-solaris10-sparcT2-gcc346-v8plus/INST/bin/orterun:_start+0x5c >>> [niagara1:29881] *** End of error message *** >>> Segmentation Fault - core dumped >>> >>> On Thu, Dec 11, 2014 at 3:29 PM, Ralph Castain <r...@open-mpi.org> wrote: >>> >>>> Ah crud - incomplete commit means we didn't send the topo string. Will >>>> roll rc3 in a few minutes. >>>> >>>> Thanks, Paul >>>> Ralph >>>> >>>> On Dec 11, 2014, at 3:08 PM, Paul Hargrove <phhargr...@lbl.gov> wrote: >>>> >>>> Testing the 1.8.4rc2 tarball on my x86-64 Solaris-11 systems I am >>>> getting the following crash for both "-m32" and "-m64" builds: >>>> >>>> $ mpirun -mca btl sm,self,openib -np 2 -host pcp-j-19,pcp-j-20 >>>> examples/ring_c' >>>> [pcp-j-19:18762] *** Process received signal *** >>>> [pcp-j-19:18762] Signal: Segmentation Fault (11) >>>> [pcp-j-19:18762] Signal code: Address not mapped (1) >>>> [pcp-j-19:18762] Failing at address: 0 >>>> /shared/OMPI/openmpi-1.8.4rc2-solaris11-x64-ib-gcc452/INST/lib/libopen-pal.so.6.2.1'opal_backtrace_print+0x26 >>>> [0xfffffd7ffaf237ba] >>>> /shared/OMPI/openmpi-1.8.4rc2-solaris11-x64-ib-gcc452/INST/lib/libopen-pal.so.6.2.1'show_stackframe+0x833 >>>> [0xfffffd7ffaf20ba1] >>>> /lib/amd64/libc.so.1'__sighndlr+0x6 [0xfffffd7fff202cc6] >>>> /lib/amd64/libc.so.1'call_user_handler+0x2aa [0xfffffd7fff1f648e] >>>> /lib/amd64/libc.so.1'strcmp+0x1a [0xfffffd7fff170fda] [Signal 11 (SEGV)] >>>> /shared/OMPI/openmpi-1.8.4rc2-solaris11-x64-ib-gcc452/INST/bin/orted'main+0x90 >>>> [0x4010b7] >>>> /shared/OMPI/openmpi-1.8.4rc2-solaris11-x64-ib-gcc452/INST/bin/orted'_start+0x6c >>>> [0x400f2c] >>>> [pcp-j-19:18762] *** End of error message *** >>>> bash: line 1: 18762 Segmentation Fault (core dumped) >>>> /shared/OMPI/openmpi-1.8.4rc2-solaris11-x64-ib-gcc452/INST/bin/orted -mca >>>> ess "env" -mca orte_ess_jobid "911343616" -mca orte_ess_vpid 1 -mca >>>> orte_ess_num_procs "2" -mca orte_hnp_uri "911343616.0;tcp:// >>>> 172.16.0.120,172.18.0.120:50362" --tree-spawn -mca btl >>>> "sm,self,openib" -mca plm "rsh" -mca shmem_mmap_enable_nfs_warning "0" >>>> >>>> Running gdb against a core generated by the 32-bit build gives line >>>> numbers: >>>> #0 0xfea1cb45 in strcmp () from /lib/libc.so.1 >>>> #1 0xfeef4900 in orte_daemon (argc=26, argv=0x80479b0) >>>> at >>>> /shared/OMPI/openmpi-1.8.4rc2-solaris11-x86-ib-gcc452/openmpi-1.8.4rc2/orte/orted/orted_main.c:789 >>>> #2 0x08050fb1 in main (argc=26, argv=0x80479b0) >>>> at >>>> /shared/OMPI/openmpi-1.8.4rc2-solaris11-x86-ib-gcc452/openmpi-1.8.4rc2/orte/tools/orted/orted.c:62 >>>> >>>> -Paul >>>> >>>> -- >>>> Paul H. Hargrove phhargr...@lbl.gov >>>> Computer Languages & Systems Software (CLaSS) Group >>>> Computer Science Department Tel: +1-510-495-2352 >>>> Lawrence Berkeley National Laboratory Fax: +1-510-486-6900 >>>> _______________________________________________ >>>> devel mailing list >>>> de...@open-mpi.org >>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >>>> Link to this post: >>>> http://www.open-mpi.org/community/lists/devel/2014/12/16514.php >>>> >>>> >>>> >>>> _______________________________________________ >>>> devel mailing list >>>> de...@open-mpi.org >>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >>>> Link to this post: >>>> http://www.open-mpi.org/community/lists/devel/2014/12/16515.php >>>> >>> >>> >>> >>> -- >>> Paul H. Hargrove phhargr...@lbl.gov >>> Computer Languages & Systems Software (CLaSS) Group >>> Computer Science Department Tel: +1-510-495-2352 >>> Lawrence Berkeley National Laboratory Fax: +1-510-486-6900 >>> _______________________________________________ >>> devel mailing list >>> de...@open-mpi.org >>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >>> Link to this post: >>> http://www.open-mpi.org/community/lists/devel/2014/12/16521.php >>> >>> >>> >>> _______________________________________________ >>> devel mailing list >>> de...@open-mpi.org >>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >>> Link to this post: >>> http://www.open-mpi.org/community/lists/devel/2014/12/16522.php >>> >> >> >> >> -- >> Paul H. Hargrove phhargr...@lbl.gov >> Computer Languages & Systems Software (CLaSS) Group >> Computer Science Department Tel: +1-510-495-2352 >> Lawrence Berkeley National Laboratory Fax: +1-510-486-6900 >> _______________________________________________ >> devel mailing list >> de...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >> Link to this post: >> http://www.open-mpi.org/community/lists/devel/2014/12/16524.php >> >> >> >> >> _______________________________________________ >> devel mailing list >> de...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >> Link to this post: >> http://www.open-mpi.org/community/lists/devel/2014/12/16541.php >> > > > > -- > Paul H. Hargrove phhargr...@lbl.gov > Computer Languages & Systems Software (CLaSS) Group > Computer Science Department Tel: +1-510-495-2352 > Lawrence Berkeley National Laboratory Fax: +1-510-486-6900 > _______________________________________________ > devel mailing list > de...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > Link to this post: > http://www.open-mpi.org/community/lists/devel/2014/12/16552.php > > > > _______________________________________________ > devel mailing list > de...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > Link to this post: > http://www.open-mpi.org/community/lists/devel/2014/12/16562.php > -- Paul H. Hargrove phhargr...@lbl.gov Computer Languages & Systems Software (CLaSS) Group Computer Science Department Tel: +1-510-495-2352 Lawrence Berkeley National Laboratory Fax: +1-510-486-6900
