Ralph, thanks. I'll try it on Tuseday.
Let me confirm one thing. I don't put "-with-libevent" when I build openmpi. Is there any possibility to build with external libevent automatically? Tetsuya Mishima > Not entirely sure - add "-mca rmaps_base_verbose 10 --display-map" to your cmd line and let's see if it finishes the mapping. > > Unless you specifically built with an external libevent (which I doubt), there is no conflict. The connection issue is unlikely to be a factor here as it works when not using the lama mapper. > > > On Dec 21, 2013, at 3:43 PM, tmish...@jcity.maeda.co.jp wrote: > > > > > > > Thank you, Ralph. > > > > Then, this problem should depend on our environment. > > But, at least, inversion problem is not the cause because > > node05 has normal hier order. > > > > I can not connect to our cluster now. Tuesday, going > > back to my office, I'll send you further report. > > > > Before that, please let me know your configuration. I will > > follow your configuation as much as possible. Our configuraion > > is very simple, only -with-tm -with-ibverbs -disable-ipv6. > > (on CentOS 5.8) > > > > The 1.7 series is a llite bit unstable on our cluster yet. > > > > Similar freezing(hang up) was observed with 1.7.3. At that > > time, lama worked well but putting "-rank-by something" caused > > same freezing (curiously, rank-by works with 1.7.4rc1). > > I checked where it stopped using gdb, then I found that it > > stopped to wait for event in a function of libevent(I can not > > recall the name). > > > > Is this related to your "connection issue in the OOB > > subsystem"? Or libevent version conflict? I guess these two > > problems are related each other. They stopped at very early > > stage before reaching mapping function because no message > > appeared before freezing, which is my random guess. > > > > Could you give me any hint or comment? > > > > Regards, > > Tetsuya Mishima > > > > > >> It seems to be working fine for me: > >> > >> [rhc@bend001 tcp]$ mpirun -np 2 -host bend001 -report-bindings -mca > > rmaps_lama_bind 1c -mca rmaps lama hostname > >> bend001 > >> [bend001:17005] MCW rank 1 bound to socket 0[core 1[hwt 0-1]]: > > [../BB/../../../..][../../../../../..] > >> [bend001:17005] MCW rank 0 bound to socket 0[core 0[hwt 0-1]]: > > [BB/../../../../..][../../../../../..] > >> bend001 > >> [rhc@bend001 tcp]$ > >> > >> (I also checked the internals using "-mca rmaps_base_verbose 10") so it > > could be your hier inversion causing problems again. Or it could be that > > you are hitting a connection issue we are seeing in > >> some scenarios in the OOB subsystem - though if you are able to run using > > a non-lama mapper, that would seem unlikely. > >> > >> > >> On Dec 20, 2013, at 8:09 PM, tmish...@jcity.maeda.co.jp wrote: > >> > >> > >> > >> Hi Ralph, > >> > >> Thank you very much. I tried many things such as: > >> > >> mpirun -np 2 -host node05 -report-bindings -mca rmaps lama -mca > >> rmaps_lama_bind 1c myprog > >> > >> But every try failed. At least they were accepted by openmpi-1.7.3 as far > >> as I remember. > >> Anyway, please check it when you have a time, because using lama comes > > from > >> my curiosity. > >> > >> Regards, > >> Tetsuya Mishima > >> > >> > >> I'll try to take a look at it - my expectation is that lama might get > >> stuck because you didn't tell it a pattern to map, and I doubt that code > >> path has seen much testing. > >> > >> > >> On Dec 20, 2013, at 5:52 PM, tmish...@jcity.maeda.co.jp wrote: > >> > >> > >> > >> Hi Ralph, I'm glad to hear that, thanks. > >> > >> By the way, yesterday I tried to check how lama in 1.7.4rc treat numa > >> node. > >> > >> Then, even wiht this simple command line, it freezed without any > >> massage: > >> > >> mpirun -np 2 -host node05 -mca rmaps lama myprog > >> > >> Could you check what happened? > >> > >> Is it better to open new thread or continue this thread? > >> > >> Regards, > >> Tetsuya Mishima > >> > >> > >> I'll make it work so that NUMA can be either above or below socket > >> > >> On Dec 20, 2013, at 2:57 AM, tmish...@jcity.maeda.co.jp wrote: > >> > >> > >> > >> Hi Brice, > >> > >> Thank you for your comment. I understand what you mean. > >> > >> My opinion was made just considering easy way to adjust the code for > >> inversion of hierarchy in object tree. > >> > >> Tetsuya Mishima > >> > >> > >> I don't think there's any such difference. > >> Also, all these NUMA architectures are reported the same by hwloc, > >> and > >> therefore used the same in Open MPI. > >> > >> And yes, L3 and NUMA are topologically-identical on AMD Magny-Cours > >> (and > >> most recent AMD and Intel platforms). > >> > >> Brice > >> > >> > >> > >> Le 20/12/2013 11:33, tmish...@jcity.maeda.co.jp a écrit : > >> > >> Hi Ralph, > >> > >> The numa-node in AMD Mangy-Cours/Interlagos is so called cc(cache > >> coherent)NUMA, > >> which seems to be a little bit different from the traditional numa > >> defined > >> in openmpi. > >> > >> I notice that ccNUMA object is almost same as L3cache object. > >> So "-bind-to l3cache" or "-map-by l3cache" is valid for what I want > >> to > >> do. > >> Therefore, "do not touch it" is one of the solution, I think ... > >> > >> Anyway, mixing up these two types of numa is the problem. > >> > >> Regards, > >> Tetsuya Mishima > >> > >> I can wait it'll be fixed in 1.7.5 or later, because putting > >> "-bind-to > >> numa" > >> and "-map-by numa" at the same time works as a workaround. > >> > >> Thanks, > >> Tetsuya Mishima > >> > >> Yeah, it will impact everything that uses hwloc topology maps, I > >> fear. > >> > >> One side note: you'll need to add --hetero-nodes to your cmd > >> line. > >> If > >> we > >> don't see that, we assume that all the node topologies are > >> identical > >> - > >> which clearly isn't true here. > >> I'll try to resolve the hier inversion over the holiday - won't > >> be > >> for > >> 1.7.4, but hopefully for 1.7.5 > >> Thanks > >> Ralph > >> > >> On Dec 18, 2013, at 9:44 PM, tmish...@jcity.maeda.co.jp wrote: > >> > >> > >> I think it's normal for AMD opteron having 8/16 cores such as > >> magny cours or interlagos. Because it usually has 2 numa nodes > >> in a cpu(socket), numa-node can not include a socket. This type > >> of hierarchy would be natural. > >> > >> (node03 is Dell PowerEdge R815 and maybe quite common, I guess) > >> > >> By the way, I think this inversion should affect rmaps_lama > >> mapping. > >> > >> Tetsuya Mishima > >> > >> Ick - yeah, that would be a problem. I haven't seen that type > >> of > >> hierarchical inversion before - is node03 a different type of > >> chip? > >> Might take awhile for me to adjust the code to handle hier > >> inversion... :-( > >> On Dec 18, 2013, at 9:05 PM, tmish...@jcity.maeda.co.jp wrote: > >> > >> > >> Hi Ralph, > >> > >> I found the reason. I attached the main part of output with 32 > >> core node(node03) and 8 core node(node05) at the bottom. > >> > >> From this information, socket of node03 includes numa-node. > >> On the other hand, numa-node of node05 includes socket. > >> The direction of object tree is opposite. > >> > >> Since "-map-by socket" may be assumed as default, > >> for node05, "-bind-to numa and -map-by socket" means > >> upward search. For node03, this should be downward. > >> > >> I guess that openmpi-1.7.4rc1 will always assume numa-node > >> includes socket. Is it right? Then, upward search is assumed > >> in orte_rmaps_base_compute_bindings even for node03 when I > >> put "-bind-to numa and -map-by socket" option. > >> > >> [node03.cluster:15508] [[38286,0],0] rmaps:base:compute_usage > >> [node03.cluster:15508] mca:rmaps: compute bindings for job > >> [38286,1] > >> with > >> policy NUMA > >> [node03.cluster:15508] mca:rmaps: bind upwards for job > >> [38286,1] > >> with > >> bindings NUMA > >> [node03.cluster:15508] [[38286,0],0] bind:upward target > >> NUMANode > >> type > >> Machine > >> > >> That's the reason of this trouble. Therefore, adding "-map-by > >> core" > >> works. > >> (mapping pattern seems to be strange ...) > >> > >> [mishima@node03 demos]$ mpirun -np 8 -bind-to numa -map-by > >> core > >> -report-bindings myprog > >> [node03.cluster:15885] [[38679,0],0] bind:upward target > >> NUMANode > >> type > >> Cache > >> [node03.cluster:15885] [[38679,0],0] bind:upward target > >> NUMANode > >> type > >> Cache > >> [node03.cluster:15885] [[38679,0],0] bind:upward target > >> NUMANode > >> type > >> Cache > >> [node03.cluster:15885] [[38679,0],0] bind:upward target > >> NUMANode > >> type > >> NUMANode > >> [node03.cluster:15885] [[38679,0],0] bind:upward target > >> NUMANode > >> type > >> Cache > >> [node03.cluster:15885] [[38679,0],0] bind:upward target > >> NUMANode > >> type > >> Cache > >> [node03.cluster:15885] [[38679,0],0] bind:upward target > >> NUMANode > >> type > >> Cache > >> [node03.cluster:15885] [[38679,0],0] bind:upward target > >> NUMANode > >> type > >> NUMANode > >> [node03.cluster:15885] [[38679,0],0] bind:upward target > >> NUMANode > >> type > >> Cache > >> [node03.cluster:15885] [[38679,0],0] bind:upward target > >> NUMANode > >> type > >> Cache > >> [node03.cluster:15885] [[38679,0],0] bind:upward target > >> NUMANode > >> type > >> Cache > >> [node03.cluster:15885] [[38679,0],0] bind:upward target > >> NUMANode > >> type > >> NUMANode > >> [node03.cluster:15885] [[38679,0],0] bind:upward target > >> NUMANode > >> type > >> Cache > >> [node03.cluster:15885] [[38679,0],0] bind:upward target > >> NUMANode > >> type > >> Cache > >> [node03.cluster:15885] [[38679,0],0] bind:upward target > >> NUMANode > >> type > >> Cache > >> [node03.cluster:15885] [[38679,0],0] bind:upward target > >> NUMANode > >> type > >> NUMANode > >> [node03.cluster:15885] [[38679,0],0] bind:upward target > >> NUMANode > >> type > >> Cache > >> [node03.cluster:15885] [[38679,0],0] bind:upward target > >> NUMANode > >> type > >> Cache > >> [node03.cluster:15885] [[38679,0],0] bind:upward target > >> NUMANode > >> type > >> Cache > >> [node03.cluster:15885] [[38679,0],0] bind:upward target > >> NUMANode > >> type > >> NUMANode > >> [node03.cluster:15885] [[38679,0],0] bind:upward target > >> NUMANode > >> type > >> Cache > >> [node03.cluster:15885] [[38679,0],0] bind:upward target > >> NUMANode > >> type > >> Cache > >> [node03.cluster:15885] [[38679,0],0] bind:upward target > >> NUMANode > >> type > >> Cache > >> [node03.cluster:15885] [[38679,0],0] bind:upward target > >> NUMANode > >> type > >> NUMANode > >> [node03.cluster:15885] [[38679,0],0] bind:upward target > >> NUMANode > >> type > >> Cache > >> [node03.cluster:15885] [[38679,0],0] bind:upward target > >> NUMANode > >> type > >> Cache > >> [node03.cluster:15885] [[38679,0],0] bind:upward target > >> NUMANode > >> type > >> Cache > >> [node03.cluster:15885] [[38679,0],0] bind:upward target > >> NUMANode > >> type > >> NUMANode > >> [node03.cluster:15885] [[38679,0],0] bind:upward target > >> NUMANode > >> type > >> Cache > >> [node03.cluster:15885] [[38679,0],0] bind:upward target > >> NUMANode > >> type > >> Cache > >> [node03.cluster:15885] [[38679,0],0] bind:upward target > >> NUMANode > >> type > >> Cache > >> [node03.cluster:15885] [[38679,0],0] bind:upward target > >> NUMANode > >> type > >> NUMANode > >> [node03.cluster:15885] MCW rank 2 bound to socket 0[core 0[hwt > >> 0]], > >> socket > >> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so > >> cket 0[core 3[hwt 0]]: > >> > >> > >> [B/B/B/B/./././.][./././././././.][./././././././.][./././././././.] > >> [node03.cluster:15885] MCW rank 3 bound to socket 0[core 0[hwt > >> 0]], > >> socket > >> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so > >> cket 0[core 3[hwt 0]]: > >> > >> > >> [B/B/B/B/./././.][./././././././.][./././././././.][./././././././.] > >> [node03.cluster:15885] MCW rank 4 bound to socket 0[core 4[hwt > >> 0]], > >> socket > >> 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], so > >> cket 0[core 7[hwt 0]]: > >> > >> > >> [././././B/B/B/B][./././././././.][./././././././.][./././././././.] > >> [node03.cluster:15885] MCW rank 5 bound to socket 0[core 4[hwt > >> 0]], > >> socket > >> 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], so > >> cket 0[core 7[hwt 0]]: > >> > >> > >> [././././B/B/B/B][./././././././.][./././././././.][./././././././.] > >> [node03.cluster:15885] MCW rank 6 bound to socket 0[core 4[hwt > >> 0]], > >> socket > >> 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], so > >> cket 0[core 7[hwt 0]]: > >> > >> > >> [././././B/B/B/B][./././././././.][./././././././.][./././././././.] > >> [node03.cluster:15885] MCW rank 7 bound to socket 0[core 4[hwt > >> 0]], > >> socket > >> 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], so > >> cket 0[core 7[hwt 0]]: > >> > >> > >> [././././B/B/B/B][./././././././.][./././././././.][./././././././.] > >> [node03.cluster:15885] MCW rank 0 bound to socket 0[core 0[hwt > >> 0]], > >> socket > >> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so > >> cket 0[core 3[hwt 0]]: > >> > >> > >> [B/B/B/B/./././.][./././././././.][./././././././.][./././././././.] > >> [node03.cluster:15885] MCW rank 1 bound to socket 0[core 0[hwt > >> 0]], > >> socket > >> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so > >> cket 0[core 3[hwt 0]]: > >> > >> > >> [B/B/B/B/./././.][./././././././.][./././././././.][./././././././.] > >> Hello world from process 6 of 8 > >> Hello world from process 5 of 8 > >> Hello world from process 0 of 8 > >> Hello world from process 7 of 8 > >> Hello world from process 3 of 8 > >> Hello world from process 4 of 8 > >> Hello world from process 2 of 8 > >> Hello world from process 1 of 8 > >> > >> Regards, > >> Tetsuya Mishima > >> > >> [node03.cluster:15508] Type: Machine Number of child objects: > >> 4 > >> Name=NULL > >> total=132358820KB > >> Backend=Linux > >> OSName=Linux > >> OSRelease=2.6.18-308.16.1.el5 > >> OSVersion="#1 SMP Tue Oct 2 22:01:43 EDT 2012" > >> Architecture=x86_64 > >> Cpuset: 0xffffffff > >> Online: 0xffffffff > >> Allowed: 0xffffffff > >> Bind CPU proc: TRUE > >> Bind CPU thread: TRUE > >> Bind MEM proc: FALSE > >> Bind MEM thread: TRUE > >> Type: Socket Number of child objects: 2 > >> Name=NULL > >> total=33071780KB > >> CPUModel="AMD Opteron(tm) Processor 6136" > >> Cpuset: 0x000000ff > >> Online: 0x000000ff > >> Allowed: 0x000000ff > >> Type: NUMANode Number of child objects: 1 > >> > >> > >> [node05.cluster:21750] Type: Machine Number of child objects: > >> 2 > >> Name=NULL > >> total=33080072KB > >> Backend=Linux > >> OSName=Linux > >> OSRelease=2.6.18-308.16.1.el5 > >> OSVersion="#1 SMP Tue Oct 2 22:01:43 EDT 2012" > >> Architecture=x86_64 > >> Cpuset: 0x000000ff > >> Online: 0x000000ff > >> Allowed: 0x000000ff > >> Bind CPU proc: TRUE > >> Bind CPU thread: TRUE > >> Bind MEM proc: FALSE > >> Bind MEM thread: TRUE > >> Type: NUMANode Number of child objects: 1 > >> Name=NULL > >> local=16532232KB > >> total=16532232KB > >> Cpuset: 0x0000000f > >> Online: 0x0000000f > >> Allowed: 0x0000000f > >> Type: Socket Number of child objects: 1 > >> > >> > >> Hmm...try adding "-mca rmaps_base_verbose 10 -mca > >> ess_base_verbose > >> 5" > >> to > >> your cmd line and let's see what it thinks it found. > >> > >> On Dec 18, 2013, at 6:55 PM, tmish...@jcity.maeda.co.jp > >> wrote: > >> > >> > >> Hi, I report one more problem with openmpi-1.7.4rc1, > >> which is more serious. > >> > >> For our 32 core nodes(AMD magny cours based) which has > >> 8 numa-nodes, "-bind-to numa" does not work. Without > >> this option, it works. For your infomation, at the > >> bottom of this mail, I added the lstopo information > >> of the node. > >> > >> Regards, > >> Tetsuya Mishima > >> > >> [mishima@manage ~]$ qsub -I -l nodes=1:ppn=32>> qsub: waiting for job 8352.manage.cluster to start > >> qsub: job 8352.manage.cluster ready > >> > >> [mishima@node03 demos]$ mpirun -np 8 -report-bindings > >> -bind-to > >> numa > >> myprog > >> [node03.cluster:15316] [[37582,0],0] bind:upward target > >> NUMANode > >> type > >> Machine > >> > >> > >> > >> > >> > > -------------------------------------------------------------------------- > >> A request was made to bind to NUMA, but an appropriate > >> target > >> could > >> not > >> be found on node node03. > >> > >> > >> > >> > >> > > -------------------------------------------------------------------------- > >> [mishima@node03 ~]$ cd ~/Desktop/openmpi-1.7/demos/ > >> [mishima@node03 demos]$ mpirun -np 8 -report-bindings myprog > >> [node03.cluster:15282] MCW rank 2 bound to socket 1[core 8 > >> [hwt > >> 0]]: > >> [./././././././.][B/././././././.][./././././././.][ > >> ./././././././.]>>>>>>>>>>>> [node03.cluster:15282] MCW rank > >> 3 bound to socket 1[core 9[hwt > >> 0]]: > >> [./././././././.][./B/./././././.][./././././././.][ > >> ./././././././.] > >> [node03.cluster:15282] MCW rank 4 bound to socket 2[core 16 > >> [hwt > >> 0]]: > >> [./././././././.][./././././././.][B/././././././.] > >> [./././././././.] > >> [node03.cluster:15282] MCW rank 5 bound to socket 2[core 17 > >> [hwt > >> 0]]: > >> [./././././././.][./././././././.][./B/./././././.] > >> [./././././././.] > >> [node03.cluster:15282] MCW rank 6 bound to socket 3[core 24 > >> [hwt > >> 0]]: > >> [./././././././.][./././././././.][./././././././.] > >> [B/././././././.] > >> [node03.cluster:15282] MCW rank 7 bound to socket 3[core 25 > >> [hwt > >> 0]]: > >> [./././././././.][./././././././.][./././././././.] > >> [./B/./././././.] > >> [node03.cluster:15282] MCW rank 0 bound to socket 0[core 0 > >> [hwt > >> 0]]: > >> [B/././././././.][./././././././.][./././././././.][ > >> ./././././././.] > >> [node03.cluster:15282] MCW rank 1 bound to socket 0[core 1 > >> [hwt > >> 0]]: > >> [./B/./././././.][./././././././.][./././././././.][ > >> ./././././././.] > >> Hello world from process 2 of 8 > >> Hello world from process 5 of 8 > >> Hello world from process 4 of 8 > >> Hello world from process 3 of 8>>>>>>>>>> Hello world from > >> process 1 of 8 > >> Hello world from process 7 of 8 > >> Hello world from process 6 of 8 > >> Hello world from process 0 of 8 > >> [mishima@node03 demos]$ ~/opt/hwloc/bin/lstopo-no-graphics > >> Machine (126GB) > >> Socket L#0 (32GB) > >> NUMANode L#0 (P#0 16GB) + L3 L#0 (5118KB) > >> L2 L#0 (512KB) + L1d L#0 (64KB) + L1i L#0 (64KB) + Core L#0 > >> + > >> PU > >> L#0 > >> (P#0) > >> L2 L#1 (512KB) + L1d L#1 (64KB) + L1i L#1 (64KB) + Core L#1 > >> + > >> PU > >> L#1 > >> (P#1) > >> L2 L#2 (512KB) + L1d L#2 (64KB) + L1i L#2 (64KB) + Core L#2 > >> + > >> PU > >> L#2 > >> (P#2) > >> L2 L#3 (512KB) + L1d L#3 (64KB) + L1i L#3 (64KB) + Core L#3 > >> + > >> PU > >> L#3 > >> (P#3) > >> NUMANode L#1 (P#1 16GB) + L3 L#1 (5118KB) > >> L2 L#4 (512KB) + L1d L#4 (64KB) + L1i L#4 (64KB) + Core L#4 > >> + > >> PU > >> L#4 > >> (P#4) > >> L2 L#5 (512KB) + L1d L#5 (64KB) + L1i L#5 (64KB) + Core L#5 > >> + > >> PU > >> L#5 > >> (P#5) > >> L2 L#6 (512KB) + L1d L#6 (64KB) + L1i L#6 (64KB) + Core L#6 > >> + > >> PU > >> L#6 > >> (P#6) > >> L2 L#7 (512KB) + L1d L#7 (64KB) + L1i L#7 (64KB) + Core L#7 > >> + > >> PU>>>>>> L#7 > >> (P#7) > >> Socket L#1 (32GB) > >> NUMANode L#2 (P#6 16GB) + L3 L#2 (5118KB) > >> L2 L#8 (512KB) + L1d L#8 (64KB) + L1i L#8 (64KB) + Core L#8 > >> + > >> PU > >> L#8 > >> (P#8) > >> L2 L#9 (512KB) + L1d L#9 (64KB) + L1i L#9 (64KB) + Core L#9 > >> + > >> PU > >> L#9 > >> (P#9) > >> L2 L#10 (512KB) + L1d L#10 (64KB) + L1i L#10 (64KB) + Core > >> L#10 > >> + > >> PU > >> L#10 (P#10) > >> L2 L#11 (512KB) + L1d L#11 (64KB) + L1i L#11 (64KB) + Core > >> L#11 > >> + > >> PU > >> L#11 (P#11) > >> NUMANode L#3 (P#7 16GB) + L3 L#3 (5118KB) > >> L2 L#12 (512KB) + L1d L#12 (64KB) + L1i L#12 (64KB) + Core > >> L#12 > >> + > >> PU > >> L#12 (P#12) > >> L2 L#13 (512KB) + L1d L#13 (64KB) + L1i L#13 (64KB) + Core > >> L#13 > >> + > >> PU > >> L#13 (P#13) > >> L2 L#14 (512KB) + L1d L#14 (64KB) + L1i L#14 (64KB) + Core > >> L#14 > >> + > >> PU > >> L#14 (P#14) > >> L2 L#15 (512KB) + L1d L#15 (64KB) + L1i L#15 (64KB) + Core > >> L#15 > >> + > >> PU > >> L#15 (P#15) > >> Socket L#2 (32GB) > >> NUMANode L#4 (P#4 16GB) + L3 L#4 (5118KB) > >> L2 L#16 (512KB) + L1d L#16 (64KB) + L1i L#16 (64KB) + Core > >> L#16 > >> + > >> PU > >> L#16 (P#16) > >> L2 L#17 (512KB) + L1d L#17 (64KB) + L1i L#17 (64KB) + Core > >> L#17 > >> + > >> PU > >> L#17 (P#17)> >>>>> L2 L#18 (512KB) + L1d L#18 (64KB) + > >> L1i > >> L#18 (64KB) + Core L#18 > >> + > >> PU > >> L#18 (P#18) > >> L2 L#19 (512KB) + L1d L#19 (64KB) + L1i L#19 (64KB) + Core > >> L#19 > >> + > >> PU > >> L#19 (P#19) > >> NUMANode L#5 (P#5 16GB) + L3 L#5 (5118KB) > >> L2 L#20 (512KB) + L1d L#20 (64KB) + L1i L#20 (64KB) + Core > >> L#20 > >> + > >> PU > >> L#20 (P#20) > >> L2 L#21 (512KB) + L1d L#21 (64KB) + L1i L#21 (64KB) + Core > >> L#21 > >> + > >> PU > >> L#21 (P#21) > >> L2 L#22 (512KB) + L1d L#22 (64KB) + L1i L#22 (64KB) + Core > >> L#22 > >> + > >> PU > >> L#22 (P#22) > >> L2 L#23 (512KB) + L1d L#23 (64KB) + L1i L#23 (64KB) + Core > >> L#23 > >> + > >> PU > >> L#23 (P#23) > >> Socket L#3 (32GB) > >> NUMANode L#6 (P#2 16GB) + L3 L#6 (5118KB) > >> L2 L#24 (512KB) + L1d L#24 (64KB) + L1i L#24 (64KB) + Core > >> L#24 > >> + > >> PU > >> L#24 (P#24)>>>>> L2 L#25 (512KB) + L1d L#25 (64KB) + L1i > >> L#25 > >> (64KB) + Core L#25 + > >> PU > >> L#25 (P#25) > >> L2 L#26 (512KB) + L1d L#26 (64KB) + L1i L#26 (64KB) + Core > >> L#26 > >> + > >> PU > >> L#26 (P#26) > >> L2 L#27 (512KB) + L1d L#27 (64KB) + L1i L#27 (64KB) + Core > >> L#27 > >> + > >> PU > >> L#27 (P#27) > >> NUMANode L#7 (P#3 16GB) + L3 L#7 (5118KB) > >> L2 L#28 (512KB) + L1d L#28 (64KB) + L1i L#28 (64KB) + Core > >> L#28 > >> + > >> PU > >> L#28 (P#28) > >> L2 L#29 (512KB) + L1d L#29 (64KB) + L1i L#29 (64KB) + Core > >> L#29 > >> + > >> PU > >> L#29 (P#29) > >> L2 L#30 (512KB) + L1d L#30 (64KB) + L1i L#30 (64KB) + Core > >> L#30 > >> + > >> PU > >> L#30 (P#30) > >> L2 L#31 (512KB) + L1d L#31 (64KB) + L1i L#31 (64KB) + Core > >> L#31 > >> + > >> PU > >> L#31 (P#31) > >> HostBridge L#0 > >> PCIBridge > >> PCI 14e4:1639 > >> Net L#0 "eth0" > >> PCI 14e4:1639 > >> Net L#1 "eth1" > >> PCIBridge > >> PCI 14e4:1639 > >> Net L#2 "eth2" > >> PCI 14e4:1639 > >> Net L#3 "eth3" > >> PCIBridge > >> PCIBridge > >> PCIBridge > >> PCI 1000:0072 > >> Block L#4 "sdb" > >> Block L#5 "sda" > >> PCI 1002:4390 > >> Block L#6 "sr0" > >> PCIBridge > >> PCI 102b:0532 > >> HostBridge L#7 > >> PCIBridge > >> PCI 15b3:6274 > >> Net L#7 "ib0" > >> OpenFabrics L#8 "mthca0" > >> > >> _______________________________________________ > >> users mailing list > >> us...@open-mpi.org > >> http://www.open-mpi.org/mailman/listinfo.cgi/users > >> _______________________________________________ > >> users mailing list > >> us...@open-mpi.org>> > >> http://www.open-mpi.org/mailman/listinfo.cgi/users > >> _______________________________________________ > >> users mailing list > >> us...@open-mpi.org > >> http://www.open-mpi.org/mailman/listinfo.cgi/users > >> _______________________________________________ > >> users mailing list > >> us...@open-mpi.org > >> http://www.open-mpi.org/mailman/listinfo.cgi/users > >> _______________________________________________ > >> users mailing list > >> us...@open-mpi.org > >> http://www.open-mpi.org/mailman/listinfo.cgi/users > >> _______________________________________________ > >> users mailing list > >> us...@open-mpi.org > >> http://www.open-mpi.org/mailman/listinfo.cgi/users > >> _______________________________________________ > >> users mailing list > >> us...@open-mpi.org > >> http://www.open-mpi.org/mailman/listinfo.cgi/users > >> _______________________________________________ > >> users mailing list > >> us...@open-mpi.org > >> http://www.open-mpi.org/mailman/listinfo.cgi/users > >> > >> _______________________________________________ > >> users mailing list > >> us...@open-mpi.org > >> http://www.open-mpi.org/mailman/listinfo.cgi/users > >> > >> _______________________________________________ > >> users mailing list > >> us...@open-mpi.org > >> http://www.open-mpi.org/mailman/listinfo.cgi/users > >> > >> _______________________________________________ > >> users mailing list > >> us...@open-mpi.org > >> http://www.open-mpi.org/mailman/listinfo.cgi/users > >> > >> _______________________________________________ > >> users mailing list > >> us...@open-mpi.org > >> http://www.open-mpi.org/mailman/listinfo.cgi/users > >> > >> _______________________________________________ > >> users mailing list > >> us...@open-mpi.org > >> http://www.open-mpi.org/mailman/listinfo.cgi/users > >> > >> _______________________________________________ > >> users mailing list > >> us...@open-mpi.org > >> > > http://www.open-mpi.org/mailman/listinfo.cgi/users_______________________________________________ > > > >> users mailing list > >> users@open-mpi.orghttp://www.open-mpi.org/mailman/listinfo.cgi/users > > > > _______________________________________________ > > users mailing list > > us...@open-mpi.org > > http://www.open-mpi.org/mailman/listinfo.cgi/users > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users
