On Dec 21, 2013, at 8:16 PM, tmish...@jcity.maeda.co.jp wrote: > > > Ralph, thanks. I'll try it on Tuseday. > > Let me confirm one thing. I don't put "-with-libevent" when I build > openmpi. > Is there any possibility to build with external libevent automatically?
No - only happens if you direct it > > Tetsuya Mishima > > >> Not entirely sure - add "-mca rmaps_base_verbose 10 --display-map" to > your cmd line and let's see if it finishes the mapping. >> >> Unless you specifically built with an external libevent (which I doubt), > there is no conflict. The connection issue is unlikely to be a factor here > as it works when not using the lama mapper. >> >> >> On Dec 21, 2013, at 3:43 PM, tmish...@jcity.maeda.co.jp wrote: >> >>> >>> >>> Thank you, Ralph. >>> >>> Then, this problem should depend on our environment. >>> But, at least, inversion problem is not the cause because >>> node05 has normal hier order. >>> >>> I can not connect to our cluster now. Tuesday, going >>> back to my office, I'll send you further report. >>> >>> Before that, please let me know your configuration. I will >>> follow your configuation as much as possible. Our configuraion >>> is very simple, only -with-tm -with-ibverbs -disable-ipv6. >>> (on CentOS 5.8) >>> >>> The 1.7 series is a llite bit unstable on our cluster yet. >>> >>> Similar freezing(hang up) was observed with 1.7.3. At that >>> time, lama worked well but putting "-rank-by something" caused >>> same freezing (curiously, rank-by works with 1.7.4rc1). >>> I checked where it stopped using gdb, then I found that it >>> stopped to wait for event in a function of libevent(I can not >>> recall the name). >>> >>> Is this related to your "connection issue in the OOB >>> subsystem"? Or libevent version conflict? I guess these two >>> problems are related each other. They stopped at very early >>> stage before reaching mapping function because no message >>> appeared before freezing, which is my random guess. >>> >>> Could you give me any hint or comment? >>> >>> Regards, >>> Tetsuya Mishima >>> >>> >>>> It seems to be working fine for me: >>>> >>>> [rhc@bend001 tcp]$ mpirun -np 2 -host bend001 -report-bindings -mca >>> rmaps_lama_bind 1c -mca rmaps lama hostname >>>> bend001 >>>> [bend001:17005] MCW rank 1 bound to socket 0[core 1[hwt 0-1]]: >>> [../BB/../../../..][../../../../../..] >>>> [bend001:17005] MCW rank 0 bound to socket 0[core 0[hwt 0-1]]: >>> [BB/../../../../..][../../../../../..] >>>> bend001 >>>> [rhc@bend001 tcp]$ >>>> >>>> (I also checked the internals using "-mca rmaps_base_verbose 10") so > it >>> could be your hier inversion causing problems again. Or it could be > that >>> you are hitting a connection issue we are seeing in >>>> some scenarios in the OOB subsystem - though if you are able to run > using >>> a non-lama mapper, that would seem unlikely. >>>> >>>> >>>> On Dec 20, 2013, at 8:09 PM, tmish...@jcity.maeda.co.jp wrote: >>>> >>>> >>>> >>>> Hi Ralph, >>>> >>>> Thank you very much. I tried many things such as: >>>> >>>> mpirun -np 2 -host node05 -report-bindings -mca rmaps lama -mca >>>> rmaps_lama_bind 1c myprog >>>> >>>> But every try failed. At least they were accepted by openmpi-1.7.3 as > far >>>> as I remember. >>>> Anyway, please check it when you have a time, because using lama comes >>> from >>>> my curiosity. >>>> >>>> Regards, >>>> Tetsuya Mishima >>>> >>>> >>>> I'll try to take a look at it - my expectation is that lama might get >>>> stuck because you didn't tell it a pattern to map, and I doubt that > code >>>> path has seen much testing. >>>> >>>> >>>> On Dec 20, 2013, at 5:52 PM, tmish...@jcity.maeda.co.jp wrote: >>>> >>>> >>>> >>>> Hi Ralph, I'm glad to hear that, thanks. >>>> >>>> By the way, yesterday I tried to check how lama in 1.7.4rc treat numa >>>> node. >>>> >>>> Then, even wiht this simple command line, it freezed without any >>>> massage: >>>> >>>> mpirun -np 2 -host node05 -mca rmaps lama myprog >>>> >>>> Could you check what happened? >>>> >>>> Is it better to open new thread or continue this thread? >>>> >>>> Regards, >>>> Tetsuya Mishima >>>> >>>> >>>> I'll make it work so that NUMA can be either above or below socket >>>> >>>> On Dec 20, 2013, at 2:57 AM, tmish...@jcity.maeda.co.jp wrote: >>>> >>>> >>>> >>>> Hi Brice, >>>> >>>> Thank you for your comment. I understand what you mean. >>>> >>>> My opinion was made just considering easy way to adjust the code for >>>> inversion of hierarchy in object tree. >>>> >>>> Tetsuya Mishima >>>> >>>> >>>> I don't think there's any such difference. >>>> Also, all these NUMA architectures are reported the same by hwloc, >>>> and >>>> therefore used the same in Open MPI. >>>> >>>> And yes, L3 and NUMA are topologically-identical on AMD Magny-Cours >>>> (and >>>> most recent AMD and Intel platforms). >>>> >>>> Brice >>>> >>>> >>>> >>>> Le 20/12/2013 11:33, tmish...@jcity.maeda.co.jp a écrit : >>>> >>>> Hi Ralph, >>>> >>>> The numa-node in AMD Mangy-Cours/Interlagos is so called cc(cache >>>> coherent)NUMA, >>>> which seems to be a little bit different from the traditional numa >>>> defined >>>> in openmpi. >>>> >>>> I notice that ccNUMA object is almost same as L3cache object. >>>> So "-bind-to l3cache" or "-map-by l3cache" is valid for what I want >>>> to >>>> do. >>>> Therefore, "do not touch it" is one of the solution, I think ... >>>> >>>> Anyway, mixing up these two types of numa is the problem. >>>> >>>> Regards, >>>> Tetsuya Mishima >>>> >>>> I can wait it'll be fixed in 1.7.5 or later, because putting >>>> "-bind-to >>>> numa" >>>> and "-map-by numa" at the same time works as a workaround. >>>> >>>> Thanks, >>>> Tetsuya Mishima >>>> >>>> Yeah, it will impact everything that uses hwloc topology maps, I >>>> fear. >>>> >>>> One side note: you'll need to add --hetero-nodes to your cmd >>>> line. >>>> If >>>> we >>>> don't see that, we assume that all the node topologies are >>>> identical >>>> - >>>> which clearly isn't true here. >>>> I'll try to resolve the hier inversion over the holiday - won't >>>> be >>>> for >>>> 1.7.4, but hopefully for 1.7.5 >>>> Thanks >>>> Ralph >>>> >>>> On Dec 18, 2013, at 9:44 PM, tmish...@jcity.maeda.co.jp wrote: >>>> >>>> >>>> I think it's normal for AMD opteron having 8/16 cores such as >>>> magny cours or interlagos. Because it usually has 2 numa nodes >>>> in a cpu(socket), numa-node can not include a socket. This type >>>> of hierarchy would be natural. >>>> >>>> (node03 is Dell PowerEdge R815 and maybe quite common, I guess) >>>> >>>> By the way, I think this inversion should affect rmaps_lama >>>> mapping. >>>> >>>> Tetsuya Mishima >>>> >>>> Ick - yeah, that would be a problem. I haven't seen that type >>>> of >>>> hierarchical inversion before - is node03 a different type of >>>> chip? >>>> Might take awhile for me to adjust the code to handle hier >>>> inversion... :-( >>>> On Dec 18, 2013, at 9:05 PM, tmish...@jcity.maeda.co.jp wrote: >>>> >>>> >>>> Hi Ralph, >>>> >>>> I found the reason. I attached the main part of output with 32 >>>> core node(node03) and 8 core node(node05) at the bottom. >>>> >>>> From this information, socket of node03 includes numa-node. >>>> On the other hand, numa-node of node05 includes socket. >>>> The direction of object tree is opposite. >>>> >>>> Since "-map-by socket" may be assumed as default, >>>> for node05, "-bind-to numa and -map-by socket" means >>>> upward search. For node03, this should be downward. >>>> >>>> I guess that openmpi-1.7.4rc1 will always assume numa-node >>>> includes socket. Is it right? Then, upward search is assumed >>>> in orte_rmaps_base_compute_bindings even for node03 when I >>>> put "-bind-to numa and -map-by socket" option. >>>> >>>> [node03.cluster:15508] [[38286,0],0] rmaps:base:compute_usage >>>> [node03.cluster:15508] mca:rmaps: compute bindings for job >>>> [38286,1] >>>> with >>>> policy NUMA >>>> [node03.cluster:15508] mca:rmaps: bind upwards for job >>>> [38286,1] >>>> with >>>> bindings NUMA >>>> [node03.cluster:15508] [[38286,0],0] bind:upward target >>>> NUMANode >>>> type >>>> Machine >>>> >>>> That's the reason of this trouble. Therefore, adding "-map-by >>>> core" >>>> works. >>>> (mapping pattern seems to be strange ...) >>>> >>>> [mishima@node03 demos]$ mpirun -np 8 -bind-to numa -map-by >>>> core >>>> -report-bindings myprog >>>> [node03.cluster:15885] [[38679,0],0] bind:upward target >>>> NUMANode >>>> type >>>> Cache >>>> [node03.cluster:15885] [[38679,0],0] bind:upward target >>>> NUMANode >>>> type >>>> Cache >>>> [node03.cluster:15885] [[38679,0],0] bind:upward target >>>> NUMANode >>>> type >>>> Cache >>>> [node03.cluster:15885] [[38679,0],0] bind:upward target >>>> NUMANode >>>> type >>>> NUMANode >>>> [node03.cluster:15885] [[38679,0],0] bind:upward target >>>> NUMANode >>>> type >>>> Cache >>>> [node03.cluster:15885] [[38679,0],0] bind:upward target >>>> NUMANode >>>> type >>>> Cache >>>> [node03.cluster:15885] [[38679,0],0] bind:upward target >>>> NUMANode >>>> type >>>> Cache >>>> [node03.cluster:15885] [[38679,0],0] bind:upward target >>>> NUMANode >>>> type >>>> NUMANode >>>> [node03.cluster:15885] [[38679,0],0] bind:upward target >>>> NUMANode >>>> type >>>> Cache >>>> [node03.cluster:15885] [[38679,0],0] bind:upward target >>>> NUMANode >>>> type >>>> Cache >>>> [node03.cluster:15885] [[38679,0],0] bind:upward target >>>> NUMANode >>>> type >>>> Cache >>>> [node03.cluster:15885] [[38679,0],0] bind:upward target >>>> NUMANode >>>> type >>>> NUMANode >>>> [node03.cluster:15885] [[38679,0],0] bind:upward target >>>> NUMANode >>>> type >>>> Cache >>>> [node03.cluster:15885] [[38679,0],0] bind:upward target >>>> NUMANode >>>> type >>>> Cache >>>> [node03.cluster:15885] [[38679,0],0] bind:upward target >>>> NUMANode >>>> type >>>> Cache >>>> [node03.cluster:15885] [[38679,0],0] bind:upward target >>>> NUMANode >>>> type >>>> NUMANode >>>> [node03.cluster:15885] [[38679,0],0] bind:upward target >>>> NUMANode >>>> type >>>> Cache >>>> [node03.cluster:15885] [[38679,0],0] bind:upward target >>>> NUMANode >>>> type >>>> Cache >>>> [node03.cluster:15885] [[38679,0],0] bind:upward target >>>> NUMANode >>>> type >>>> Cache >>>> [node03.cluster:15885] [[38679,0],0] bind:upward target >>>> NUMANode >>>> type >>>> NUMANode >>>> [node03.cluster:15885] [[38679,0],0] bind:upward target >>>> NUMANode >>>> type >>>> Cache >>>> [node03.cluster:15885] [[38679,0],0] bind:upward target >>>> NUMANode >>>> type >>>> Cache >>>> [node03.cluster:15885] [[38679,0],0] bind:upward target >>>> NUMANode >>>> type >>>> Cache >>>> [node03.cluster:15885] [[38679,0],0] bind:upward target >>>> NUMANode >>>> type >>>> NUMANode >>>> [node03.cluster:15885] [[38679,0],0] bind:upward target >>>> NUMANode >>>> type >>>> Cache >>>> [node03.cluster:15885] [[38679,0],0] bind:upward target >>>> NUMANode >>>> type >>>> Cache >>>> [node03.cluster:15885] [[38679,0],0] bind:upward target >>>> NUMANode >>>> type >>>> Cache >>>> [node03.cluster:15885] [[38679,0],0] bind:upward target >>>> NUMANode >>>> type >>>> NUMANode >>>> [node03.cluster:15885] [[38679,0],0] bind:upward target >>>> NUMANode >>>> type >>>> Cache >>>> [node03.cluster:15885] [[38679,0],0] bind:upward target >>>> NUMANode >>>> type >>>> Cache >>>> [node03.cluster:15885] [[38679,0],0] bind:upward target >>>> NUMANode >>>> type >>>> Cache >>>> [node03.cluster:15885] [[38679,0],0] bind:upward target >>>> NUMANode >>>> type >>>> NUMANode >>>> [node03.cluster:15885] MCW rank 2 bound to socket 0[core 0[hwt >>>> 0]], >>>> socket >>>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so >>>> cket 0[core 3[hwt 0]]: >>>> >>>> >>>> [B/B/B/B/./././.][./././././././.][./././././././.][./././././././.] >>>> [node03.cluster:15885] MCW rank 3 bound to socket 0[core 0[hwt >>>> 0]], >>>> socket >>>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so >>>> cket 0[core 3[hwt 0]]: >>>> >>>> >>>> [B/B/B/B/./././.][./././././././.][./././././././.][./././././././.] >>>> [node03.cluster:15885] MCW rank 4 bound to socket 0[core 4[hwt >>>> 0]], >>>> socket >>>> 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], so >>>> cket 0[core 7[hwt 0]]: >>>> >>>> >>>> [././././B/B/B/B][./././././././.][./././././././.][./././././././.] >>>> [node03.cluster:15885] MCW rank 5 bound to socket 0[core 4[hwt >>>> 0]], >>>> socket >>>> 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], so >>>> cket 0[core 7[hwt 0]]: >>>> >>>> >>>> [././././B/B/B/B][./././././././.][./././././././.][./././././././.] >>>> [node03.cluster:15885] MCW rank 6 bound to socket 0[core 4[hwt >>>> 0]], >>>> socket >>>> 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], so >>>> cket 0[core 7[hwt 0]]: >>>> >>>> >>>> [././././B/B/B/B][./././././././.][./././././././.][./././././././.] >>>> [node03.cluster:15885] MCW rank 7 bound to socket 0[core 4[hwt >>>> 0]], >>>> socket >>>> 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], so >>>> cket 0[core 7[hwt 0]]: >>>> >>>> >>>> [././././B/B/B/B][./././././././.][./././././././.][./././././././.] >>>> [node03.cluster:15885] MCW rank 0 bound to socket 0[core 0[hwt >>>> 0]], >>>> socket >>>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so >>>> cket 0[core 3[hwt 0]]: >>>> >>>> >>>> [B/B/B/B/./././.][./././././././.][./././././././.][./././././././.] >>>> [node03.cluster:15885] MCW rank 1 bound to socket 0[core 0[hwt >>>> 0]], >>>> socket >>>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so >>>> cket 0[core 3[hwt 0]]: >>>> >>>> >>>> [B/B/B/B/./././.][./././././././.][./././././././.][./././././././.] >>>> Hello world from process 6 of 8 >>>> Hello world from process 5 of 8 >>>> Hello world from process 0 of 8 >>>> Hello world from process 7 of 8 >>>> Hello world from process 3 of 8 >>>> Hello world from process 4 of 8 >>>> Hello world from process 2 of 8 >>>> Hello world from process 1 of 8 >>>> >>>> Regards, >>>> Tetsuya Mishima >>>> >>>> [node03.cluster:15508] Type: Machine Number of child objects: >>>> 4 >>>> Name=NULL >>>> total=132358820KB >>>> Backend=Linux >>>> OSName=Linux >>>> OSRelease=2.6.18-308.16.1.el5 >>>> OSVersion="#1 SMP Tue Oct 2 22:01:43 EDT 2012" >>>> Architecture=x86_64 >>>> Cpuset: 0xffffffff >>>> Online: 0xffffffff >>>> Allowed: 0xffffffff >>>> Bind CPU proc: TRUE >>>> Bind CPU thread: TRUE >>>> Bind MEM proc: FALSE >>>> Bind MEM thread: TRUE >>>> Type: Socket Number of child objects: 2 >>>> Name=NULL >>>> total=33071780KB >>>> CPUModel="AMD Opteron(tm) Processor 6136" >>>> Cpuset: 0x000000ff >>>> Online: 0x000000ff >>>> Allowed: 0x000000ff >>>> Type: NUMANode Number of child objects: 1 >>>> >>>> >>>> [node05.cluster:21750] Type: Machine Number of child objects: >>>> 2 >>>> Name=NULL >>>> total=33080072KB >>>> Backend=Linux >>>> OSName=Linux >>>> OSRelease=2.6.18-308.16.1.el5 >>>> OSVersion="#1 SMP Tue Oct 2 22:01:43 EDT 2012" >>>> Architecture=x86_64 >>>> Cpuset: 0x000000ff >>>> Online: 0x000000ff >>>> Allowed: 0x000000ff >>>> Bind CPU proc: TRUE >>>> Bind CPU thread: TRUE >>>> Bind MEM proc: FALSE >>>> Bind MEM thread: TRUE >>>> Type: NUMANode Number of child objects: 1 >>>> Name=NULL >>>> local=16532232KB >>>> total=16532232KB >>>> Cpuset: 0x0000000f >>>> Online: 0x0000000f >>>> Allowed: 0x0000000f >>>> Type: Socket Number of child objects: 1 >>>> >>>> >>>> Hmm...try adding "-mca rmaps_base_verbose 10 -mca >>>> ess_base_verbose >>>> 5" >>>> to >>>> your cmd line and let's see what it thinks it found. >>>> >>>> On Dec 18, 2013, at 6:55 PM, tmish...@jcity.maeda.co.jp >>>> wrote: >>>> >>>> >>>> Hi, I report one more problem with openmpi-1.7.4rc1, >>>> which is more serious. >>>> >>>> For our 32 core nodes(AMD magny cours based) which has >>>> 8 numa-nodes, "-bind-to numa" does not work. Without >>>> this option, it works. For your infomation, at the >>>> bottom of this mail, I added the lstopo information >>>> of the node. >>>> >>>> Regards, >>>> Tetsuya Mishima >>>> >>>> [mishima@manage ~]$ qsub -I -l nodes=1:ppn=32>> qsub: waiting for job > 8352.manage.cluster to start >>>> qsub: job 8352.manage.cluster ready >>>> >>>> [mishima@node03 demos]$ mpirun -np 8 -report-bindings >>>> -bind-to >>>> numa >>>> myprog >>>> [node03.cluster:15316] [[37582,0],0] bind:upward target >>>> NUMANode >>>> type >>>> Machine >>>> >>>> >>>> >>>> >>>> >>> > -------------------------------------------------------------------------- >>>> A request was made to bind to NUMA, but an appropriate >>>> target >>>> could >>>> not >>>> be found on node node03. >>>> >>>> >>>> >>>> >>>> >>> > -------------------------------------------------------------------------- >>>> [mishima@node03 ~]$ cd ~/Desktop/openmpi-1.7/demos/ >>>> [mishima@node03 demos]$ mpirun -np 8 -report-bindings myprog >>>> [node03.cluster:15282] MCW rank 2 bound to socket 1[core 8 >>>> [hwt >>>> 0]]: >>>> [./././././././.][B/././././././.][./././././././.][ >>>> ./././././././.]>>>>>>>>>>>> [node03.cluster:15282] MCW rank >>>> 3 bound to socket 1[core 9[hwt >>>> 0]]: >>>> [./././././././.][./B/./././././.][./././././././.][ >>>> ./././././././.] >>>> [node03.cluster:15282] MCW rank 4 bound to socket 2[core 16 >>>> [hwt >>>> 0]]: >>>> [./././././././.][./././././././.][B/././././././.] >>>> [./././././././.] >>>> [node03.cluster:15282] MCW rank 5 bound to socket 2[core 17 >>>> [hwt >>>> 0]]: >>>> [./././././././.][./././././././.][./B/./././././.] >>>> [./././././././.] >>>> [node03.cluster:15282] MCW rank 6 bound to socket 3[core 24 >>>> [hwt >>>> 0]]: >>>> [./././././././.][./././././././.][./././././././.] >>>> [B/././././././.] >>>> [node03.cluster:15282] MCW rank 7 bound to socket 3[core 25 >>>> [hwt >>>> 0]]: >>>> [./././././././.][./././././././.][./././././././.] >>>> [./B/./././././.] >>>> [node03.cluster:15282] MCW rank 0 bound to socket 0[core 0 >>>> [hwt >>>> 0]]: >>>> [B/././././././.][./././././././.][./././././././.][ >>>> ./././././././.] >>>> [node03.cluster:15282] MCW rank 1 bound to socket 0[core 1 >>>> [hwt >>>> 0]]: >>>> [./B/./././././.][./././././././.][./././././././.][ >>>> ./././././././.] >>>> Hello world from process 2 of 8 >>>> Hello world from process 5 of 8 >>>> Hello world from process 4 of 8 >>>> Hello world from process 3 of 8>>>>>>>>>> Hello world from >>>> process 1 of 8 >>>> Hello world from process 7 of 8 >>>> Hello world from process 6 of 8 >>>> Hello world from process 0 of 8 >>>> [mishima@node03 demos]$ ~/opt/hwloc/bin/lstopo-no-graphics >>>> Machine (126GB) >>>> Socket L#0 (32GB) >>>> NUMANode L#0 (P#0 16GB) + L3 L#0 (5118KB) >>>> L2 L#0 (512KB) + L1d L#0 (64KB) + L1i L#0 (64KB) + Core L#0 >>>> + >>>> PU >>>> L#0 >>>> (P#0) >>>> L2 L#1 (512KB) + L1d L#1 (64KB) + L1i L#1 (64KB) + Core L#1 >>>> + >>>> PU >>>> L#1 >>>> (P#1) >>>> L2 L#2 (512KB) + L1d L#2 (64KB) + L1i L#2 (64KB) + Core L#2 >>>> + >>>> PU >>>> L#2 >>>> (P#2) >>>> L2 L#3 (512KB) + L1d L#3 (64KB) + L1i L#3 (64KB) + Core L#3 >>>> + >>>> PU >>>> L#3 >>>> (P#3) >>>> NUMANode L#1 (P#1 16GB) + L3 L#1 (5118KB) >>>> L2 L#4 (512KB) + L1d L#4 (64KB) + L1i L#4 (64KB) + Core L#4 >>>> + >>>> PU >>>> L#4 >>>> (P#4) >>>> L2 L#5 (512KB) + L1d L#5 (64KB) + L1i L#5 (64KB) + Core L#5 >>>> + >>>> PU >>>> L#5 >>>> (P#5) >>>> L2 L#6 (512KB) + L1d L#6 (64KB) + L1i L#6 (64KB) + Core L#6 >>>> + >>>> PU >>>> L#6 >>>> (P#6) >>>> L2 L#7 (512KB) + L1d L#7 (64KB) + L1i L#7 (64KB) + Core L#7 >>>> + >>>> PU>>>>>> L#7 >>>> (P#7) >>>> Socket L#1 (32GB) >>>> NUMANode L#2 (P#6 16GB) + L3 L#2 (5118KB) >>>> L2 L#8 (512KB) + L1d L#8 (64KB) + L1i L#8 (64KB) + Core L#8 >>>> + >>>> PU >>>> L#8 >>>> (P#8) >>>> L2 L#9 (512KB) + L1d L#9 (64KB) + L1i L#9 (64KB) + Core L#9 >>>> + >>>> PU >>>> L#9 >>>> (P#9) >>>> L2 L#10 (512KB) + L1d L#10 (64KB) + L1i L#10 (64KB) + Core >>>> L#10 >>>> + >>>> PU >>>> L#10 (P#10) >>>> L2 L#11 (512KB) + L1d L#11 (64KB) + L1i L#11 (64KB) + Core >>>> L#11 >>>> + >>>> PU >>>> L#11 (P#11) >>>> NUMANode L#3 (P#7 16GB) + L3 L#3 (5118KB) >>>> L2 L#12 (512KB) + L1d L#12 (64KB) + L1i L#12 (64KB) + Core >>>> L#12 >>>> + >>>> PU >>>> L#12 (P#12) >>>> L2 L#13 (512KB) + L1d L#13 (64KB) + L1i L#13 (64KB) + Core >>>> L#13 >>>> + >>>> PU >>>> L#13 (P#13) >>>> L2 L#14 (512KB) + L1d L#14 (64KB) + L1i L#14 (64KB) + Core >>>> L#14 >>>> + >>>> PU >>>> L#14 (P#14) >>>> L2 L#15 (512KB) + L1d L#15 (64KB) + L1i L#15 (64KB) + Core >>>> L#15 >>>> + >>>> PU >>>> L#15 (P#15) >>>> Socket L#2 (32GB) >>>> NUMANode L#4 (P#4 16GB) + L3 L#4 (5118KB) >>>> L2 L#16 (512KB) + L1d L#16 (64KB) + L1i L#16 (64KB) + Core >>>> L#16 >>>> + >>>> PU >>>> L#16 (P#16) >>>> L2 L#17 (512KB) + L1d L#17 (64KB) + L1i L#17 (64KB) + Core >>>> L#17 >>>> + >>>> PU >>>> L#17 (P#17)> >>>>> L2 L#18 (512KB) + L1d L#18 (64KB) + >>>> L1i >>>> L#18 (64KB) + Core L#18 >>>> + >>>> PU >>>> L#18 (P#18) >>>> L2 L#19 (512KB) + L1d L#19 (64KB) + L1i L#19 (64KB) + Core >>>> L#19 >>>> + >>>> PU >>>> L#19 (P#19) >>>> NUMANode L#5 (P#5 16GB) + L3 L#5 (5118KB) >>>> L2 L#20 (512KB) + L1d L#20 (64KB) + L1i L#20 (64KB) + Core >>>> L#20 >>>> + >>>> PU >>>> L#20 (P#20) >>>> L2 L#21 (512KB) + L1d L#21 (64KB) + L1i L#21 (64KB) + Core >>>> L#21 >>>> + >>>> PU >>>> L#21 (P#21) >>>> L2 L#22 (512KB) + L1d L#22 (64KB) + L1i L#22 (64KB) + Core >>>> L#22 >>>> + >>>> PU >>>> L#22 (P#22) >>>> L2 L#23 (512KB) + L1d L#23 (64KB) + L1i L#23 (64KB) + Core >>>> L#23 >>>> + >>>> PU >>>> L#23 (P#23) >>>> Socket L#3 (32GB) >>>> NUMANode L#6 (P#2 16GB) + L3 L#6 (5118KB) >>>> L2 L#24 (512KB) + L1d L#24 (64KB) + L1i L#24 (64KB) + Core >>>> L#24 >>>> + >>>> PU >>>> L#24 (P#24)>>>>> L2 L#25 (512KB) + L1d L#25 (64KB) + L1i >>>> L#25 >>>> (64KB) + Core L#25 + >>>> PU >>>> L#25 (P#25) >>>> L2 L#26 (512KB) + L1d L#26 (64KB) + L1i L#26 (64KB) + Core >>>> L#26 >>>> + >>>> PU >>>> L#26 (P#26) >>>> L2 L#27 (512KB) + L1d L#27 (64KB) + L1i L#27 (64KB) + Core >>>> L#27 >>>> + >>>> PU >>>> L#27 (P#27) >>>> NUMANode L#7 (P#3 16GB) + L3 L#7 (5118KB) >>>> L2 L#28 (512KB) + L1d L#28 (64KB) + L1i L#28 (64KB) + Core >>>> L#28 >>>> + >>>> PU >>>> L#28 (P#28) >>>> L2 L#29 (512KB) + L1d L#29 (64KB) + L1i L#29 (64KB) + Core >>>> L#29 >>>> + >>>> PU >>>> L#29 (P#29) >>>> L2 L#30 (512KB) + L1d L#30 (64KB) + L1i L#30 (64KB) + Core >>>> L#30 >>>> + >>>> PU >>>> L#30 (P#30) >>>> L2 L#31 (512KB) + L1d L#31 (64KB) + L1i L#31 (64KB) + Core >>>> L#31 >>>> + >>>> PU >>>> L#31 (P#31) >>>> HostBridge L#0 >>>> PCIBridge >>>> PCI 14e4:1639 >>>> Net L#0 "eth0" >>>> PCI 14e4:1639 >>>> Net L#1 "eth1" >>>> PCIBridge >>>> PCI 14e4:1639 >>>> Net L#2 "eth2" >>>> PCI 14e4:1639 >>>> Net L#3 "eth3" >>>> PCIBridge >>>> PCIBridge >>>> PCIBridge >>>> PCI 1000:0072 >>>> Block L#4 "sdb" >>>> Block L#5 "sda" >>>> PCI 1002:4390 >>>> Block L#6 "sr0" >>>> PCIBridge >>>> PCI 102b:0532 >>>> HostBridge L#7 >>>> PCIBridge >>>> PCI 15b3:6274 >>>> Net L#7 "ib0" >>>> OpenFabrics L#8 "mthca0" >>>> >>>> _______________________________________________ >>>> users mailing list >>>> us...@open-mpi.org >>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>> _______________________________________________ >>>> users mailing list >>>> us...@open-mpi.org>> >>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>> _______________________________________________ >>>> users mailing list >>>> us...@open-mpi.org >>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>> _______________________________________________ >>>> users mailing list >>>> us...@open-mpi.org >>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>> _______________________________________________ >>>> users mailing list >>>> us...@open-mpi.org >>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>> _______________________________________________ >>>> users mailing list >>>> us...@open-mpi.org >>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>> _______________________________________________ >>>> users mailing list >>>> us...@open-mpi.org >>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>> _______________________________________________ >>>> users mailing list >>>> us...@open-mpi.org >>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>> >>>> _______________________________________________ >>>> users mailing list >>>> us...@open-mpi.org >>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>> >>>> _______________________________________________ >>>> users mailing list >>>> us...@open-mpi.org >>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>> >>>> _______________________________________________ >>>> users mailing list >>>> us...@open-mpi.org >>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>> >>>> _______________________________________________ >>>> users mailing list >>>> us...@open-mpi.org >>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>> >>>> _______________________________________________ >>>> users mailing list >>>> us...@open-mpi.org >>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>> >>>> _______________________________________________ >>>> users mailing list >>>> us...@open-mpi.org >>>> >>> > http://www.open-mpi.org/mailman/listinfo.cgi/users_______________________________________________ > >>> >>>> users mailing list >>>> users@open-mpi.orghttp://www.open-mpi.org/mailman/listinfo.cgi/users >>> >>> _______________________________________________ >>> users mailing list >>> us...@open-mpi.org >>> http://www.open-mpi.org/mailman/listinfo.cgi/users >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users
