Re: [hwloc-users] Netloc feature suggestion
Don't forget that network topologies can also be complex -- it's not always a simple, single-path hierarchy. There can be multiple paths between any pair of hosts on the network. Sometimes the hosts are aware of the multiple paths, sometimes they are not (e.g., sometimes the fabric routing changes during the course of a single MPI job, and the hosts/MPI applications are unaware). Meaning: the information about which network paths are taken for a given host-A-to-host-B traversal may be both distributed and transient. On Aug 14, 2019, at 11:05 AM, Rigel Falcao do Couto Alves mailto:rigel.al...@tu-dresden.de>> wrote: Hi, I am doing a PhD in performance analysis of highly parallel CFD codes and would like to suggest a feature for Netloc: from topic Build Scotch sub-architectures (at https://www.open-mpi.org/projects/hwloc/doc/v2.0.3/a00329.php), create a function-version of netloc_get_resources, which could retrieve at runtime the network details of the available cluster resources (i.e. the nodes allocated to the job). I am mostly interested about how many switches (the gray circles in the figure below) need to be traversed in order for any pair of allocated nodes to communicate with each other: For example, suppose my job is running within 4 nodes in the cluster, illustrated by the numbers above. All I would love to get from Netloc - at runtime - is some sort of classification of the nodes, like: 1: aa 2: ab 3: ba 4: ca The difference between nodes 1 and 2 is on the last digit, which means their MPI communications only need to traverse 1 switch; however, between any of them and nodes 3 or 4, the difference starts on the second-last digit, which means their communications need to traverse two switches. More digits may be left-added to the string, per necessity; i.e. if the central gray circle on the above figure is connected to another switch, which in turnleads to another part of the cluster's structure (with its own switches, nodes etc.). For me, it is at the present moment irrelevant whether e.g. nodes 1 and 2 are physically - or logically - consecutive to each other: a, b, c etc. would be just arbitrary identifiers. I would then use this data to plot the process placement, using open-source tools developed here in the University of Dresden (Germany); i.e. Scotch is not an option for me. The results of my study will be open-source as well and I can gladly share them with you once the thesis is finished. I hope I have clearly explained what I have in mind; please let me know if there are any questions. Finally, it is important that this feature is part of Netloc's API (as it is supposed to be integrated with the tools we develop here), works at runtime and doesn't require root privileges (as those tools are used by our cluster's costumers on their every-day job submissions). Kind regards, -- Dipl.-Ing. Rigel Alves researcher Technische Universität Dresden Center for Information Services and High Performance Computing (ZIH) Zellescher Weg 12 A 218, 01069 Dresden | Germany �� +49 (351) 463.42418 �� https://tu-dresden.de/zih/die-einrichtung/struktur/rigel-alves ___ hwloc-users mailing list hwloc-users@lists.open-mpi.org<mailto:hwloc-users@lists.open-mpi.org> https://lists.open-mpi.org/mailman/listinfo/hwloc-users -- Jeff Squyres jsquy...@cisco.com<mailto:jsquy...@cisco.com> ___ hwloc-users mailing list hwloc-users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/hwloc-users
Re: [hwloc-users] Hwloc command not working
Jeyaraj -- I think what we need is a bit more specific information in order to help you. Everyone's system is setup differently; we don't know how yours is setup. For example: - What version of hwloc did you install? - Where did you get the RPM for hwloc? - How exactly are you testing? - You mentioned that you installed an hwloc RPM -- did you look at the RPM contents to see if it includes the hwloc-dump-hwdata command? (e.g., "rpm -ql hwloc") - If you're using hwloc from that RPM and something is not there that you expect to be there, you might want to contact the maintainer of that RPM (e.g., look at "rpm -qi hwloc") - ...etc. When you check the contents of the hwloc RPM, you might notice that hwloc-dump-hwdata might well be installed in the sbin directory, not the bin directory. A wild guess: perhaps your PATH does not include the sbin directory...? > On Mar 2, 2017, at 8:25 AM, Marco Atzeri <marco.atz...@gmail.com> wrote: > > On 02/03/2017 11:56, jeyaraj wrote: >> Hi, >> All rpm file and tar file also installed. Not working >> > > a bit vague, and we have no crystal ball to look on your system. > > Are you sure the package is properly installed ? > How you did the verification ? > Should you not ask on the mailing list for help of your system, > what ever it is, instead of asking here ? > > > > > > > > ___ > hwloc-users mailing list > hwloc-users@lists.open-mpi.org > https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users -- Jeff Squyres jsquy...@cisco.com ___ hwloc-users mailing list hwloc-users@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users
Re: [hwloc-users] hello world can't run in Ubuntu 12.04
+1 Please try upgrading to Open MPI v1.8.x and see if that solves your problem. > On Apr 15, 2015, at 12:06 AM, Christopher Samuel <sam...@unimelb.edu.au> > wrote: > > On 15/04/15 12:19, Li Li wrote: > >>I am installed openmpi 1.5 and test it with a simple program > > Umm, Open-MPI 1.5 is ancient! > > Open-MPI 1.8.x is the current stable release branch, 1.6 was the > previous stable release branch (we're still on that here). > > 1.5 was the old feature branch that led up to the 1.6 stable series. > > All the best, > Chris > -- > Christopher SamuelSenior Systems Administrator > VLSCI - Victorian Life Sciences Computation Initiative > Email: sam...@unimelb.edu.au Phone: +61 (0)3 903 55545 > http://www.vlsci.org.au/ http://twitter.com/vlsci > > ___ > hwloc-users mailing list > hwloc-us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-users > Link to this post: > http://www.open-mpi.org/community/lists/hwloc-users/2015/04/1163.php -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
Re: [hwloc-users] Selecting real cores vs HT cores
On Dec 11, 2014, at 2:03 PM, Brice Goglin <brice.gog...@inria.fr> wrote: > By the way, if you can't in the BIOS, you may want to disable the > hyperthread in the kernel: > > for i in $(hwloc-calc --whole-system --po -I pu core:all.pu:0) ; do echo 0 > > /sys/devices/system/cpu/cpu$i/online ; done > > (write 1 instead of 0 to reenable them). But keep in mind that this is the semantic equivalent of using hwloc-bind to bind to the first HT in each core. I.e., disabling HT in the Linux kernel just disables scheduling on the 2nd HT. -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
Re: [hwloc-users] Selecting real cores vs HT cores
I'm not sure you're asking a well-formed question. When the BIOS is set to enable hyper threading, then several resources on the core are split when the machine is booted up (e.g., some of the queue depths for various processing units in the core are half the length that they are when hyperthreading is disabled in the BIOS). Hence, running a process on a core that only uses a single hyperthread (when HT is enabled) is not quite the same thing as booting up with HT disabled and running that same job on the core. Make sense? Meaning: if you want to test HT vs. non-HT performance, you really need to change the BIOS settings and reboot, sorry. Also, note that if you have HT enabled and you run a single-threaded app bound to a core, it will only use 1 of those HTs -- the other HT will be largely dormant. Meaning: don't expect that running a single-threaded app on a core that has HT enabled will magically take advantage of some performance benefit of aggressive automatic parallelization. You really need multiple threads in a process to get performance advantages out of HT. On Dec 11, 2014, at 12:51 PM, Brock Palen <bro...@umich.edu> wrote: > When a system has HT enabled is one core presented the real one and one the > fake partner? Or is that not the case? > > If wanting to test behavior without messing with the bios how do I select > just the 'real cores' if this is the case? > > I am looking for the equivelent of > > hwloc-bind ALLREALCORES my.exe > > Doing some performance study type things. > > Thanks, > > Brock Palen > www.umich.edu/~brockp > CAEN Advanced Computing > XSEDE Campus Champion > bro...@umich.edu > (734)936-1985 > > > > ___ > hwloc-users mailing list > hwloc-us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-users > Link to this post: > http://www.open-mpi.org/community/lists/hwloc-users/2014/12/1126.php -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
Re: [hwloc-users] Using hwloc to detect Hard Disks
On Aug 28, 2014, at 7:27 PM, Samuel Thibault <samuel.thiba...@inria.fr> wrote: >> I am not able to figure out how to read Hard drive details, for e.g., >> the content provided by hdparm application. >> >> My first question is, is it possible to read this using hwloc? If yes, can >> anyone direct me to the documentation which describes how to use it? > > Well, hwloc's goal is to describe the hardware _locality_, not its > precise content. So we don't provide that level of detail, we only > provide where the pieces of hardware reside. Can you be a bit more specific about what information you want to query? I ask because it strikes me that hwloc does gather some kinds of hardware information and put them as attributes on existing hwloc topology objects. -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
Re: [hwloc-users] CPU info on ARM
I passed this on to my OMPI ARM contact (Leif Lindholm). Here's what he said: "It gets a bit trickier on ARM... since we may also have (implementation time) configurable cache sizes and also big.LITTLE (different processor models executing in the same SMP system)." He passed the question on to another ARM guy, asking for further detail. I'll pass on what he says. On Jan 28, 2014, at 3:39 AM, Brice Goglin <brice.gog...@inria.fr> wrote: > Hello, > > Is anybody familiar with ARM CPUs? > > I am adding more CPU information because Intel needs more: > CPUVendor=GenuineIntel > CPUModel=Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz > CPUModelNumber=45 > CPUFamilyNumber=6 > > Would something similar be useful for ARM? What are the fields below > from /proc/cpuinfo on ARM that would be useful to developers? > Processor: Marvell PJ4Bv7 Processor rev 1 (v7l) > BogoMIPS: 1196.85 > Features: swp half thumb fastmult vfp edsp vfpv3 vfpv3d16 tls > CPU implementer: 0x56 > CPU architecture: 7 > CPU variant: 0x1 > CPU part: 0x581 > CPU revision: 1 > Hardware: Marvell Armada-370 > Revision: > Serial: > > thanks > Brice > > ___ > hwloc-users mailing list > hwloc-us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-users -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
[hwloc-users] hwloc problem on SGI machine
Jeff Becker (CC'ed) reported to me a failure with hwloc 1.7.2 (in OMPI trunk). I had him verify this with a standalone hwloc 1.7.2, and then had him try standalone hwloc 1.8 as well -- all got the same failure. Here's what he's seeing in 1.7.2: $ lstopo Different OS indexes lstopo: topology-linux.c:2731: look_sysfsnode: Assertion `node == res_obj' failed. Aborted (core dumped) In 1.8, the issue is the same, but a different line number (2741). It's an SGI x86_64 server, running SLES 11. Is this an hwloc issue, or a hardware issue? -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
[hwloc-users] Migrating www.open-mpi.org
All -- Our hosting provider will be migrating www.open-mpi.org<http://www.open-mpi.org> to a new machine on Wednesday. See message below for details. Begin forwarded message: From: DongInn Kim <di...@cs.indiana.edu<mailto:di...@cs.indiana.edu>> Subject: Migrating www.open-mpi.org<http://www.open-mpi.org> from milliways to lion List-Post: hwloc-users@lists.open-mpi.org Date: August 5, 2013 11:53:38 AM PDT Dear Open MPI developers and users, We are planning to move all the services under www.open-mpi.org<http://www.open-mpi.org/> to the new server on Wednesday, Aug 7th, 2013. This migration may need some outage time of web services (e.g., http://www.open-mpi.org<http://www.open-mpi.org/>) and mailing list services (e.g., us...@open-mpi.org<mailto:us...@open-mpi.org>, de...@open-mpi.org<mailto:de...@open-mpi.org>, …). The migration schedule is following: - Date: Wednesday, Aug 7th, 2013 - Time: 6:00am-8:00am Pacific US time 7:00am-9:00am Mountain US time 8:00am-10:00am Central US time 9:00am-11:00am Eastern US time 1:00pm-3:00pm GMT The following services would not be available during the migration. - Web services (e.g., www.open-mpi.org<http://www.open-mpi.org/>) - mailing lists: ad...@open-mpi.org<mailto:ad...@open-mpi.org> announce bugs devel devel-core docs ft hwloc-announce hwloc-bugs hwloc-devel hwloc-svn hwloc-users llamas mtt-announce mtt-bugs mtt-devel mtt-devel-core mtt-results mtt-svn mtt-users ompi-user-docs-bugs ompi-user-docs-svn svn svn-docs svn-docs-full svn-full svn-private svn-private-full users - Mail archives http://www.open-mpi.org/community/lists/ - Mercurial mirror Will disappear (it has long-since moved out to Bitbucket) I hope that we will not lose any mails sent to the above mailing lists even during the migration but it would be really appreciated if you hold up sending emails and svn commit until the migration is done. Please let me know if you have any questions or issues about this migration. Regards, -- - DongInn --- CREST System administrator Indiana University Bloomington, IN -- Jeff Squyres jsquy...@cisco.com<mailto:jsquy...@cisco.com> For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
Re: [hwloc-users] Solaris and hwloc
On Sep 12, 2012, at 6:42 PM, Samuel Thibault wrote: > No, we have it, but not all solaris systems have it. Ah, I see. So if Siegmar had done "hwloc-bind socket:0 ..." -- assuming his system has lgrp support -- that should work. right? -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
Re: [hwloc-users] Thread binding problem
Perhaps you simply have run out of memory on that NUMA node, and therefore the malloc failed. Check "numactl --hardware", for example. You might want to check the output of numastat to see if one or more of your NUMA nodes have run out of memory. On Sep 5, 2012, at 12:58 PM, Gabriele Fatigati wrote: > I've reproduced the problem in a small MPI + OpenMP code. > > The error is the same: after some memory bind, gives "Cannot allocate memory". > > Thanks. > > 2012/9/5 Gabriele Fatigati <g.fatig...@cineca.it> > Downscaling the matrix size, binding works well, but the memory available is > enought also using more big matrix, so I'm a bit confused. > > Using the same big matrix size without binding the code works well, so how I > can explain this behaviour? > > Maybe hwloc_set_area_membind_nodeset introduces other extra allocation that > are resilient after the call? > > > > 2012/9/5 Brice Goglin <brice.gog...@inria.fr> > An internal malloc failed then. That would explain why your malloc failed too. > It looks like you malloc'ed too much memory in your program? > > Brice > > > > > Le 05/09/2012 15:56, Gabriele Fatigati a écrit : >> An update: >> >> placing strerror(errno) after hwloc_set_area_membind_nodeset gives: "Cannot >> allocate memory" >> >> 2012/9/5 Gabriele Fatigati <g.fatig...@cineca.it> >> Hi, >> >> I've noted that hwloc_set_area_membind_nodeset return -1 but errno is not >> equal to EXDEV or ENOSYS. I supposed that these two case was the two unique >> possibly. >> >> From the hwloc documentation: >> >> -1 with errno set to ENOSYS if the action is not supported >> -1 with errno set to EXDEV if the binding cannot be enforced >> >> >> Any other binding failure reason? The memory available is enought. >> >> 2012/9/5 Brice Goglin <brice.gog...@inria.fr> >> Hello Gabriele, >> >> The only limit that I would think of is the available physical memory on >> each NUMA node (numactl -H will tell you how much of each NUMA node memory >> is still available). >> malloc usually only fails (it returns NULL?) when there no *virtual* memory >> anymore, that's different. If you don't allocate tons of terabytes of >> virtual memory, this shouldn't happen easily. >> >> Brice >> >> >> >> >> Le 05/09/2012 14:27, Gabriele Fatigati a écrit : >>> Dear Hwloc users and developers, >>> >>> >>> I'm using hwloc 1.4.1 on a multithreaded program in a Linux platform, where >>> each thread bind many non contiguos pieces of a big matrix using in a very >>> intensive way hwloc_set_area_membind_nodeset function: >>> >>> hwloc_set_area_membind_nodeset(topology, punt+offset, len, nodeset, >>> HWLOC_MEMBIND_BIND, HWLOC_MEMBIND_THREAD | HWLOC_MEMBIND_MIGRATE); >>> >>> Binding seems works well, since the returned code from function is 0 for >>> every calls. >>> >>> The problems is that after binding, a simple little new malloc fails, >>> without any apparent reason. >>> >>> Disabling memory binding, the allocations works well. Is there any knows >>> problem if hwloc_set_area_membind_nodeset is used intensively? >>> >>> Is there some operating system limit for memory pages binding? >>> >>> Thanks in advance. >>> >>> -- >>> Ing. Gabriele Fatigati >>> >>> HPC specialist >>> >>> SuperComputing Applications and Innovation Department >>> >>> Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy >>> >>> www.cineca.itTel: +39 051 6171722 >>> >>> g.fatigati [AT] cineca.it >>> >>> >>> ___ >>> hwloc-users mailing list >>> >>> hwloc-us...@open-mpi.org >>> http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-users >> >> >> >> >> -- >> Ing. Gabriele Fatigati >> >> HPC specialist >> >> SuperComputing Applications and Innovation Department >> >> Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy >> >> www.cineca.itTel: +39 051 6171722 >> >> g.fatigati [AT] cineca.it >> >> >> >> -- >> Ing. Gabriele Fatigati >> >> HPC specialist >> >> SuperComputing Applications and Innovation Department >> >&
Re: [hwloc-users] HWLoc Documentation pages 404's
Try looking here: http://www.open-mpi.org/projects/hwloc/doc/ You have an extra "projects" in your URL. How did you get to that URL? Do we have a bug in our web pages somewhere? On Aug 10, 2012, at 3:56 PM, Brock Palen wrote: > http://www.open-mpi.org/projects/projects/hwloc/doc/ > > Oh noooss!!! > > Brock Palen > www.umich.edu/~brockp > CAEN Advanced Computing > bro...@umich.edu > (734)936-1985 > > > > ___ > hwloc-users mailing list > hwloc-us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-users -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
[hwloc-users] #tgfh (thank God for hwloc)
Yesterday, I was installing 2 machines in a physically remote location -- meaning that I did not have access to see or touch the machines. Although the 2 machines are slightly different models from each other, they both have multiple Ethernet ports: some LOM, and 2 ports on a PCI 10GB Ethernet NIC. All the Ethernet ports are live and connected to different networks. I was working on setting up the 2 ports on the PCI card. #tgfh, because hwloc clearly showed me which ports were on a PCI device (by grouping and by vendor ID) and told me exactly what their ethX devices were. And, by extension, it showed me which ports were LOM (and what their ethX devices were). See the 2 PDFs attached for what hwloc showed me on each machine. This allowed me to go edit the relevant /etc/sysconfig/network-scripts/ifcfg-ethX scripts and be up and running within minutes. Yay hwloc!! (sorry; I just felt the need to share this story :-) ) -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ svbu-mpi058.pdf Description: Adobe PDF document svbu-mpi059.pdf Description: Adobe PDF document
Re: [hwloc-users] GPU/NIC/CPU locality
On Nov 29, 2011, at 12:01 PM, Brice Goglin wrote: > Yes, always installed. There are some configure checks for verbs, but > it's only used for enabling verbs-related helper testing. Ok, how's this for output at the end of configure? Linux: - Hwloc optional build support status (more details can be found above): Probe / display PCI devices: yes Graphical output (Cairo):yes XML output: full Memory support: binding, set policy, migrate pages - OS X: - Hwloc optional build support status (more details can be found above): Probe / display PCI devices: no Graphical output (Cairo):yes XML output: full Memory support: none - XML support will show "basic" if libxml2 is not found. -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
Re: [hwloc-users] GPU/NIC/CPU locality
On Nov 29, 2011, at 11:53 AM, Brice Goglin wrote: >> What about MX, verbs, Cuda, ...? > > MX and verbs are not used internally, we just have public helpers to > interoperate with them (and tests). I forget -- are the helpers installed/available even if the MX headers/libraries are not found at configure time? (ditto for verbs, cuda, etc.) > Same for cuda in trunk (until Samuel's cuda branch gets merged). > > Brice > > ___ > hwloc-users mailing list > hwloc-us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-users -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
Re: [hwloc-users] GPU/NIC/CPU locality
On Nov 29, 2011, at 7:25 AM, Stefan Eilemann wrote: >> You are probably missing the libpci-devel package. > > Thanks, that either doesn't exist or wasn't installed on Redhat. It works now. > > I think messages of found/not found optional modules could be more prominent > at the end of the configure process. FWIW, I've traditionally been against such things for two reasons: 1. The information *was* displayed above (i.e., that pci-devel wasn't found/wasn't usable/whatever). I realize that most people don't read the stdout of configure at all, but all the information you need is already there. 2. A list of what will/will not be built at the end tends to grow lengthy such that it dilutes the value of repeating the information at the end. That being said, I can *somewhat* see the value of displaying a user-friendly "PCI device support will not be built" vs. the output of a configure test, which might be somewhat obscure. However, in hwloc's case, the configure test output is pretty self-evident. Examples: checking for PCI... no checking pci/pci.h usability... no checking pci/pci.h presence... no checking for pci/pci.h... no checking for LIBXML2... yes checking for xmlNewDoc... yes checking for final LIBXML2 support... yes A simple string search for "pci" and "xml" will find these lines in the configure output. Assumedly, if you're building from source, you've likely got at least *some* experience and it shouldn't be unreasonable to ask you to go look in the output of configure. Don't get me wrong -- I'm not dead-set against a listing at the bottom. I just find it redundant and somewhat of a maintenance hassle. -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
Re: [hwloc-users] Re : lstopo on multiple machines
On Aug 17, 2011, at 2:26 AM, Brice Goglin wrote: >> What about an MPI version of lstopo ? > > If you want to see the entire MPI job topology within a single topology, > doing it inside hwloc would likely require to check for mpirun/mpiexec > parameters and so on at configure... big mess. Something like below with > the previously proposed API/utility may be enough: >mpirun lstopo .xml >hwloc_xml_agregate cluster.xml *.xml >export HWLOC_XMLFILE=cluster.xml Much as an MPI version sounds interesting (even potentially as a 3rd-party tool), I have to agree that a shell script similar to what Brice typed might be much easier. -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
Re: [hwloc-users] Re : lstopo on multiple machines
I'd be against hwloc automatically spreading across multiple machines. I think there are plenty of tools to do that already. That being said, having better support for being able to aggregate data from multiple hwloc instances (e.g., lstopo) on multiple machines into a single, cohesive map, would be great (waving hands here; I have no specific suggestions). On Aug 16, 2011, at 10:02 AM, Brice Goglin wrote: > > Hello Seb, > Hwloc only looks at the local machine, there's no support for multinode > topology detection so far. We are considering adding it but we don't know yet > what users want to do with it, if it should be in the core or not, automatic > or nor. Your feedback is welcome. > Brice > > - Reply message - > De : "PULVERAIL S?bastien" <sebastien.pulver...@sogeti.com> > Pour?: <hwloc-us...@open-mpi.org> > Objet : [hwloc-users] lstopo on multiple machines > Date : mar., ao?t 16, 2011 15:04 > > > > > Hello, > > > > I have two machines I use for running my programs on multiple nodes (with > hydra or slurm). > > When I launch my lstopo command, only one machine characteristics are > printed. > > How can I tell HWLOC to look for those two machines ? > > > > -- > > Seb > > > > _______ > hwloc-users mailing list > hwloc-us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-users -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
[hwloc-users] Article about hwloc published in Linux Pro Magazine
Woo hoo! Brice, Samuel, and I wrote an article about hwloc for Linux Pro Magazine. My copy just showed up in the mail: http://blogs.cisco.com/performance/hwloc-article-published-in-linux-pro-magazine/ -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
Re: [hwloc-users] Patch to disable GCC __builtin_ operations
On Jun 8, 2011, at 5:30 PM, Dave Goodell wrote: >> Is there a reason we wouldn't disable it in OMPI's hwloc by default? > > Performance will be better when left enabled on platforms where the compiler > and the architecture are in agreement... I'm not too concerned about hwloc's performance in OMPI -- it'll be used during initialization only. Unless there's a dramatic difference for, say, large-core-count machines, I'd be inclined to just disable it unless there's some reason to leave it on. It's one less thing that a user will have to know/remember to --disable, even in Josh's exotic case. > IMO Josh's use case is a bit exotic. He's using one system's compiler as an > approximation of an appropriate compiler for another system instead of using > a cross compiler or compiling in an identical environment. That viewpoint > may or may not be shared by the OMPI developers. > > -Dave > > > ___ > hwloc-users mailing list > hwloc-us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-users -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
Re: [hwloc-users] Patch to disable GCC __builtin_ operations
Is there a reason we wouldn't disable it in OMPI's hwloc by default? On Jun 8, 2011, at 5:14 PM, Josh Hursey wrote: > In short, I haven't yet. I figured out the problem was in hwloc, and > started with the hwloc branch by itself. > > In Open MPI, we should be able to pass the --disable-gcc-builtin from > the main configure, right (since we pull in config/hwloc_internal.m4)? > So we would pass it similar to how we had to pass --disable-xml to > turn off that feature in the builtin hwloc (before it was turned off > by default). > > -- Josh > > On Wed, Jun 8, 2011 at 4:50 PM, Jeff Squyres <jsquy...@cisco.com> wrote: >> Josh -- >> >> How did you get this disabled from within OMPI? We don't invoke hwloc's >> configure via sub-shell; we directly invoke its m4, so we don't have an >> opportunity to pass --disable-gcc-builtin. Unless you passed that to the >> top-level OMPI configure script...? >> >> >> On Jun 8, 2011, at 4:28 PM, Josh Hursey wrote: >> >>> (This should have gone to the devel list) >>> >>> The attached patch adds a configure option (--disable-gcc-builtin) to >>> disable the use of GCC __builtin_ operations, even if the GCC compiler >>> supports them. The patch is a diff from the r3509 revision of the >>> hwloc trunk. >>> >>> I hit a problem when installing hwloc statically on a machine with a >>> slightly different gcc support libraries and OSs on the head/compile >>> node versus the compute nodes. The builtin functions would cause hwloc >>> to segfault when run on the compute nodes. By disabling the builtin >>> operations, and using the more portable techniques seemed to do the >>> trick. >>> >>> This problem first became apparent when using hwloc as part of Open >>> MPI. In Open MPI the mpirun process runs on the headnode, so the hwloc >>> install would work in the mpirun process but cause the compute >>> processes to segv. >>> >>> Can you review the patch, and apply it to the trunk? Once the patch is >>> in the trunk, then I'll work on the Open MPI folks to update their >>> revision. >>> >>> Thanks, >>> Josh >>> >>> -- >>> Joshua Hursey >>> Postdoctoral Research Associate >>> Oak Ridge National Laboratory >>> http://users.nccs.gov/~jjhursey >>> ___ >>> hwloc-users mailing list >>> hwloc-us...@open-mpi.org >>> http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-users >> >> >> -- >> Jeff Squyres >> jsquy...@cisco.com >> For corporate legal information go to: >> http://www.cisco.com/web/about/doing_business/legal/cri/ >> >> >> _______ >> hwloc-users mailing list >> hwloc-us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-users >> >> > > > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > > ___ > hwloc-users mailing list > hwloc-us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-users -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
Re: [hwloc-users] Patch to disable GCC __builtin_ operations
Josh -- How did you get this disabled from within OMPI? We don't invoke hwloc's configure via sub-shell; we directly invoke its m4, so we don't have an opportunity to pass --disable-gcc-builtin. Unless you passed that to the top-level OMPI configure script...? On Jun 8, 2011, at 4:28 PM, Josh Hursey wrote: > (This should have gone to the devel list) > > The attached patch adds a configure option (--disable-gcc-builtin) to > disable the use of GCC __builtin_ operations, even if the GCC compiler > supports them. The patch is a diff from the r3509 revision of the > hwloc trunk. > > I hit a problem when installing hwloc statically on a machine with a > slightly different gcc support libraries and OSs on the head/compile > node versus the compute nodes. The builtin functions would cause hwloc > to segfault when run on the compute nodes. By disabling the builtin > operations, and using the more portable techniques seemed to do the > trick. > > This problem first became apparent when using hwloc as part of Open > MPI. In Open MPI the mpirun process runs on the headnode, so the hwloc > install would work in the mpirun process but cause the compute > processes to segv. > > Can you review the patch, and apply it to the trunk? Once the patch is > in the trunk, then I'll work on the Open MPI folks to update their > revision. > > Thanks, > Josh > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > ___ > hwloc-users mailing list > hwloc-us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-users -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
Re: [hwloc-users] HWLOC problem
(brought over from the OMPI user's list) This likely means you installed hwloc to a non-standard location (meaning that your system is not looking for shared libraries in $hwloc_prefix/lib by default). If you prepend/append your LD_LIBRARY_PATH environment variable (or set it, if it's not already set) to include $hwloc_prefix/lib, it should find hwloc's shared library and lstopo -- and all of its friends -- should work fine. On Jun 7, 2011, at 12:51 PM, vaibhav dutt wrote: > Hi, > > I have installed HWLOC 1.2 on my cluster , each node has two Intel Xeon E5450 > quad cores. > When I try to execute the command "lstopo" to determine the hardware topology > of my system, > I get an error like: > > ./lstopo: error while loading shared libraries: libhwloc.so.3: cannot open > shared object file: No such file or directory > > > Can anyone please help me as to what is the reason for this error and where > can I find this shared > library. > > Thanks. > > > ___ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
[hwloc-users] Fwd: [OMPI devel] problem with absent L3 on AMD CPU
Moving this patch over to the hwloc users list... Begin forwarded message: > From: Andriy Gapon <a...@icyb.net.ua> > Date: April 10, 2011 3:47:37 AM EDT > To: de...@open-mpi.org > Subject: [OMPI devel] problem with absent L3 on AMD CPU > Reply-To: Open MPI Developers <de...@open-mpi.org> > > > It seems that lstopo can get mightly confused with AMD Athlon II processor > (family 10h) that doesn't have L3 cache. > > I believe that the following patch should fix that: > --- src/topology-x86.c.orig 2011-04-10 10:38:39.370239628 +0300 > +++ src/topology-x86.c2011-04-10 10:38:44.573256245 +0300 > @@ -59,10 +59,6 @@ > unsigned cachenum; > unsigned size = 0; > > - cachenum = infos->numcaches++; > - infos->cache = realloc(infos->cache, > infos->numcaches*sizeof(*infos->cache)); > - cache = >cache[cachenum]; > - > if (level == 1) > size = ((cpuid >> 24)) << 10; > else if (level == 2) > @@ -72,6 +68,10 @@ > if (!size) > return; > > + cachenum = infos->numcaches++; > + infos->cache = realloc(infos->cache, > infos->numcaches*sizeof(*infos->cache)); > + cache = >cache[cachenum]; > + > cache->type = 1; > cache->level = level; > if (level <= 2) > > > Otherwise, numcaches gets incremented and the cache array grows a new entry, > but > that new entry is not initialized. Maybe this is an OS or envrionment > specific > problem, but at least here on FreeBSD the new memory is not zero-ed out and > POSIX doesn't require realloc to do that. > > This report is for the version 1.1.2. > Apologies for the noise if this problem is already fixed in newer code. > > Thanks! > -- > Andriy Gapon > ___ > devel mailing list > de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/devel -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
Re: [hwloc-users] hwloc-ps output - how to verify process binding on the core level?
On Feb 14, 2011, at 9:35 AM, Siew Yin Chan wrote: > 1. I tried Open MPI 1.5.1 before turning to hwloc-bind. Yep. Open MPI 1.5.1 > does provide the --bycore and --bind-to-core option, but this option seems to > bind processes to cores on my machine according to the *physical* indexes: FWIW, you might want to try one of the OMPI 1.5.2 nightly tarballs -- we switched the process affinity stuff to hwloc in 1.5.2 (the 1.5.1 stuff uses a different mechanism). > FYI, my testing environment and application imposes these requirements for > optimum performance: > > i. Different binaries optimized for heterogeneous machines. This necessitates > MIMD, and can be done in OMPI using the -app option (providing an > application context file). > ii. The application is communication-sensitive. Thus, fine-grained process > mapping on *machines* and on *cores* is required to minimize inter-machine > and inter-socket communication costs occurring on the network and on the > system bus. Specifically, processes should be mapped onto successive cores of > one socket before the next socket is considered, i.e., socket.0:core0-3, then > socket.1:core0-3. In this case, the communication among neighboring rank 0-3 > will be confined to socket 0 without going through the system bus. Same for > rank 4-7 on socket 1. As such, the order of the cores should follow the > *logical* indexes. I think that OMPI 1.5.2 should do this for you -- rather than following and logical/physical ordering, it does what you describe: traverses successive cores on a socket before going to the next socket (which happens to correspond to hwloc's logical ordering, but that was not the intent). FWIW, we have a huge revamp of OMPI's affinity support on the mpirun command line that will offer much more flexible binding choices. > Initially, I tried combining the features of rankfile and appfile, e.g., > > $ cat rankfile8np4 > rank 0=compute-0-8 slot=0:0 > rank 1=compute-0-8 slot=0:1 > rank 2=compute-0-8 slot=0:2 > rank 3=compute-0-8 slot=0:3 > $ cat rankfile9np4 > rank 0=compute-0-9 slot=0:0 > rank 1=compute-0-9 slot=0:1 > rank 2=compute-0-9 slot=0:2 > rank 3=compute-0-9 slot=0:3 > $ cat my_appfile_rankfile > --host compute-0-8 -rf rankfile8np4 -np 4 ./test1 > --host compute-0-9 -rf rankfile9np4 -np 4 ./test2 > $ mpirun -app my_appfile_rankfile > > but found out that only the rankfile stated on the first line took effect; > the second was ignored completely. After some time of googling and trial and > error, I decided to try an external binder, and this direction led me to > hwloc-bind. > > Maybe I should bring the issue of rankfile + appfile to the OMPI mailing list. Yes. I'd have to look at it more closely, but it's possible that we only allow one rankfile per job -- i.e., that the rankfile should specify all the procs in the job, not on a per-host basis. But perhaps we don't warn/error if multiple rankfiles are used; I would consider that a bug. -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
Re: [hwloc-users] hwloc@SC10
Brice will also be giving a ~10 min short talk on hwloc in the Cisco booth; stop by and say hello! You can hear the "right" way to pronounce "hwloc". :-) Cisco is also hosting some Open MPI/MPI-related short talks in our booth; I just posted about this on the Open MPI lists: http://www.open-mpi.org/community/lists/users/2010/11/14741.php Drop by the Cisco booth for the exact schedule; we're right next to the main SciNet NOC. See you there! On Nov 8, 2010, at 11:22 AM, Brice Goglin wrote: > Hello, > For those of you going to SC10 @ New Orleans next week, you should know > that hwloc will be there too. I will be hanging around the INRIA Booth > (#2751, between TACC and Microsoft) and Jeff Squyres will be near the > Cisco Booth (#3247, on the other side of Microsoft). Feel free to visit > us and request new features for hwloc 1.2 :) > Brice > > ___ > hwloc-users mailing list > hwloc-us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-users -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
Re: [hwloc-users] hwloc_set/get_thread_cpubind
Fixed -- thanks for the heads-up! On Jul 14, 2010, at 2:28 PM, Αλέξανδρος Παπαδογιαννάκης wrote: > > hwloc_set_thread_cpubind and hwloc_get_thread_cpubind are missing from the > html documentation > http://www.open-mpi.org/projects/hwloc/doc/v1.0.1/group__hwlocality__binding.php > > _ > Your E-mail and More On-the-Go. Get Windows Live Hotmail Free. > https://signup.live.com/signup.aspx?id=60969 > ___ > hwloc-users mailing list > hwloc-us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-users > -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
Re: [hwloc-users] hwloc sockets support on solaris
On Jun 23, 2010, at 4:29 PM, Brice Goglin wrote: > Don't you want hwloc_cpuset_set(set, i) instead ? > hwloc_cpuset_cpu(set, i) changes the cpuset into a single CPU, i.e. it's > zero(set) + set(set, i). Ah. Well, that would do it. :-) -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
Re: [hwloc-users] hwloc sockets support on solaris
Hm. We should be. Here's the hwloc plugin code for setting CPU affinity (it's
static because it's invoked by function pointer):
static int module_set(opal_paffinity_base_cpu_set_t mask)
{
int i, ret = OPAL_SUCCESS;
hwloc_cpuset_t set;
hwloc_topology_t *t = _paffinity_hwloc_component.topology;
set = hwloc_cpuset_alloc();
hwloc_cpuset_zero(set);
for (i = 0; ((unsigned int) i) < OPAL_PAFFINITY_BITMASK_T_NUM_BITS; ++i) {
if (OPAL_PAFFINITY_CPU_ISSET(i, mask) &&
i < mca_paffinity_hwloc_component.cpuset_max_size) {
hwloc_cpuset_cpu(set, i);
}
}
if (0 != hwloc_set_cpubind(*t, set, 0)) {
ret = OPAL_ERR_IN_ERRNO;
}
hwloc_cpuset_free(set);
return ret;
}
On Jun 23, 2010, at 4:14 PM, Brice Goglin wrote:
> I see this in the solaris binding core:
>
> if (hwloc_cpuset_weight(hwloc_set) != 1) {
> errno = EXDEV;
> return -1;
> }
>
> OMPI doesn't get this error ?
>
> Brice
>
>
>
>
> Le 23/06/2010 21:56, Terry Dontje a écrit :
>> Does hwloc think it supports binding processes to sockets or multiple cpus?
>> I am asking because I believe there are no current Solaris accessors that
>> support this (processor_bind only binds a pid or a set of pids to a *single*
>> processor).
>>
>> I bring this up because in testing OMPI with hwloc support it looks like
>> -bind-to-socket is acting like -bind-to-core on Solaris. I believe the
>> issue is hwloc should be returning an error to tell OMPI it cannot
>> bind-to-socket or multiple cpus at that.
>>
>> --
>>
>> Terry D. Dontje | Principal Software Engineer
>> Developer Tools Engineering | +1.650.633.7054
>> Oracle - Performance Technologies
>> 95 Network Drive, Burlington, MA 01803
>> Email terry.don...@oracle.com
>> --
>>
>> Terry D. Dontje | Principal Software Engineer
>> Developer Tools Engineering | +1.650.633.7054
>> Oracle - Performance Technologies
>> 95 Network Drive, Burlington, MA 01803
>> Email terry.don...@oracle.com
>>
>>
>> ___
>> hwloc-users mailing list
>>
>> hwloc-us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-users
>>
>>
>>
>
> ___
> hwloc-users mailing list
> hwloc-us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-users
--
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/
Re: [hwloc-users] Getting a graphics view for anon graphic system...
On Jun 6, 2010, at 4:03 PM, Olivier Cessenat wrote: > What you write is clear to computer scientists, but I failed to figure > out what it meant. Sorry, it is clear now ! FWIW, there's a section about "output formats" in the hwloc-ls.1 man page. It's probably worth adding a sentence in there that the list in the man page applies to the filenames; i.e., that the filename determines the output format. Here's a snipit from the man page: OUTPUT FORMATS - Send a text summary to stdout. /dev/stdout Send a text summary to stdout. It is effectively the same as specifying "-". .txt If the filename ends in ".txt", lstopo outputs an ASCII art rep- resentation of the map. -.txt If the entire filename is "-.txt", lstopo outputs the same ASCII art representation as other ".txt" filenames, but with two exe- ceptions: 1) the output is sent to stdout, and 2) if colors are supported on the terminal, the ASCII art will be colorized. .fig If the filename ends in ".fig", lstopo outputs a representation of the map that can be loaded in Xfig. .pdf If the filename ends in ".pdf" and lstopo was compiled with the proper support, lstopo outputs a PDF representation of the map. .ps If the filename ends in ".ps" and lstopo was compiled with the proper support, lstopo outputs a Postscript representation of the map. .png If the filename ends in ".png" and lstopo was compiled with the proper support, lstopo outputs a PNG representation of the map. .svg If the filename ends in ".svn" and lstopo was compiled with the proper support, lstopo outputs an SVG representation of the map. .xml If the filename ends in ".xml" and lstopo was compiled with the proper support, lstopo outputs an XML representation of the map. It may be reused later, even on another machine, with lstopo --xml, the HWLOC_XMLFILE environment variable, or the hwloc_topology_set_xml() function. See the output of "lstopo --help" for a specific list of what graphical output formats are supported in your hwloc installation. -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
Re: [hwloc-users] hwloc on systems with more than 64 cpus?
I believe that Brice / Samuel (the two main developers) have tested hwloc on an old Altix 4700 with 256 itanium cores. I don't have their exact results, and I don't see them on IM right now, so I don't know if they're around today or not... On May 14, 2010, at 8:57 AM, Jirka Hladky wrote: > Hello, > > I have tested hwloc on several systems and I was very impressed with results. > It's a great tool! > > The biggest box I have tested it on had 64 CPUs. (32 cores + hyper threading > enabled). > > I wonder if somebody has tested it on box with more than 64 CPUs. If so, can > you please share your results? > > Thanks a lot! > Jirka > > ___ > hwloc-users mailing list > hwloc-us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-users > -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
[hwloc-users] 1.0rc2
A bunch of changes/fixes have gone in, and I figured out what was causing Badness when trying to embed hwloc into Open MPI. So I took the liberty of rolling 1.0rc2. Please give it a whirl and let us know how it goes: http://www.open-mpi.org/~jsquyres/www.open-mpi.org/software/hwloc/v1.0/ -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
