Re: [OMPI users] Gigabit ethernet (PCI Express) and openmpi v1.2.4
Hi Allan, This suggest that your chipset is not able to handle the full PCI-E speed on more than 3 ports. This usually depends on the way the PCI-E links are wired trough the ports and the capacity of the chipset itself. As an exemple we were never able to reach fullspeed performance with Myrinet 10g on IBM e325 nodes, because of chipset limitations. We had to make the node changed to solve the issue. Running several instances of NPtcp should somewhat show the bandwith limit of the PCI-E bus on your machine. Aurelien Hi Aurelien or anybody else, How do you run several instances of NPtcp simultaneouly between two idential nodes a1, a2 through 3 similat gigabit etherenet cards with different subnets and switches. a1: eth0 - 192.168.1.1 eth2 - 192.168.5.1 eth3 - 192.168.8.1 a2: eth0 - 192.168.1.2 eth2 - 192.168.5.2 eth3 - 192.168.8.2 This is the way i do it currently. on a1:>./NPtcp ssh a2 a2> ./NPtcp -h 192.168.1.1 -n 50 for tcp b/w of eth0 and so on sor eth2 and eth3 I do not how to do it simultaneosly ie check the total b/w of eth0+eth2+eth3 at the same time in one invocation of ./NPtcp. I can do it with mpirun and NPmpi Can some one please tell me how to do it with NPtcp because i do not know. Regards, Allan Menezes
Re: [OMPI users] Gigabit ethernet (PCI Express) and openmpi v1.2.4
Just to add. My whole cluster is intel em64t or x86_64 and with openmpiv1.2.4 i was getting for two pc express intel gigabit and a pciexpress gigabit ethernet Syskonnect @ 888, 892 and 892 Mbps measured using NPtcp a sum total bandwidth of 1950Mbps on two identical different systems connected by three gigabit switches. But by changing to the beta version of openmpi, version 1.3a1r16973 nightly and recompiling NPtcp( which does not matter since it uses gcc) and NPmpi which uses the newer mpicc I get for the same setting between two seperate identical nodes 2583Mbps which is a 3 fold increase in bandwidth! The MTU in all was default of 1500 for all eth cards and both trials. I am using Fedora Core 8, x86_64 for the operating system. Allan Menezes Hi, I found the problem. It's a bug with openmpi v 1.2.4 i think. As below tests confirm(AND an big THANKS to George!) I compiled openmpi v 1.3a1r16973 and tried the same tests with the same mca-params.conf file and got for three pci express gigabit ethernet cards a total bandwidth of 2583Mbps which is close to 892+892+888=2672Mbps for a linear increase in b/w everything else the same except for a recompilation of NPmpi and Nptcp of netpipe. NPmpi uses mpicc to compile NPmpi whereas NPtcp is compiled with gcc! I am now going to do some benchmarking with hpl of my basement cluster with openmpi v 1.3a1r16973 for increase in performnce and stability. V 1.2.4 is stable and completes all 18 hpl tests without errors! With openmpi v1.24 and NPmpi compiled wit's mpicc and using the shared memory commands below in --(a) I get for ./NPmpi -u 1 negative numbers for performance above approx 200Mbytes. Some sort of overflow in v1.2.4. Thank you, Regards, Allan Menezes Hi George, The following test peaks at 8392Mpbs: mpirun --prefix /opt/opnmpi124b --host a1,a1 -mca btl tcp,sm,self -np 2 ./NPmpi on a1 and on a2 mpirun --prefix /opt/opnmpi124b --host a2,a2 -mca btl tcp,sm,self -np 2 ./NPmpi gives 8565Mbps --(a) on a1: mpirun --prefix /opt/opnmpi124b --host a1,a1 -np 2 ./NPmpi gives 8424Mbps on a2: mpirun --prefix /opt/opnmpi124b --host a2,a2 -np 2 ./NPmpi gives 8372Mbps So theres enough memory and processor b/w to give 2.7Gbps for 3 pci express eth cards especially from --(a) between a1 and a2? Thank you for your help. Any assistance would be greatly apprectiated! Regards, Allan Menezes You should run a shared memory test, to see what's the max memory bandwidth you can get. Thanks, george. On Dec 17, 2007, at 7:14 AM, Gleb Natapov wrote: On Sun, Dec 16, 2007 at 06:49:30PM -0500, Allan Menezes wrote: Hi, How many PCI-Express Gigabit ethernet cards does OpenMPI version 1.2.4 support with a corresponding linear increase in bandwith measured with netpipe NPmpi and openmpi mpirun? With two PCI express cards I get a B/W of 1.75Gbps for 892Mbps each ans for three pci express cards ( one built into the motherboard) i get 1.95Gbps. They all are around 890Mbs indiviually measured with netpipe and NPtcp and NPmpi and openmpi. For two it seems there is a linear increase in b/w but not for three pci express gigabit eth cards. I have tune the cards using netpipe and $HOME/.openmpi/mca- params.conf file for latency and percentage b/w . Please advise. What is in your $HOME/.openmpi/mca-params.conf? May be are hitting your chipset limit here. What is your HW configuration? Can you try to run NPtcp on each interface simultaneously and see what BW do you get. -- Gleb.
Re: [OMPI users] Gigabit ethernet (PCI Express) and openmpi v1.2.4
Hi, I found the problem. It's a bug with openmpi v 1.2.4 i think. As below tests confirm(AND an big THANKS to George!) I compiled openmpi v 1.3a1r16973 and tried the same tests with the same mca-params.conf file and got for three pci express gigabit ethernet cards a total bandwidth of 2583Mbps which is close to 892+892+888=2672Mbps for a linear increase in b/w everything else the same except for a recompilation of NPmpi and Nptcp of netpipe. NPmpi uses mpicc to compile NPmpi whereas NPtcp is compiled with gcc! I am now going to do some benchmarking with hpl of my basement cluster with openmpi v 1.3a1r16973 for increase in performnce and stability. V 1.2.4 is stable and completes all 18 hpl tests without errors! With openmpi v1.24 and NPmpi compiled wit's mpicc and using the shared memory commands below in --(a) I get for ./NPmpi -u 1 negative numbers for performance above approx 200Mbytes. Some sort of overflow in v1.2.4. Thank you, Regards, Allan Menezes Hi George, The following test peaks at 8392Mpbs: mpirun --prefix /opt/opnmpi124b --host a1,a1 -mca btl tcp,sm,self -np 2 ./NPmpi on a1 and on a2 mpirun --prefix /opt/opnmpi124b --host a2,a2 -mca btl tcp,sm,self -np 2 ./NPmpi gives 8565Mbps --(a) on a1: mpirun --prefix /opt/opnmpi124b --host a1,a1 -np 2 ./NPmpi gives 8424Mbps on a2: mpirun --prefix /opt/opnmpi124b --host a2,a2 -np 2 ./NPmpi gives 8372Mbps So theres enough memory and processor b/w to give 2.7Gbps for 3 pci express eth cards especially from --(a) between a1 and a2? Thank you for your help. Any assistance would be greatly apprectiated! Regards, Allan Menezes You should run a shared memory test, to see what's the max memory bandwidth you can get. Thanks, george. On Dec 17, 2007, at 7:14 AM, Gleb Natapov wrote: On Sun, Dec 16, 2007 at 06:49:30PM -0500, Allan Menezes wrote: Hi, How many PCI-Express Gigabit ethernet cards does OpenMPI version 1.2.4 support with a corresponding linear increase in bandwith measured with netpipe NPmpi and openmpi mpirun? With two PCI express cards I get a B/W of 1.75Gbps for 892Mbps each ans for three pci express cards ( one built into the motherboard) i get 1.95Gbps. They all are around 890Mbs indiviually measured with netpipe and NPtcp and NPmpi and openmpi. For two it seems there is a linear increase in b/w but not for three pci express gigabit eth cards. I have tune the cards using netpipe and $HOME/.openmpi/mca- params.conf file for latency and percentage b/w . Please advise. What is in your $HOME/.openmpi/mca-params.conf? May be are hitting your chipset limit here. What is your HW configuration? Can you try to run NPtcp on each interface simultaneously and see what BW do you get. -- Gleb.
Re: [OMPI users] Gigabit ethernet (PCI Express) and openmpi v1.2.4
Hi George, The following test peaks at 8392Mpbs: mpirun --prefix /opt/opnmpi124b --host a1,a1 -mca btl tcp,sm,self -np 2 ./NPmpi on a1 and on a2 mpirun --prefix /opt/opnmpi124b --host a2,a2 -mca btl tcp,sm,self -np 2 ./NPmpi gives 8565Mbps --(a) on a1: mpirun --prefix /opt/opnmpi124b --host a1,a1 -np 2 ./NPmpi gives 8424Mbps on a2: mpirun --prefix /opt/opnmpi124b --host a2,a2 -np 2 ./NPmpi gives 8372Mbps So theres enough memory and processor b/w to give 2.7Gbps for 3 pci express eth cards especially from --(a) between a1 and a2? Thank you for your help. Any assistance would be greatly apprectiated! Regards, Allan Menezes You should run a shared memory test, to see what's the max memory bandwidth you can get. Thanks, george. On Dec 17, 2007, at 7:14 AM, Gleb Natapov wrote: On Sun, Dec 16, 2007 at 06:49:30PM -0500, Allan Menezes wrote: Hi, How many PCI-Express Gigabit ethernet cards does OpenMPI version 1.2.4 support with a corresponding linear increase in bandwith measured with netpipe NPmpi and openmpi mpirun? With two PCI express cards I get a B/W of 1.75Gbps for 892Mbps each ans for three pci express cards ( one built into the motherboard) i get 1.95Gbps. They all are around 890Mbs indiviually measured with netpipe and NPtcp and NPmpi and openmpi. For two it seems there is a linear increase in b/w but not for three pci express gigabit eth cards. I have tune the cards using netpipe and $HOME/.openmpi/mca- params.conf file for latency and percentage b/w . Please advise. What is in your $HOME/.openmpi/mca-params.conf? May be are hitting your chipset limit here. What is your HW configuration? Can you try to run NPtcp on each interface simultaneously and see what BW do you get. -- Gleb.
Re: [OMPI users] Gigabit ethernet (PCI Express) and openmpi v1.2.4
You should run a shared memory test, to see what's the max memory bandwidth you can get. Thanks, george. On Dec 17, 2007, at 7:14 AM, Gleb Natapov wrote: On Sun, Dec 16, 2007 at 06:49:30PM -0500, Allan Menezes wrote: Hi, How many PCI-Express Gigabit ethernet cards does OpenMPI version 1.2.4 support with a corresponding linear increase in bandwith measured with netpipe NPmpi and openmpi mpirun? With two PCI express cards I get a B/W of 1.75Gbps for 892Mbps each ans for three pci express cards ( one built into the motherboard) i get 1.95Gbps. They all are around 890Mbs indiviually measured with netpipe and NPtcp and NPmpi and openmpi. For two it seems there is a linear increase in b/w but not for three pci express gigabit eth cards. I have tune the cards using netpipe and $HOME/.openmpi/mca- params.conf file for latency and percentage b/w . Please advise. What is in your $HOME/.openmpi/mca-params.conf? May be are hitting your chipset limit here. What is your HW configuration? Can you try to run NPtcp on each interface simultaneously and see what BW do you get. -- Gleb. ___ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users smime.p7s Description: S/MIME cryptographic signature
Re: [OMPI users] Gigabit ethernet (PCI Express) and openmpi v1.2.4
On Sun, Dec 16, 2007 at 06:49:30PM -0500, Allan Menezes wrote: > Hi, > How many PCI-Express Gigabit ethernet cards does OpenMPI version 1.2.4 > support with a corresponding linear increase in bandwith measured with > netpipe NPmpi and openmpi mpirun? > With two PCI express cards I get a B/W of 1.75Gbps for 892Mbps each ans > for three pci express cards ( one built into the motherboard) i get > 1.95Gbps. They all are around 890Mbs indiviually measured with netpipe > and NPtcp and NPmpi and openmpi. For two it seems there is a linear > increase in b/w but not for three pci express gigabit eth cards. > I have tune the cards using netpipe and $HOME/.openmpi/mca-params.conf > file for latency and percentage b/w . > Please advise. What is in your $HOME/.openmpi/mca-params.conf? May be are hitting your chipset limit here. What is your HW configuration? Can you try to run NPtcp on each interface simultaneously and see what BW do you get. -- Gleb.
[OMPI users] Gigabit ethernet (PCI Express) and openmpi v1.2.4
Hi, How many PCI-Express Gigabit ethernet cards does OpenMPI version 1.2.4 support with a corresponding linear increase in bandwith measured with netpipe NPmpi and openmpi mpirun? With two PCI express cards I get a B/W of 1.75Gbps for 892Mbps each ans for three pci express cards ( one built into the motherboard) i get 1.95Gbps. They all are around 890Mbs indiviually measured with netpipe and NPtcp and NPmpi and openmpi. For two it seems there is a linear increase in b/w but not for three pci express gigabit eth cards. I have tune the cards using netpipe and $HOME/.openmpi/mca-params.conf file for latency and percentage b/w . Please advise. Regards, Allan Menezes
