Re: mdrun on 8-core AMD + GTX TITAN (was: Re: [gmx-users] Re: Gromacs-4.6 on two Titans GPUs)
Hi Mark and Szilard Thanks for your both suggestions. They are very helpful. > > Neither run had a PP-PME work distribution suitable for the hardware it > was > running on (and fixing that for each run requires opposite changes). > Adding > a GPU and hoping to see scaling requires that there be proportionately > more > GPU work available to do, *and* enough absolute work to do. mdrun tries to > do this, and reports early in the log file, which is one of the reasons > Szilard asked to see whole log files - please use a file sharing service > to > do that. > This task involves GPU calculation. We would not see PP-PME work distribution. This is a good hint from the angle of PP-PME work distribution. And I guessed that two GPUs' calculations are fast / or no enough work for GPU calculation, which is aligned with your explanation. Please see logs below again. ONE GPU## http://pastebin.com/B6bRUVSa TWO GPUs## http://pastebin.com/SLAYnejP > > As you can see, this test was made on the same node regardless of > > networking. Can the performance be improved say 50% more when 2 GPUs > are > > used on a general task ? If yes, how ? > > > > >Indeed, as Richard pointed out, I was asking for *full* logs, these > > >summaries can't tell much, the table above the summary entitled "R E A > > >L C Y C L E A N D T I M E A C C O U N T I N G" as well as > > >other reported information across the log file is what I need to make > > >an assessment of your simulations' performance. > > > > Please see below. > > > > >>However, in your case I suspect that the > > >>bottleneck is multi-threaded scaling on the AMD CPUs and you should > > >>probably decrease the number of threads per MPI rank and share GPUs > > >>between 2-4 ranks. > > > > After I test all three clusters, I found it may NOT be an issue of AMD > > cpus. > > Intel cpus has the SAME scaling issue. > > > > However, I am curious as to how you justify the setup of 2-4 ranks > sharing > > GPUs ? Can you please explain it a bit more ? > > > > NUMA effects on multi-socket AMD processors are particularly severe; the > way GROMACS uses OpenMP is not well suited to them. Using a rank (or two) > per socket will greatly reduce those effects, but introduces different > algorithmic overhead from the need to do DD and explicitly communicate > between ranks. (You can see the latter in your .log file snippets below.) > Also, that means the parcel of PP work available from a rank to give to > the > GPU is smaller, which is the opposite of what you'd like for GPU > performance and/or scaling. We are working on a general solution for this > and lots of related issues in the post-5.0 space, but there is a very hard > limitation imposed by the need to amortize the cost of CPU-GPU transfer by > having lots of PP work available to do. > Is this reason why the scaling of two GPUs won't happen because of smaller PP workload ? >From the implication, I am wondering if we can increase PP workload through parameters in a mdp file. The question is what parameters are mostly related to PP workload ? Would you please give more specific suggestions ? > > > NOTE: The GPU has >20% more load than the CPU. This imbalance causes > > performance loss, consider using a shorter cut-off and a finer PME > > grid. > > > > This note needs to be addressed before maximum throughput is achieved and > the question of scaling is worth considering. Ideally, "Wait GPU local" > should be nearly zero, achieved as suggested above. Note that > launch+force+mesh+wait is the sum of gpu total! But much of the > information > needed is higher up the log file, and the whole question is constrained by > things like rvdw. > >From the note, it clearly suggested a shorter cut-off and a finer PME grid. I am not sure how to set up a finer PME grid but I am able to set up shorter cut-offs . However, it is risky to do so based on others' reports. Indeed, I see differences among tests for 1 GPU. Here cutoffs refer to rlist, rvdw and rcoulomb. I found that the smaller values of cutoffs, the faster computations. The question is how small they can go because it is interesting to know if these different cutoffs generate equally good simulations. As to two GPUs, when I set up larger cut-offs, these two GPUs in the same node had been very busy. However, the outcome in such a configuration is worse in terms of ns/day and time. So what dose "a finer PME grid" mean, with respect to GPU workload ? You mention the sum of GPU total is launch + force + mesh + wait.I thought PME mesh is carried out by CPU instead of GPU. Do I miss something here ? I thought GPU is responsible for the calculation of short-ranged non-bonded force whereas CPU is responsible for that of bonded and PME long-ranged force. Can you clarify it here ? Also, would rvdw play an important role in improving the performance of GPU calculation ? > > > Unfortunately you didn't copy the GPU timing stuff here!
Re: mdrun on 8-core AMD + GTX TITAN (was: Re: [gmx-users] Re: Gromacs-4.6 on two Titans GPUs)
As Mark said, please share the *entire* log file. Among other important things, the result of PP-PME tuning is not included above. However, I suspect that in this case scaling is strongly affected or by the small size of the system you are simulating. -- Szilárd On Sun, Nov 10, 2013 at 5:28 AM, Dwey Kauffman wrote: > Hi Szilard, > > Thank you very much for your suggestions. > >>Actually, I was jumping to conclusions too early, as you mentioned AMD >>"cluster", I assumed you must have 12-16-core Opteron CPUs. If you >>have an 8-core (desktop?) AMD CPU, than you may not need to run more >>than one rank per GPU. > > Yes, we do have independent clusters of AMD, AMD opteron, Intel Corei7. All > nodes of three clusters are installed with (at least) 1 GPU card. I have > run the same test on these three clusters. > > Let's focus on a basic scaling issue: One GPU v.s Two GPUs within the same > node of 8-core AMD cpu. > Using 1 GPU, we can have a performance of ~32 ns/day. Using two GPU, we > gain not much more ( ~38.5 ns/day ). It is about ~20% more performance. > However, this is not really true because in some tests, I also saw only 2-5% > more, which really surprised me. > > As you can see, this test was made on the same node regardless of > networking. Can the performance be improved say 50% more when 2 GPUs are > used on a general task ? If yes, how ? > >>Indeed, as Richard pointed out, I was asking for *full* logs, these >>summaries can't tell much, the table above the summary entitled "R E A >>L C Y C L E A N D T I M E A C C O U N T I N G" as well as >>other reported information across the log file is what I need to make >>an assessment of your simulations' performance. > > Please see below. > >>>However, in your case I suspect that the >>>bottleneck is multi-threaded scaling on the AMD CPUs and you should >>>probably decrease the number of threads per MPI rank and share GPUs >>>between 2-4 ranks. > > After I test all three clusters, I found it may NOT be an issue of AMD cpus. > Intel cpus has the SAME scaling issue. > > However, I am curious as to how you justify the setup of 2-4 ranks sharing > GPUs ? Can you please explain it a bit more ? > > >>You could try running >>mpirun -np 4 mdrun -ntomp 2 -gpu_id 0011 >>but I suspect this won't help because your scaling issue > > Your guess is correct but why is that ? it is worse. The more nodes are > involved in a task, the performance is worse. > > >>> in my >>>experience even reaction field runs don't scale across nodes with 10G >>>ethernet if you have more than 4-6 ranks per node trying to >>>communicate (let alone with PME). > > What dose it mean " let alone with PME" ? how to do so ? by mdrun ? > I do know " mdrun -npme to specify PME process. > > Thank you. > > Dwey > > > > ### One GPU > > R E A L C Y C L E A N D T I M E A C C O U N T I N G > > Computing: Nodes Th. Count Wall t (s) G-Cycles % > - > Neighbor search18 11 431.81713863.390 1.6 > Launch GPU ops.18501 472.90615182.556 1.7 > Force 185011328.61142654.785 4.9 > PME mesh 18501 11561.327 371174.09042.8 > Wait GPU local 185016888.008 221138.11125.5 > NB X/F buffer ops. 189911216.49939055.455 4.5 > Write traj.18 1030 12.741 409.039 0.0 > Update 185011696.35854461.226 6.3 > Constraints185011969.72663237.647 7.3 > Rest 11458.82046835.133 5.4 > - > Total 1 27036.812 868011.431 100.0 > - > - > PME spread/gather 18 10026975.086 223933.73925.8 > PME 3D-FFT 18 10023928.259 126115.97614.5 > PME solve 18501 636.48820434.327 2.4 > - > GPU timings > - > Computing: Count Wall t (s) ms/step % > - > Pair list H2D 11 43.4350.434 0.2 > X / q H2D501 567.1680.113 2.8 > Nonbonded F kernel 400 14174.3163.54470.8 > Nonbonded F+ene k.904314.4384.794
Re: mdrun on 8-core AMD + GTX TITAN (was: Re: [gmx-users] Re: Gromacs-4.6 on two Titans GPUs)
On Sun, Nov 10, 2013 at 5:28 AM, Dwey Kauffman wrote: > Hi Szilard, > > Thank you very much for your suggestions. > > >Actually, I was jumping to conclusions too early, as you mentioned AMD > >"cluster", I assumed you must have 12-16-core Opteron CPUs. If you > >have an 8-core (desktop?) AMD CPU, than you may not need to run more > >than one rank per GPU. > > Yes, we do have independent clusters of AMD, AMD opteron, Intel Corei7. All > nodes of three clusters are installed with (at least) 1 GPU card. I have > run the same test on these three clusters. > > Let's focus on a basic scaling issue: One GPU v.s Two GPUs within the > same > node of 8-core AMD cpu. > Using 1 GPU, we can have a performance of ~32 ns/day. Using two GPU, we > gain not much more ( ~38.5 ns/day ). It is about ~20% more performance. > However, this is not really true because in some tests, I also saw only > 2-5% > more, which really surprised me. Neither run had a PP-PME work distribution suitable for the hardware it was running on (and fixing that for each run requires opposite changes). Adding a GPU and hoping to see scaling requires that there be proportionately more GPU work available to do, *and* enough absolute work to do. mdrun tries to do this, and reports early in the log file, which is one of the reasons Szilard asked to see whole log files - please use a file sharing service to do that. As you can see, this test was made on the same node regardless of > networking. Can the performance be improved say 50% more when 2 GPUs are > used on a general task ? If yes, how ? > > >Indeed, as Richard pointed out, I was asking for *full* logs, these > >summaries can't tell much, the table above the summary entitled "R E A > >L C Y C L E A N D T I M E A C C O U N T I N G" as well as > >other reported information across the log file is what I need to make > >an assessment of your simulations' performance. > > Please see below. > > >>However, in your case I suspect that the > >>bottleneck is multi-threaded scaling on the AMD CPUs and you should > >>probably decrease the number of threads per MPI rank and share GPUs > >>between 2-4 ranks. > > After I test all three clusters, I found it may NOT be an issue of AMD > cpus. > Intel cpus has the SAME scaling issue. > > However, I am curious as to how you justify the setup of 2-4 ranks sharing > GPUs ? Can you please explain it a bit more ? > NUMA effects on multi-socket AMD processors are particularly severe; the way GROMACS uses OpenMP is not well suited to them. Using a rank (or two) per socket will greatly reduce those effects, but introduces different algorithmic overhead from the need to do DD and explicitly communicate between ranks. (You can see the latter in your .log file snippets below.) Also, that means the parcel of PP work available from a rank to give to the GPU is smaller, which is the opposite of what you'd like for GPU performance and/or scaling. We are working on a general solution for this and lots of related issues in the post-5.0 space, but there is a very hard limitation imposed by the need to amortize the cost of CPU-GPU transfer by having lots of PP work available to do. >You could try running > >mpirun -np 4 mdrun -ntomp 2 -gpu_id 0011 > >but I suspect this won't help because your scaling issue > > Your guess is correct but why is that ? it is worse. The more nodes are > involved in a task, the performance is worse. > > > >> in my > >>experience even reaction field runs don't scale across nodes with 10G > >>ethernet if you have more than 4-6 ranks per node trying to > >>communicate (let alone with PME). > > What dose it mean " let alone with PME" ? how to do so ? by mdrun ? > I do know " mdrun -npme to specify PME process. > If using PME (rather than RF), network demands are more severe. > Thank you. > > Dwey > > > > ### One GPU > > R E A L C Y C L E A N D T I M E A C C O U N T I N G > > Computing: Nodes Th. Count Wall t (s) G-Cycles % > > - > Neighbor search18 11 431.81713863.390 1.6 > Launch GPU ops.18501 472.90615182.556 1.7 > Force 185011328.61142654.785 4.9 > PME mesh 18501 11561.327 371174.09042.8 > Wait GPU local 185016888.008 221138.11125.5 > NB X/F buffer ops. 189911216.49939055.455 4.5 > Write traj.18 1030 12.741 409.039 0.0 > Update 185011696.35854461.226 6.3 > Constraints185011969.72663237.647 7.3 > Rest 11458.82046835.133 5.4 > > - > Total 1
Re: mdrun on 8-core AMD + GTX TITAN (was: Re: [gmx-users] Re: Gromacs-4.6 on two Titans GPUs)
Hi Szilard, Thank you very much for your suggestions. >Actually, I was jumping to conclusions too early, as you mentioned AMD >"cluster", I assumed you must have 12-16-core Opteron CPUs. If you >have an 8-core (desktop?) AMD CPU, than you may not need to run more >than one rank per GPU. Yes, we do have independent clusters of AMD, AMD opteron, Intel Corei7. All nodes of three clusters are installed with (at least) 1 GPU card. I have run the same test on these three clusters. Let's focus on a basic scaling issue: One GPU v.s Two GPUs within the same node of 8-core AMD cpu. Using 1 GPU, we can have a performance of ~32 ns/day. Using two GPU, we gain not much more ( ~38.5 ns/day ). It is about ~20% more performance. However, this is not really true because in some tests, I also saw only 2-5% more, which really surprised me. As you can see, this test was made on the same node regardless of networking. Can the performance be improved say 50% more when 2 GPUs are used on a general task ? If yes, how ? >Indeed, as Richard pointed out, I was asking for *full* logs, these >summaries can't tell much, the table above the summary entitled "R E A >L C Y C L E A N D T I M E A C C O U N T I N G" as well as >other reported information across the log file is what I need to make >an assessment of your simulations' performance. Please see below. >>However, in your case I suspect that the >>bottleneck is multi-threaded scaling on the AMD CPUs and you should >>probably decrease the number of threads per MPI rank and share GPUs >>between 2-4 ranks. After I test all three clusters, I found it may NOT be an issue of AMD cpus. Intel cpus has the SAME scaling issue. However, I am curious as to how you justify the setup of 2-4 ranks sharing GPUs ? Can you please explain it a bit more ? >You could try running >mpirun -np 4 mdrun -ntomp 2 -gpu_id 0011 >but I suspect this won't help because your scaling issue Your guess is correct but why is that ? it is worse. The more nodes are involved in a task, the performance is worse. >> in my >>experience even reaction field runs don't scale across nodes with 10G >>ethernet if you have more than 4-6 ranks per node trying to >>communicate (let alone with PME). What dose it mean " let alone with PME" ? how to do so ? by mdrun ? I do know " mdrun -npme to specify PME process. Thank you. Dwey ### One GPU R E A L C Y C L E A N D T I M E A C C O U N T I N G Computing: Nodes Th. Count Wall t (s) G-Cycles % - Neighbor search18 11 431.81713863.390 1.6 Launch GPU ops.18501 472.90615182.556 1.7 Force 185011328.61142654.785 4.9 PME mesh 18501 11561.327 371174.09042.8 Wait GPU local 185016888.008 221138.11125.5 NB X/F buffer ops. 189911216.49939055.455 4.5 Write traj.18 1030 12.741 409.039 0.0 Update 185011696.35854461.226 6.3 Constraints185011969.72663237.647 7.3 Rest 11458.82046835.133 5.4 - Total 1 27036.812 868011.431 100.0 - - PME spread/gather 18 10026975.086 223933.73925.8 PME 3D-FFT 18 10023928.259 126115.97614.5 PME solve 18501 636.48820434.327 2.4 - GPU timings - Computing: Count Wall t (s) ms/step % - Pair list H2D 11 43.4350.434 0.2 X / q H2D501 567.1680.113 2.8 Nonbonded F kernel 400 14174.3163.54470.8 Nonbonded F+ene k.904314.4384.79421.5 Nonbonded F+ene+prune k. 11 572.3705.724 2.9 F D2H501 358.1200.072 1.8 - Total 20029.8464.006 100.0 - Force evaluation time GPU/CPU: 4.006 ms/2.578 ms = 1.554 For optimal performance
Re: [gmx-users] Re: Gromacs-4.6 on two Titans GPUs
On Thu, Nov 7, 2013 at 6:34 AM, James Starlight wrote: > I've gone to conclusion that simulation with 1 or 2 GPU simultaneously gave > me the same performance > mdrun -ntmpi 2 -ntomp 6 -gpu_id 01 -v -deffnm md_CaM_test, > > mdrun -ntmpi 2 -ntomp 6 -gpu_id 0 -v -deffnm md_CaM_test, > > Doest it be due to the small CPU cores or addition RAM ( this system has 32 > gb) is needed ? OR may be some extra options are needed in the config? GROMACS does not really need much (or fast) ram and it's most probably not configuration settings that are causing the lack of scaling. Given your setup my guess is that your hardware for your system is simply imbalanced to be efficiently used in GROMACS runs. Please post *full* log files (FYI use e.g. http://pastebin.com), that will help explain what is going on. > > James > > > > > 2013/11/6 Richard Broadbent > >> Hi Dwey, >> >> >> On 05/11/13 22:00, Dwey Kauffman wrote: >> >>> Hi Szilard, >>> >>> Thanks for your suggestions. I am indeed aware of this page. In a >>> 8-core >>> AMD with 1GPU, I am very happy about its performance. See below. My >>> intention is to obtain a even better one because we have multiple nodes. >>> >>> ### 8 core AMD with 1 GPU, >>> Force evaluation time GPU/CPU: 4.006 ms/2.578 ms = 1.554 >>> For optimal performance this ratio should be close to 1! >>> >>> >>> NOTE: The GPU has >20% more load than the CPU. This imbalance causes >>>performance loss, consider using a shorter cut-off and a finer PME >>> grid. >>> >>> Core t (s) Wall t (s)(%) >>> Time: 216205.51027036.812 799.7 >>> 7h30:36 >>> (ns/day)(hour/ns) >>> Performance: 31.9560.751 >>> >>> ### 8 core AMD with 2 GPUs >>> >>> Core t (s) Wall t (s)(%) >>> Time: 178961.45022398.880 799.0 >>> 6h13:18 >>> (ns/day)(hour/ns) >>> Performance: 38.5730.622 >>> Finished mdrun on node 0 Sat Jul 13 09:24:39 2013 >>> >>> >> I'm almost certain that Szilard meant the lines above this that give the >> breakdown of where the time is spent in the simulation. >> >> Richard >> >> >>> However, in your case I suspect that the bottleneck is multi-threaded scaling on the AMD CPUs and you should probably decrease the number of threads per MPI rank and share GPUs between 2-4 ranks. >>> >>> >>> OK but can you give a example of mdrun command ? given a 8 core AMD with 2 >>> GPUs. >>> I will try to run it again. >>> >>> >>> Regarding scaling across nodes, you can't expect much from gigabit ethernet - especially not from the cheaper cards/switches, in my experience even reaction field runs don't scale across nodes with 10G ethernet if you have more than 4-6 ranks per node trying to communicate (let alone with PME). However, on infiniband clusters we have seen scaling to 100 atoms/core (at peak). >>> >>> From your comments, it sounds like a cluster of AMD cpus is difficult to >>> scale across nodes in our current setup. >>> >>> Let's assume we install Infiniband (20 or 40GB/s) in the same system of 16 >>> nodes of 8 core AMD with 1 GPU only. Considering the same AMD system, what >>> is a good way to obtain better performance when we run a task across >>> nodes >>> ? in other words, what dose mudrun_mpi look like ? >>> >>> Thanks, >>> Dwey >>> >>> >>> >>> >>> -- >>> View this message in context: http://gromacs.5086.x6.nabble. >>> com/Gromacs-4-6-on-two-Titans-GPUs-tp5012186p5012279.html >>> Sent from the GROMACS Users Forum mailing list archive at Nabble.com. >>> >>> -- >> gmx-users mailing [email protected] >> http://lists.gromacs.org/mailman/listinfo/gmx-users >> * Please search the archive at http://www.gromacs.org/ >> Support/Mailing_Lists/Search before posting! >> * Please don't post (un)subscribe requests to the list. Use the www >> interface or send it to [email protected]. >> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists >> > -- > gmx-users mailing [email protected] > http://lists.gromacs.org/mailman/listinfo/gmx-users > * Please search the archive at > http://www.gromacs.org/Support/Mailing_Lists/Search before posting! > * Please don't post (un)subscribe requests to the list. Use the > www interface or send it to [email protected]. > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists -- gmx-users mailing [email protected] http://lists.gromacs.org/mailman/listinfo/gmx-users * Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/Search before posting! * Please don't post (un)subscribe requests to the list. Use the www interface or send it to [email protected]. * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
mdrun on 8-core AMD + GTX TITAN (was: Re: [gmx-users] Re: Gromacs-4.6 on two Titans GPUs)
Let's not hijack James' thread as your hardware is different from his. On Tue, Nov 5, 2013 at 11:00 PM, Dwey Kauffman wrote: > Hi Szilard, > >Thanks for your suggestions. I am indeed aware of this page. In a 8-core > AMD with 1GPU, I am very happy about its performance. See below. My Actually, I was jumping to conclusions too early, as you mentioned AMD "cluster", I assumed you must have 12-16-core Opteron CPUs. If you have an 8-core (desktop?) AMD CPU, than you may not need to run more than one rank per GPU. > intention is to obtain a even better one because we have multiple nodes. Btw, I'm not sure it's an economically viable solution to install Infiniband network - especially if you have desktop-class machines. Such a network will end up costing >$500 per machine just for a single network card, let alone cabling and switches. > > ### 8 core AMD with 1 GPU, > Force evaluation time GPU/CPU: 4.006 ms/2.578 ms = 1.554 > For optimal performance this ratio should be close to 1! > > > NOTE: The GPU has >20% more load than the CPU. This imbalance causes > performance loss, consider using a shorter cut-off and a finer PME > grid. > >Core t (s) Wall t (s)(%) >Time: 216205.51027036.812 799.7 > 7h30:36 > (ns/day)(hour/ns) > Performance: 31.9560.751 > > ### 8 core AMD with 2 GPUs > >Core t (s) Wall t (s)(%) >Time: 178961.45022398.880 799.0 > 6h13:18 > (ns/day)(hour/ns) > Performance: 38.5730.622 > Finished mdrun on node 0 Sat Jul 13 09:24:39 2013 > Indeed, as Richard pointed out, I was asking for *full* logs, these summaries can't tell much, the table above the summary entitled "R E A L C Y C L E A N D T I M E A C C O U N T I N G" as well as other reported information across the log file is what I need to make an assessment of your simulations' performance. >>However, in your case I suspect that the >>bottleneck is multi-threaded scaling on the AMD CPUs and you should >>probably decrease the number of threads per MPI rank and share GPUs >>between 2-4 ranks. > > > OK but can you give a example of mdrun command ? given a 8 core AMD with 2 > GPUs. > I will try to run it again. You could try running mpirun -np 4 mdrun -ntomp 2 -gpu_id 0011 but I suspect this won't help because your scaling issue > > >>Regarding scaling across nodes, you can't expect much from gigabit >>ethernet - especially not from the cheaper cards/switches, in my >>experience even reaction field runs don't scale across nodes with 10G >>ethernet if you have more than 4-6 ranks per node trying to >>communicate (let alone with PME). However, on infiniband clusters we >>have seen scaling to 100 atoms/core (at peak). > > >From your comments, it sounds like a cluster of AMD cpus is difficult to > scale across nodes in our current setup. > > Let's assume we install Infiniband (20 or 40GB/s) in the same system of 16 > nodes of 8 core AMD with 1 GPU only. Considering the same AMD system, what > is a good way to obtain better performance when we run a task across nodes > ? in other words, what dose mudrun_mpi look like ? > > Thanks, > Dwey > > > > > -- > View this message in context: > http://gromacs.5086.x6.nabble.com/Gromacs-4-6-on-two-Titans-GPUs-tp5012186p5012279.html > Sent from the GROMACS Users Forum mailing list archive at Nabble.com. > -- > gmx-users mailing [email protected] > http://lists.gromacs.org/mailman/listinfo/gmx-users > * Please search the archive at > http://www.gromacs.org/Support/Mailing_Lists/Search before posting! > * Please don't post (un)subscribe requests to the list. Use the > www interface or send it to [email protected]. > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists -- gmx-users mailing [email protected] http://lists.gromacs.org/mailman/listinfo/gmx-users * Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/Search before posting! * Please don't post (un)subscribe requests to the list. Use the www interface or send it to [email protected]. * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
Re: [gmx-users] Re: Gromacs-4.6 on two Titans GPUs
First, there is no value in ascribing problems to the hardware if the simulation setup is not yet balanced, or not large enough to provide enough atoms and long enough rlist to saturate the GPUs, etc. Look at the log files and see what complaints mdrun makes about things like PME load balance, and the times reported for different components of the simulation, because these must differ between the two runs you report. diff -y -W 160 *log |less is your friend. Some (non-GPU-specific) background information in part 5 here http://www.gromacs.org/Documentation/Tutorials/GROMACS_USA_Workshop_and_Conference_2013/Topology_preparation%2c_%22What's_in_a_log_file%22%2c_basic_performance_improvements%3a_Mark_Abraham%2c_Session_1A (though I recommend the PDF version) Mark On Thu, Nov 7, 2013 at 6:34 AM, James Starlight wrote: > I've gone to conclusion that simulation with 1 or 2 GPU simultaneously gave > me the same performance > mdrun -ntmpi 2 -ntomp 6 -gpu_id 01 -v -deffnm md_CaM_test, > > mdrun -ntmpi 2 -ntomp 6 -gpu_id 0 -v -deffnm md_CaM_test, > > Doest it be due to the small CPU cores or addition RAM ( this system has 32 > gb) is needed ? OR may be some extra options are needed in the config? > > James > > > > > 2013/11/6 Richard Broadbent > > > Hi Dwey, > > > > > > On 05/11/13 22:00, Dwey Kauffman wrote: > > > >> Hi Szilard, > >> > >> Thanks for your suggestions. I am indeed aware of this page. In a > >> 8-core > >> AMD with 1GPU, I am very happy about its performance. See below. My > >> intention is to obtain a even better one because we have multiple nodes. > >> > >> ### 8 core AMD with 1 GPU, > >> Force evaluation time GPU/CPU: 4.006 ms/2.578 ms = 1.554 > >> For optimal performance this ratio should be close to 1! > >> > >> > >> NOTE: The GPU has >20% more load than the CPU. This imbalance causes > >>performance loss, consider using a shorter cut-off and a finer > PME > >> grid. > >> > >> Core t (s) Wall t (s)(%) > >> Time: 216205.51027036.812 799.7 > >> 7h30:36 > >> (ns/day)(hour/ns) > >> Performance: 31.9560.751 > >> > >> ### 8 core AMD with 2 GPUs > >> > >> Core t (s) Wall t (s)(%) > >> Time: 178961.45022398.880 799.0 > >> 6h13:18 > >> (ns/day)(hour/ns) > >> Performance: 38.5730.622 > >> Finished mdrun on node 0 Sat Jul 13 09:24:39 2013 > >> > >> > > I'm almost certain that Szilard meant the lines above this that give the > > breakdown of where the time is spent in the simulation. > > > > Richard > > > > > >> However, in your case I suspect that the > >>> bottleneck is multi-threaded scaling on the AMD CPUs and you should > >>> probably decrease the number of threads per MPI rank and share GPUs > >>> between 2-4 ranks. > >>> > >> > >> > >> OK but can you give a example of mdrun command ? given a 8 core AMD > with 2 > >> GPUs. > >> I will try to run it again. > >> > >> > >> Regarding scaling across nodes, you can't expect much from gigabit > >>> ethernet - especially not from the cheaper cards/switches, in my > >>> experience even reaction field runs don't scale across nodes with 10G > >>> ethernet if you have more than 4-6 ranks per node trying to > >>> communicate (let alone with PME). However, on infiniband clusters we > >>> have seen scaling to 100 atoms/core (at peak). > >>> > >> > >> From your comments, it sounds like a cluster of AMD cpus is difficult > to > >>> > >> scale across nodes in our current setup. > >> > >> Let's assume we install Infiniband (20 or 40GB/s) in the same system of > 16 > >> nodes of 8 core AMD with 1 GPU only. Considering the same AMD system, > what > >> is a good way to obtain better performance when we run a task across > >> nodes > >> ? in other words, what dose mudrun_mpi look like ? > >> > >> Thanks, > >> Dwey > >> > >> > >> > >> > >> -- > >> View this message in context: http://gromacs.5086.x6.nabble. > >> com/Gromacs-4-6-on-two-Titans-GPUs-tp5012186p5012279.html > >> Sent from the GROMACS Users Forum mailing list archive at Nabble.com. > >> > >> -- > > gmx-users mailing [email protected] > > http://lists.gromacs.org/mailman/listinfo/gmx-users > > * Please search the archive at http://www.gromacs.org/ > > Support/Mailing_Lists/Search before posting! > > * Please don't post (un)subscribe requests to the list. Use the www > > interface or send it to [email protected]. > > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists > > > -- > gmx-users mailing [email protected] > http://lists.gromacs.org/mailman/listinfo/gmx-users > * Please search the archive at > http://www.gromacs.org/Support/Mailing_Lists/Search before posting! > * Please don't post (un)subscribe requests to the list. Use the > www interface or send it to [email protected]. > * Can't post? Read
Re: [gmx-users] Re: Gromacs-4.6 on two Titans GPUs
I've gone to conclusion that simulation with 1 or 2 GPU simultaneously gave me the same performance mdrun -ntmpi 2 -ntomp 6 -gpu_id 01 -v -deffnm md_CaM_test, mdrun -ntmpi 2 -ntomp 6 -gpu_id 0 -v -deffnm md_CaM_test, Doest it be due to the small CPU cores or addition RAM ( this system has 32 gb) is needed ? OR may be some extra options are needed in the config? James 2013/11/6 Richard Broadbent > Hi Dwey, > > > On 05/11/13 22:00, Dwey Kauffman wrote: > >> Hi Szilard, >> >> Thanks for your suggestions. I am indeed aware of this page. In a >> 8-core >> AMD with 1GPU, I am very happy about its performance. See below. My >> intention is to obtain a even better one because we have multiple nodes. >> >> ### 8 core AMD with 1 GPU, >> Force evaluation time GPU/CPU: 4.006 ms/2.578 ms = 1.554 >> For optimal performance this ratio should be close to 1! >> >> >> NOTE: The GPU has >20% more load than the CPU. This imbalance causes >>performance loss, consider using a shorter cut-off and a finer PME >> grid. >> >> Core t (s) Wall t (s)(%) >> Time: 216205.51027036.812 799.7 >> 7h30:36 >> (ns/day)(hour/ns) >> Performance: 31.9560.751 >> >> ### 8 core AMD with 2 GPUs >> >> Core t (s) Wall t (s)(%) >> Time: 178961.45022398.880 799.0 >> 6h13:18 >> (ns/day)(hour/ns) >> Performance: 38.5730.622 >> Finished mdrun on node 0 Sat Jul 13 09:24:39 2013 >> >> > I'm almost certain that Szilard meant the lines above this that give the > breakdown of where the time is spent in the simulation. > > Richard > > >> However, in your case I suspect that the >>> bottleneck is multi-threaded scaling on the AMD CPUs and you should >>> probably decrease the number of threads per MPI rank and share GPUs >>> between 2-4 ranks. >>> >> >> >> OK but can you give a example of mdrun command ? given a 8 core AMD with 2 >> GPUs. >> I will try to run it again. >> >> >> Regarding scaling across nodes, you can't expect much from gigabit >>> ethernet - especially not from the cheaper cards/switches, in my >>> experience even reaction field runs don't scale across nodes with 10G >>> ethernet if you have more than 4-6 ranks per node trying to >>> communicate (let alone with PME). However, on infiniband clusters we >>> have seen scaling to 100 atoms/core (at peak). >>> >> >> From your comments, it sounds like a cluster of AMD cpus is difficult to >>> >> scale across nodes in our current setup. >> >> Let's assume we install Infiniband (20 or 40GB/s) in the same system of 16 >> nodes of 8 core AMD with 1 GPU only. Considering the same AMD system, what >> is a good way to obtain better performance when we run a task across >> nodes >> ? in other words, what dose mudrun_mpi look like ? >> >> Thanks, >> Dwey >> >> >> >> >> -- >> View this message in context: http://gromacs.5086.x6.nabble. >> com/Gromacs-4-6-on-two-Titans-GPUs-tp5012186p5012279.html >> Sent from the GROMACS Users Forum mailing list archive at Nabble.com. >> >> -- > gmx-users mailing [email protected] > http://lists.gromacs.org/mailman/listinfo/gmx-users > * Please search the archive at http://www.gromacs.org/ > Support/Mailing_Lists/Search before posting! > * Please don't post (un)subscribe requests to the list. Use the www > interface or send it to [email protected]. > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists > -- gmx-users mailing [email protected] http://lists.gromacs.org/mailman/listinfo/gmx-users * Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/Search before posting! * Please don't post (un)subscribe requests to the list. Use the www interface or send it to [email protected]. * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
Re: [gmx-users] Re: Gromacs-4.6 on two Titans GPUs
Hi Dwey, On 05/11/13 22:00, Dwey Kauffman wrote: Hi Szilard, Thanks for your suggestions. I am indeed aware of this page. In a 8-core AMD with 1GPU, I am very happy about its performance. See below. My intention is to obtain a even better one because we have multiple nodes. ### 8 core AMD with 1 GPU, Force evaluation time GPU/CPU: 4.006 ms/2.578 ms = 1.554 For optimal performance this ratio should be close to 1! NOTE: The GPU has >20% more load than the CPU. This imbalance causes performance loss, consider using a shorter cut-off and a finer PME grid. Core t (s) Wall t (s)(%) Time: 216205.51027036.812 799.7 7h30:36 (ns/day)(hour/ns) Performance: 31.9560.751 ### 8 core AMD with 2 GPUs Core t (s) Wall t (s)(%) Time: 178961.45022398.880 799.0 6h13:18 (ns/day)(hour/ns) Performance: 38.5730.622 Finished mdrun on node 0 Sat Jul 13 09:24:39 2013 I'm almost certain that Szilard meant the lines above this that give the breakdown of where the time is spent in the simulation. Richard However, in your case I suspect that the bottleneck is multi-threaded scaling on the AMD CPUs and you should probably decrease the number of threads per MPI rank and share GPUs between 2-4 ranks. OK but can you give a example of mdrun command ? given a 8 core AMD with 2 GPUs. I will try to run it again. Regarding scaling across nodes, you can't expect much from gigabit ethernet - especially not from the cheaper cards/switches, in my experience even reaction field runs don't scale across nodes with 10G ethernet if you have more than 4-6 ranks per node trying to communicate (let alone with PME). However, on infiniband clusters we have seen scaling to 100 atoms/core (at peak). From your comments, it sounds like a cluster of AMD cpus is difficult to scale across nodes in our current setup. Let's assume we install Infiniband (20 or 40GB/s) in the same system of 16 nodes of 8 core AMD with 1 GPU only. Considering the same AMD system, what is a good way to obtain better performance when we run a task across nodes ? in other words, what dose mudrun_mpi look like ? Thanks, Dwey -- View this message in context: http://gromacs.5086.x6.nabble.com/Gromacs-4-6-on-two-Titans-GPUs-tp5012186p5012279.html Sent from the GROMACS Users Forum mailing list archive at Nabble.com. -- gmx-users mailing [email protected] http://lists.gromacs.org/mailman/listinfo/gmx-users * Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/Search before posting! * Please don't post (un)subscribe requests to the list. Use the www interface or send it to [email protected]. * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
Re: [gmx-users] Re: Gromacs-4.6 on two Titans GPUs
Hi Dwey, First and foremost, make sure to read the http://www.gromacs.org/Documentation/Acceleration_and_parallelization page, in particular the "Multiple MPI ranks per GPU" section which applies in your case. Secondly, please do post log files (pastebin is your friend), the performance table at the end of the log tells much the performance story and based on that I/we can make suggestions. Using multiple GPU requires domain-decomposition which does have a considerable overhead, especially comparing no DD with DD (i.e. 1 GPU run with 2 GPU run). However, in your case I suspect that the bottleneck is multi-threaded scaling on the AMD CPUs and you should probably decrease the number of threads per MPI rank and share GPUs between 2-4 ranks. Regarding scaling across nodes, you can't expect much from gigabit ethernet - especially not from the cheaper cards/switches, in my experience even reaction field runs don't scale across nodes with 10G ethernet if you have more than 4-6 ranks per node trying to communicate (let alone with PME). However, on infiniband clusters we have seen scaling to 100 atoms/core (at peak). Cheers, -- Szilárd On Tue, Nov 5, 2013 at 9:29 PM, Dwey wrote: > Hi Mike, > > > I have similar configurations except a cluster of AMD-based linux > platforms with 2 GPU cards. > > Your suggestion works. However, the performance of 2 GPU discourages > me because , for example, with 1 GPU, our computer node can easily > obtain a simulation of 31ns/day for a protein of 300 amino acids but > with 2 GPUs, it goes as far as 38 ns/day. I am very curious as to why > the performance of 2 GPUs is under expectation. Is there any overhead > that we should pay attention to ? Note that these 2GPU cards are > linked by a SLI bridge within the same node. > > Since the computer nodes of our cluster have at least one GPU but > they are connected by slow network cards ( 1GB/sec), unfortunately, I > reasonably doubt that the performance will not be proportional to the > total number of GPU cards. I am wondering if you have any suggestion > about a cluster of GPU nodes. For example, will a infiniband > networking help increase a final performance when we execute a mpi > task ? or what else ? or forget about mpi and use single GPU instead. > > Any suggestion is highly appreciated. > Thanks. > > Dwey > >> Date: Tue, 5 Nov 2013 16:20:39 +0100 >> From: Mark Abraham >> Subject: Re: [gmx-users] Gromacs-4.6 on two Titans GPUs >> To: Discussion list for GROMACS users >> Message-ID: >> >> Content-Type: text/plain; charset=ISO-8859-1 >> >> On Tue, Nov 5, 2013 at 12:55 PM, James Starlight >> wrote: >> >>> Dear Richard, >>> >>> >>> 1) mdrun -ntmpi 1 -ntomp 12 -gpu_id 0 -v -deffnm md_CaM_test >>> gave me performance about 25ns/day for the explicit solved system consisted >>> of 68k atoms (charmm ff. 1.0 cutoofs) >>> >>> gaves slightly worse performation in comparison to the 1) >>> >>> >> Richard suggested >> >> mdrun -ntmpi 2 -ntomp 6 -gpu_id 01 -v -deffnm md_CaM_test, >> >> which looks correct to me. -ntomp 6 is probably superfluous >> >> Mark >> >> >>> finally >>> >>> 3) mdrun -deffnm md_CaM_test >>> running in the same regime as in the 2) so its also gave me 22ns/day for >>> the same system. >>> >>> How the efficacy of using of dual-GPUs could be increased? >>> >>> James >>> >>> >>> 2013/11/5 Richard Broadbent >>> >>> > Dear James, >>> > >>> > >>> > On 05/11/13 11:16, James Starlight wrote: >>> > >>> >> My suggestions: >>> >> >>> >> 1) During compilstion using -march=corei7-avx-i I have obtained error >>> that >>> >> somethng now found ( sorry I didnt save log) so I compile gromacs >>> without >>> >> this flag >>> >> >>> >> 2) I have twice as better performance using just 1 gpu by means of >>> >> >>> >> mdrun -ntmpi 1 -ntomp 12 -gpu_id 0 -v -deffnm md_CaM_test >>> >> >>> >> than using of both gpus >>> >> >>> >> mdrun -ntmpi 2 -ntomp 12 -gpu_id 01 -v -deffnm md_CaM_test >>> >> >>> >> in the last case I have obtained warning >>> >> >>> >> WARNING: Oversubscribing the available 12 logical CPU cores with 24 >>> >> threads. >>> >> This will cause considerable performance loss! >>> >> >>> >> here you are requesting 2 thread mpi processes each with 12 openmp >>> > threads, hence a total of 24 threads however even with hyper threading >>> > enabled there are only 12 threads on your machine. Therefore, only >>> allocate >>> > 12. Try >>> > >>> > mdrun -ntmpi 2 -ntomp 6 -gpu_id 01 -v -deffnm md_CaM_test >>> > >>> > or even >>> > >>> > mdrun -v -deffnm md_CaM_test >>> > >>> > I believe it should autodetect the GPUs and run accordingly for details >>> of >>> > how to use gromacs with mpi/thread mpi openmp and GPUs see >>> > >>> > http://www.gromacs.org/Documentation/Acceleration_and_parallelization >>> > >>> > Which describes how to use these systems >>> > >>> > Richard >>> > >>> > >>> > How it could be fixed? >>> >> All gpu are recognized correctly >>> >> >>> >> >>> >> 2 GPUs detected:
