Hello all, specially Dr. Matt, On 4/16/08, Matthew Knepley <knepley at gmail.com> wrote: > > On Tue, Apr 15, 2008 at 7:19 PM, Randall Mackie <rlmackie862 at gmail.com> > wrote: > > I'm running my PETSc code on a cluster of quad core Xeon's connected > > by Infiniband. I hadn't much worried about the performance, because > > everything seemed to be working quite well, but today I was actually > > comparing performance (wall clock time) for the same problem, but on > > different combinations of CPUS. > > > > I find that my PETSc code is quite scalable until I start to use > > multiple cores/cpu. > > > > For example, the run time doesn't improve by going from 1 core/cpu > > to 4 cores/cpu, and I find this to be very strange, especially since > > looking at top or Ganglia, all 4 cpus on each node are running at 100% > > almost > > all of the time. I would have thought if the cpus were going all out, > > that I would still be getting much more scalable results. > > Those a really coarse measures. There is absolutely no way that all cores > are going 100%. Its easy to show by hand. Take the peak flop rate and > this gives you the bandwidth needed to sustain that computation (if > everything is perfect, like axpy). You will find that the chip bandwidth > is far below this. A nice analysis is in > > http://www.mcs.anl.gov/~kaushik/Papers/pcfd99_gkks.pdf > > > We are using mvapich-0.9.9 with infiniband. So, I don't know if > > this is a cluster/Xeon issue, or something else. > > This is actually mathematics! How satisfying. The only way to improve > this is to change the data structure (e.g. use blocks) or change the > algorithm (e.g. use spectral elements and unassembled structures)
Would you please explain a bit about "unassembled structures"? Does Discontinuous Galerkin Method falls into this category? Thanks and Regrads, Amjad Ali. Matt > > > Anybody with experience on this? > > > > Thanks, Randy M. > > > > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener > > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20080523/eb482d9c/attachment.htm>
