Hi Jed, This is useful to know. My matrices are all very sparse but just may not be ordered optimally (there's a problem-specific reason why I order them in a certain way). That said, this is the first time in many years of similar computations with similar matrices that I've encountered this problem. It may just be peculiar to the XC30's.
Thanks, Samar On May 9, 2014, at 8:30 AM, Jed Brown <[email protected]> wrote: > Samar Khatiwala <[email protected]> writes: >> CRAY has finally gotten back with a solution, >> which is to use the run-time option -vecscatter_alltoall. Apparently >> this is a known issue and according to the HLRN folks passing this >> command line option to PETSc seems to work nicely. > > This option is good when you have nearly-dense rows or columns (in terms > of processors depended on). For problems with actual dense rows or > columns, it is good to formulate as a sparse matrix plus a low-rank > correction. The other cases are usually poor dof layout, and reordering > will make the graph sparser. Sparse problems with good layout usually > run faster with the default VecScatter, though there are exceptions > (mostly non-PDE problems).
