So, I have tested both PaStiX and MUMPS solvers. Tests were run on 4 inifinibanded nodes, each equipped with two 12 core AMD Opteron and 64 GB RAM. Intel Compiler 11.1 + MKL + OpenMPI was the tool-chain.
The problem is 3D Helmholtz equation, 1.4 Mio of unknowns. The matrix is symmetric thus I used LDL^T for both. First of all, both PaStiX and MUMPS gave correct solution with the relative residual < 1e-12, although the test case was not numerically difficult. Below are tables, showing time for analysis+factorization (seconds) and overall memory usage (megabytes). PASTIX: N_cpus T_fac memory 1 9.27E+03 27900 4 5.28E+03 33200 16 1.44E+03 77700 32 755 131377 64 471 225399 MUMPS: N_cpus T_fac memory 1 8009 49689 4 2821 63501 16 1375 84115 32 1081 86583 64 733 98235 According to this test, PaStiX is slightly faster when run on more cores, but also consumes much more memory. Which is opposite to what Garth said. Either I did something wrong or our matrices are very different. PS Can anyone explain why direct solvers require more memory when run in parallel? On 10.11.2012 14:14, Alexander Grayver wrote: > Garth, > > At the time I was tested PaStiX it failed for my problem: > https://lists.mcs.anl.gov/mailman/htdig/petsc-dev/2011-December/006887.html > > Since then PaStiX has been updated with several critical bug fixes, so > I should consider testing new version. > > The memory scalability of the MUMPS is not nice, that is true. > Running MUMPS with default parameters on large amount of cores is > often not optimal. I don't how much you spent tweaking parameters. > MUMPS is among the most robust distributed solvers nowadays and it is > still being developed and hopefully will improve. > > /*To petsc developers:* /are there plans to update PaStiX supplied > with PETSc? The current version is 5.2 from 2012-06-08 and > PETSc-3.3-p3 uses 5.1.8 from 2011-02-23. > > Here is changelog: > https://gforge.inria.fr/frs/shownotes.php?group_id=186&release_id=7096 > -- Regards, Alexander -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20121112/ea87ac12/attachment.html>
