On Thu, 14 Jan 2010 13:40:35 -0600 (CST), Roy Stogner <[email protected]> wrote: > Besides, Ben's code (with an expensive assembly combined with > experimenting with a matrix-free solve) isn't typical. Jed was right > that, for most users, threading the assembly isn't going to help much > when you've got an un-threaded solve going on in the background.
Even just insertion won't scale great, locking would be tricky since PETSc doesn't require you to preallocate correctly. We could make MatSetValues thread-safe when allocation is correct, but this has not been done, and taking the locks would have nontrivial cost for the people who typically insert very few values per row. > Although, doesn't PETSc end up doing most of it's work in BLAS? No, some time is spent in BLAS1 (which offers limited benefit from threading (without extra sockets) because it's so overwhelmingly bandwidth limited), but most is spent in sparse matrix kernels. Unless you are using MatDense, BLAS and Lapack have very little impact on peformance. Sparse matrix-vector products could be made to use OpenMP fairly easily, but the benefit is pretty dubious since separate memory spaces often perform better anyway (SpMV can be done somewhat faster with shared memory threading, but not very portably because the algorithm has to be aware of the architecture and practice cooperative threading (i.e. each thread knows which pages the neighbors have in cache) to see much benefit). And the most important preconditioning kernels (triangular solves) are fundamentally serial, so threading would necessarily change the algorithm (adding another level to the domain decomposition). Now, if you have an algorithm that can be put on a GPU, (e.g. you can precondition with FMM, and you do matrix-vector products unassembled), then changing the programming model from plain MPI looks worthwhile (PETSc can support this too, the kernels just move to the GPU). Jed ------------------------------------------------------------------------------ Throughout its 18-year history, RSA Conference consistently attracts the world's best and brightest in the field, creating opportunities for Conference attendees to learn about information security's most important issues through interactions with peers, luminaries and emerging and established companies. http://p.sf.net/sfu/rsaconf-dev2dev _______________________________________________ Libmesh-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/libmesh-users
