Did you renumber your on-rank nodes after partitioning? T
On 06/28/2012 08:59 AM, Thomas DE-SOZA wrote: > > Dear PETSc Users, > > We're experiencing performances issues after having switched to fully > distributed meshes in our in-house code and would like your opinion on > the matter. > > In the current version of our structural mechanics FEA software > (http://www.code-aster.org/), all MPI processes have knowledge of the > whole matrix and therefore can easily pass it to PETSc without the > need for any communication. In a nutshell, stash is empty after > MatSetValues and no mallocs occur during assembly. > We're now building a distributed version of the software with each > process reading its own subdomain in order to save memory. The mesh > was partitioned with Metis and as a first approach we built a simple > partition of the degrees of freedom based on the gradual subdomains. > This eliminates the need for Application Ordering but yields an > unbalanced decomposition in terms of rows. If we take an example with > 2 MPI processes : processor 0 will have more unknowns than processor 1 > and will receive entries lying on the interface whereas processor 1 > will have all entries locally. > > PETSc manual states that "It is fine to generate some entries on the > "wrong" process. Often this can lead to cleaner, simpler, less buggy > codes. One should never make code overly complicated in order to > generate all values locally. Rather, one should organize the code in > such a way that most values are generated locally." > Judging from the performance we obtain on a simple cube with two > processes, it seems we have generated too much entries on the wrong > process. Indeed our distributed code runs slower than the current one. > However the stash does not seem to contain that much (650 000 over a > total of 50 000 000 nnz). We have attached the output obtained with > "-info" as well as the "-log_summary" profiling. Most of the time is > spent in the assembly and a lot of mallocs occur. > > > > > What's your advice on this ? Is working with ghost cells the only > option ? > We were wondering if we could preallocate the stash for example to > decrease the number of mallocs. > > Regards, > > Thomas > > > Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') sont > ?tablis ? l'intention exclusive des destinataires et les informations > qui y figurent sont strictement confidentielles. Toute utilisation de > ce Message non conforme ? sa destination, toute diffusion ou toute > publication totale ou partielle, est interdite sauf autorisation expresse. > > Si vous n'?tes pas le destinataire de ce Message, il vous est interdit > de le copier, de le faire suivre, de le divulguer ou d'en utiliser > tout ou partie. Si vous avez re?u ce Message par erreur, merci de le > supprimer de votre syst?me, ainsi que toutes ses copies, et de n'en > garder aucune trace sur quelque support que ce soit. Nous vous > remercions ?galement d'en avertir imm?diatement l'exp?diteur par > retour du message. > > Il est impossible de garantir que les communications par messagerie > ?lectronique arrivent en temps utile, sont s?curis?es ou d?nu?es de > toute erreur ou virus. > ____________________________________________________ > > This message and any attachments (the 'Message') are intended solely > for the addressees. The information contained in this Message is > confidential. Any use of information contained in this Message not in > accord with its purpose, any dissemination or disclosure, either whole > or partial, is prohibited except formal approval. > > If you are not the addressee, you may not copy, forward, disclose or > use any part of it. If you have received this message in error, please > delete it and all copies from your system and notify the sender > immediately by return message. > > E-mail communication cannot be guaranteed to be timely secure, error > or virus-free. > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120628/8ba01ccd/attachment.html>
