Thanks Matt. When I use -start_in_debugger, it just executes the program without any extra message whatsoever! so its like I did not use the flag at all.
Also, thought to share my conclusions on using ParMetis with others. The two important things are: 1- Make sure you have a symmetric adjacency matrix for partitioning. Chaning my non-symmetric matrix into a symmetric one solved the "Key 10716 not found!" problem. 2- Unlike PETSc, ParMetis does not equal number of nodes/elements to each processors. The numbers are close, but not the same as you start partitioning the grid with. It, however, does produce contigious distribution of nodes/elements. You can use ISPartitioningCount to figure out how many nodes/elements are assigned to each processor. Mohammad On Wed, Apr 11, 2012 at 5:43 PM, Matthew Knepley <knepley at gmail.com> wrote: > On Wed, Apr 11, 2012 at 10:49 PM, Mohammad Mirzadeh <mirzadeh at > gmail.com>wrote: > >> I see. Well the reason for non-symmetry is that I'm using a finite >> difference discretization on quadtree AMR grids. When discretizing at a >> hanging node, i.e. a node that is missing a direct neighbor, I use >> interpolation from nodes of the parent cell. When it comes to discretizing >> at those nodes (i.e. nodes of the parent), however, they are on a coarser >> grid and will have direct neighbors so they would not see the hanging node >> in their discretization. This leads to a non-symmetric discretization >> matrix which I was using (structurally) to define the adjacency matrix of >> the graph. I think I can fix this by requiring the parent nodes to also >> have a link to the hanging node (although they are not used in the >> discretization anyway) >> >> I'm just surprised that ParMetis does not complain about non-symmetric >> graphs at lower resolutions and partitions them just fine! >> > > ParMetis does not burden the user with a whole lot of diagnostic output :) > > >> As for the debugger, -start_in_debugger does not start the debugger for >> me :(. I'm using mpirun -np 2 ./a.out -start_in_debugger. Is this the >> correct way? >> > > Send the entire output. It is likely that at the beginning there are lines > showing a failure to open an X window. > > Matt > > >> >> On Wed, Apr 11, 2012 at 3:07 PM, Matthew Knepley <knepley at gmail.com>wrote: >> >>> On Wed, Apr 11, 2012 at 8:24 PM, Mohammad Mirzadeh <mirzadeh at >>> gmail.com>wrote: >>> >>>> I was reading the FAQ list when I came across the following: >>>> >>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#key >>>> >>>> When calling MatPartitioningApply() you get a message Error! Key 16615 >>>> not found The graph of the matrix you are using is not symmetric. You >>>> must use symmetric matrices for partitioning. >>>> >>>> Is this a limitation on ParMetis side? I set up the adjacency matrix >>>> based on the discretization that I will be performing on the grid which is >>>> non-symmetric; both numerically and structurally. What's the solution here? >>>> Make an "approximate" adjacency matrix that sort of looks like >>>> (structurally) my discretization but is symmetric? What I don't understand >>>> is my matrix IS non-symmetric when the code runs on coarser grids! >>>> >>> >>> I don't quite understand how you can have a non-symmetric adjacency >>> description. But Metis/ParMetis partitions undirected graphs, which >>> by definition have symmetric adjacency matrices. Non-symmetric adjacency >>> would seem to imply a directed graph of some sort, or more plainly, >>> something is a adjacent to another thing which is not adjacent to it. >>> That is a very strange concept. >>> >>> >>>> Also, I was reading the FAQ hoping I can find something regarding using >>>> gdb in parallel. I found this: >>>> http://scicomp.stackexchange.com/a/410/485 but I'm not sure how I >>>> should be using gdb in parallel. Could you (maybe Matt?) please explain a >>>> little bit? >>>> >>> >>> -start_in_debugger spawns a gdb windows for EVERY process and attaches it >>> >>> -debbuger_nodes a,b,c spawns gdb windows ONLY for ranks a, b, and c >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Thanks >>>> >>>> On Tue, Apr 10, 2012 at 12:08 AM, Mohammad Mirzadeh <mirzadeh at gmail.com >>>> > wrote: >>>> >>>>> Just built petsc-dev and it did not help. I'm going to look into the >>>>> code to see if my graph is "ill-formed" in some sense. Just hope the >>>>> problem is from my side not a real bug in ParMetis! >>>>> >>>>> >>>>> >>>>> On Mon, Apr 9, 2012 at 8:46 PM, Mohammad Mirzadeh <mirzadeh at >>>>> gmail.com>wrote: >>>>> >>>>>> ok. Thanks Jed. I'll try petsc-dev to see if it fixes the problem. >>>>>> >>>>>> Thanks everyone. >>>>>> >>>>>> >>>>>> On Mon, Apr 9, 2012 at 8:42 PM, Jed Brown <jedbrown at mcs.anl.gov>wrote: >>>>>> >>>>>>> On Mon, Apr 9, 2012 at 22:37, Mohammad Mirzadeh <mirzadeh at >>>>>>> gmail.com>wrote: >>>>>>> >>>>>>>> Thanks Sean. I'm using Petsc 3.2-p6 along with ParMetis 4.0.2. >>>>>>>> Since this was not supported with 3.2-p6, and previous versions had >>>>>>>> bugs, I >>>>>>>> built parmetis myself and used --with-parmetis-include and >>>>>>>> --with-parmetis-lib flags to build petsc. >>>>>>>> >>>>>>>> Should I switch to petsc-dev? >>>>>>>> >>>>>>> >>>>>>> Yes, and use --download-metis --download-parmetis because the >>>>>>> version upstream has some bugs for which the patches have not been >>>>>>> applied. >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120412/c388e7a0/attachment.htm>
