Dmitry,
finally, I am able to make a simple program and reproduce the error. Since I
had moved to
petsc-dev (before I used petsc-3.2), the error is a little different.
Download the program attached and run the run.sh. You will see the output from
PCASM.
There is a submatrix (submat[1] on rank 0) which has a row like
row 1: (0, -1) (1, 0 + 0.4 i) (4, -1) (2, -1)
Note that the column 2 is placed in the last and the column numbers are not in
the normal order.
Following that, we can also see the error message like
[0]PETSC ERROR: New nonzero at (1,2) caused a malloc!
You can also try with GASM, use
mpirun -np 2 ./gasm_test -dd_type gasm -n 4 -px 1 -py 2 -sx 2 -sy 1
which partitions the domain to 1 * 2 and mapped to the two processors, and for
each processor
we further partition its local domain to 2 * 1 subdomains.
Then, you will see that the row 1 of submat[1] on rank 1 becomes
row 1: (0, -1) (1, 0 + 0.4 i) (3, -1) (6, -1)
which is totally wrong because from the IS for this overlapping subdomain and
the big matrix A,
row 1 should have non-zeros like the output from PCASM.
I guess the problem is due to that we set PCASMSetSortIndices to FALSE and
something goes wrong
in asm.c and gasm.c.
Thanks!
Hui
> On Sun, May 20, 2012 at 5:06 AM, Hui Zhang <mike.hui.zhang at hotmail.com>
> wrote:
> I have the following MatView output:
>
> row 0: (0, -0.0351719 - 0.638814 i) (1, -0.517586 - 0.617823 i) (4, -0.517586
> - 0.617823 i) (1, -1.03517) (5, -0.508793)
>
> Note that there are two values for the column 1, is this normal?
>
> The above matrix is obtained from PCASMGetSubKSP(pc,subksp) and
> KSPGetOperators(subksp[0], ...).
>
> This is odd. Can you explain a bit more what leads up to this, so we can try
> and reproduce the problem?
> Thanks.
> Dmitry.
>
> Thanks!
>
>
>
> On May 11, 2012, at 7:17 PM, Dmitry Karpeev wrote:
>
>> You can call PCSetUp(pc) on either ASM or GASM, and that will destroy and
>> recreate the matrices (including calling
>> your modification subroutine), but not the subdomains or the subdomain
>> solvers.
>> If you just want to modify the submatrices, you can call
>> PC(G)ASMGetSubmatrices() and modify the matrices it returns
>> (in the same order as the subdomains were set). That's a bit of a hack,
>> since you will essentially be modifying the PC's internal data structures.
>> As long as you are careful, you should be okay, since you already
>> effectively have the same type of access to the submatrices through the
>> Modify callback.
>>
>> Dmitry.
>>
>> On Fri, May 11, 2012 at 11:52 AM, Hui Zhang <mike.hui.zhang at hotmail.com>
>> wrote:
>> I just have a question about reuse of PCASM or PCGASM.
>> Suppose I have seted up the PCASM and related KSP and I solved one time.
>> Next for the same linear system (matrix and RHS), I just want PCASM modify
>> the submatrices (PCSetModifySubmatrices) in a different way, using the same
>> routine for modifying but with
>> different user context for the modifying routine.
>>
>> What can I do for this task? Currently, I destroy the KSP and re-construct
>> it. I guess
>> even for PCASM I can re-use it because the partition of subdomains remain
>> the same.
>>
>> Thanks!
>>
>>
>> On May 10, 2012, at 6:37 PM, Dmitry Karpeev wrote:
>>
>>> Hui,
>>> There've been several changes to PCGASM ahead of the new release.
>>> Let me go back and see if it affected the convergence problem.
>>> Dmitry.
>>>
>>> On Thu, May 10, 2012 at 4:16 AM, Hui Zhang <mike.hui.zhang at hotmail.com>
>>> wrote:
>>> Hi Dmitry,
>>>
>>> is there any news about PCGASM?
>>>
>>> thanks,
>>> Hui
>>>
>>> On Feb 20, 2012, at 6:38 PM, Dmitry Karpeev wrote:
>>>
>>>> Okay, thanks.
>>>> I'll take a look.
>>>>
>>>> Dmitry.
>>>>
>>>> On Mon, Feb 20, 2012 at 11:30 AM, Hui Zhang <mike.hui.zhang at
>>>> hotmail.com> wrote:
>>>> For reference, my results are attached.
>>>>
>>>> asm1.txt for asm with 1 process,
>>>> asm2.txt for asm with 2 processes,
>>>> gasm1.txt for gasm with 1 process, (with the iteration numbers different
>>>> from others)
>>>> gasm2.txt for gasm with 2 processes
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> thank you,
>>>> Hui
>>>>
>>>> On Feb 20, 2012, at 3:06 PM, Dmitry Karpeev wrote:
>>>>
>>>>>
>>>>>
>>>>> On Mon, Feb 20, 2012 at 12:59 AM, Hui Zhang <mike.hui.zhang at
>>>>> hotmail.com> wrote:
>>>>>
>>>>> On Feb 20, 2012, at 12:41 AM, Dmitry Karpeev wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Sun, Feb 19, 2012 at 3:08 PM, Hui Zhang <mike.hui.zhang at
>>>>>> hotmail.com> wrote:
>>>>>> I have a new problem: the results from ASM and GASM are different and it
>>>>>> seems
>>>>>> GASM has something wrong with SetModifySubMatrices. Numerical tests are
>>>>>> with
>>>>>> each subdomain supported only by one subdomain. There are no problems
>>>>>> when
>>>>>> I did not modify submatrices. But when I modify submatrices, there are
>>>>>> problems
>>>>>> with GASM but no problems with ASM.
>>>>>>
>>>>>> For example, I use two subdomains. In the first case each subdomain is
>>>>>> supported by
>>>>>> one processor and there seems no problem with GASM. But when I use run
>>>>>> my program
>>>>>> with only one proc. so that it supports both of the two subdomains, the
>>>>>> iteration
>>>>>> number is different from the first case and is much larger. On the
>>>>>> other hand
>>>>>> ASM has no such problem.
>>>>>>
>>>>>> Are the solutions the same?
>>>>>> What problem are you solving?
>>>>>
>>>>> Yes, the solutions are the same. That's why ASM gives the same results
>>>>> with one or
>>>>> two processors. But GASM did not.
>>>>> Sorry, I wasn't clear: ASM and GASM produced different solutions in the
>>>>> case of two domains per processor?
>>>>> I'm solving the Helmholtz equation. Maybe
>>>>> I can prepare a simpler example to show this difference.
>>>>> That would be helpful.
>>>>> Thanks.
>>>>>
>>>>> Dmitry.
>>>>>
>>>>>>
>>>>>> Dmitry.
>>>>>>
>>>>>>
>>>>>> On Feb 15, 2012, at 6:46 PM, Dmitry Karpeev wrote:
>>>>>>
>>>>>>> You should be able to.
>>>>>>> This behavior is the same as in PCASM,
>>>>>>> except in GASM the matrices live on subcommunicators.
>>>>>>> I am in transit right now, but I can take a closer look in Friday.
>>>>>>>
>>>>>>> Dmitry
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Feb 15, 2012, at 8:07, Hui Zhang <mike.hui.zhang at hotmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
>>>>>>>>
>>>>>>>>> Hi Dmitry,
>>>>>>>>>
>>>>>>>>> thanks a lot! Currently, I'm not using ISColoring. Just comes another
>>>>>>>>> question
>>>>>>>>> on PCGASMSetModifySubMatrices(). The user provided function has the
>>>>>>>>> prototype
>>>>>>>>>
>>>>>>>>> func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
>>>>>>>>>
>>>>>>>>> I think the coloumns from the parameter 'col' are always the same as
>>>>>>>>> the rows
>>>>>>>>> from the parameter 'row'. Because PCGASMSetLocalSubdomains() only
>>>>>>>>> accepts
>>>>>>>>> index sets but not rows and columns. Has I misunderstood something?
>>>>>>>>
>>>>>>>> As I tested, the row and col are always the same.
>>>>>>>>
>>>>>>>> I have a new question. Am I allowed to SetLocalToGlobalMapping() for
>>>>>>>> the submat's
>>>>>>>> in the above func()?
>>>>>>>>
>>>>>>>> thanks,
>>>>>>>> Hui
>>>>>>>>
>>>>>>>>>
>>>>>>>>> thanks,
>>>>>>>>> Hui
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
>>>>>>>>>
>>>>>>>>>> Yes, that's right.
>>>>>>>>>> There is no good way to help the user assemble the subdomains at the
>>>>>>>>>> moment beyond the 2D stuff.
>>>>>>>>>> It is expected that they are generated from mesh subdomains.
>>>>>>>>>> Each IS does carry the subdomains subcomm.
>>>>>>>>>>
>>>>>>>>>> There is ISColoringToList() that is supposed to convert a "coloring"
>>>>>>>>>> of indices to an array of ISs,
>>>>>>>>>> each having the indices with the same color and the subcomm that
>>>>>>>>>> supports that color. It is
>>>>>>>>>> largely untested, though. You could try using it and give us
>>>>>>>>>> feedback on any problems you encounter.
>>>>>>>>>>
>>>>>>>>>> Dmitry.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <mike.hui.zhang at
>>>>>>>>>> hotmail.com> wrote:
>>>>>>>>>> About PCGASMSetLocalSubdomains(), in the case of one subdomain
>>>>>>>>>> supported by
>>>>>>>>>> multiple processors, shall I always create the arguments 'is[s]' and
>>>>>>>>>> 'is_local[s]'
>>>>>>>>>> in a subcommunicator consisting of processors supporting the
>>>>>>>>>> subdomain 's'?
>>>>>>>>>>
>>>>>>>>>> The source code of PCGASMCreateSubdomains2D() seemingly does so.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Hui
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120521/ea19bf1d/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: GASM_test.zip
Type: application/zip
Size: 7088 bytes
Desc: not available
URL:
<http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120521/ea19bf1d/attachment.zip>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120521/ea19bf1d/attachment-0001.html>
