subject:"\[DuMux\] Singular matrix and MPI"

Re: [DuMux] Singular matrix and MPI

2022-06-30 Thread Dmitry Pavlov


Etienne,

Thank you. I just did what Timo advised (to follow your approach).

It might be out of Dumux hands. If Dune doesn't handle recovering from 
the exception and deadlocks before we get back to Dumux, we cannot 
really do anything. Here is a paper about this general problem with 
MPI and C++ exceptions from the Dune community 
(https://arxiv.org/abs/1804.04481).


In this case, the only thing that helps (without implementing some 
error propagation framework) is Etienne's suggestion of manually 
removing exception throwing where you consider it acceptable failures.




Best regards,

Dmitry



On 6/30/22 10:22, Etienne Ahusborde wrote:

Hi Dmitry ,

We already had the same problem.

We understood that it is due to some "DUNE_THROW" commands.

Our solution (probably not the best) was to remove several 
"DUNE_THROW" commands and to replace them with a simple std::cout to 
keep the information that something is wrong.


Regards

Etienne

Le 29/06/2022 à 19:36, Dmitry Pavlov a écrit :


Hello,

In one porous flow simulation, I get this message:

Newton iteration  6 done, residual = 3.1654e+13
Solve: M deltax^k = rNewton: Caught exception: "FMatrixError 
[luDecomposition:/home/dpavlov/DUMUX/dune-common/dune/common/densematrix.hh:939]: 
matrix is singular"


... and it is fine. Matrix can be singular, no objection here. After 
this, the Newton solver reports that it did not converge and restarts 
with a smaller time step, and eventually it proceeds.


However, when I run the same simulation with MPI, I get this

Newton iteration  8 done, residual = 7.2437e+13
Solve: M deltax^k =

and the program hangs.

So it seems that it misses the singularity in the MPI mode and tries 
to proceed with the current time step, to no avail.


How this can be cured?

Best regards,

Dmitry


___
DuMux mailing list
DuMux@listserv.uni-stuttgart.de
https://listserv.uni-stuttgart.de/mailman/listinfo/dumux

___
DuMux mailing list
DuMux@listserv.uni-stuttgart.de
https://listserv.uni-stuttgart.de/mailman/listinfo/dumux

___
DuMux mailing list
DuMux@listserv.uni-stuttgart.de
https://listserv.uni-stuttgart.de/mailman/listinfo/dumux

Re: [DuMux] Singular matrix and MPI

2022-06-30 Thread Dmitry Pavlov

Timo,

I just did, and wrote a reply there.

Regards,

Dmitry

On 6/30/22 14:17, Timo Koch wrote:

Dear Dmitry,

thank you! Can you try if this
https://git.iws.uni-stuttgart.de/dumux-repositories/dumux/-/merge_requests/3178

fixes your issue?

Best
Timo

On 30 Jun 2022, at 13:14, Dmitry Pavlov
wrote:

Timo,

Thank you. I filed a bug report:

https://git.iws.uni-stuttgart.de/dumux-repositories/dumux/-/issues/1171

Unfortunately, I can not prepare a proper test at present.

Best regards,

Dmitry

On 6/30/22 13:55, Timo Koch wrote:

Hi Dmitry,

looks like you are running into a deadlock. Looks like a bug in the
exception handler of the Newton
(I assume
https://git.iws.uni-stuttgart.de/dumux-repositories/dumux/-/blob/master/dumux/nonlinear/newtonsolver.hh#L548 is
wrong because the exception cannot be assumed to be thrown on all
processes).

Thanks for reporting this. Would you open a bug report?

Best wishes
Timo

On 29 Jun 2022, at 19:36, Dmitry Pavlov
wrote:

Hello,

In one porous flow simulation, I get this message:

Newton iteration 6 done, residual = 3.1654e+13
Solve: M deltax^k = rNewton: Caught exception: "FMatrixError
[luDecomposition:/home/dpavlov/DUMUX/dune-common/dune/common/densematrix.hh:939]:
matrix is singular"

... and it is fine. Matrix can be singular, no objection here.
After this, the Newton solver reports that it did not converge and
restarts with a smaller time step, and eventually it proceeds.

However, when I run the same simulation with MPI, I get this

Newton iteration 8 done, residual = 7.2437e+13
Solve: M deltax^k =

and the program hangs.

So it seems that it misses the singularity in the MPI mode and
tries to proceed with the current time step, to no avail.

How this can be cured?

Best regards,

Dmitry

___
DuMux mailing list
DuMux@listserv.uni-stuttgart.de
https://listserv.uni-stuttgart.de/mailman/listinfo/dumux

___
DuMux mailing list
DuMux@listserv.uni-stuttgart.de
https://listserv.uni-stuttgart.de/mailman/listinfo/dumux
___
DuMux mailing list
DuMux@listserv.uni-stuttgart.de
https://listserv.uni-stuttgart.de/mailman/listinfo/dumux

Re: [DuMux] Singular matrix and MPI

2022-06-30 Thread Timo Koch

Dear Dmitry,

thank you! Can you try if this
https://git.iws.uni-stuttgart.de/dumux-repositories/dumux/-/merge_requests/3178
fixes your issue?

Best
Timo

On 30 Jun 2022, at 13:14, Dmitry Pavlov
mailto:dmitry.pav...@outlook.com>> wrote:

Timo,

Thank you. I filed a bug report:

https://git.iws.uni-stuttgart.de/dumux-repositories/dumux/-/issues/1171

Unfortunately, I can not prepare a proper test at present.

Best regards,

Dmitry

On 6/30/22 13:55, Timo Koch wrote:
Hi Dmitry,

looks like you are running into a deadlock. Looks like a bug in the exception
handler of the Newton
(I assume
https://git.iws.uni-stuttgart.de/dumux-repositories/dumux/-/blob/master/dumux/nonlinear/newtonsolver.hh#L548
is wrong because the exception cannot be assumed to be thrown on all
processes).
Thanks for reporting this. Would you open a bug report?

Best wishes
Timo

On 29 Jun 2022, at 19:36, Dmitry Pavlov
mailto:dmitry.pav...@outlook.com>> wrote:

Hello,

In one porous flow simulation, I get this message:

... and it is fine. Matrix can be singular, no objection here. After this, the
Newton solver reports that it did not converge and restarts with a smaller time
step, and eventually it proceeds.

However, when I run the same simulation with MPI, I get this

Newton iteration 8 done, residual = 7.2437e+13
Solve: M deltax^k =

and the program hangs.

So it seems that it misses the singularity in the MPI mode and tries to proceed
with the current time step, to no avail.

How this can be cured?

Best regards,

Dmitry

___
DuMux mailing list
DuMux@listserv.uni-stuttgart.de
https://listserv.uni-stuttgart.de/mailman/listinfo/dumux

Re: [DuMux] Singular matrix and MPI

2022-06-30 Thread Dmitry Pavlov

Timo,

Thank you. I filed a bug report:

https://git.iws.uni-stuttgart.de/dumux-repositories/dumux/-/issues/1171

Unfortunately, I can not prepare a proper test at present.

Best regards,

Dmitry

On 6/30/22 13:55, Timo Koch wrote:

Hi Dmitry,

Thanks for reporting this. Would you open a bug report?

Best wishes
Timo

On 29 Jun 2022, at 19:36, Dmitry Pavlov
wrote:

Hello,

In one porous flow simulation, I get this message:

... and it is fine. Matrix can be singular, no objection here. After
this, the Newton solver reports that it did not converge and restarts
with a smaller time step, and eventually it proceeds.

However, when I run the same simulation with MPI, I get this

Newton iteration 8 done, residual = 7.2437e+13
Solve: M deltax^k =

and the program hangs.

So it seems that it misses the singularity in the MPI mode and tries
to proceed with the current time step, to no avail.

How this can be cured?

Best regards,

Dmitry

Re: [DuMux] Singular matrix and MPI

2022-06-30 Thread Timo Koch

Hi Dmitry,

looks like you are running into a deadlock. Looks like a bug in the exception 
handler of the Newton
(I assume 
https://git.iws.uni-stuttgart.de/dumux-repositories/dumux/-/blob/master/dumux/nonlinear/newtonsolver.hh#L548
 is wrong because the exception cannot be assumed to be thrown on all 
processes).
Thanks for reporting this. Would you open a bug report?

Best wishes
Timo

On 29 Jun 2022, at 19:36, Dmitry Pavlov 
mailto:dmitry.pav...@outlook.com>> wrote:

Hello,

In one porous flow simulation, I get this message:

Newton iteration  6 done, residual = 3.1654e+13
Solve: M deltax^k = rNewton: Caught exception: "FMatrixError 
[luDecomposition:/home/dpavlov/DUMUX/dune-common/dune/common/densematrix.hh:939]:
 matrix is singular"

... and it is fine. Matrix can be singular, no objection here. After this, the 
Newton solver reports that it did not converge and restarts with a smaller time 
step, and eventually it proceeds.

However, when I run the same simulation with MPI, I get this

Newton iteration  8 done, residual = 7.2437e+13
Solve: M deltax^k =

and the program hangs.

So it seems that it misses the singularity in the MPI mode and tries to proceed 
with the current time step, to no avail.

How this can be cured?

Best regards,

Dmitry


___
DuMux mailing list
DuMux@listserv.uni-stuttgart.de
https://listserv.uni-stuttgart.de/mailman/listinfo/dumux

___
DuMux mailing list
DuMux@listserv.uni-stuttgart.de
https://listserv.uni-stuttgart.de/mailman/listinfo/dumux

Re: [DuMux] Singular matrix and MPI

2022-06-30 Thread Etienne Ahusborde


Hi Dmitry ,

We already had the same problem.

We understood that it is due to some "DUNE_THROW" commands.

Our solution (probably not the best) was to remove several "DUNE_THROW" 
commands and to replace them with a simple std::cout to keep the 
information that something is wrong.


Regards

Etienne

Le 29/06/2022 à 19:36, Dmitry Pavlov a écrit :


Hello,

In one porous flow simulation, I get this message:

Newton iteration  6 done, residual = 3.1654e+13
Solve: M deltax^k = rNewton: Caught exception: "FMatrixError 
[luDecomposition:/home/dpavlov/DUMUX/dune-common/dune/common/densematrix.hh:939]: 
matrix is singular"


... and it is fine. Matrix can be singular, no objection here. After 
this, the Newton solver reports that it did not converge and restarts 
with a smaller time step, and eventually it proceeds.


However, when I run the same simulation with MPI, I get this

Newton iteration  8 done, residual = 7.2437e+13
Solve: M deltax^k =

and the program hangs.

So it seems that it misses the singularity in the MPI mode and tries 
to proceed with the current time step, to no avail.


How this can be cured?

Best regards,

Dmitry


___
DuMux mailing list
DuMux@listserv.uni-stuttgart.de
https://listserv.uni-stuttgart.de/mailman/listinfo/dumux

___
DuMux mailing list
DuMux@listserv.uni-stuttgart.de
https://listserv.uni-stuttgart.de/mailman/listinfo/dumux

[DuMux] Singular matrix and MPI

2022-06-29 Thread Dmitry Pavlov


Hello,

In one porous flow simulation, I get this message:

Newton iteration  6 done, residual = 3.1654e+13
Solve: M deltax^k = rNewton: Caught exception: "FMatrixError 
[luDecomposition:/home/dpavlov/DUMUX/dune-common/dune/common/densematrix.hh:939]: 
matrix is singular"


... and it is fine. Matrix can be singular, no objection here. After 
this, the Newton solver reports that it did not converge and restarts 
with a smaller time step, and eventually it proceeds.


However, when I run the same simulation with MPI, I get this

Newton iteration  8 done, residual = 7.2437e+13
Solve: M deltax^k =

and the program hangs.

So it seems that it misses the singularity in the MPI mode and tries to 
proceed with the current time step, to no avail.


How this can be cured?

Best regards,

Dmitry


___
DuMux mailing list
DuMux@listserv.uni-stuttgart.de
https://listserv.uni-stuttgart.de/mailman/listinfo/dumux

Re: [DuMux] Singular matrix and MPI

Re: [DuMux] Singular matrix and MPI

Re: [DuMux] Singular matrix and MPI

Re: [DuMux] Singular matrix and MPI

Re: [DuMux] Singular matrix and MPI

Re: [DuMux] Singular matrix and MPI

[DuMux] Singular matrix and MPI

7 matches

Site Navigation

Mail list logo

Footer information