[Libmesh-users] A problem on memory blow up when using build_cube() for 3D mesh

[li . luo]

Dear developers,

I tried a 3D mesh of 200x200x200 by using build_cube(), on 128 cores.
However, it runs out of memory. 

I read the details of build_cube(), it seems that a global 200x200x200 mesh is 
generated on every processor.
If this is true, then the meshes are redundant.
I tested that the memory used had already blowed up before the calling of  
prepare_for_use().

Another question is, I did test that, the calling of delete_remote_elements() 
can not reduce memory.

Regards,
--
Li Luo







--
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
___
Libmesh-users mailing list
Libmesh-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/libmesh-users

Re: [Libmesh-users] dof numbering

[Manav Bhatia]

Ok. Just so that I understand your point:  you are saying that because petsc 
(and other advanced direct solvers) will calculate a new ordering for the 
matrix factorization, it does not help to calculate it in advance. However, for 
a more basic direct solver, which does not calculate the reordering as a part 
of its factorization, it may be beneficial to calculate this reordering in 
advance. correct? 

-Manav

> On Feb 6, 2015, at 10:11 PM, Jed Brown  wrote:
> 
> 
> One of those orderings is already being computed for your matrix as part
> of the analysis (or "symbolic factorization") step.

--
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
___
Libmesh-users mailing list
Libmesh-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/libmesh-users

Re: [Libmesh-users] dof numbering

[Manav Bhatia]

> On Feb 6, 2015, at 9:40 PM, Jed Brown  wrote:
> 
> Manav Bhatia  writes:
>> I am not sure if I understand your point. Are you saying that using a
>> fill-reducing ordering for the dofs would not change the performance
>> of a direct solver?
> 
> Correct.  The solver is already computing an ordering (e.g., with nested
> dissection).  You can change the ordering algorithm as a run-time option
> to most solvers.  "Natural" ordering is an option with the PETSc
> solvers, but it's pretty much always super slow.
> 
> Either ABAQUS is building a different matrix (e.g., different
> discretization) or using a different ordering algorithm, but there is no
> point reordering your dofs to "help out".


Ok. Then how does the discussion about the fill-reduction using something like 
metis fit into this:

http://en.wikipedia.org/wiki/Sparse_matrix#Reducing_fill-in 

http://matrixprogramming.com/2008/05/metis 


Does that not qualify as a reordering algorithm? Or are you talking about a 
matrix that is already available in dense format and the dof connectivity is 
not know? One could still deduce some connectivity information from the sparse 
matrix storage, and use that to run through METIS. Or not?

-Manav

--
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
___
Libmesh-users mailing list
Libmesh-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/libmesh-users

Re: [Libmesh-users] dof numbering

[Manav Bhatia]

> On Feb 6, 2015, at 9:29 PM, Jed Brown  wrote:
> 
> Manav Bhatia  writes:
> 
>> I am in agreement with that. However my situation is more rudimentary than 
>> that. 
>> 
>> If I run the same problem in ABAQUS, it uses a multifraontal direct solver 
>> and gets done in two minutes. 
>> 
>> If I use the -ksp_type preonly -pc_type lu on my code built on libmesh and 
>> petsc, it takes ~16GB in storage and half hour for solution. 
> 
> Do you have the same matrix in both cases?  Direct solvers compute an
> ordering (e.g., by nested dissection), so changing the ordering of your
> matrices is not likely to change much.  What direct solver is ABAQUS
> calling?

All that I could find in its manuals is that it uses a multifront technique 
(http://www.maths.cam.ac.uk/computing/software/abaqus_docs/docs/v6.12/books/usb/default.htm?startat=pt03ch06s01aus46.html#usb-anl-asolveroverview).
 

I am not sure if I understand your point. Are you saying that using a 
fill-reducing ordering for the dofs would not change the performance of a 
direct solver? 

-Manav


--
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
___
Libmesh-users mailing list
Libmesh-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/libmesh-users

Re: [Libmesh-users] dof numbering

[Manav Bhatia]

I am in agreement with that. However my situation is more rudimentary than 
that. 

If I run the same problem in ABAQUS, it uses a multifraontal direct solver and 
gets done in two minutes. 

If I use the -ksp_type preonly -pc_type lu on my code built on libmesh and 
petsc, it takes ~16GB in storage and half hour for solution. 

I am looking for ways to bring my solvers performance to that of ABAQUS, snd I 
think dof numbering is key.

Hence this discussion.

-Manav  


> On Feb 6, 2015, at 9:15 PM, Jed Brown  wrote:
> 
> David Knezevic  writes:
>> One of my colleagues has used it. Apparently it works, but it's very slow
>> (not surprising).
> 
> I propose that global storage as an algorithmic device (out-of-core
> factorization, checkpointing to disk for transient adjoints, etc) is
> nearly dead and will be soon.  The time required to write the full
> contents of memory to disk and read it back is about an hour on today's
> large machines.
> 
> If you don't have enough memory, use more nodes, up to the entire
> machine; it will be more efficient than writing to global storage.  If
> you are running on the entire machine and still don't have enough
> memory, you also don't have enough time in your allocation to run an
> out-of-core algorithm (unless you're going to burn the entire annual
> budget of an INCITE award on a single run).


--
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
___
Libmesh-users mailing list
Libmesh-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/libmesh-users

Re: [Libmesh-users] dof numbering

[David Knezevic]

I'm not sure why MUMPS wouldn't be available on your machine, it works fine
for me (I just use the --download-mumps option when configuring PETSc).


But you seem to be suggesting that MUMPS has a builtin fill-reduction
> algorithm. Correct?
>

Yes, have a look at ICNTL(7) in the MUMPS manual. It provides different
reordering schemes which can reduce fill-in in the LU factors.



> Also, do you have experience with the out-of-core option of MUMPS?
>


One of my colleagues has used it. Apparently it works, but it's very slow
(not surprising).

David





On Feb 6, 2015, at 8:47 PM, David Knezevic 
> wrote:
>
> It sounds like you're right at the limit of what your computer can handle
> with a direct solver. Which direct solver are you using? I generally use
> MUMPS or SuperLU instead of PETSc's built in direct solver (e.g. -ksp_type
> preonly -pc_type lu -pc_factor_mat_solver_package superlu_dist). I think
> SuperLU typically uses less RAM than MUMPS, so maybe you should try that?
>
> Also, I know that with MUMPS you can try different fill-reduction
> algorithms, which can be controlled via PETSc's ictnl flags
> 
> .
>
> David
>
>
>
>
> On Sat, Feb 7, 2015 at 10:33 AM, Manav Bhatia 
> wrote:
>
>> Hi,
>>
>>I am trying to understand the dof numbering a bit better.
>>
>>The simulation I have at hand is that of a solid mechanics problem
>> with thermal load. This has a total of ~770,000 dofs, and so far none of
>> the iterative solves in petsc has provided good convergence property.
>> Hence, I am stuck with using direct solvers.
>>
>> As a result, it is important for me to have a fill reducing dof
>> numbering for my problems. Some of my dofs are coupled only through the dof
>> constraints, not through a physical element connectivity. I have two
>> different sets of such coupling constraints. One set is applied
>> independently, and the second set is applied in addition to the first set.
>>
>> What I have observed is that the -ksp_type preonly -pc_type lu is
>> able to solve with only the first set of dof constraints applied (but takes
>> a lot of memory ~16 GB, which is all that my computer has). However, petsc
>> complains about insufficient memory when I apply the second set in addition
>> to the first.
>>
>> I suspect that renumbering the dofs using a fill-reducing criteria by
>> accounting for these dof constraints could address the problem. However, I
>> am not sure if the current code setup allows this to be done in libMesh.
>> What I see is that the partitioners are associated with the MeshBase class,
>> without any information from the DofMap.
>>
>> I would appreciate any insight in this matter.
>>
>> Thanks,
>> Manav
>>
>>
>>
>> --
>> Dive into the World of Parallel Programming. The Go Parallel Website,
>> sponsored by Intel and developed in partnership with Slashdot Media, is
>> your
>> hub for all things parallel software development, from weekly thought
>> leadership blogs to news, videos, case studies, tutorials and more. Take a
>> look and join the conversation now. http://goparallel.sourceforge.net/
>> ___
>> Libmesh-users mailing list
>> Libmesh-users@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/libmesh-users
>>
>
>
>
--
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
___
Libmesh-users mailing list
Libmesh-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/libmesh-users

Re: [Libmesh-users] dof numbering

[David Knezevic]

It sounds like you're right at the limit of what your computer can handle
with a direct solver. Which direct solver are you using? I generally use
MUMPS or SuperLU instead of PETSc's built in direct solver (e.g. -ksp_type
preonly -pc_type lu -pc_factor_mat_solver_package superlu_dist). I think
SuperLU typically uses less RAM than MUMPS, so maybe you should try that?

Also, I know that with MUMPS you can try different fill-reduction
algorithms, which can be controlled via PETSc's ictnl flags

.

David




On Sat, Feb 7, 2015 at 10:33 AM, Manav Bhatia  wrote:

> Hi,
>
>I am trying to understand the dof numbering a bit better.
>
>The simulation I have at hand is that of a solid mechanics problem with
> thermal load. This has a total of ~770,000 dofs, and so far none of the
> iterative solves in petsc has provided good convergence property. Hence, I
> am stuck with using direct solvers.
>
> As a result, it is important for me to have a fill reducing dof
> numbering for my problems. Some of my dofs are coupled only through the dof
> constraints, not through a physical element connectivity. I have two
> different sets of such coupling constraints. One set is applied
> independently, and the second set is applied in addition to the first set.
>
> What I have observed is that the -ksp_type preonly -pc_type lu is able
> to solve with only the first set of dof constraints applied (but takes a
> lot of memory ~16 GB, which is all that my computer has). However, petsc
> complains about insufficient memory when I apply the second set in addition
> to the first.
>
> I suspect that renumbering the dofs using a fill-reducing criteria by
> accounting for these dof constraints could address the problem. However, I
> am not sure if the current code setup allows this to be done in libMesh.
> What I see is that the partitioners are associated with the MeshBase class,
> without any information from the DofMap.
>
> I would appreciate any insight in this matter.
>
> Thanks,
> Manav
>
>
>
> --
> Dive into the World of Parallel Programming. The Go Parallel Website,
> sponsored by Intel and developed in partnership with Slashdot Media, is
> your
> hub for all things parallel software development, from weekly thought
> leadership blogs to news, videos, case studies, tutorials and more. Take a
> look and join the conversation now. http://goparallel.sourceforge.net/
> ___
> Libmesh-users mailing list
> Libmesh-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/libmesh-users
>
--
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
___
Libmesh-users mailing list
Libmesh-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/libmesh-users

Re: [Libmesh-users] dof numbering

[Manav Bhatia]

I tried the MUMPS option through petsc using this set of options: 

-ksp_type preonly -pc_type lu -pc_factor_mat_solver_package mumps 
-mat_mumps_icntl_ 22 1 -mat_mumps_icntl_ 23 1 

However, MUMPS does not seems to be running on my machine with the PETSC build. 
I build 3.5.3 today with the download mumps option, and it has not worked even 
for small problems. 

But you seem to be suggesting that MUMPS has a builtin fill-reduction 
algorithm. Correct? Also, do you have experience with the out-of-core option of 
MUMPS? 

Manav



> On Feb 6, 2015, at 8:47 PM, David Knezevic  wrote:
> 
> It sounds like you're right at the limit of what your computer can handle 
> with a direct solver. Which direct solver are you using? I generally use 
> MUMPS or SuperLU instead of PETSc's built in direct solver (e.g. -ksp_type 
> preonly -pc_type lu -pc_factor_mat_solver_package superlu_dist). I think 
> SuperLU typically uses less RAM than MUMPS, so maybe you should try that?
> 
> Also, I know that with MUMPS you can try different fill-reduction algorithms, 
> which can be controlled via PETSc's ictnl flags 
> .
> 
> David
> 
> 
> 
> 
> On Sat, Feb 7, 2015 at 10:33 AM, Manav Bhatia  > wrote:
> Hi,
> 
>I am trying to understand the dof numbering a bit better.
> 
>The simulation I have at hand is that of a solid mechanics problem with 
> thermal load. This has a total of ~770,000 dofs, and so far none of the 
> iterative solves in petsc has provided good convergence property. Hence, I am 
> stuck with using direct solvers.
> 
> As a result, it is important for me to have a fill reducing dof numbering 
> for my problems. Some of my dofs are coupled only through the dof 
> constraints, not through a physical element connectivity. I have two 
> different sets of such coupling constraints. One set is applied 
> independently, and the second set is applied in addition to the first set.
> 
> What I have observed is that the -ksp_type preonly -pc_type lu is able to 
> solve with only the first set of dof constraints applied (but takes a lot of 
> memory ~16 GB, which is all that my computer has). However, petsc complains 
> about insufficient memory when I apply the second set in addition to the 
> first.
> 
> I suspect that renumbering the dofs using a fill-reducing criteria by 
> accounting for these dof constraints could address the problem. However, I am 
> not sure if the current code setup allows this to be done in libMesh. What I 
> see is that the partitioners are associated with the MeshBase class, without 
> any information from the DofMap.
> 
> I would appreciate any insight in this matter.
> 
> Thanks,
> Manav
> 
> 
> --
> Dive into the World of Parallel Programming. The Go Parallel Website,
> sponsored by Intel and developed in partnership with Slashdot Media, is your
> hub for all things parallel software development, from weekly thought
> leadership blogs to news, videos, case studies, tutorials and more. Take a
> look and join the conversation now. http://goparallel.sourceforge.net/ 
> 
> ___
> Libmesh-users mailing list
> Libmesh-users@lists.sourceforge.net 
> 
> https://lists.sourceforge.net/lists/listinfo/libmesh-users 
> 
> 

--
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
___
Libmesh-users mailing list
Libmesh-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/libmesh-users

[Libmesh-users] dof numbering

[Manav Bhatia]

Hi, 

   I am trying to understand the dof numbering a bit better. 

   The simulation I have at hand is that of a solid mechanics problem with 
thermal load. This has a total of ~770,000 dofs, and so far none of the 
iterative solves in petsc has provided good convergence property. Hence, I am 
stuck with using direct solvers. 

As a result, it is important for me to have a fill reducing dof numbering 
for my problems. Some of my dofs are coupled only through the dof constraints, 
not through a physical element connectivity. I have two different sets of such 
coupling constraints. One set is applied independently, and the second set is 
applied in addition to the first set. 

What I have observed is that the -ksp_type preonly -pc_type lu is able to 
solve with only the first set of dof constraints applied (but takes a lot of 
memory ~16 GB, which is all that my computer has). However, petsc complains 
about insufficient memory when I apply the second set in addition to the first. 

I suspect that renumbering the dofs using a fill-reducing criteria by 
accounting for these dof constraints could address the problem. However, I am 
not sure if the current code setup allows this to be done in libMesh. What I 
see is that the partitioners are associated with the MeshBase class, without 
any information from the DofMap. 

I would appreciate any insight in this matter. 

Thanks,
Manav


--
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
___
Libmesh-users mailing list
Libmesh-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/libmesh-users

Re: [Libmesh-users] DofConstraintRow

[Roy Stogner]

On Fri, 6 Feb 2015, Manav Bhatia wrote:

> If the dofs constraint row is used to couple two dofs that otherwise
> do not have any coupling in the mesh (meaning, they belong to
> independent elements), then is this information taken into account
> in defining the sparsity pattern?

Yes.  Likewise for the definition of "ghost" dofs in parallel vectors.
---
Roy

--
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
___
Libmesh-users mailing list
Libmesh-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/libmesh-users

Re: [Libmesh-users] DofConstraintRow

[Manav Bhatia]

Question about dof constraints and sparsity pattern: 

If the dofs constraint row is used to couple two dofs that otherwise do not 
have any coupling in the mesh (meaning, they belong to independent elements), 
then is this information taken into account in defining the sparsity pattern? 

Thanks,
Manav


> On Jan 20, 2015, at 2:47 PM, Roy Stogner  wrote:
> 
> 
> On Tue, 20 Jan 2015, Manav Bhatia wrote:
> 
>> Am  going to give the multipoint constraint a try.
>> 
>> I am curious. Is this handled internally by simply replacing the equation 
>> for the constrained dof with the specified row?
> 
> The two options are "replace the old equation" and "scream and die if
> there was already a constraint", yeah.  The latter is a nice check if
> you think you've found an algorithm that avoids conflicting with other
> constraints but you're not certain.
> ---
> Roy


--
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
___
Libmesh-users mailing list
Libmesh-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/libmesh-users

Re: [Libmesh-users] Code stuck without any motion

[Paul T. Bauman]

On Fri, Feb 6, 2015 at 3:07 AM, subramanya sadasiva 
wrote:

> Hi,
> I built libmesh with intel compilers and intel mpi.


I would avoid Intel MPI. I've found it to be buggy, including hanging on a
broadcast (I'm not the only one to have this happen).


> The code built successfully however, when I run the code, the code gets
> stuck without doing anything. Is there anything I can look at to diagnose
> this situation?
>

Attach a debugger to see where it's hanging.

http://dirac.org/linux/gdb/06-Debugging_A_Running_Process.php
--
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
___
Libmesh-users mailing list
Libmesh-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/libmesh-users

[Libmesh-users] Code stuck without any motion

[subramanya sadasiva]

Hi, 
I built libmesh with intel compilers and intel mpi.  The code built 
successfully however, when I run the code, the code gets stuck without doing 
anything. Is there anything I can look at to diagnose this situation? I will be 
grateful for any suggestions. THanks, Subramanya
 
--
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
___
Libmesh-users mailing list
Libmesh-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/libmesh-users