On Sat, 2 Mar 2019, Xiang LI wrote:
> I would like to create a mesh in parallel. I configured libMesh with
> --enable-parmesh and did some experiments. For example, I run an example
> introduction_ex4.C which solves the Poisson's equation in parallel with 48
> processors. I create a 80*80*80 grid and what I got is as follows:
>
> Mesh Information:
> elem_dimensions()={3}
> spatial_dimension()=3
> n_nodes()=4173281
> n_local_nodes()=91821
Not bad; you only have 5% over optimum load imbalance (assuming proc 0
is the worst, which is likely with our default settings). If that
gets worse with future problems, the git head has some better node
partitioning options you can try.
> n_elem()=512000
> n_local_elem()=10631
> n_active_elem()=512000
> n_subdomains()=1
> n_partitions()=48
> n_processors()=48
> n_threads()=1
> processor_id()=0
All looks good.
> I observed that the code is successfully parallelized, but in
> each processor the mesh use more than 1G memory.
That definitely shouldn't be. I forget exactly what we're using per
node now, but 1e5 nodes with one variable and a narrow stencil should
surely be less than 1e7 bytes, not more than 1e9.
> Seems that each processor stored a copy of the whole mesh
> (the memory usage of mesh is unchanged if I use 96 processors
> instead).
We should probably print out Replicated vs Distributed vs
Distributed-but-not-currently-distributed status from print_info(),
but it certainly sounds like you accidentally got a replicated mesh
somehow.
> My questions are as follows:
> 1. Why the mesh is not stored in parallel? Is it because that
> my configuration is not successful, or I should modify the
> code? If I should modify the code, I was wondering how to do
> it. Do I need to initialize ParallelMesh or DistributedMesh
> instead of simply use Mesh mesh(init.comm()) at first?
In theory you shouldn't have to modify the code; anything (like nearly
all our examples) that creates a Mesh will get a replicated or a
distributed mesh under the hood depending on configuration. You can
also manually specify ReplicatedMesh vs DistributedMesh in the code
itself, but in your case that shouldn't have been necessary.
> 2. Is there a primer or an example on how to use ParallelMesh
> or DistributedMesh?
I'm afraid not.
> Is it OK to use a ParallelMesh in the same way as a normal
> Mesh?
Usually. Out of 64 examples you'll only find 9 that are having to
manually create a ReplicatedMesh.
But the more sophisticated your code is, the harder it is to make it
work on a distributed mesh. If you're solving Poisson there's
probably no changes to make. If you're doing ray tracing or
integrating boundary layer quantities for a turbulent transition model
or something else that has one processor needing information about a
distant processor's elements then it's harder when that information
can't be acquired without manual communication code.
> 3.For some reason I am still using libMesh 0.9.5. Is it
> crucial to update libMesh if I have to use ParallelMesh?
YES.
That release is so old I honestly don't even remember how much of
what-was-then-ParallelMesh was working and how much wasn't.
---
Roy
_______________________________________________
Libmesh-users mailing list
Libmesh-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/libmesh-users
