[deal.II] PETSc vs Trilinos for step41

[Mohammadreza Yaghoobi]

Hi,

I was wondering why Trilinos is used in Step41 and not Petsc?

Thanks
Reza

-- 
The deal.II project is located at http://www.dealii.org/
For mailing list/forum options, see 
https://groups.google.com/d/forum/dealii?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"deal.II User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to dealii+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/dealii/e4666e1b-ce96-4c4f-b28b-b3d9b930f76bn%40googlegroups.com.

Re: [deal.II] Removing cells in parallel distributed triangulation

[Mohammadreza Yaghoobi]

Thanks Jean-Paul and Wolfgang,

I'll give a try on parallel::fullydistributed::Triangulation as adding
FE_nothing would still have that issue of memory. If I don't get success,
I'll circle back with you on FE_nothing.

Thanks
Reza

On Tue, Apr 20, 2021 at 5:19 PM Jean-Paul Pelteret 
wrote:

> Hi Reza,
>
> With no context as to how element deletion affects the geometry itself,
> may I ask if you’ve considered element deactivation using the FE_Nothing
> element? If you can afford to store the mesh but primarily want to ensure
> that no DoFs are assigned to a region outside of the physical domain
> boundaries, then this might be an alternate approach to adjusting the mesh
> directly.
>
> Best,
> Jean-Paul
>
> On 20. Apr 2021, at 21:58, mohammadreza yaghoobi <
> mohammadreza.yagho...@gmail.com> wrote:
>
> Thanks Marc.
>
> My biggest challenges are both time and memory. I'm trying to model a
> crystal plasticity sample with ~20 million elements. In the case of no
> element deletion, I can simply use global refinement and I won't face
> memory issues while generating mesh and running the code (and of course no
> time on element deletion).
>
> However, when I'm trying to add the element deletion feature, then I
> cannot use global refinement,  and I'll face the memory issue, since all of
> the triangulation will be on the coarsest level. On top of that, for every
> processor, the code has to search the element deletion pattern for all of
> the triangulation (~20 million elements). As I increase the sample size,
> each processor still needs to handle all the elements which becomes a big
> bottle-neck, and also uses the resources very inefficiently. Actually,
> parallelization cannot help at all with this part of the code, which
> actually serves as the serial with large memory requirements. Just to give
> you an idea, it will take about 12 hrs in my example to bypass the element
> deletion search part running on 1500 processors, each has 38gb memory. I
> actually have to underutilize my node (instead of 24, I'm using 5) because
> of memory issues.
>
> The ideal case would be each processor just handling their own part and
> some reduction happened throughout the processors.
>
> I hope I was able to clarify the bottle neck.
>
> Thanks
> Reza
>
>
>
> On Tue, Apr 20, 2021 at 3:47 PM Marc Fehling  wrote:
>
>> Hello Reza,
>>
>> `parallel::distributed::Triangulation` objects store the entire coarse
>> mesh, see also the module documentation
>> <https://www.dealii.org/current/doxygen/deal.II/group__distributed.html>.
>> This allows for a quick execution of adaptive refinement. So from my
>> understanding, you need to perform this operation on *all* active cells,
>> including ghost and artificial ones. So the working code you have might be
>> the only way to use
>> `GridGenerator::create_triangulation_with_removed_cells()` in the parallel
>> distributed context.
>>
>> Can you quantify how inefficient your code is? For example did you time
>> how long your mesh creation takes and compare it to the overall runtime?
>>
>> Best,
>> Marc
>>
>> On Tuesday, April 20, 2021 at 12:14:10 PM UTC-6 mohammadre...@gmail.com
>> wrote:
>>
>>> Hi,
>>>
>>> I'm trying to subtract some elements based on a user-defined criteria.
>>> Ideally, I want to use each processor in a way that they are responsible
>>> only for their partition (using is_locally_owned()) to remove the cells
>>> from the triangulation
>>> using GridGenerator::create_triangulation_with_removed_cells. However, when
>>> I used  this, it will not proceed after
>>> GridGenerator::create_triangulation_with_removed_cells.
>>> Very inefficient solution is each processor will process the whole
>>> triangulation space and remove the elements. This works. However, as the
>>> simulation size increases, this becomes really a bottle neck.
>>>
>>> Is there any way I can tackle this smarter?
>>>
>>> Thanks
>>> Reza
>>>
>>> *Below is the version of the code which works but very inefficiently.*
>>> *I added the red lines so every processor be responsible just for their
>>> own parts. It stops at
>>> GridGenerator::create_triangulation_with_removed_cells.*
>>>
>>>
>>>
>>> GridGenerator::subdivided_hyper_rectangle (triangulation2,
>>> userInputs.subdivisions, Point(),
>>> Point(userInputs.span[0],userInputs.span[1],userInputs.span[2]), true);
>>>
>>> typename Triangulation< dim, dim >::active_cell_iterator cell =
>>> triangulation2

Re: [deal.II] Re: Removing cells in parallel distributed triangulation

[mohammadreza yaghoobi]

Thanks Wolfgang.

That's a really good news as this is our biggest current bottleneck.
Just a quick question to warm up! does
GridGenerator::create_triangulation_with_removed_cells or something similar
is available for this class?

Thanks
Reza

On Tue, Apr 20, 2021 at 4:05 PM Wolfgang Bangerth 
wrote:

> On 4/20/21 1:58 PM, mohammadreza yaghoobi wrote:
> >
> > The ideal case would be each processor just handling their own part and
> some
> > reduction happened throughout the processors.
>
> Reza -- the parallel::distributed::Triangulation class was written under
> the
> assumption that storing artificial cells is not expensive, which in
> practice
> boils down to the assumption that the mesh is refined globally or
> adaptively
> sufficiently many times and/or that the coarse mesh is relatively small.
> In
> your case, your coarse mesh has 20M cells, and that is a case that the
> p::d::T
> class simply was not designed for.
>
> But, the good news is that that there is a different class,
> parallel::fullydistributed::Triangulation, that is meant for this kind of
> case. This class is meant to be used for meshes where the coarse mesh is
> so
> large that it can not be replicated on all processes. (I don't recall
> right
> now whether it allows mesh refinement in its current state, though.)
>
> There is currently no tutorial program that illustrates its use, but maybe
> you
> can gather enough information from what exists in the class documentation
> and
> what you can find in the test suite. Feel free to also ask on the mailing
> list
> if you have questions about this class!
>
> Best
>   Wolfgang
>
> --
> 
> Wolfgang Bangerth  email: bange...@colostate.edu
> www: http://www.math.colostate.edu/~bangerth/
>
> --
> The deal.II project is located at http://www.dealii.org/
> For mailing list/forum options, see
> https://groups.google.com/d/forum/dealii?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "deal.II User Group" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to dealii+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/dealii/3a8ea65c-06c1-a7f9-60b2-c5e0383d3e6d%40colostate.edu
> .
>

-- 
The deal.II project is located at http://www.dealii.org/
For mailing list/forum options, see 
https://groups.google.com/d/forum/dealii?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"deal.II User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to dealii+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/dealii/CAGxJ-SoKDbnpRvj%2BofdSpaZDaC%3D1n_QR-E3H4tsyU%2B0Ldf%2BgQw%40mail.gmail.com.

Re: [deal.II] Re: Removing cells in parallel distributed triangulation

[mohammadreza yaghoobi]

Thanks Marc.

My biggest challenges are both time and memory. I'm trying to model a
crystal plasticity sample with ~20 million elements. In the case of no
element deletion, I can simply use global refinement and I won't face
memory issues while generating mesh and running the code (and of course no
time on element deletion).

However, when I'm trying to add the element deletion feature, then I cannot
use global refinement,  and I'll face the memory issue, since all of the
triangulation will be on the coarsest level. On top of that, for every
processor, the code has to search the element deletion pattern for all of
the triangulation (~20 million elements). As I increase the sample size,
each processor still needs to handle all the elements which becomes a big
bottle-neck, and also uses the resources very inefficiently. Actually,
parallelization cannot help at all with this part of the code, which
actually serves as the serial with large memory requirements. Just to give
you an idea, it will take about 12 hrs in my example to bypass the element
deletion search part running on 1500 processors, each has 38gb memory. I
actually have to underutilize my node (instead of 24, I'm using 5) because
of memory issues.

The ideal case would be each processor just handling their own part and
some reduction happened throughout the processors.

I hope I was able to clarify the bottle neck.

Thanks
Reza



On Tue, Apr 20, 2021 at 3:47 PM Marc Fehling  wrote:

> Hello Reza,
>
> `parallel::distributed::Triangulation` objects store the entire coarse
> mesh, see also the module documentation
> .
> This allows for a quick execution of adaptive refinement. So from my
> understanding, you need to perform this operation on *all* active cells,
> including ghost and artificial ones. So the working code you have might be
> the only way to use
> `GridGenerator::create_triangulation_with_removed_cells()` in the parallel
> distributed context.
>
> Can you quantify how inefficient your code is? For example did you time
> how long your mesh creation takes and compare it to the overall runtime?
>
> Best,
> Marc
>
> On Tuesday, April 20, 2021 at 12:14:10 PM UTC-6 mohammadre...@gmail.com
> wrote:
>
>> Hi,
>>
>> I'm trying to subtract some elements based on a user-defined criteria.
>> Ideally, I want to use each processor in a way that they are responsible
>> only for their partition (using is_locally_owned()) to remove the cells
>> from the triangulation
>> using GridGenerator::create_triangulation_with_removed_cells. However, when
>> I used  this, it will not proceed after
>> GridGenerator::create_triangulation_with_removed_cells.
>> Very inefficient solution is each processor will process the whole
>> triangulation space and remove the elements. This works. However, as the
>> simulation size increases, this becomes really a bottle neck.
>>
>> Is there any way I can tackle this smarter?
>>
>> Thanks
>> Reza
>>
>> *Below is the version of the code which works but very inefficiently.*
>> *I added the red lines so every processor be responsible just for their
>> own parts. It stops at
>> GridGenerator::create_triangulation_with_removed_cells.*
>>
>>
>>
>> GridGenerator::subdivided_hyper_rectangle (triangulation2,
>> userInputs.subdivisions, Point(),
>> Point(userInputs.span[0],userInputs.span[1],userInputs.span[2]), true);
>>
>> typename Triangulation< dim, dim >::active_cell_iterator cell =
>> triangulation2.begin_active(), endc = triangulation2.end();
>>
>> for (; cell!=endc; ++cell) {
>> if (cell->is_locally_owned()){
>> double pnt3[3];
>> const Point pnt2=cell->center();
>> for (unsigned int i=0; i> pnt3[i]=pnt2[i];
>> }
>>
>> gID=orientations_Mesh.getMaterialID(pnt3);
>> cellOrientationMap_Mesh.push_back(gID);
>> }
>> }
>>
>> unsigned int materialID;
>> unsigned int cell2=0;
>> cell = triangulation2.begin_active(); endc = triangulation2.end();
>> std::set::active_cell_iterator>
>> cells_to_remove;
>> unsigned int NumberOwned=0;
>> for (; cell!=endc; ++cell) {
>> if (cell->is_locally_owned()){
>> materialID=cellOrientationMap_Mesh[cell2];
>> NumberOwned=NumberOwned+1;
>> if (materialID ==userInputs.deletionGrainID){
>> cells_to_remove.insert (cell);
>> }
>> cell2=cell2+1;
>> }
>> }
>>
>> char buffer[200];
>>
>> if (cells_to_remove.size()>0){
>>
>> GridGenerator::create_triangulation_with_removed_cells(triangulation2,cells_to_remove,triangulation);
>> }
>> else{
>> triangulation.copy_triangulation(triangulation2);
>> }
>> }
>> else {
>> triangulation.copy_triangulation(triangulation2);
>> }
>>
> --
> The deal.II project is located at http://www.dealii.org/
> For mailing list/forum options, see
> https://groups.google.com/d/forum/dealii?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "deal.II User Group" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to 

Re: [deal.II] Assertion ref_count!=0 error

[mohammadreza yaghoobi]

Hi,

The simplest program has the issue was step-2 and step-3 of deal.ii.
However, step-1 runs successfully.

Maybe the issue is another version of TBB. Let me check it.

Thanks
Reza

On Tue, Feb 2, 2021 at 10:32 AM Wolfgang Bangerth 
wrote:

> On 2/1/21 5:44 PM, Mohammadreza ya'ghoobi wrote:
> >
> > We tried to install the deal.ii on a workstation with Ubuntu 18.04 OS
> and used
> > candi to compile it. It successfully compile the deal.ii and all
> packages.
> > it runs the step-1 successfully. But when we run step-2 and 3, it shows
> this
> > error:
> > Assertion ref_count!=0 failed on line 563 of file
> > .../dealii-9.2.0/bundled/tbb-2018_U2/tbb/scheduler.cpp
> > description: attempt to spawn task whose parent has a ref_count==0
> (forgot to
> > set_ref_count?)
>
> I don't think I've ever seen this error. Do you have another version of
> the
> TBB installed on your system?
>
> The image you show is for a crystal plasticity programs. What is the
> simplest
> program that illustrates the problem reproducibly?
>
> Best
>   W.
>
>
> --
> 
> Wolfgang Bangerth  email: bange...@colostate.edu
> www: http://www.math.colostate.edu/~bangerth/
>
> --
> The deal.II project is located at http://www.dealii.org/
> For mailing list/forum options, see
> https://groups.google.com/d/forum/dealii?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "deal.II User Group" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to dealii+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/dealii/40f1c94a-03ee-e5ef-23c5-ff6e301e24b2%40colostate.edu
> .
>

-- 
The deal.II project is located at http://www.dealii.org/
For mailing list/forum options, see 
https://groups.google.com/d/forum/dealii?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"deal.II User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to dealii+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/dealii/CAGxJ-Sq3uCXEmniz-pSAKWe2peL4Ki6WOej6GsUOwM877zYQWQ%40mail.gmail.com.