Hi Matthew,
In the worst case, every node/cell may have different label.
Below is one of the worst scenario with 102299 nodes and 102299
different labels for test. I found the time cost increase during the
loop. The first 9300 loop takes least time (<0.5) while the last 9300
loops takes much more time (>7.7), as shown below. If I use larger mesh
with >1 million nodes, it runs very very slowly in this part. The PETSc
is configured with optimization on.
Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran
--download-mpich --download-scalapack --download-parmetis
--download-metis --download-ptscotch --download-fblaslapack
--download-hypre --download-superlu_dist --download-hdf5=yes
--download-ctetgen --with-debugging=0 COPTFLAGS="-O3 -march=native
-mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native"
FOPTFLAGS="-O3 -march=native -mtune=native"
istart iend progress CPU_Time time cost - old (sec) time cost - new
(sec)
0 9299 0 1524670045.51166
9300 18599 0.100010753 1524670045.99605 0.4843890667
0.497246027
18600 27899 0.200010747 1524670047.32635 1.330302
1.3820912838
27900 37199 0.300010741 1524670049.3066 1.9802515507
2.2439446449
37200 46499 0.400010765 1524670052.1594 2.852804184
3.0739262104
46500 55799 0.500010729 1524670055.90961 3.7502081394
3.9270553589
55800 65099 0.600010753 1524670060.47654 4.5669286251
4.7571902275
65100 74399 0.700010777 1524670066.0941 5.6175630093
5.7428796291
74400 83699 0.800010741 1524670072.53886 6.44475317
6.5761549473
83700 92998 0.900010765 1524670079.99072 7.4518604279
7.4606924057
92999 102298 1 1524670087.71066 7.7199423313 8.2424075603
old code
do ipoint = 0, istart-1
!c output time cost, use 1 processor to test
if (b_enable_output .and. rank == 0) then
if (mod(ipoint,iprogress) == 0 .or. ipoint == istart-1) then
!write(*,'(f3.1,1x)',advance="no") (ipoint+1.0)/istart
write(*,*) ipoint, (ipoint+1.0)/istart,"time",MPI_Wtime()
end if
end if
call DMSetLabelValue(dmda_flow%da,"cid_lg2g",ipoint, &
ipoint+1,ierr)
CHKERRQ(ierr)
end do
new code
call DMCreateLabel(dmda_flow%da,'cid_lg2g',ierr)
CHKERRQ(ierr)
call DMGetLabel(dmda_flow%da,'cid_lg2g',label, ierr)
CHKERRQ(ierr)
do ipoint = 0, istart-1
!c output time cost, use 1 processor to test
if (b_enable_output .and. rank == 0) then
if (mod(ipoint,iprogress) == 0 .or. ipoint == istart-1) then
!write(*,'(f3.1,1x)',advance="no") (ipoint+1.0)/istart
write(*,*) ipoint, (ipoint+1.0)/istart,"time",MPI_Wtime()
end if
end if
call DMLabelSetValue(label,ipoint,ipoint+1,ierr)
CHKERRQ(ierr)
end do
Thanks,
Danyang
On 2018-04-25 03:16 AM, Matthew Knepley wrote:
On Tue, Apr 24, 2018 at 11:57 PM, Danyang Su <[email protected]
<mailto:[email protected]>> wrote:
Hi All,
I use DMPlex in unstructured grid code and recently found
DMSetLabelValue takes a lot of time for large problem, e.g., num.
of cells > 1 million. In my code, I use
I read your code wrong. For large loop, you should not use the
convenience function. You should use
DMPlexCreateFromCellList ()
DMGetLabel(dm, name, &label)
Loop over all cells/nodes{
DMSetLabelValue
Replace this by DMLabelSetValue(label, point, val)
}
DMPlexDistribute
The code works fine except DMSetLabelValue takes a lot of time for
large problem. I use DMSetLabelValue to set material id for all
the nodes or cells so that each subdomain has a copy of material
id. Is there any other functions that can be used more efficient,
e.g. set labels by array, not 1 by 1?
That should take much less time.
Thanks,
Matt
Thanks,
Danyang
--
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which
their experiments lead.
-- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/ <http://www.caam.rice.edu/%7Emk51/>
