On Mon, Jan 12, 2015 at 10:31 AM, Roy Stogner <royst...@ices.utexas.edu>
wrote:
The failure seems to be within a ParallelMesh-only code path.
>
OK, I see this with ParallelMesh enabled.
Environment:
Currently Loaded Modules:
1) texlive/2013 3) asciidoc/8.6.9 5) subversion/1.7.17 7)
emacs/24.3 9) autotools/1.0 11) epd/7.3.2 13) gdb/7.6.2
15) paraview/4.1.0 17) mpich/3.0.4 19) vtk/5.10.1 21)
petsc/3.5.2-debug
2) texinfo/5.2 4) git/1.8.5.3 6) svn2git/2.3.2 8) cmake/
2.8.12.2 10) doxygen/1.8.6 12) graphviz/2.36.0 14) valgrind/3.9.0 16)
gcc/4.8.2 18) openblas/0.2.9.rc1 20) tbb/4.2
configure line:
../../libMesh/configure
--prefix=/fry1/data/users/pbauman/software/libs/libmesh/master
--enable-everything --with-metis=PETSc --enable-parmesh
Building and running the installed introduction/ex4 as is in dbg mode gives
the attached dbg_output.txt
If I run the devel version, it hangs. Here's the stack trace attached to
one of the two processes:
(gdb) where
#0 0x00007f14008350f3 in MPIDI_CH3I_Progress () from
/fry1/data/users/pbauman/software/libs/mpich/3.0.4/gcc/4.8.2/lib/libmpich.so.10
#1 0x00007f14008279a5 in MPID_Probe () from
/fry1/data/users/pbauman/software/libs/mpich/3.0.4/gcc/4.8.2/lib/libmpich.so.10
#2 0x00007f14008de396 in PMPI_Probe () from
/fry1/data/users/pbauman/software/libs/mpich/3.0.4/gcc/4.8.2/lib/libmpich.so.10
#3 0x00007f14063c9818 in probe (tag=..., src_processor_id=1,
this=0x7fff31c58078) at ./include/libmesh/parallel_implementation.h:1931
#4 receive<unsigned int> (tag=..., type=..., buf=..., src_processor_id=1,
this=0x7fff31c58078) at ./include/libmesh/parallel_implementation.h:2504
#5 send_receive<unsigned int, unsigned int> (recv_tag=..., send_tag=...,
type2=..., recv=..., source_processor_id=1, type1=..., sendvec=...,
dest_processor_id=<optimized out>, this=0x7fff31c58078)
at ./include/libmesh/parallel_implementation.h:2632
#6 libMesh::Parallel::Communicator::send_receive<unsigned int>
(this=0x7fff31c58078, dest_processor_id=<optimized out>, sendvec=...,
source_processor_id=1, recv=..., send_tag=..., recv_tag=...)
at ./include/libmesh/parallel_implementation.h:2697
#7 0x00007f14063f1499 in libMesh::DofMap::scatter_constraints
(this=this@entry=0xa625d0, mesh=...) at
../../libMesh/src/base/dof_map_constraints.C:3402
#8 0x00007f14063f42d8 in libMesh::DofMap::process_constraints
(this=0xa625d0, mesh=...) at
../../libMesh/src/base/dof_map_constraints.C:3229
#9 0x00007f1406b263d7 in libMesh::System::init_data (this=this@entry=0xa62240)
at ../../libMesh/src/systems/system.C:268
#10 0x00007f1406ae8c52 in libMesh::ExplicitSystem::init_data
(this=this@entry=0xa62240) at ../../libMesh/src/systems/explicit_system.C:72
#11 0x00007f1406b0d8a4 in libMesh::ImplicitSystem::init_data
(this=0xa62240) at ../../libMesh/src/systems/implicit_system.C:89
#12 0x00007f1406b102c9 in libMesh::LinearImplicitSystem::init_data
(this=0xa62240) at ../../libMesh/src/systems/linear_implicit_system.C:77
#13 0x00007f1406b26819 in libMesh::System::init (this=0xa62240) at
../../libMesh/src/systems/system.C:235
#14 0x00007f1406ae03e7 in libMesh::EquationSystems::init
(this=this@entry=0x7fff31c58130)
at ../../libMesh/src/systems/equation_systems.C:121
#15 0x000000000042aa1c in main (argc=<optimized out>, argv=<optimized out>)
at introduction_ex4.C:287
(gdb) quit
opt also hangs.
Note I see the same behavior minus the PARMETIS ERROR is I switch to a
different partitioner (MetisPartitioner and LinearPartitioner both worked
in dbg and hung in devel and opt).
I'm going to try with OpenMPI and see if this is replicated or not.
Running ./example-dbg -d 2 -n 15 -ksp_monitor
PARMETIS ERROR: The sum of tpwgts for constraint #0 is not 1.0
PARMETIS ERROR: The sum of tpwgts for constraint #0 is not 1.0
Mesh Information:
mesh_dimension()=2
spatial_dimension()=3
n_nodes()=961
n_local_nodes()=513
n_elem()=225
n_local_elem()=114
n_active_elem()=225
n_subdomains()=1
n_partitions()=2
n_processors()=2
n_threads()=1
processor_id()=0
EquationSystems
n_systems()=1
System #0, "Poisson"
Type "LinearImplicit"
Variables="u"
Finite Element Types="LAGRANGE", "JACOBI_20_00"
Infinite Element Mapping="CARTESIAN"
Approximation Orders="SECOND", "THIRD"
n_dofs()=961
n_local_dofs()=513
n_constrained_dofs()=120
n_local_constrained_dofs()=62
n_vectors()=1
n_matrices()=1
DofMap Sparsity
Average On-Processor Bandwidth <= 14.7336
Average Off-Processor Bandwidth <= 0.720083
Maximum On-Processor Bandwidth <= 26
Maximum Off-Processor Bandwidth <= 15
DofMap Constraints
Number of DoF Constraints = 120
Number of Heterogenous Constraints= 118
Average DoF Constraint Length= 0
Number of Node Constraints = 0
Mesh Information:
mesh_dimension()=2
spatial_dimension()=3
n_nodes()=961
n_local_nodes()=513
n_elem()=225
n_local_elem()=114
n_active_elem()=225
n_subdomains()=1
n_partitions()=2
n_processors()=2
n_threads()=1
processor_id()=0
-------------------------------------------------------------------------------------------
| Processor id: 0
|
| Num Processors: 2
|
| Time: Mon Jan 12 11:22:46 2015
|
| OS: Linux
|
| HostName: fry.eng.buffalo.edu
|
| OS Release: 2.6.32-504.3.3.el6.x86_64
|
| OS Version: #1 SMP Fri Dec 12 16:05:43 EST 2014
|
| Machine: x86_64
|
| Username: pbauman
|
| Configuration: ../../libMesh/configure
'--prefix=/fry1/data/users/pbauman/software/libs/libmesh/master'|
| '--enable-everything'
|
| '--with-metis=PETSc'
|
| '--enable-parmesh'
|
| 'CXX=g++'
|
| 'CC=gcc'
|
| 'FC=gfortran'
|
| 'F77=gfortran'
|
| 'PETSC_DIR=/fry1/data/users/pbauman/software/libs/petsc/petsc-3.5.2'
|
| 'PETSC_ARCH=gcc-4.8.2-mpich-3.0.4-openblas-0.2.9.rc1-cxx-opt'
|
|
'VTK_INCLUDE=/fry1/data/users/pbauman/software/libs/vtk/5.10.1/gcc/4.8.2/include/vtk-5.10'|
| 'VTK_DIR=/fry1/data/users/pbauman/software/libs/vtk/5.10.1/gcc/4.8.2'
|
-------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------
| Matrix Assembly Performance: Alive time=0.032835, Active time=0.025345
|
------------------------------------------------------------------------------------------------------------
| Event nCalls Total Time Avg Time Total Time
Avg Time % of Active Time |
| w/o Sub w/o Sub With Sub
With Sub w/o S With S |
|------------------------------------------------------------------------------------------------------------|
|
|
| Fe 114 0.0012 0.000010 0.0012
0.000010 4.65 4.65 |
| Ke 114 0.0068 0.000059 0.0068
0.000059 26.71 26.71 |
| elem init 114 0.0167 0.000147 0.0167
0.000147 66.06 66.06 |
| matrix insertion 114 0.0007 0.000006 0.0007
0.000006 2.59 2.59 |
------------------------------------------------------------------------------------------------------------
| Totals: 456 0.0253
100.00 |
------------------------------------------------------------------------------------------------------------
0 KSP Residual norm 6.281555363893e+00
1 KSP Residual norm 1.441578986136e+00
2 KSP Residual norm 8.657661935565e-01
3 KSP Residual norm 6.298483075757e-01
4 KSP Residual norm 4.584797396291e-01
5 KSP Residual norm 3.672004215548e-01
6 KSP Residual norm 2.854933103510e-01
7 KSP Residual norm 2.207387889462e-01
8 KSP Residual norm 1.507528183709e-01
9 KSP Residual norm 7.609993007381e-02
10 KSP Residual norm 3.870496969236e-02
11 KSP Residual norm 1.694470333105e-02
12 KSP Residual norm 1.056205718874e-02
13 KSP Residual norm 7.962944594936e-03
14 KSP Residual norm 6.247891445290e-03
15 KSP Residual norm 5.142557174263e-03
16 KSP Residual norm 4.064335778374e-03
17 KSP Residual norm 2.986839198661e-03
18 KSP Residual norm 2.212248591674e-03
19 KSP Residual norm 1.711006399209e-03
20 KSP Residual norm 1.423591127897e-03
21 KSP Residual norm 1.228658042495e-03
22 KSP Residual norm 1.021344123622e-03
23 KSP Residual norm 8.969047514369e-04
24 KSP Residual norm 7.995662891498e-04
25 KSP Residual norm 6.652930701941e-04
26 KSP Residual norm 5.304353279729e-04
27 KSP Residual norm 4.030242592245e-04
28 KSP Residual norm 2.730993578525e-04
29 KSP Residual norm 1.645810058886e-04
30 KSP Residual norm 1.103538427762e-04
31 KSP Residual norm 8.610454940155e-05
32 KSP Residual norm 6.498404359506e-05
33 KSP Residual norm 4.556983417930e-05
34 KSP Residual norm 3.231154166980e-05
35 KSP Residual norm 2.577303633035e-05
36 KSP Residual norm 2.113582798346e-05
37 KSP Residual norm 1.752446809300e-05
38 KSP Residual norm 1.484672477927e-05
39 KSP Residual norm 1.302629184089e-05
40 KSP Residual norm 1.172471062290e-05
41 KSP Residual norm 1.006971964980e-05
42 KSP Residual norm 8.230366349966e-06
43 KSP Residual norm 6.171634470124e-06
44 KSP Residual norm 4.117409795018e-06
45 KSP Residual norm 2.611987673662e-06
46 KSP Residual norm 1.745591647676e-06
47 KSP Residual norm 1.190676355730e-06
48 KSP Residual norm 9.028453572394e-07
49 KSP Residual norm 7.224248002419e-07
50 KSP Residual norm 5.339195753159e-07
51 KSP Residual norm 3.748508543720e-07
52 KSP Residual norm 2.677517396278e-07
53 KSP Residual norm 1.840889093657e-07
54 KSP Residual norm 1.207493202252e-07
55 KSP Residual norm 7.670458622698e-08
56 KSP Residual norm 4.744749826056e-08
57 KSP Residual norm 2.888924255267e-08
58 KSP Residual norm 1.855668558305e-08
59 KSP Residual norm 1.225261973163e-08
60 KSP Residual norm 7.274693375029e-09
61 KSP Residual norm 5.678840027999e-09
62 KSP Residual norm 4.272621003973e-09
63 KSP Residual norm 3.297977189053e-09
64 KSP Residual norm 2.507083460763e-09
65 KSP Residual norm 1.834020100457e-09
66 KSP Residual norm 1.337490896343e-09
67 KSP Residual norm 1.047810076369e-09
68 KSP Residual norm 7.828493744498e-10
69 KSP Residual norm 5.516829101298e-10
70 KSP Residual norm 3.616568657065e-10
71 KSP Residual norm 2.576313931394e-10
72 KSP Residual norm 2.089089652561e-10
73 KSP Residual norm 1.754670989786e-10
74 KSP Residual norm 1.516055264043e-10
75 KSP Residual norm 1.314547789622e-10
76 KSP Residual norm 1.151571443845e-10
77 KSP Residual norm 1.002170334358e-10
78 KSP Residual norm 8.399470485732e-11
79 KSP Residual norm 6.720961716070e-11
80 KSP Residual norm 4.385649839608e-11
81 KSP Residual norm 2.932111979815e-11
82 KSP Residual norm 1.808693464627e-11
83 KSP Residual norm 1.155116888517e-11
84 KSP Residual norm 7.787886447474e-12
85 KSP Residual norm 5.539837062396e-12
Warning: This MeshOutput subclass only supports meshes which have been
serialized!
Warning: This MeshOutput subclass only supports meshes which have been
serialized!
----------------------------------------------------------------------------
| Reference count information |
----------------------------------------------------------------------------
| N7libMesh10FEAbstractE reference count information:
| Creations: 4
| Destructions: 4
| N7libMesh10Parameters5ValueE reference count information:
| Creations: 2
| Destructions: 2
| N7libMesh12LinearSolverIdEE reference count information:
| Creations: 1
| Destructions: 1
| N7libMesh12SparseMatrixIdEE reference count information:
| Creations: 1
| Destructions: 1
| N7libMesh13NumericVectorIdEE reference count information:
| Creations: 5
| Destructions: 5
| N7libMesh15EquationSystemsE reference count information:
| Creations: 1
| Destructions: 1
| N7libMesh4ElemE reference count information:
| Creations: 2632
| Destructions: 2632
| N7libMesh4NodeE reference count information:
| Creations: 1333
| Destructions: 1333
| N7libMesh5QBaseE reference count information:
| Creations: 5
| Destructions: 5
| N7libMesh6DofMapE reference count information:
| Creations: 1
| Destructions: 1
| N7libMesh6SystemE reference count information:
| Creations: 1
| Destructions: 1
| N7libMesh9DofObjectE reference count information:
| Creations: 3965
| Destructions: 3965
----------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------------------------
| libMesh Performance: Alive time=1.62748, Active time=1.36299
|
---------------------------------------------------------------------------------------------------------------------
| Event nCalls Total Time Avg Time
Total Time Avg Time % of Active Time |
| w/o Sub w/o Sub
With Sub With Sub w/o S With S |
|---------------------------------------------------------------------------------------------------------------------|
|
|
|
|
| DofMap
|
| add_neighbors_to_send_list() 1 0.0050 0.005017
0.0073 0.007321 0.37 0.54 |
| build_constraint_matrix_and_vector() 114 0.0018 0.000016
0.0018 0.000016 0.13 0.13 |
| build_sparsity() 1 0.0313 0.031347
0.0378 0.037773 2.30 2.77 |
| create_dof_constraints() 1 0.0072 0.007207
0.0170 0.017019 0.53 1.25 |
| distribute_dofs() 1 0.1193 0.119323
0.6241 0.624148 8.75 45.79 |
| dof_indices() 524 0.0257 0.000049
0.0257 0.000049 1.89 1.89 |
| hetero_cnstrn_elem_mat_vec() 114 0.0011 0.000010
0.0011 0.000010 0.08 0.08 |
| prepare_send_list() 1 0.0006 0.000619
0.0006 0.000619 0.05 0.05 |
| reinit() 1 0.0148 0.014765
0.0148 0.014765 1.08 1.08 |
|
|
| EquationSystems
|
| build_solution_vector() 1 0.0019 0.001909
0.0077 0.007734 0.14 0.57 |
|
|
| ExodusII_IO
|
| write_nodal_data() 1 0.0057 0.005673
0.0060 0.005965 0.42 0.44 |
|
|
| FE
|
| compute_shape_functions() 144 0.0058 0.000040
0.0058 0.000040 0.43 0.43 |
| init_shape_functions() 31 0.0002 0.000007
0.0002 0.000007 0.02 0.02 |
| inverse_map() 90 0.0009 0.000010
0.0009 0.000010 0.07 0.07 |
|
|
| FEMap
|
| compute_affine_map() 144 0.0025 0.000018
0.0025 0.000018 0.19 0.19 |
| compute_face_map() 30 0.0009 0.000028
0.0019 0.000063 0.06 0.14 |
| init_face_shape_functions() 30 0.0002 0.000007
0.0002 0.000007 0.02 0.02 |
| init_reference_to_physical_map() 31 0.0009 0.000029
0.0009 0.000029 0.07 0.07 |
|
|
| Mesh
|
| find_neighbors() 2 0.0289 0.014471
0.0317 0.015828 2.12 2.32 |
|
|
| MeshCommunication
|
| (all)gather() 1 0.0438 0.043773
0.0580 0.057992 3.21 4.25 |
| compute_hilbert_indices() 4 0.0049 0.001213
0.0049 0.001213 0.36 0.36 |
| delete_remote_elements() 3 0.0104 0.003473
0.0119 0.003965 0.76 0.87 |
| find_global_indices() 4 0.0106 0.002642
0.0217 0.005420 0.78 1.59 |
| parallel_sort() 4 0.0042 0.001053
0.0054 0.001358 0.31 0.40 |
|
|
| MeshOutput
|
| write_equation_systems() 1 0.0132 0.013242
0.1180 0.117971 0.97 8.66 |
|
|
| MeshTools::Generation
|
| build_cube() 1 0.0141 0.014134
0.0141 0.014134 1.04 1.04 |
|
|
| Parallel
|
| allgather() 24 0.0013 0.000052
0.0018 0.000074 0.09 0.13 |
| broadcast() 1 0.0000 0.000009
0.0000 0.000009 0.00 0.00 |
| max(bool) 8310 0.0544 0.000007
0.0544 0.000007 3.99 3.99 |
| max(scalar) 19698 0.1212 0.000006
0.1212 0.000006 8.90 8.90 |
| max(vector) 3763 0.0506 0.000013
0.1410 0.000037 3.71 10.34 |
| max(vector<bool>) 3 0.0007 0.000224
0.0008 0.000252 0.05 0.06 |
| min(bool) 8158 0.0525 0.000006
0.0525 0.000006 3.85 3.85 |
| min(scalar) 61241 0.3972 0.000006
0.3972 0.000006 29.14 29.14 |
| min(vector) 3763 0.0518 0.000014
0.1450 0.000039 3.80 10.64 |
| probe() 67 0.0016 0.000024
0.0016 0.000024 0.12 0.12 |
| receive() 67 0.0006 0.000009
0.0023 0.000035 0.04 0.17 |
| send() 67 0.0003 0.000004
0.0003 0.000004 0.02 0.02 |
| send_receive() 98 0.0011 0.000012
0.0042 0.000043 0.08 0.31 |
| sum() 52 0.0008 0.000016
0.0016 0.000030 0.06 0.11 |
|
|
| Parallel::Request
|
| wait() 70 0.0002 0.000003
0.0002 0.000003 0.01 0.01 |
|
|
| ParallelMesh
|
| renumber_nodes_and_elements() 2 0.0722 0.036124
0.2182 0.109118 5.30 16.01 |
|
|
| ParmetisPartitioner
|
| repartition() 1 0.1673 0.167309
0.1818 0.181825 12.28 13.34 |
|
|
| Partitioner
|
| set_node_processor_ids() 1 0.0097 0.009723
0.0265 0.026458 0.71 1.94 |
| set_parent_processor_ids() 1 0.0007 0.000657
0.0007 0.000657 0.05 0.05 |
|
|
| PetscLinearSolver
|
| solve() 1 0.0069 0.006875
0.0069 0.006875 0.50 0.50 |
|
|
| System
|
| assemble() 1 0.0158 0.015771
0.0331 0.033131 1.16 2.43 |
---------------------------------------------------------------------------------------------------------------------
| Totals: 106669 1.3630
100.00 |
---------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------
New Year. New Location. New Benefits. New Data Center in Ashburn, VA.
GigeNET is offering a free month of service with a new server in Ashburn.
Choose from 2 high performing configs, both with 100TB of bandwidth.
Higher redundancy.Lower latency.Increased capacity.Completely compliant.
www.gigenet.com
_______________________________________________
Libmesh-devel mailing list
Libmesh-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/libmesh-devel
