Hi Jed,
As to your previous comment, I am attaching the full error message
running under valgrind. All valgrind errors seem to have to do with the
low-level Petsc functions; none of the valgrind errors are related to my
application code.
Another odd behavior of this error is that it depends on when
SNESSetFunctionDomainError() is invoked. I called the function in the
libmesh residual_and_jacobian() function in the following way:
if( my_metric > a_constant ){
libMesh::PetscNonlinearSolver<Real> *tp =
dynamic_cast<libMesh::PetscNonlinearSolver<Real>*>(system.nonlinear_solver.get());
SNESSetFunctionDomainError(tp->snes() );
}
where my_metric and a_constant are two numeric values. When a_constant <
1e3, SNESSetFunctionDomainError() terminates the ; however, when
a_constant > 1e3, I got the aforementioned Petsc error. Although this
if-condition is specific to my own program, I believe that *the numeric
value should not affect the logical operation of
SNESSetFunctionDomainError()*. Any comments about this odd behavior?
*_The error reported by valgrind:_*
[5]PETSC ERROR: VecAssemblyBegin_MPI() line 1012 in
/home/dwang/Program/petsc/3.4.3/src/vec/vec/impls/mpi/pdvec.c
[5]PETSC ERROR: VecAssemblyBegin() line 220 in
/home/dwang/Program/petsc/3.4.3/src/vec/vec/interface/vector.c
[5]PETSC ERROR: close() line 953 in
"unknowndirectory/"/home/dwang/Program/libmesh/0.9.2.2/RelWithDebInfo/include/libmesh/petsc_vector.h
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 5 in communicator MPI_COMM_WORLD
with errorcode 1.
NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
[0]PETSC ERROR: VecAssemblyBegin_MPI() line 1012 in
/home/dwang/Program/petsc/3.4.3/src/vec/vec/impls/mpi/pdvec.c
[0]PETSC ERROR: VecAssemblyBegin() line 220 in
/home/dwang/Program/petsc/3.4.3/src/vec/vec/interface/vector.c
[0]PETSC ERROR: close() line 953 in
"unknowndirectory/"/home/dwang/Program/libmesh/0.9.2.2/RelWithDebInfo/include/libmesh/petsc_vector.h
--------------------------------------------------------------------------
mpiexec has exited due to process rank 5 with PID 14474 on
node wonderful exiting without calling "finalize". This may
have caused other processes in the application to be
terminated by signals sent by mpiexec (as reported here).
--------------------------------------------------------------------------
[wonderful:14454] 2 more processes have sent help message
help-mpi-api.txt / mpi-abort
[wonderful:14454] Set MCA parameter "orte_base_help_aggregate" to 0 to
see all help / error messages
==14454==
==14454== HEAP SUMMARY:
==14454== in use at exit: 299,640 bytes in 1,642 blocks
==14454== total heap usage: 10,415 allocs, 8,773 frees, 15,363,253
bytes allocated
==14454==
==14454== 4 bytes in 1 blocks are definitely lost in loss record 118 of
1,431
==14454== at 0x4C2B6CD: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14454== by 0x60BC010: ???
==14454== by 0x60BD716: ???
==14454== by 0x60BB8E8: ???
==14454== by 0x60BC3C9: ???
==14454== by 0x60BB37F: ???
==14454== by 0x50BC48C: opal_paffinity_base_select (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x5092963: opal_init (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x4E3FA0C: orte_init (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x402FE4: ??? (in /usr/bin/orterun)
==14454== by 0x402B33: ??? (in /usr/bin/orterun)
==14454== by 0x551576C: (below main) (libc-start.c:226)
==14454==
==14454== 4 bytes in 1 blocks are definitely lost in loss record 119 of
1,431
==14454== at 0x4C2B6CD: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14454== by 0x60BC030: ???
==14454== by 0x60BD716: ???
==14454== by 0x60BB8E8: ???
==14454== by 0x60BC3C9: ???
==14454== by 0x60BB37F: ???
==14454== by 0x50BC48C: opal_paffinity_base_select (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x5092963: opal_init (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x4E3FA0C: orte_init (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x402FE4: ??? (in /usr/bin/orterun)
==14454== by 0x402B33: ??? (in /usr/bin/orterun)
==14454== by 0x551576C: (below main) (libc-start.c:226)
==14454==
==14454== 4 bytes in 1 blocks are definitely lost in loss record 120 of
1,431
==14454== at 0x4C2B6CD: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14454== by 0x557CAB1: strdup (strdup.c:43)
==14454== by 0x50A3AC5: ??? (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x50A475D: ??? (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x50A4B1F: mca_base_param_reg_string (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x8F458DF: ???
==14454== by 0x50A1E24: mca_base_components_open (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x4E58E44: orte_filem_base_open (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x70CDE85: ???
==14454== by 0x4E3FB72: orte_init (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x402FE4: ??? (in /usr/bin/orterun)
==14454== by 0x402B33: ??? (in /usr/bin/orterun)
==14454==
==14454== 4 bytes in 1 blocks are definitely lost in loss record 121 of
1,431
==14454== at 0x4C2B6CD: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14454== by 0x557CAB1: strdup (strdup.c:43)
==14454== by 0x50A3AC5: ??? (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x50A475D: ??? (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x50A4B1F: mca_base_param_reg_string (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x8F4590C: ???
==14454== by 0x50A1E24: mca_base_components_open (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x4E58E44: orte_filem_base_open (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x70CDE85: ???
==14454== by 0x4E3FB72: orte_init (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x402FE4: ??? (in /usr/bin/orterun)
==14454== by 0x402B33: ??? (in /usr/bin/orterun)
==14454==
==14454== 5 bytes in 1 blocks are definitely lost in loss record 201 of
1,431
==14454== at 0x4C2B6CD: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14454== by 0x50B4737: opal_crs_base_open (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x50930E2: opal_cr_init (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x5092A58: opal_init (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x4E3FA0C: orte_init (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x402FE4: ??? (in /usr/bin/orterun)
==14454== by 0x402B33: ??? (in /usr/bin/orterun)
==14454== by 0x551576C: (below main) (libc-start.c:226)
==14454==
==14454== 5 bytes in 1 blocks are definitely lost in loss record 202 of
1,431
==14454== at 0x4C2B6CD: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14454== by 0x557CAB1: strdup (strdup.c:43)
==14454== by 0x50A3AC5: ??? (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x50A475D: ??? (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x50A3C56: mca_base_param_reg_string_name (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x4E6869E: orte_rmaps_base_open (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x70CDAFB: ???
==14454== by 0x4E3FB72: orte_init (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x402FE4: ??? (in /usr/bin/orterun)
==14454== by 0x402B33: ??? (in /usr/bin/orterun)
==14454== by 0x551576C: (below main) (libc-start.c:226)
==14454==
==14454== 5 bytes in 1 blocks are definitely lost in loss record 203 of
1,431
==14454== at 0x4C2B6CD: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14454== by 0x557CAB1: strdup (strdup.c:43)
==14454== by 0x4E4A7AC: orte_session_dir_get_name (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x4E4A961: orte_session_dir (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x70CDC79: ???
==14454== by 0x4E3FB72: orte_init (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x402FE4: ??? (in /usr/bin/orterun)
==14454== by 0x402B33: ??? (in /usr/bin/orterun)
==14454== by 0x551576C: (below main) (libc-start.c:226)
==14454==
==14454== 5 bytes in 1 blocks are definitely lost in loss record 204 of
1,431
==14454== at 0x4C2B6CD: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14454== by 0x4E6B840: orte_snapc_base_select (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x70CDF16: ???
==14454== by 0x4E3FB72: orte_init (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x402FE4: ??? (in /usr/bin/orterun)
==14454== by 0x402B33: ??? (in /usr/bin/orterun)
==14454== by 0x551576C: (below main) (libc-start.c:226)
==14454==
==14454== 6 bytes in 1 blocks are definitely lost in loss record 259 of
1,431
==14454== at 0x4C2B6CD: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14454== by 0x557CAB1: strdup (strdup.c:43)
==14454== by 0x50A3AC5: ??? (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x50A475D: ??? (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x50A4B1F: mca_base_param_reg_string (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x76E22FE: ???
==14454== by 0x50A1E24: mca_base_components_open (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x4E63D10: mca_oob_base_open (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x74D9D18: ???
==14454== by 0x50A1E24: mca_base_components_open (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x4E6A2BD: orte_rml_base_open (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x70CD952: ???
==14454==
==14454== 10 bytes in 1 blocks are definitely lost in loss record 371 of
1,431
==14454== at 0x4C2B6CD: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14454== by 0x5097481: ??? (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x5096340: opal_dss_unpack_buffer (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x50972FC: ??? (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x4E53303: orte_util_decode_nodemap (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x4E5FFD2: orte_odls_base_default_update_daemon_info (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x4E57390: orte_daemon_cmd_processor (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x509D9F1: ??? (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x5092220: opal_progress (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x4E65794: orte_plm_base_launch_apps (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x72D465E: ???
==14454== by 0x403769: ??? (in /usr/bin/orterun)
==14454==
==14454== 16 bytes in 1 blocks are definitely lost in loss record 503 of
1,431
==14454== at 0x4C2B6CD: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14454== by 0x557CAB1: strdup (strdup.c:43)
==14454== by 0x50A3AC5: ??? (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x50A475D: ??? (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x50A4B1F: mca_base_param_reg_string (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x80F6DFC: ???
==14454== by 0x50A1E24: mca_base_components_open (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x4E67B11: orte_ras_base_open (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x70CDAA5: ???
==14454== by 0x4E3FB72: orte_init (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x402FE4: ??? (in /usr/bin/orterun)
==14454== by 0x402B33: ??? (in /usr/bin/orterun)
==14454==
==14454== 16 bytes in 1 blocks are definitely lost in loss record 504 of
1,431
==14454== at 0x4C2B6CD: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14454== by 0x5097728: ??? (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x5096340: opal_dss_unpack_buffer (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x50972FC: ??? (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x4E5FF6B: orte_odls_base_default_update_daemon_info (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x4E57390: orte_daemon_cmd_processor (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x509D9F1: ??? (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x5092220: opal_progress (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x4E65794: orte_plm_base_launch_apps (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x72D465E: ???
==14454== by 0x403769: ??? (in /usr/bin/orterun)
==14454== by 0x402B33: ??? (in /usr/bin/orterun)
==14454==
==14454== 16 bytes in 1 blocks are definitely lost in loss record 505 of
1,431
==14454== at 0x4C2B6CD: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14454== by 0x5097728: ??? (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x5096340: opal_dss_unpack_buffer (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x50972FC: ??? (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x4E60045: orte_odls_base_default_update_daemon_info (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x4E57390: orte_daemon_cmd_processor (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x509D9F1: ??? (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x5092220: opal_progress (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x4E65794: orte_plm_base_launch_apps (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x72D465E: ???
==14454== by 0x403769: ??? (in /usr/bin/orterun)
==14454== by 0x402B33: ??? (in /usr/bin/orterun)
==14454==
==14454== 16 bytes in 1 blocks are definitely lost in loss record 506 of
1,431
==14454== at 0x4C2B6CD: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14454== by 0x5097728: ??? (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x5096340: opal_dss_unpack_buffer (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x50972FC: ??? (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x4E607BC: orte_odls_base_default_construct_child_list
(in /usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x80F694D: ???
==14454== by 0x4E5553C: ??? (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x4E571D6: orte_daemon_cmd_processor (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x509D9F1: ??? (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x5092220: opal_progress (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x4E65794: orte_plm_base_launch_apps (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x72D465E: ???
==14454==
==14454== 20 bytes in 1 blocks are definitely lost in loss record 558 of
1,431
==14454== at 0x4C2B6CD: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14454== by 0x557CAB1: strdup (strdup.c:43)
==14454== by 0x50A6705: opal_basename (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x4E4C1B9: orte_util_check_context_app (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x4E614B3: orte_odls_base_default_launch_local (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x80F6961: ???
==14454== by 0x4E5553C: ??? (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x4E571D6: orte_daemon_cmd_processor (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x509D9F1: ??? (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x5092220: opal_progress (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x4E65794: orte_plm_base_launch_apps (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x72D465E: ???
==14454==
==14454== 27 bytes in 1 blocks are definitely lost in loss record 623 of
1,431
==14454== at 0x4C2B6CD: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14454== by 0x55FE887: __vasprintf_chk (vasprintf_chk.c:82)
==14454== by 0x55FE731: __asprintf_chk (asprintf_chk.c:34)
==14454== by 0x50AC59E: opal_setenv (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x402FC0: ??? (in /usr/bin/orterun)
==14454== by 0x402B33: ??? (in /usr/bin/orterun)
==14454== by 0x551576C: (below main) (libc-start.c:226)
==14454==
==14454== 32 bytes in 1 blocks are definitely lost in loss record 759 of
1,431
==14454== at 0x4C2B6CD: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14454== by 0x55FE887: __vasprintf_chk (vasprintf_chk.c:82)
==14454== by 0x55FE731: __asprintf_chk (asprintf_chk.c:34)
==14454== by 0x4E6C437: orte_snapc_base_unique_global_snapshot_name
(in /usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x4E6B78B: orte_snapc_base_open (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x70CDEDB: ???
==14454== by 0x4E3FB72: orte_init (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x402FE4: ??? (in /usr/bin/orterun)
==14454== by 0x402B33: ??? (in /usr/bin/orterun)
==14454== by 0x551576C: (below main) (libc-start.c:226)
==14454==
==14454== 48 bytes in 1 blocks are definitely lost in loss record 1,094
of 1,431
==14454== at 0x4C2B6CD: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14454== by 0x74DA592: ???
==14454== by 0x8D403E1: ???
==14454== by 0x50A24BF: mca_base_select (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x4E5C465: orte_iof_base_select (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x70CDE57: ???
==14454== by 0x4E3FB72: orte_init (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x402FE4: ??? (in /usr/bin/orterun)
==14454== by 0x402B33: ??? (in /usr/bin/orterun)
==14454== by 0x551576C: (below main) (libc-start.c:226)
==14454==
==14454== 55 bytes in 1 blocks are definitely lost in loss record 1,151
of 1,431
==14454== at 0x4C2B6CD: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14454== by 0x557CAB1: strdup (strdup.c:43)
==14454== by 0x50A3AC5: ??? (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x50A475D: ??? (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x50A3C56: mca_base_param_reg_string_name (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x50ABF7C: opal_net_init (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x509282C: opal_init_util (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x402D90: ??? (in /usr/bin/orterun)
==14454== by 0x402B33: ??? (in /usr/bin/orterun)
==14454== by 0x551576C: (below main) (libc-start.c:226)
==14454==
==14454== 64 bytes in 1 blocks are definitely lost in loss record 1,206
of 1,431
==14454== at 0x4C2B6CD: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14454== by 0x50AD1AD: opal_os_path (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x4E62B6E: orte_base_default_waitpid_fired (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x4E5640C: ??? (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x4E57035: orte_daemon_cmd_processor (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x509D9F1: ??? (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x403783: ??? (in /usr/bin/orterun)
==14454== by 0x402B33: ??? (in /usr/bin/orterun)
==14454== by 0x551576C: (below main) (libc-start.c:226)
==14454==
==14454== 80 bytes in 8 blocks are definitely lost in loss record 1,236
of 1,431
==14454== at 0x4C2B6CD: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14454== by 0x557CAB1: strdup (strdup.c:43)
==14454== by 0x4080D6: ??? (in /usr/bin/orterun)
==14454== by 0x40377E: ??? (in /usr/bin/orterun)
==14454== by 0x402B33: ??? (in /usr/bin/orterun)
==14454== by 0x551576C: (below main) (libc-start.c:226)
==14454==
==14454== 88 (16 direct, 72 indirect) bytes in 1 blocks are definitely
lost in loss record 1,247 of 1,431
==14454== at 0x4C2B6CD: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14454== by 0x4E45C2C: ??? (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x5096340: opal_dss_unpack_buffer (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x50972FC: ??? (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x4E6076F: orte_odls_base_default_construct_child_list
(in /usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x80F694D: ???
==14454== by 0x4E5553C: ??? (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x4E571D6: orte_daemon_cmd_processor (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x509D9F1: ??? (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x5092220: opal_progress (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x4E65794: orte_plm_base_launch_apps (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x72D465E: ???
==14454==
==14454== 128 bytes in 1 blocks are definitely lost in loss record 1,348
of 1,431
==14454== at 0x4C2B6CD: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14454== by 0x74D9944: ???
==14454== by 0x4E6A3E4: orte_rml_base_select (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x70CD95E: ???
==14454== by 0x4E3FB72: orte_init (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x402FE4: ??? (in /usr/bin/orterun)
==14454== by 0x402B33: ??? (in /usr/bin/orterun)
==14454== by 0x551576C: (below main) (libc-start.c:226)
==14454==
==14454== 128 bytes in 1 blocks are definitely lost in loss record 1,349
of 1,431
==14454== at 0x4C2B6CD: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14454== by 0x5093C52: ??? (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x5094AF1: ??? (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x5094B81: ??? (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x4E63397: orte_odls_base_default_collect_data (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x4E55C07: ??? (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x4E57035: orte_daemon_cmd_processor (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x509D9F1: ??? (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x403783: ??? (in /usr/bin/orterun)
==14454== by 0x402B33: ??? (in /usr/bin/orterun)
==14454== by 0x551576C: (below main) (libc-start.c:226)
==14454==
==14454== 128 bytes in 1 blocks are definitely lost in loss record 1,350
of 1,431
==14454== at 0x4C2B6CD: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14454== by 0x5093C52: ??? (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x5094AF1: ??? (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x5094B81: ??? (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x4E5E8F2: odls_base_default_wait_local_proc (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x4E47377: ??? (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x509D9F1: ??? (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x403783: ??? (in /usr/bin/orterun)
==14454== by 0x402B33: ??? (in /usr/bin/orterun)
==14454== by 0x551576C: (below main) (libc-start.c:226)
==14454==
==14454== 176 (32 direct, 144 indirect) bytes in 1 blocks are definitely
lost in loss record 1,369 of 1,431
==14454== at 0x4C2B6CD: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14454== by 0x60BC20C: ???
==14454== by 0x60BD716: ???
==14454== by 0x60BB8E8: ???
==14454== by 0x60BC3C9: ???
==14454== by 0x60BB37F: ???
==14454== by 0x50BC48C: opal_paffinity_base_select (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x5092963: opal_init (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x4E3FA0C: orte_init (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x402FE4: ??? (in /usr/bin/orterun)
==14454== by 0x402B33: ??? (in /usr/bin/orterun)
==14454== by 0x551576C: (below main) (libc-start.c:226)
==14454==
==14454== 176 bytes in 1 blocks are definitely lost in loss record 1,370
of 1,431
==14454== at 0x4C2B6CD: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14454== by 0x76E3743: ???
==14454== by 0x4E63EE2: mca_oob_base_init (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x74D97FB: ???
==14454== by 0x4E6A3E4: orte_rml_base_select (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x70CD95E: ???
==14454== by 0x4E3FB72: orte_init (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x402FE4: ??? (in /usr/bin/orterun)
==14454== by 0x402B33: ??? (in /usr/bin/orterun)
==14454== by 0x551576C: (below main) (libc-start.c:226)
==14454==
==14454== 213 bytes in 4 blocks are definitely lost in loss record 1,376
of 1,431
==14454== at 0x4C2B6CD: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14454== by 0x557CAB1: strdup (strdup.c:43)
==14454== by 0x50AF3F8: opal_show_help_vstring (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x4E4BF2E: orte_show_help (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x404656: ??? (in /usr/bin/orterun)
==14454== by 0x40382D: ??? (in /usr/bin/orterun)
==14454== by 0x509D9F1: ??? (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x5092220: opal_progress (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x4E480C1: orte_trigger_event (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x4E6655F: orte_plm_base_check_job_completed (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x4E64A0A: orte_plm_base_receive_process_msg (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x509D9F1: ??? (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454==
==14454== 300 (60 direct, 240 indirect) bytes in 1 blocks are definitely
lost in loss record 1,382 of 1,431
==14454== at 0x4C2B6CD: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14454== by 0x55F9BC4: nss_parse_service_list (nsswitch.c:678)
==14454== by 0x55FA685: __nss_database_lookup (nsswitch.c:175)
==14454== by 0x8506623: ???
==14454== by 0x55B317C: getpwuid_r@@GLIBC_2.2.5 (getXXbyYY_r.c:256)
==14454== by 0x55B2A72: getpwuid (getXXbyYY.c:117)
==14454== by 0x4E4A3B4: orte_session_dir_get_name (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x4E4A961: orte_session_dir (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x70CDC79: ???
==14454== by 0x4E3FB72: orte_init (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x402FE4: ??? (in /usr/bin/orterun)
==14454== by 0x402B33: ??? (in /usr/bin/orterun)
==14454==
==14454== 300 (60 direct, 240 indirect) bytes in 1 blocks are definitely
lost in loss record 1,383 of 1,431
==14454== at 0x4C2B6CD: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14454== by 0x55F9BC4: nss_parse_service_list (nsswitch.c:678)
==14454== by 0x55FA685: __nss_database_lookup (nsswitch.c:175)
==14454== by 0x8504169: ???
==14454== by 0x55B19EC: getgrnam_r@@GLIBC_2.2.5 (getXXbyYY_r.c:256)
==14454== by 0x5622481: grantpt (grantpt.c:153)
==14454== by 0x5AB8480: openpty (openpty.c:102)
==14454== by 0x4E5D122: orte_iof_base_setup_prefork (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x80F6A3F: ???
==14454== by 0x4E61926: orte_odls_base_default_launch_local (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x80F6961: ???
==14454== by 0x4E5553C: ??? (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454==
==14454== 304 (48 direct, 256 indirect) bytes in 1 blocks are definitely
lost in loss record 1,386 of 1,431
==14454== at 0x4C2B6CD: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14454== by 0x76E4377: ???
==14454== by 0x74DA4D6: ???
==14454== by 0x4E6ACA3: orte_rml_base_update_contact_info (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x78F0C9F: ???
==14454== by 0x4E600B3: orte_odls_base_default_update_daemon_info (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x4E57390: orte_daemon_cmd_processor (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x509D9F1: ??? (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x5092220: opal_progress (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x4E65794: orte_plm_base_launch_apps (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x72D465E: ???
==14454== by 0x403769: ??? (in /usr/bin/orterun)
==14454==
==14454== 320 (288 direct, 32 indirect) bytes in 1 blocks are definitely
lost in loss record 1,387 of 1,431
==14454== at 0x4C2B6CD: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14454== by 0x74DB01D: ???
==14454== by 0x4E64A89: orte_plm_base_comm_start (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x72D36E8: ???
==14454== by 0x70CDA7A: ???
==14454== by 0x4E3FB72: orte_init (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x402FE4: ??? (in /usr/bin/orterun)
==14454== by 0x402B33: ??? (in /usr/bin/orterun)
==14454== by 0x551576C: (below main) (libc-start.c:226)
==14454==
==14454== 320 (288 direct, 32 indirect) bytes in 1 blocks are definitely
lost in loss record 1,388 of 1,431
==14454== at 0x4C2B6CD: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14454== by 0x74DB01D: ???
==14454== by 0x4E6AA19: orte_rml_base_comm_start (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x74D9F11: ???
==14454== by 0x70CDBEE: ???
==14454== by 0x4E3FB72: orte_init (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x402FE4: ??? (in /usr/bin/orterun)
==14454== by 0x402B33: ??? (in /usr/bin/orterun)
==14454== by 0x551576C: (below main) (libc-start.c:226)
==14454==
==14454== 320 (288 direct, 32 indirect) bytes in 1 blocks are definitely
lost in loss record 1,389 of 1,431
==14454== at 0x4C2B6CD: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14454== by 0x74DB01D: ???
==14454== by 0x4E6B519: orte_routed_base_comm_start (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x78F0D33: ???
==14454== by 0x70CDE01: ???
==14454== by 0x4E3FB72: orte_init (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x402FE4: ??? (in /usr/bin/orterun)
==14454== by 0x402B33: ??? (in /usr/bin/orterun)
==14454== by 0x551576C: (below main) (libc-start.c:226)
==14454==
==14454== 320 (288 direct, 32 indirect) bytes in 1 blocks are definitely
lost in loss record 1,390 of 1,431
==14454== at 0x4C2B6CD: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14454== by 0x74DB01D: ???
==14454== by 0x8D403CC: ???
==14454== by 0x50A24BF: mca_base_select (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x4E5C465: orte_iof_base_select (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x70CDE57: ???
==14454== by 0x4E3FB72: orte_init (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x402FE4: ??? (in /usr/bin/orterun)
==14454== by 0x402B33: ??? (in /usr/bin/orterun)
==14454== by 0x551576C: (below main) (libc-start.c:226)
==14454==
==14454== 320 (288 direct, 32 indirect) bytes in 1 blocks are definitely
lost in loss record 1,391 of 1,431
==14454== at 0x4C2B6CD: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14454== by 0x74DB01D: ???
==14454== by 0x8F46090: ???
==14454== by 0x4E58FCA: orte_filem_base_select (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x70CDEB0: ???
==14454== by 0x4E3FB72: orte_init (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x402FE4: ??? (in /usr/bin/orterun)
==14454== by 0x402B33: ??? (in /usr/bin/orterun)
==14454== by 0x551576C: (below main) (libc-start.c:226)
==14454==
==14454== 320 (288 direct, 32 indirect) bytes in 1 blocks are definitely
lost in loss record 1,392 of 1,431
==14454== at 0x4C2B6CD: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14454== by 0x74DB01D: ???
==14454== by 0x4E596B4: orte_filem_base_comm_start (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x8F4609B: ???
==14454== by 0x4E58FCA: orte_filem_base_select (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x70CDEB0: ???
==14454== by 0x4E3FB72: orte_init (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x402FE4: ??? (in /usr/bin/orterun)
==14454== by 0x402B33: ??? (in /usr/bin/orterun)
==14454== by 0x551576C: (below main) (libc-start.c:226)
==14454==
==14454== 320 (288 direct, 32 indirect) bytes in 1 blocks are definitely
lost in loss record 1,393 of 1,431
==14454== at 0x4C2B6CD: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14454== by 0x74DB01D: ???
==14454== by 0x4E48D01: orte_data_server_init (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x4035AB: ??? (in /usr/bin/orterun)
==14454== by 0x402B33: ??? (in /usr/bin/orterun)
==14454== by 0x551576C: (below main) (libc-start.c:226)
==14454==
==14454== 320 (288 direct, 32 indirect) bytes in 1 blocks are definitely
lost in loss record 1,394 of 1,431
==14454== at 0x4C2B6CD: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14454== by 0x74DB01D: ???
==14454== by 0x4E6547D: orte_plm_base_setup_job (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x72D4405: ???
==14454== by 0x403769: ??? (in /usr/bin/orterun)
==14454== by 0x402B33: ??? (in /usr/bin/orterun)
==14454== by 0x551576C: (below main) (libc-start.c:226)
==14454==
==14454== 320 (288 direct, 32 indirect) bytes in 1 blocks are definitely
lost in loss record 1,395 of 1,431
==14454== at 0x4C2B6CD: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14454== by 0x74DB01D: ???
==14454== by 0x4E65764: orte_plm_base_launch_apps (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x72D465E: ???
==14454== by 0x403769: ??? (in /usr/bin/orterun)
==14454== by 0x402B33: ??? (in /usr/bin/orterun)
==14454== by 0x551576C: (below main) (libc-start.c:226)
==14454==
==14454== 320 (288 direct, 32 indirect) bytes in 1 blocks are definitely
lost in loss record 1,396 of 1,431
==14454== at 0x4C2B6CD: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14454== by 0x74DB01D: ???
==14454== by 0x4E60F1D: orte_odls_base_default_launch_local (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x80F6961: ???
==14454== by 0x4E5553C: ??? (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x4E571D6: orte_daemon_cmd_processor (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x509D9F1: ??? (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x5092220: opal_progress (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x4E65794: orte_plm_base_launch_apps (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x72D465E: ???
==14454== by 0x403769: ??? (in /usr/bin/orterun)
==14454== by 0x402B33: ??? (in /usr/bin/orterun)
==14454==
==14454== 320 (288 direct, 32 indirect) bytes in 1 blocks are definitely
lost in loss record 1,397 of 1,431
==14454== at 0x4C2B6CD: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14454== by 0x74DB01D: ???
==14454== by 0x4E57096: orte_daemon_cmd_processor (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x509D9F1: ??? (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x403B6D: ??? (in /usr/bin/orterun)
==14454== by 0x509D9F1: ??? (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x5092220: opal_progress (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x4E480C1: orte_trigger_event (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x4E6655F: orte_plm_base_check_job_completed (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x4E64A0A: orte_plm_base_receive_process_msg (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x509D9F1: ??? (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x403783: ??? (in /usr/bin/orterun)
==14454==
==14454== 320 (288 direct, 32 indirect) bytes in 1 blocks are definitely
lost in loss record 1,398 of 1,431
==14454== at 0x4C2B6CD: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14454== by 0x74DB01D: ???
==14454== by 0x4E4BCA0: orte_show_help_recv (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x74DA80A: ???
==14454== by 0x76E4B3D: ???
==14454== by 0x76E561A: ???
==14454== by 0x76E6A20: ???
==14454== by 0x509D9F1: ??? (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x5092220: opal_progress (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x4E480C1: orte_trigger_event (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x4E562D6: ??? (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x4E571D6: orte_daemon_cmd_processor (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454==
==14454== 448 (64 direct, 384 indirect) bytes in 1 blocks are definitely
lost in loss record 1,404 of 1,431
==14454== at 0x4C2B6CD: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14454== by 0x4E47558: ??? (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x4E47EB1: orte_wait_cb (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x4E61C96: orte_odls_base_default_launch_local (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x80F6961: ???
==14454== by 0x4E5553C: ??? (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x4E571D6: orte_daemon_cmd_processor (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x509D9F1: ??? (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x5092220: opal_progress (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x4E65794: orte_plm_base_launch_apps (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x72D465E: ???
==14454== by 0x403769: ??? (in /usr/bin/orterun)
==14454==
==14454== 534 (136 direct, 398 indirect) bytes in 1 blocks are
definitely lost in loss record 1,416 of 1,431
==14454== at 0x4C2B6CD: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14454== by 0x4E4B98F: ??? (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x4E4BD6D: orte_show_help_recv (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x74DA80A: ???
==14454== by 0x76E4B3D: ???
==14454== by 0x76E561A: ???
==14454== by 0x76E6A20: ???
==14454== by 0x509D9F1: ??? (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x403783: ??? (in /usr/bin/orterun)
==14454== by 0x402B33: ??? (in /usr/bin/orterun)
==14454== by 0x551576C: (below main) (libc-start.c:226)
==14454==
==14454== 544 bytes in 8 blocks are definitely lost in loss record 1,418
of 1,431
==14454== at 0x4C2B6CD: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14454== by 0x50AD1AD: opal_os_path (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x40812A: ??? (in /usr/bin/orterun)
==14454== by 0x40377E: ??? (in /usr/bin/orterun)
==14454== by 0x402B33: ??? (in /usr/bin/orterun)
==14454== by 0x551576C: (below main) (libc-start.c:226)
==14454==
==14454== 752 (48 direct, 704 indirect) bytes in 1 blocks are definitely
lost in loss record 1,420 of 1,431
==14454== at 0x4C2B6CD: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14454== by 0x4E61B6A: orte_odls_base_default_launch_local (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x80F6961: ???
==14454== by 0x4E5553C: ??? (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x4E571D6: orte_daemon_cmd_processor (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x509D9F1: ??? (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x5092220: opal_progress (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x4E65794: orte_plm_base_launch_apps (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x72D465E: ???
==14454== by 0x403769: ??? (in /usr/bin/orterun)
==14454== by 0x402B33: ??? (in /usr/bin/orterun)
==14454== by 0x551576C: (below main) (libc-start.c:226)
==14454==
==14454== 1,024 bytes in 1 blocks are definitely lost in loss record
1,421 of 1,431
==14454== at 0x4C2B7B2: realloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14454== by 0x5093BD6: ??? (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x50946D5: ??? (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x4E633FD: orte_odls_base_default_collect_data (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x4E55C07: ??? (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x4E57035: orte_daemon_cmd_processor (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x509D9F1: ??? (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x403783: ??? (in /usr/bin/orterun)
==14454== by 0x402B33: ??? (in /usr/bin/orterun)
==14454== by 0x551576C: (below main) (libc-start.c:226)
==14454==
==14454== 1,072 (88 direct, 984 indirect) bytes in 1 blocks are
definitely lost in loss record 1,423 of 1,431
==14454== at 0x4C2B6CD: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14454== by 0x4E60CBF: orte_odls_base_default_construct_child_list
(in /usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x80F694D: ???
==14454== by 0x4E5553C: ??? (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x4E571D6: orte_daemon_cmd_processor (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x509D9F1: ??? (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x5092220: opal_progress (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x4E65794: orte_plm_base_launch_apps (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x72D465E: ???
==14454== by 0x403769: ??? (in /usr/bin/orterun)
==14454== by 0x402B33: ??? (in /usr/bin/orterun)
==14454== by 0x551576C: (below main) (libc-start.c:226)
==14454==
==14454== 1,920 (1,728 direct, 192 indirect) bytes in 6 blocks are
definitely lost in loss record 1,425 of 1,431
==14454== at 0x4C2B6CD: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14454== by 0x74DB01D: ???
==14454== by 0x4E57096: orte_daemon_cmd_processor (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==14454== by 0x509D9F1: ??? (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x403783: ??? (in /usr/bin/orterun)
==14454== by 0x402B33: ??? (in /usr/bin/orterun)
==14454== by 0x551576C: (below main) (libc-start.c:226)
==14454==
==14454== 57,750 bytes in 14 blocks are definitely lost in loss record
1,429 of 1,431
==14454== at 0x4C2B6CD: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14454== by 0x76E5047: ???
==14454== by 0x76E6A02: ???
==14454== by 0x509D9F1: ??? (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==14454== by 0x403783: ??? (in /usr/bin/orterun)
==14454== by 0x402B33: ??? (in /usr/bin/orterun)
==14454== by 0x551576C: (below main) (libc-start.c:226)
==14454==
==14454== LEAK SUMMARY:
==14454== definitely lost: 66,269 bytes in 86 blocks
==14454== indirectly lost: 3,998 bytes in 82 blocks
==14454== possibly lost: 0 bytes in 0 blocks
==14454== still reachable: 229,373 bytes in 1,474 blocks
==14454== suppressed: 0 bytes in 0 blocks
==14454== Reachable blocks (those to which a pointer was found) are not
shown.
==14454== To see them, rerun with: --leak-check=full --show-reachable=yes
==14454==
==14454== For counts of detected and suppressed errors, rerun with: -v
==14454== ERROR SUMMARY: 536 errors from 53 contexts (suppressed: 2 from 2)
valgrind --leak-check=full mpiexec -n 8
~/Research/JHU/LibmeshMechanic/build/standaloneMechanics
heart_LagIncmp_Circ.mechpar 2>&1 >|logtmp.txt
Cheers,
Dafang
On 12/30/2014 02:14 AM, Jed Brown wrote:
> Dafang Wang <[email protected]> writes:
>
>> Hi Jed,
>>
>> The full error message is attached below. Furthermore, I have reached
>> the SNES nonlinear solve termination by the printing of the message:
>> "Nonlinear solve did not converge due to DIVERGED_FUNCTION_DOMAIN
>> iterations 2"
>>
>> After this, I was executing the following code where close() was called.
>> However, I never reached Point 1 (it was never printed).
>>
>> if( SNES_converged_reason<0){
>> if(processor_id()==0){cout << "Point 1 "<< endl; }
>> fflush(NULL);
>> MPI_Barrier(PETSC_COMM_WORLD);
>> system.solution->close();
> Is it already "closed" in case of a domain error?
>
>> }
>>
>> The full error message:
>>
>> Nonlinear solve did not converge due to DIVERGED_FUNCTION_DOMAIN
>> iterations 2
>> [13]PETSC ERROR: VecAssemblyBegin_MPI() line 1012 in
>> /data1/trayanova/petsc/3.4.3/src/vec/vec/impls/mpi/pdvec.c
> The error message indicates a memory corruption problem. Can you
> reproduce running under valgrind?
>
>> [13]PETSC ERROR: VecAssemblyBegin() line 220 in
>> /data1/trayanova/petsc/3.4.3/src/vec/vec/interface/vector.c
>> [13]PETSC ERROR: close() line 953 in
>> "unknowndirectory/"/data1/trayanova/libmesh/0.9.2.2/install/include/libmesh/petsc_vector.h
>> --------------------------------------------------------------------------
>> MPI_ABORT was invoked on rank 13 in communicator MPI_COMM_WORLD
>> with errorcode 1.
>>
>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
>> You may or may not see output from other processes, depending on
>> exactly when Open MPI kills them.
>> --------------------------------------------------------------------------
>> [0]PETSC ERROR: VecAssemblyBegin_MPI() line 1012 in
>> /data1/trayanova/petsc/3.4.3/src/vec/vec/impls/mpi/pdvec.c
>> [0]PETSC ERROR: VecAssemblyBegin() line 220 in
>> /data1/trayanova/petsc/3.4.3/src/vec/vec/interface/vector.c
>> [0]PETSC ERROR: close() line 953 in
>> "unknowndirectory/"/data1/trayanova/libmesh/0.9.2.2/install/include/libmesh/petsc_vector.h
>> --------------------------------------------------------------------------
>> mpirun has exited due to process rank 13 with PID 24354 on
>> node ln236 exiting without calling "finalize". This may
>> have caused other processes in the application to be
>> terminated by signals sent by mpirun (as reported here).
>> --------------------------------------------------------------------------
>> [2]PETSC ERROR:
>> ------------------------------------------------------------------------
>> [2]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the
>> batch system) has told this process to end
>> [2]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>> [2]PETSC ERROR: or see
>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[2]PETSC
>> ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to
>> find memory corruption errors
>> [2]PETSC ERROR: configure using --with-debugging=yes, recompile, link,
>> and run
>> [1]PETSC ERROR:
>> ------------------------------------------------------------------------
>> [1]PETSC ERROR: [2]PETSC ERROR: to get more information on the crash.
>> [2]PETSC ERROR: --------------------- Error Message
>> ------------------------------------
>> [2]PETSC ERROR: [6]PETSC ERROR: [ln280:08397] 1 more process has sent
>> help message help-mpi-api.txt / mpi-abort
>> [ln280:08397] Set MCA parameter "orte_base_help_aggregate" to 0 to see
>> all help / error messages
--
Dafang Wang, Ph.D
Postdoctoral Fellow
Institute of Computational Medicine
Department of Biomedical Engineering
Johns Hopkins University
Hackerman Hall Room 218
Baltimore, MD, 21218
------------------------------------------------------------------------------
Dive into the World of Parallel Programming! The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net
_______________________________________________
Libmesh-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/libmesh-users
