Here's the info you requested:
OS: Ubuntu 12.04.4 LTS, fully upgraded.
Instruction Set: x86_64
MPI: OpenMPI from repositories, 1.4.3-2.1ubuntu3
CPU: Intel Core i7-2600K CPU @ 3.40GHz × 8
PETSc: 3.4.3
PETSc configure line: --PETSC_ARCH=RelWithDebInfo --with-pic=1 --with-debugging=0 --COPTFLAGS=\"-O2 -g\" --CXXOPTFLAGS=\"-O2 -g\" --FOPTFLAGS=\"-O2 -g\" --download-superlu=yes --download-metis=yes --download-superlu_dist=yes --download-ml=yes --download-hypre=yes --download-spai=yes --download-scalapack=yes --download-parmetis=yes --download-parms=yes --download-mumps=yes
Libmesh: 0.9.2.2
Libmesh configure line: --prefix=/home/dwang/Program/libmesh/0.9.2.2/RelWithDebInfo --with-methods=devel --with-metis=PETSc

Attachments:
* hilbertProblem.cpp - working C++ libmesh code that replicates the problem. Running this with

mpiexec -n 7 ./hilbertProblem

... seems to reproduce the problem fairly reliably. You should be able to reproduce this on a virtual machine.

* hibertProblemConfig.txt - all the above information, along with complete compile lines and ldd dumps of the executable * hilbertValgrindErrors.txt - most recently, I was able to generate valgrind errors by running the program. Attached are those errors which seem to originate in parallel_sort.C. These errors do /not/ originate where the deadlock happens, but earlier when converting a linear tet mesh to a second-order tet mesh.

Strangely enough, I can't seem to replicate the problem with 14.04.


On 10/29/2014 11:10 PM, Roy Stogner wrote:

On Wed, 29 Oct 2014, Robert Blake wrote:

We're running into a intermittent deadlock when calling
assign_global_indices().  The problem only appears on >= 5 processors,
and even then it only appears intermittently.

I'd like to try and replicate, but if it's an intermittent problem
then every bit of information you've got might help: OS, compiler,
configure settings, METHOD, config summary (which isn't redundant
w/settings, since a lot of options are autodetected), libMesh version
(release number or git hash), etc.

Here's the error we see:

 Assertion `hi <= node_upper_bounds[communicator.rank()]' failed.
hi = 887794606_2212644264_162978473
node_upper_bounds[communicator.rank()] = 583640776_2377824377_539889346
[0] src/mesh/mesh_communication_global_indices.C, line 412, compiled Oct
29 2013 at 08:06:24

Valgrind for this code returns no problems, both with and without the
intermittent error.

The code generating this error is as follows:

int main(void) {

I assume a LibMeshInit goes here?

  //create a mesh
  const int dim=3;
  Mesh mesh(dim);
  MeshTools::Generation::build_cube(mesh,
  5000, 5000, 5000, //increase this if needed to generate the bug.

You're configuring with 8 byte dof indices, I assume?  Otherwise
there's no way you're going to be running successfully with more than
a few billion elements.

Actually, wait a minute.  Glancing at our libHilbert code it looks
like it's hard-coded to use 32-bit integers.  That's not *necessarily*
the end of the world (it still means using 96 bits total, there
should be no collisions on a simple 5000^3 cube, and IIRC we have code
for handling collisions) but it still worries me.  Ben, thoughts?
---
Roy

OS: Ubuntu 12.04.4 LTS, fully upgraded.  
Instruction Set: x86_64
MPI: OpenMPI from repositories.
CPU: Intel Core i7-2600K CPU @ 3.40GHz × 8
PETSc: 3.4.3
PETSc configure line:  --PETSC_ARCH=RelWithDebInfo --with-pic=1 
--with-debugging=0 --COPTFLAGS=\"-O2 -g\" --CXXOPTFLAGS=\"-O2 -g\" 
--FOPTFLAGS=\"-O2 -g\" --download-superlu=yes --download-metis=yes 
--download-superlu_dist=yes --download-ml=yes --download-hypre=yes 
--download-spai=yes --download-scalapack=yes --download-parmetis=yes 
--download-parms=yes --download-mumps=yes
Libmesh: 0.9.2.2
Libmesh configure line: 
--prefix=/home/dwang/Program/libmesh/0.9.2.2/RelWithDebInfo 
--with-methods=devel --with-metis=PETSc

compile line (Full, generated from Cmake):

/usr/bin/mpicxx     -O2 -felide-constructors -g -ansi -pedantic -W -Wall 
-Wextra -Wno-long-long -Wunused -Wpointer-arith -Wformat -Wparentheses 
-Wuninitialized -funroll-loops -fstrict-aliasing -std=c++0x 
-Woverloaded-virtual -Wdisabled-optimization -fopenmp -O2 -g    
CMakeFiles/hilbertProblem.dir/hilbertProblem.cpp.o  -o hilbertProblem -rdynamic 
/home/dwang/Program/petsc/3.4.3/RelWithDebInfo/lib/libpetsc.so 
/home/dwang/Program/petsc/3.4.3/RelWithDebInfo/lib/libHYPRE.a 
/home/dwang/Program/petsc/3.4.3/RelWithDebInfo/lib/libsuperlu_4.3.a 
/home/dwang/Program/petsc/3.4.3/RelWithDebInfo/lib/libparms.a 
/home/dwang/Program/petsc/3.4.3/RelWithDebInfo/lib/libml.a 
/home/dwang/Program/petsc/3.4.3/RelWithDebInfo/lib/libcmumps.a 
/home/dwang/Program/petsc/3.4.3/RelWithDebInfo/lib/libdmumps.a 
/home/dwang/Program/petsc/3.4.3/RelWithDebInfo/lib/libsmumps.a 
/home/dwang/Program/petsc/3.4.3/RelWithDebInfo/lib/libzmumps.a 
/home/dwang/Program/petsc/3.4.3/RelWithDebInfo/lib/libmumps_common.a 
/home/dwang/Program/petsc/3.4.3/RelWithDebInfo/lib/libpord.a 
/home/dwang/Program/petsc/3.4.3/RelWithDebInfo/lib/libscalapack.a 
/home/dwang/Program/petsc/3.4.3/RelWithDebInfo/lib/libspai.a 
/home/dwang/Program/petsc/3.4.3/RelWithDebInfo/lib/libsuperlu_dist_3.3.a 
-llapack -lblas -lX11 
/home/dwang/Program/petsc/3.4.3/RelWithDebInfo/lib/libparmetis.so 
/home/dwang/Program/petsc/3.4.3/RelWithDebInfo/lib/libmetis.so -lmpi_f90 
-lmpi_f77 -lgfortran -lquadmath -lm -lmpi_cxx -lstdc++ -lmpi -lopen-rte 
-lopen-pal -lnsl -lutil -lgcc_s -lpthread -ldl 
/home/dwang/Program/libmesh/0.9.2.2/RelWithDebInfo/lib/libmesh_devel.so 
/home/dwang/Program/petsc/3.4.3/RelWithDebInfo/lib/libpetsc.so 
/home/dwang/Program/petsc/3.4.3/RelWithDebInfo/lib/libHYPRE.a 
/home/dwang/Program/petsc/3.4.3/RelWithDebInfo/lib/libsuperlu_4.3.a 
/home/dwang/Program/petsc/3.4.3/RelWithDebInfo/lib/libparms.a 
/home/dwang/Program/petsc/3.4.3/RelWithDebInfo/lib/libml.a 
/home/dwang/Program/petsc/3.4.3/RelWithDebInfo/lib/libcmumps.a 
/home/dwang/Program/petsc/3.4.3/RelWithDebInfo/lib/libdmumps.a 
/home/dwang/Program/petsc/3.4.3/RelWithDebInfo/lib/libsmumps.a 
/home/dwang/Program/petsc/3.4.3/RelWithDebInfo/lib/libzmumps.a 
/home/dwang/Program/petsc/3.4.3/RelWithDebInfo/lib/libmumps_common.a 
/home/dwang/Program/petsc/3.4.3/RelWithDebInfo/lib/libpord.a 
/home/dwang/Program/petsc/3.4.3/RelWithDebInfo/lib/libscalapack.a 
/home/dwang/Program/petsc/3.4.3/RelWithDebInfo/lib/libspai.a 
/home/dwang/Program/petsc/3.4.3/RelWithDebInfo/lib/libsuperlu_dist_3.3.a 
-llapack -lblas -lX11 
/home/dwang/Program/petsc/3.4.3/RelWithDebInfo/lib/libparmetis.so 
/home/dwang/Program/petsc/3.4.3/RelWithDebInfo/lib/libmetis.so -lmpi_f90 
-lmpi_f77 -lgfortran -lquadmath -lm -lmpi_cxx -lstdc++ -lmpi -lopen-rte 
-lopen-pal -lnsl -lutil -lgcc_s -lpthread -ldl 
/home/dwang/Program/libmesh/0.9.2.2/RelWithDebInfo/lib/libmesh_devel.so 
-Wl,-rpath,/home/dwang/Program/petsc/3.4.3/RelWithDebInfo/lib:/home/dwang/Program/libmesh/0.9.2.2/RelWithDebInfo/lib
 

ldd hilbertProblem:

linux-vdso.so.1 =>  (0x00007fffacbff000)
libmpi_cxx.so.0 => /usr/lib/libmpi_cxx.so.0 (0x00007f1df429f000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f1df3f9f000)
libmpi.so.0 => /usr/lib/libmpi.so.0 (0x00007f1df3ced000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f1df3ad7000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f1df38ba000)
libmesh_devel.so.0 => 
/home/dwang/Program/libmesh/0.9.2.2/RelWithDebInfo/lib/libmesh_devel.so.0 
(0x00007f1df2664000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1df22a5000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f1df1fa9000)
/lib64/ld-linux-x86-64.so.2 (0x00007f1df44d7000)
libopen-rte.so.0 => /usr/lib/libopen-rte.so.0 (0x00007f1df1d5a000)
libnetcdf.so.7 => /home/dwang/Program/libmesh/0.9.2.2/Debug/lib/libnetcdf.so.7 
(0x00007f1df18fb000)
libpetsc.so => /home/dwang/Program/petsc/3.4.3/RelWithDebInfo/lib/libpetsc.so 
(0x00007f1df078e000)
liblapack.so.3gf => /usr/lib/liblapack.so.3gf (0x00007f1defb98000)
libblas.so.3gf => /usr/lib/libblas.so.3gf (0x00007f1def8fe000)
libmetis.so => /home/dwang/Program/petsc/3.4.3/RelWithDebInfo/lib/libmetis.so 
(0x00007f1def698000)
libopen-pal.so.0 => /usr/lib/libopen-pal.so.0 (0x00007f1def441000)
libX11.so.6 => /usr/lib/x86_64-linux-gnu/libX11.so.6 (0x00007f1def10b000)
libparmetis.so => 
/home/dwang/Program/petsc/3.4.3/RelWithDebInfo/lib/libparmetis.so 
(0x00007f1deeeca000)
libmpi_f90.so.0 => /usr/lib/libmpi_f90.so.0 (0x00007f1deecc6000)
libmpi_f77.so.0 => /usr/lib/libmpi_f77.so.0 (0x00007f1deea92000)
libgfortran.so.3 => /usr/lib/x86_64-linux-gnu/libgfortran.so.3 
(0x00007f1dee77b000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f1dee577000)
libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f1dee373000)
libxcb.so.1 => /usr/lib/x86_64-linux-gnu/libxcb.so.1 (0x00007f1dee155000)
libquadmath.so.0 => /usr/lib/x86_64-linux-gnu/libquadmath.so.0 
(0x00007f1dedf1e000)
libXau.so.6 => /usr/lib/x86_64-linux-gnu/libXau.so.6 (0x00007f1dedd1b000)
libXdmcp.so.6 => /usr/lib/x86_64-linux-gnu/libXdmcp.so.6 (0x00007f1dedb15000)

$ mpiexec -n 7 valgrind ./hilbertProblem
==3685== Memcheck, a memory error detector
==3685== Copyright (C) 2002-2011, and GNU GPL'd, by Julian Seward et al.
==3685== Using Valgrind-3.7.0 and LibVEX; rerun with -h for copyright info
==3685== Command: ./hilbertProblem
==3685== 
==3691== Memcheck, a memory error detector
==3691== Copyright (C) 2002-2011, and GNU GPL'd, by Julian Seward et al.
==3691== Using Valgrind-3.7.0 and LibVEX; rerun with -h for copyright info
==3691== Command: ./hilbertProblem
==3691== 
==3683== Memcheck, a memory error detector
==3683== Copyright (C) 2002-2011, and GNU GPL'd, by Julian Seward et al.
==3683== Using Valgrind-3.7.0 and LibVEX; rerun with -h for copyright info
==3683== Command: ./hilbertProblem
==3683== 
==3684== Memcheck, a memory error detector
==3684== Copyright (C) 2002-2011, and GNU GPL'd, by Julian Seward et al.
==3684== Using Valgrind-3.7.0 and LibVEX; rerun with -h for copyright info
==3694== Memcheck, a memory error detector
==3694== Copyright (C) 2002-2011, and GNU GPL'd, by Julian Seward et al.
==3694== Using Valgrind-3.7.0 and LibVEX; rerun with -h for copyright info
==3694== Command: ./hilbertProblem
==3694== 
==3684== Command: ./hilbertProblem
==3684== 
==3682== Memcheck, a memory error detector
==3682== Copyright (C) 2002-2011, and GNU GPL'd, by Julian Seward et al.
==3682== Using Valgrind-3.7.0 and LibVEX; rerun with -h for copyright info
==3682== Command: ./hilbertProblem
==3682== 
==3687== Memcheck, a memory error detector
==3687== Copyright (C) 2002-2011, and GNU GPL'd, by Julian Seward et al.
==3687== Using Valgrind-3.7.0 and LibVEX; rerun with -h for copyright info
==3687== Command: ./hilbertProblem
==3687== 
==3685== Invalid read of size 8
==3685==    at 0x6D16BCD: _wordcopy_fwd_dest_aligned (wordcopy.c:205)
==3685==    by 0x6D107EB: __GI_memmove (memmove.c:76)
==3685==    by 0x6D17F90: argz_insert (argz-insert.c:55)
==3685==    by 0x9C67695: ??? (in /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3685==    by 0x9C678DF: ??? (in /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3685==    by 0x9C66FCE: ??? (in /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3685==    by 0x9C68017: lt_dlforeachfile (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3685==    by 0x9C6F153: mca_base_component_find (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3685==    by 0x9C6FC43: mca_base_components_open (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3685==    by 0x9C8A5C4: opal_paffinity_base_open (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3685==    by 0x9C6094D: opal_init (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3685==    by 0x734DA0C: orte_init (in 
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==3685==  Address 0xb802ac0 is 32 bytes inside a block of size 39 alloc'd
==3685==    at 0x4C2B6CD: malloc (in 
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3685==    by 0x9C66618: lt__malloc (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3685==    by 0x9C67854: ??? (in /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3685==    by 0x9C66FCE: ??? (in /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3685==    by 0x9C68017: lt_dlforeachfile (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3685==    by 0x9C6F153: mca_base_component_find (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3685==    by 0x9C6FC43: mca_base_components_open (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3685==    by 0x9C8A5C4: opal_paffinity_base_open (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3685==    by 0x9C6094D: opal_init (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3685==    by 0x734DA0C: orte_init (in 
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==3685==    by 0x53822E0: ??? (in /usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3685==    by 0x53A33FA: PMPI_Init (in /usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3685== 
==3694== Invalid read of size 8
==3694==    at 0x6D16BCD: _wordcopy_fwd_dest_aligned (wordcopy.c:205)
==3694==    by 0x6D107EB: __GI_memmove (memmove.c:76)
==3694==    by 0x6D17F90: argz_insert (argz-insert.c:55)
==3694==    by 0x9C67695: ??? (in /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3694==    by 0x9C678DF: ??? (in /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3694==    by 0x9C66FCE: ??? (in /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3694==    by 0x9C68017: lt_dlforeachfile (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3694==    by 0x9C6F153: mca_base_component_find (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3694==    by 0x9C6FC43: mca_base_components_open (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3694==    by 0x9C8A5C4: opal_paffinity_base_open (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3694==    by 0x9C6094D: opal_init (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3694==    by 0x734DA0C: orte_init (in 
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==3694==  Address 0xb802ac0 is 32 bytes inside a block of size 39 alloc'd
==3694==    at 0x4C2B6CD: malloc (in 
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3694==    by 0x9C66618: lt__malloc (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3694==    by 0x9C67854: ??? (in /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3694==    by 0x9C66FCE: ??? (in /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3694==    by 0x9C68017: lt_dlforeachfile (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3694==    by 0x9C6F153: mca_base_component_find (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3694==    by 0x9C6FC43: mca_base_components_open (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3694==    by 0x9C8A5C4: opal_paffinity_base_open (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3694==    by 0x9C6094D: opal_init (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3694==    by 0x734DA0C: orte_init (in 
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==3694==    by 0x53822E0: ??? (in /usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3694==    by 0x53A33FA: PMPI_Init (in /usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3694== 
==3685== Syscall param sched_setaffinity(mask) points to unaddressable byte(s)
==3685==    at 0x6D75539: syscall (syscall.S:39)
==3685==    by 0xBBCE4F1: ??? (in 
/usr/lib/openmpi/lib/openmpi/mca_paffinity_linux.so)
==3685==    by 0xBBCE8DC: ??? (in 
/usr/lib/openmpi/lib/openmpi/mca_paffinity_linux.so)
==3685==    by 0xBBCF3C9: ??? (in 
/usr/lib/openmpi/lib/openmpi/mca_paffinity_linux.so)
==3685==    by 0xBBCE37F: ??? (in 
/usr/lib/openmpi/lib/openmpi/mca_paffinity_linux.so)
==3685==    by 0x9C8A48C: opal_paffinity_base_select (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3685==    by 0x9C60963: opal_init (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3685==    by 0x734DA0C: orte_init (in 
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==3685==    by 0x53822E0: ??? (in /usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3685==    by 0x53A33FA: PMPI_Init (in /usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3685==    by 0x5D01374: libMesh::LibMeshInit::LibMeshInit(int, char const* 
const*, ompi_communicator_t*) (libmesh.C:370)
==3685==    by 0x40E330: main (hilbertProblem.cpp:10)
==3685==  Address 0x0 is not stack'd, malloc'd or (recently) free'd
==3685== 
==3694== Syscall param sched_setaffinity(mask) points to unaddressable byte(s)
==3694==    at 0x6D75539: syscall (syscall.S:39)
==3694==    by 0xBBCE4F1: ??? (in 
/usr/lib/openmpi/lib/openmpi/mca_paffinity_linux.so)
==3694==    by 0xBBCE8DC: ??? (in 
/usr/lib/openmpi/lib/openmpi/mca_paffinity_linux.so)
==3694==    by 0xBBCF3C9: ??? (in 
/usr/lib/openmpi/lib/openmpi/mca_paffinity_linux.so)
==3694==    by 0xBBCE37F: ??? (in 
/usr/lib/openmpi/lib/openmpi/mca_paffinity_linux.so)
==3694==    by 0x9C8A48C: opal_paffinity_base_select (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3694==    by 0x9C60963: opal_init (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3694==    by 0x734DA0C: orte_init (in 
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==3694==    by 0x53822E0: ??? (in /usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3694==    by 0x53A33FA: PMPI_Init (in /usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3694==    by 0x5D01374: libMesh::LibMeshInit::LibMeshInit(int, char const* 
const*, ompi_communicator_t*) (libmesh.C:370)
==3694==    by 0x40E330: main (hilbertProblem.cpp:10)
==3694==  Address 0x0 is not stack'd, malloc'd or (recently) free'd
==3694== 
==3687== Invalid read of size 8
==3687==    at 0x6D16BCD: _wordcopy_fwd_dest_aligned (wordcopy.c:205)
==3687==    by 0x6D107EB: __GI_memmove (memmove.c:76)
==3687==    by 0x6D17F90: argz_insert (argz-insert.c:55)
==3687==    by 0x9C67695: ??? (in /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3687==    by 0x9C678DF: ??? (in /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3687==    by 0x9C66FCE: ??? (in /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3687==    by 0x9C68017: lt_dlforeachfile (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3687==    by 0x9C6F153: mca_base_component_find (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3687==    by 0x9C6FC43: mca_base_components_open (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3687==    by 0x9C8A5C4: opal_paffinity_base_open (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3687==    by 0x9C6094D: opal_init (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3687==    by 0x734DA0C: orte_init (in 
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==3687==  Address 0xb802ac0 is 32 bytes inside a block of size 39 alloc'd
==3687==    at 0x4C2B6CD: malloc (in 
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3687==    by 0x9C66618: lt__malloc (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3687==    by 0x9C67854: ??? (in /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3687==    by 0x9C66FCE: ??? (in /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3687==    by 0x9C68017: lt_dlforeachfile (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3687==    by 0x9C6F153: mca_base_component_find (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3687==    by 0x9C6FC43: mca_base_components_open (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3687==    by 0x9C8A5C4: opal_paffinity_base_open (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3687==    by 0x9C6094D: opal_init (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3687==    by 0x734DA0C: orte_init (in 
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==3687==    by 0x53822E0: ??? (in /usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3687==    by 0x53A33FA: PMPI_Init (in /usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3687== 
==3691== Invalid read of size 8
==3691==    at 0x6D16BCD: _wordcopy_fwd_dest_aligned (wordcopy.c:205)
==3691==    by 0x6D107EB: __GI_memmove (memmove.c:76)
==3691==    by 0x6D17F90: argz_insert (argz-insert.c:55)
==3691==    by 0x9C67695: ??? (in /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3691==    by 0x9C678DF: ??? (in /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3691==    by 0x9C66FCE: ??? (in /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3691==    by 0x9C68017: lt_dlforeachfile (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3691==    by 0x9C6F153: mca_base_component_find (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3691==    by 0x9C6FC43: mca_base_components_open (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3691==    by 0x9C8A5C4: opal_paffinity_base_open (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3691==    by 0x9C6094D: opal_init (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3691==    by 0x734DA0C: orte_init (in 
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==3691==  Address 0xb802ac0 is 32 bytes inside a block of size 39 alloc'd
==3691==    at 0x4C2B6CD: malloc (in 
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3691==    by 0x9C66618: lt__malloc (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3691==    by 0x9C67854: ??? (in /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3691==    by 0x9C66FCE: ??? (in /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3691==    by 0x9C68017: lt_dlforeachfile (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3691==    by 0x9C6F153: mca_base_component_find (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3691==    by 0x9C6FC43: mca_base_components_open (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3691==    by 0x9C8A5C4: opal_paffinity_base_open (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3691==    by 0x9C6094D: opal_init (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3691==    by 0x734DA0C: orte_init (in 
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==3691==    by 0x53822E0: ??? (in /usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3691==    by 0x53A33FA: PMPI_Init (in /usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3691== 
==3684== Invalid read of size 8
==3684==    at 0x6D16BCD: _wordcopy_fwd_dest_aligned (wordcopy.c:205)
==3684==    by 0x6D107EB: __GI_memmove (memmove.c:76)
==3684==    by 0x6D17F90: argz_insert (argz-insert.c:55)
==3684==    by 0x9C67695: ??? (in /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3684==    by 0x9C678DF: ??? (in /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3684==    by 0x9C66FCE: ??? (in /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3684==    by 0x9C68017: lt_dlforeachfile (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3684==    by 0x9C6F153: mca_base_component_find (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3684==    by 0x9C6FC43: mca_base_components_open (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3684==    by 0x9C8A5C4: opal_paffinity_base_open (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3684==    by 0x9C6094D: opal_init (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3684==    by 0x734DA0C: orte_init (in 
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==3684==  Address 0xb802ac0 is 32 bytes inside a block of size 39 alloc'd
==3684==    at 0x4C2B6CD: malloc (in 
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3684==    by 0x9C66618: lt__malloc (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3684==    by 0x9C67854: ??? (in /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3684==    by 0x9C66FCE: ??? (in /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3684==    by 0x9C68017: lt_dlforeachfile (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3684==    by 0x9C6F153: mca_base_component_find (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3684==    by 0x9C6FC43: mca_base_components_open (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3684==    by 0x9C8A5C4: opal_paffinity_base_open (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3684==    by 0x9C6094D: opal_init (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3684==    by 0x734DA0C: orte_init (in 
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==3684==    by 0x53822E0: ??? (in /usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3684==    by 0x53A33FA: PMPI_Init (in /usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3684== 
==3682== Invalid read of size 8
==3682==    at 0x6D16BCD: _wordcopy_fwd_dest_aligned (wordcopy.c:205)
==3682==    by 0x6D107EB: __GI_memmove (memmove.c:76)
==3682==    by 0x6D17F90: argz_insert (argz-insert.c:55)
==3682==    by 0x9C67695: ??? (in /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3682==    by 0x9C678DF: ??? (in /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3682==    by 0x9C66FCE: ??? (in /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3682==    by 0x9C68017: lt_dlforeachfile (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3682==    by 0x9C6F153: mca_base_component_find (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3682==    by 0x9C6FC43: mca_base_components_open (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3682==    by 0x9C8A5C4: opal_paffinity_base_open (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3682==    by 0x9C6094D: opal_init (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3682==    by 0x734DA0C: orte_init (in 
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==3682==  Address 0xb802ac0 is 32 bytes inside a block of size 39 alloc'd
==3682==    at 0x4C2B6CD: malloc (in 
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3682==    by 0x9C66618: lt__malloc (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3682==    by 0x9C67854: ??? (in /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3682==    by 0x9C66FCE: ??? (in /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3682==    by 0x9C68017: lt_dlforeachfile (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3682==    by 0x9C6F153: mca_base_component_find (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3682==    by 0x9C6FC43: mca_base_components_open (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3682==    by 0x9C8A5C4: opal_paffinity_base_open (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3682==    by 0x9C6094D: opal_init (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3682==    by 0x734DA0C: orte_init (in 
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==3682==    by 0x53822E0: ??? (in /usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3682==    by 0x53A33FA: PMPI_Init (in /usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3682== 
==3687== Syscall param sched_setaffinity(mask) points to unaddressable byte(s)
==3687==    at 0x6D75539: syscall (syscall.S:39)
==3687==    by 0xBBCE4F1: ??? (in 
/usr/lib/openmpi/lib/openmpi/mca_paffinity_linux.so)
==3687==    by 0xBBCE8DC: ??? (in 
/usr/lib/openmpi/lib/openmpi/mca_paffinity_linux.so)
==3687==    by 0xBBCF3C9: ??? (in 
/usr/lib/openmpi/lib/openmpi/mca_paffinity_linux.so)
==3687==    by 0xBBCE37F: ??? (in 
/usr/lib/openmpi/lib/openmpi/mca_paffinity_linux.so)
==3687==    by 0x9C8A48C: opal_paffinity_base_select (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3687==    by 0x9C60963: opal_init (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3687==    by 0x734DA0C: orte_init (in 
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==3687==    by 0x53822E0: ??? (in /usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3687==    by 0x53A33FA: PMPI_Init (in /usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3687==    by 0x5D01374: libMesh::LibMeshInit::LibMeshInit(int, char const* 
const*, ompi_communicator_t*) (libmesh.C:370)
==3687==    by 0x40E330: main (hilbertProblem.cpp:10)
==3687==  Address 0x0 is not stack'd, malloc'd or (recently) free'd
==3687== 
==3683== Invalid read of size 8
==3683==    at 0x6D16BCD: _wordcopy_fwd_dest_aligned (wordcopy.c:205)
==3683==    by 0x6D107EB: __GI_memmove (memmove.c:76)
==3683==    by 0x6D17F90: argz_insert (argz-insert.c:55)
==3683==    by 0x9C67695: ??? (in /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3683==    by 0x9C678DF: ??? (in /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3683==    by 0x9C66FCE: ??? (in /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3683==    by 0x9C68017: lt_dlforeachfile (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3683==    by 0x9C6F153: mca_base_component_find (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3683==    by 0x9C6FC43: mca_base_components_open (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3683==    by 0x9C8A5C4: opal_paffinity_base_open (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3683==    by 0x9C6094D: opal_init (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3683==    by 0x734DA0C: orte_init (in 
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==3683==  Address 0xb802ac0 is 32 bytes inside a block of size 39 alloc'd
==3683==    at 0x4C2B6CD: malloc (in 
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3683==    by 0x9C66618: lt__malloc (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3683==    by 0x9C67854: ??? (in /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3683==    by 0x9C66FCE: ??? (in /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3683==    by 0x9C68017: lt_dlforeachfile (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3683==    by 0x9C6F153: mca_base_component_find (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3683==    by 0x9C6FC43: mca_base_components_open (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3683==    by 0x9C8A5C4: opal_paffinity_base_open (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3683==    by 0x9C6094D: opal_init (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3683==    by 0x734DA0C: orte_init (in 
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==3683==    by 0x53822E0: ??? (in /usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3683==    by 0x53A33FA: PMPI_Init (in /usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3683== 
==3691== Syscall param sched_setaffinity(mask) points to unaddressable byte(s)
==3691==    at 0x6D75539: syscall (syscall.S:39)
==3691==    by 0xBBCE4F1: ??? (in 
/usr/lib/openmpi/lib/openmpi/mca_paffinity_linux.so)
==3691==    by 0xBBCE8DC: ??? (in 
/usr/lib/openmpi/lib/openmpi/mca_paffinity_linux.so)
==3691==    by 0xBBCF3C9: ??? (in 
/usr/lib/openmpi/lib/openmpi/mca_paffinity_linux.so)
==3691==    by 0xBBCE37F: ??? (in 
/usr/lib/openmpi/lib/openmpi/mca_paffinity_linux.so)
==3691==    by 0x9C8A48C: opal_paffinity_base_select (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3691==    by 0x9C60963: opal_init (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3691==    by 0x734DA0C: orte_init (in 
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==3691==    by 0x53822E0: ??? (in /usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3691==    by 0x53A33FA: PMPI_Init (in /usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3691==    by 0x5D01374: libMesh::LibMeshInit::LibMeshInit(int, char const* 
const*, ompi_communicator_t*) (libmesh.C:370)
==3691==    by 0x40E330: main (hilbertProblem.cpp:10)
==3691==  Address 0x0 is not stack'd, malloc'd or (recently) free'd
==3691== 
==3684== Syscall param sched_setaffinity(mask) points to unaddressable byte(s)
==3684==    at 0x6D75539: syscall (syscall.S:39)
==3684==    by 0xBBCE4F1: ??? (in 
/usr/lib/openmpi/lib/openmpi/mca_paffinity_linux.so)
==3684==    by 0xBBCE8DC: ??? (in 
/usr/lib/openmpi/lib/openmpi/mca_paffinity_linux.so)
==3684==    by 0xBBCF3C9: ??? (in 
/usr/lib/openmpi/lib/openmpi/mca_paffinity_linux.so)
==3684==    by 0xBBCE37F: ??? (in 
/usr/lib/openmpi/lib/openmpi/mca_paffinity_linux.so)
==3684==    by 0x9C8A48C: opal_paffinity_base_select (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3684==    by 0x9C60963: opal_init (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3684==    by 0x734DA0C: orte_init (in 
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==3684==    by 0x53822E0: ??? (in /usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3684==    by 0x53A33FA: PMPI_Init (in /usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3684==    by 0x5D01374: libMesh::LibMeshInit::LibMeshInit(int, char const* 
const*, ompi_communicator_t*) (libmesh.C:370)
==3684==    by 0x40E330: main (hilbertProblem.cpp:10)
==3684==  Address 0x0 is not stack'd, malloc'd or (recently) free'd
==3684== 
==3682== Syscall param sched_setaffinity(mask) points to unaddressable byte(s)
==3682==    at 0x6D75539: syscall (syscall.S:39)
==3682==    by 0xBBCE4F1: ??? (in 
/usr/lib/openmpi/lib/openmpi/mca_paffinity_linux.so)
==3682==    by 0xBBCE8DC: ??? (in 
/usr/lib/openmpi/lib/openmpi/mca_paffinity_linux.so)
==3682==    by 0xBBCF3C9: ??? (in 
/usr/lib/openmpi/lib/openmpi/mca_paffinity_linux.so)
==3682==    by 0xBBCE37F: ??? (in 
/usr/lib/openmpi/lib/openmpi/mca_paffinity_linux.so)
==3682==    by 0x9C8A48C: opal_paffinity_base_select (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3682==    by 0x9C60963: opal_init (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3682==    by 0x734DA0C: orte_init (in 
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==3682==    by 0x53822E0: ??? (in /usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3682==    by 0x53A33FA: PMPI_Init (in /usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3682==    by 0x5D01374: libMesh::LibMeshInit::LibMeshInit(int, char const* 
const*, ompi_communicator_t*) (libmesh.C:370)
==3682==    by 0x40E330: main (hilbertProblem.cpp:10)
==3682==  Address 0x0 is not stack'd, malloc'd or (recently) free'd
==3682== 
==3683== Syscall param sched_setaffinity(mask) points to unaddressable byte(s)
==3683==    at 0x6D75539: syscall (syscall.S:39)
==3683==    by 0xBBCE4F1: ??? (in 
/usr/lib/openmpi/lib/openmpi/mca_paffinity_linux.so)
==3683==    by 0xBBCE8DC: ??? (in 
/usr/lib/openmpi/lib/openmpi/mca_paffinity_linux.so)
==3683==    by 0xBBCF3C9: ??? (in 
/usr/lib/openmpi/lib/openmpi/mca_paffinity_linux.so)
==3683==    by 0xBBCE37F: ??? (in 
/usr/lib/openmpi/lib/openmpi/mca_paffinity_linux.so)
==3683==    by 0x9C8A48C: opal_paffinity_base_select (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3683==    by 0x9C60963: opal_init (in 
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3683==    by 0x734DA0C: orte_init (in 
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==3683==    by 0x53822E0: ??? (in /usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3683==    by 0x53A33FA: PMPI_Init (in /usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3683==    by 0x5D01374: libMesh::LibMeshInit::LibMeshInit(int, char const* 
const*, ompi_communicator_t*) (libmesh.C:370)
==3683==    by 0x40E330: main (hilbertProblem.cpp:10)
==3683==  Address 0x0 is not stack'd, malloc'd or (recently) free'd
==3683== 
==3687== Syscall param writev(vector[...]) points to uninitialised byte(s)
==3687==    at 0x6D71C6B: writev (writev.c:51)
==3687==    by 0xCFEEDE2: ??? (in /usr/lib/openmpi/lib/openmpi/mca_oob_tcp.so)
==3687==    by 0xCFEFB9C: ??? (in /usr/lib/openmpi/lib/openmpi/mca_oob_tcp.so)
==3687==    by 0xCFF33D6: ??? (in /usr/lib/openmpi/lib/openmpi/mca_oob_tcp.so)
==3687==    by 0xCDE5557: ??? (in /usr/lib/openmpi/lib/openmpi/mca_rml_oob.so)
==3687==    by 0xCDE5B3F: ??? (in /usr/lib/openmpi/lib/openmpi/mca_rml_oob.so)
==3687==    by 0xCBDFC97: ??? (in 
/usr/lib/openmpi/lib/openmpi/mca_grpcomm_bad.so)
==3687==    by 0xCBDF2F4: ??? (in 
/usr/lib/openmpi/lib/openmpi/mca_grpcomm_bad.so)
==3687==    by 0x5382AF2: ??? (in /usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3687==    by 0x53A33FA: PMPI_Init (in /usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3687==    by 0x5D01374: libMesh::LibMeshInit::LibMeshInit(int, char const* 
const*, ompi_communicator_t*) (libmesh.C:370)
==3687==    by 0x40E330: main (hilbertProblem.cpp:10)
==3687==  Address 0xfb984d7 is 87 bytes inside a block of size 128 alloc'd
==3687==    at 0x4C2B6CD: malloc (in 
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3687==    by 0x9C61C52: ??? (in /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3687==    by 0x9C62AF1: ??? (in /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3687==    by 0x9C62B81: ??? (in /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3687==    by 0xCBDFB9C: ??? (in 
/usr/lib/openmpi/lib/openmpi/mca_grpcomm_bad.so)
==3687==    by 0xCBDF2F4: ??? (in 
/usr/lib/openmpi/lib/openmpi/mca_grpcomm_bad.so)
==3687==    by 0x5382AF2: ??? (in /usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3687==    by 0x53A33FA: PMPI_Init (in /usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3687==    by 0x5D01374: libMesh::LibMeshInit::LibMeshInit(int, char const* 
const*, ompi_communicator_t*) (libmesh.C:370)
==3687==    by 0x40E330: main (hilbertProblem.cpp:10)
==3687== 
==3694== Syscall param writev(vector[...]) points to uninitialised byte(s)
==3694==    at 0x6D71C6B: writev (writev.c:51)
==3694==    by 0xCFEEDE2: ??? (in /usr/lib/openmpi/lib/openmpi/mca_oob_tcp.so)
==3694==    by 0xCFEFB9C: ??? (in /usr/lib/openmpi/lib/openmpi/mca_oob_tcp.so)
==3694==    by 0xCFF33D6: ??? (in /usr/lib/openmpi/lib/openmpi/mca_oob_tcp.so)
==3694==    by 0xCDE5557: ??? (in /usr/lib/openmpi/lib/openmpi/mca_rml_oob.so)
==3694==    by 0xCDE5B3F: ??? (in /usr/lib/openmpi/lib/openmpi/mca_rml_oob.so)
==3694==    by 0xCBDFC97: ??? (in 
/usr/lib/openmpi/lib/openmpi/mca_grpcomm_bad.so)
==3694==    by 0xCBDF2F4: ??? (in 
/usr/lib/openmpi/lib/openmpi/mca_grpcomm_bad.so)
==3694==    by 0x5382AF2: ??? (in /usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3694==    by 0x53A33FA: PMPI_Init (in /usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3694==    by 0x5D01374: libMesh::LibMeshInit::LibMeshInit(int, char const* 
const*, ompi_communicator_t*) (libmesh.C:370)
==3694==    by 0x40E330: main (hilbertProblem.cpp:10)
==3694==  Address 0xfb984d7 is 87 bytes inside a block of size 128 alloc'd
==3694==    at 0x4C2B6CD: malloc (in 
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3694==    by 0x9C61C52: ??? (in /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3694==    by 0x9C62AF1: ??? (in /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3694==    by 0x9C62B81: ??? (in /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3694==    by 0xCBDFB9C: ??? (in 
/usr/lib/openmpi/lib/openmpi/mca_grpcomm_bad.so)
==3694==    by 0xCBDF2F4: ??? (in 
/usr/lib/openmpi/lib/openmpi/mca_grpcomm_bad.so)
==3694==    by 0x5382AF2: ??? (in /usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3694==    by 0x53A33FA: PMPI_Init (in /usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3694==    by 0x5D01374: libMesh::LibMeshInit::LibMeshInit(int, char const* 
const*, ompi_communicator_t*) (libmesh.C:370)
==3694==    by 0x40E330: main (hilbertProblem.cpp:10)
==3694== 
==3685== Syscall param writev(vector[...]) points to uninitialised byte(s)
==3685==    at 0x6D71C6B: writev (writev.c:51)
==3685==    by 0xCFEEDE2: ??? (in /usr/lib/openmpi/lib/openmpi/mca_oob_tcp.so)
==3685==    by 0xCFEFB9C: ??? (in /usr/lib/openmpi/lib/openmpi/mca_oob_tcp.so)
==3685==    by 0xCFF33D6: ??? (in /usr/lib/openmpi/lib/openmpi/mca_oob_tcp.so)
==3685==    by 0xCDE5557: ??? (in /usr/lib/openmpi/lib/openmpi/mca_rml_oob.so)
==3685==    by 0xCDE5B3F: ??? (in /usr/lib/openmpi/lib/openmpi/mca_rml_oob.so)
==3685==    by 0xCBDFC97: ??? (in 
/usr/lib/openmpi/lib/openmpi/mca_grpcomm_bad.so)
==3685==    by 0xCBDF2F4: ??? (in 
/usr/lib/openmpi/lib/openmpi/mca_grpcomm_bad.so)
==3685==    by 0x5382AF2: ??? (in /usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3685==    by 0x53A33FA: PMPI_Init (in /usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3685==    by 0x5D01374: libMesh::LibMeshInit::LibMeshInit(int, char const* 
const*, ompi_communicator_t*) (libmesh.C:370)
==3685==    by 0x40E330: main (hilbertProblem.cpp:10)
==3685==  Address 0xfb984d7 is 87 bytes inside a block of size 128 alloc'd
==3685==    at 0x4C2B6CD: malloc (in 
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3685==    by 0x9C61C52: ??? (in /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3685==    by 0x9C62AF1: ??? (in /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3685==    by 0x9C62B81: ??? (in /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3685==    by 0xCBDFB9C: ??? (in 
/usr/lib/openmpi/lib/openmpi/mca_grpcomm_bad.so)
==3685==    by 0xCBDF2F4: ??? (in 
/usr/lib/openmpi/lib/openmpi/mca_grpcomm_bad.so)
==3685==    by 0x5382AF2: ??? (in /usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3685==    by 0x53A33FA: PMPI_Init (in /usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3685==    by 0x5D01374: libMesh::LibMeshInit::LibMeshInit(int, char const* 
const*, ompi_communicator_t*) (libmesh.C:370)
==3685==    by 0x40E330: main (hilbertProblem.cpp:10)
==3685== 
==3691== Syscall param writev(vector[...]) points to uninitialised byte(s)
==3691==    at 0x6D71C6B: writev (writev.c:51)
==3691==    by 0xCFEEDE2: ??? (in /usr/lib/openmpi/lib/openmpi/mca_oob_tcp.so)
==3691==    by 0xCFEFB9C: ??? (in /usr/lib/openmpi/lib/openmpi/mca_oob_tcp.so)
==3691==    by 0xCFF33D6: ??? (in /usr/lib/openmpi/lib/openmpi/mca_oob_tcp.so)
==3691==    by 0xCDE5557: ??? (in /usr/lib/openmpi/lib/openmpi/mca_rml_oob.so)
==3691==    by 0xCDE5B3F: ??? (in /usr/lib/openmpi/lib/openmpi/mca_rml_oob.so)
==3691==    by 0xCBDFC97: ??? (in 
/usr/lib/openmpi/lib/openmpi/mca_grpcomm_bad.so)
==3691==    by 0xCBDF2F4: ??? (in 
/usr/lib/openmpi/lib/openmpi/mca_grpcomm_bad.so)
==3691==    by 0x5382AF2: ??? (in /usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3691==    by 0x53A33FA: PMPI_Init (in /usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3691==    by 0x5D01374: libMesh::LibMeshInit::LibMeshInit(int, char const* 
const*, ompi_communicator_t*) (libmesh.C:370)
==3691==    by 0x40E330: main (hilbertProblem.cpp:10)
==3691==  Address 0xfb984d7 is 87 bytes inside a block of size 128 alloc'd
==3691==    at 0x4C2B6CD: malloc (in 
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3691==    by 0x9C61C52: ??? (in /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3691==    by 0x9C62AF1: ??? (in /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3691==    by 0x9C62B81: ??? (in /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3691==    by 0xCBDFB9C: ??? (in 
/usr/lib/openmpi/lib/openmpi/mca_grpcomm_bad.so)
==3691==    by 0xCBDF2F4: ??? (in 
/usr/lib/openmpi/lib/openmpi/mca_grpcomm_bad.so)
==3691==    by 0x5382AF2: ??? (in /usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3691==    by 0x53A33FA: PMPI_Init (in /usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3691==    by 0x5D01374: libMesh::LibMeshInit::LibMeshInit(int, char const* 
const*, ompi_communicator_t*) (libmesh.C:370)
==3691==    by 0x40E330: main (hilbertProblem.cpp:10)
==3691== 
==3682== Syscall param writev(vector[...]) points to uninitialised byte(s)
==3682==    at 0x6D71C6B: writev (writev.c:51)
==3682==    by 0xCFEEDE2: ??? (in /usr/lib/openmpi/lib/openmpi/mca_oob_tcp.so)
==3682==    by 0xCFEFB9C: ??? (in /usr/lib/openmpi/lib/openmpi/mca_oob_tcp.so)
==3682==    by 0xCFF33D6: ??? (in /usr/lib/openmpi/lib/openmpi/mca_oob_tcp.so)
==3682==    by 0xCDE5557: ??? (in /usr/lib/openmpi/lib/openmpi/mca_rml_oob.so)
==3682==    by 0xCDE5B3F: ??? (in /usr/lib/openmpi/lib/openmpi/mca_rml_oob.so)
==3682==    by 0xCBDFC97: ??? (in 
/usr/lib/openmpi/lib/openmpi/mca_grpcomm_bad.so)
==3682==    by 0xCBDF2F4: ??? (in 
/usr/lib/openmpi/lib/openmpi/mca_grpcomm_bad.so)
==3682==    by 0x5382AF2: ??? (in /usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3682==    by 0x53A33FA: PMPI_Init (in /usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3682==    by 0x5D01374: libMesh::LibMeshInit::LibMeshInit(int, char const* 
const*, ompi_communicator_t*) (libmesh.C:370)
==3682==    by 0x40E330: main (hilbertProblem.cpp:10)
==3682==  Address 0xfb98667 is 87 bytes inside a block of size 256 alloc'd
==3682==    at 0x4C2B7B2: realloc (in 
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3682==    by 0x9C61BD6: ??? (in /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3682==    by 0x9C626D5: ??? (in /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3682==    by 0xCBDFC6A: ??? (in 
/usr/lib/openmpi/lib/openmpi/mca_grpcomm_bad.so)
==3682==    by 0xCBDF2F4: ??? (in 
/usr/lib/openmpi/lib/openmpi/mca_grpcomm_bad.so)
==3682==    by 0x5382AF2: ??? (in /usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3682==    by 0x53A33FA: PMPI_Init (in /usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3682==    by 0x5D01374: libMesh::LibMeshInit::LibMeshInit(int, char const* 
const*, ompi_communicator_t*) (libmesh.C:370)
==3682==    by 0x40E330: main (hilbertProblem.cpp:10)
==3682== 
==3684== Syscall param writev(vector[...]) points to uninitialised byte(s)
==3684==    at 0x6D71C6B: writev (writev.c:51)
==3684==    by 0xCFEEDE2: ??? (in /usr/lib/openmpi/lib/openmpi/mca_oob_tcp.so)
==3684==    by 0xCFEFB9C: ??? (in /usr/lib/openmpi/lib/openmpi/mca_oob_tcp.so)
==3684==    by 0xCFF33D6: ??? (in /usr/lib/openmpi/lib/openmpi/mca_oob_tcp.so)
==3684==    by 0xCDE5557: ??? (in /usr/lib/openmpi/lib/openmpi/mca_rml_oob.so)
==3684==    by 0xCDE5B3F: ??? (in /usr/lib/openmpi/lib/openmpi/mca_rml_oob.so)
==3684==    by 0xCBDFC97: ??? (in 
/usr/lib/openmpi/lib/openmpi/mca_grpcomm_bad.so)
==3684==    by 0xCBDF2F4: ??? (in 
/usr/lib/openmpi/lib/openmpi/mca_grpcomm_bad.so)
==3684==    by 0x5382AF2: ??? (in /usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3684==    by 0x53A33FA: PMPI_Init (in /usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3684==    by 0x5D01374: libMesh::LibMeshInit::LibMeshInit(int, char const* 
const*, ompi_communicator_t*) (libmesh.C:370)
==3684==    by 0x40E330: main (hilbertProblem.cpp:10)
==3684==  Address 0xfb984d7 is 87 bytes inside a block of size 128 alloc'd
==3684==    at 0x4C2B6CD: malloc (in 
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3684==    by 0x9C61C52: ??? (in /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3684==    by 0x9C62AF1: ??? (in /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3684==    by 0x9C62B81: ??? (in /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3684==    by 0xCBDFB9C: ??? (in 
/usr/lib/openmpi/lib/openmpi/mca_grpcomm_bad.so)
==3684==    by 0xCBDF2F4: ??? (in 
/usr/lib/openmpi/lib/openmpi/mca_grpcomm_bad.so)
==3684==    by 0x5382AF2: ??? (in /usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3684==    by 0x53A33FA: PMPI_Init (in /usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3684==    by 0x5D01374: libMesh::LibMeshInit::LibMeshInit(int, char const* 
const*, ompi_communicator_t*) (libmesh.C:370)
==3684==    by 0x40E330: main (hilbertProblem.cpp:10)
==3684== 
==3683== Syscall param writev(vector[...]) points to uninitialised byte(s)
==3683==    at 0x6D71C6B: writev (writev.c:51)
==3683==    by 0xCFEEDE2: ??? (in /usr/lib/openmpi/lib/openmpi/mca_oob_tcp.so)
==3683==    by 0xCFEFB9C: ??? (in /usr/lib/openmpi/lib/openmpi/mca_oob_tcp.so)
==3683==    by 0xCFF33D6: ??? (in /usr/lib/openmpi/lib/openmpi/mca_oob_tcp.so)
==3683==    by 0xCDE5557: ??? (in /usr/lib/openmpi/lib/openmpi/mca_rml_oob.so)
==3683==    by 0xCDE5B3F: ??? (in /usr/lib/openmpi/lib/openmpi/mca_rml_oob.so)
==3683==    by 0xCBDFC97: ??? (in 
/usr/lib/openmpi/lib/openmpi/mca_grpcomm_bad.so)
==3683==    by 0xCBDF2F4: ??? (in 
/usr/lib/openmpi/lib/openmpi/mca_grpcomm_bad.so)
==3683==    by 0x5382AF2: ??? (in /usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3683==    by 0x53A33FA: PMPI_Init (in /usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3683==    by 0x5D01374: libMesh::LibMeshInit::LibMeshInit(int, char const* 
const*, ompi_communicator_t*) (libmesh.C:370)
==3683==    by 0x40E330: main (hilbertProblem.cpp:10)
==3683==  Address 0xfb984d7 is 87 bytes inside a block of size 128 alloc'd
==3683==    at 0x4C2B6CD: malloc (in 
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3683==    by 0x9C61C52: ??? (in /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3683==    by 0x9C62AF1: ??? (in /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3683==    by 0x9C62B81: ??? (in /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==3683==    by 0xCBDFB9C: ??? (in 
/usr/lib/openmpi/lib/openmpi/mca_grpcomm_bad.so)
==3683==    by 0xCBDF2F4: ??? (in 
/usr/lib/openmpi/lib/openmpi/mca_grpcomm_bad.so)
==3683==    by 0x5382AF2: ??? (in /usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3683==    by 0x53A33FA: PMPI_Init (in /usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3683==    by 0x5D01374: libMesh::LibMeshInit::LibMeshInit(int, char const* 
const*, ompi_communicator_t*) (libmesh.C:370)
==3683==    by 0x40E330: main (hilbertProblem.cpp:10)
==3683== 
*** Warning, This code is deprecated, and likely to be removed in future 
library versions! src/mesh/boundary_info.C, line 752, compiled Oct 29 2013 at 
08:06:16 ***
==3694== Source and destination overlap in memcpy(0x179badd0, 0x179badd4, 24)
==3694==    at 0x4C2CFA0: memcpy@@GLIBC_2.14 (in 
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3687== Source and destination overlap in memcpy(0x179badd0, 0x179baddc, 16)
==3687==    at 0x4C2CFA0: memcpy@@GLIBC_2.14 (in 
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3687==    by 0x538EB7E: ompi_ddt_copy_content_same_ddt (in 
/usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3687==    by 0x1138848A: ??? (in 
/usr/lib/openmpi/lib/openmpi/mca_coll_tuned.so)
==3687==    by 0x53955B2: PMPI_Allgather (in 
/usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3687==    by 0x62E9E10: libMesh::Parallel::Sort<Hilbert::HilbertIndices, 
unsigned int>::communicate_bins() (parallel_sort.C:306)
==3687==    by 0x62EF9AF: libMesh::Parallel::Sort<Hilbert::HilbertIndices, 
unsigned int>::sort() (parallel_sort.C:84)
==3687==    by 0x60F36FC: void 
libMesh::MeshCommunication::find_global_indices<libMesh::MeshBase::element_iterator>(libMesh::Parallel::Communicator
 const&, libMesh::MeshTools::BoundingBox const&, 
libMesh::MeshBase::element_iterator const&, libMesh::MeshBase::element_iterator 
const&, std::vector<unsigned int, std::allocator<unsigned int> >&) const 
(mesh_communication_global_indices.C:657)
==3687==    by 0x6300EBD: 
libMesh::Partitioner::partition_unpartitioned_elements(libMesh::MeshBase&, 
unsigned int) (partitioner.C:233)
==3687==    by 0x6307DB1: libMesh::Partitioner::partition(libMesh::MeshBase&, 
unsigned int) (partitioner.C:81)
==3687==    by 0x60C4BC9: libMesh::MeshBase::prepare_for_use(bool) 
(mesh_base.h:494)
==3687==    by 0x6116F2F: 
libMesh::MeshTools::Generation::build_cube(libMesh::UnstructuredMesh&, unsigned 
int, unsigned int, unsigned int, double, double, double, double, double, 
double, libMeshEnums::ElemType, bool) (mesh_generation.C:1455)
==3687==    by 0x40E38C: main (hilbertProblem.cpp:21)
==3687== 
==3694==    by 0x538EB7E: ompi_ddt_copy_content_same_ddt (in 
/usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3694==    by 0x1138848A: ??? (in 
/usr/lib/openmpi/lib/openmpi/mca_coll_tuned.so)
==3694==    by 0x53955B2: PMPI_Allgather (in 
/usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3694==    by 0x62E9E10: libMesh::Parallel::Sort<Hilbert::HilbertIndices, 
unsigned int>::communicate_bins() (parallel_sort.C:306)
==3694==    by 0x62EF9AF: libMesh::Parallel::Sort<Hilbert::HilbertIndices, 
unsigned int>::sort() (parallel_sort.C:84)
==3694==    by 0x60F36FC: void 
libMesh::MeshCommunication::find_global_indices<libMesh::MeshBase::element_iterator>(libMesh::Parallel::Communicator
 const&, libMesh::MeshTools::BoundingBox const&, 
libMesh::MeshBase::element_iterator const&, libMesh::MeshBase::element_iterator 
const&, std::vector<unsigned int, std::allocator<unsigned int> >&) const 
(mesh_communication_global_indices.C:657)
==3694==    by 0x6300EBD: 
libMesh::Partitioner::partition_unpartitioned_elements(libMesh::MeshBase&, 
unsigned int) (partitioner.C:233)
==3694==    by 0x6307DB1: libMesh::Partitioner::partition(libMesh::MeshBase&, 
unsigned int) (partitioner.C:81)
==3694==    by 0x60C4BC9: libMesh::MeshBase::prepare_for_use(bool) 
(mesh_base.h:494)
==3694==    by 0x6116F2F: 
libMesh::MeshTools::Generation::build_cube(libMesh::UnstructuredMesh&, unsigned 
int, unsigned int, unsigned int, double, double, double, double, double, 
double, libMeshEnums::ElemType, bool) (mesh_generation.C:1455)
==3694==    by 0x40E38C: main (hilbertProblem.cpp:21)
==3694== 
==3691== Source and destination overlap in memcpy(0x179badd0, 0x179badd8, 20)
==3691==    at 0x4C2CFA0: memcpy@@GLIBC_2.14 (in 
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3691==    by 0x538EB7E: ompi_ddt_copy_content_same_ddt (in 
/usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3691==    by 0x1138848A: ??? (in 
/usr/lib/openmpi/lib/openmpi/mca_coll_tuned.so)
==3691==    by 0x53955B2: PMPI_Allgather (in 
/usr/lib/openmpi/lib/libmpi.so.0.0.2)
==3691==    by 0x62E9E10: libMesh::Parallel::Sort<Hilbert::HilbertIndices, 
unsigned int>::communicate_bins() (parallel_sort.C:306)
==3691==    by 0x62EF9AF: libMesh::Parallel::Sort<Hilbert::HilbertIndices, 
unsigned int>::sort() (parallel_sort.C:84)
==3691==    by 0x60F36FC: void 
libMesh::MeshCommunication::find_global_indices<libMesh::MeshBase::element_iterator>(libMesh::Parallel::Communicator
 const&, libMesh::MeshTools::BoundingBox const&, 
libMesh::MeshBase::element_iterator const&, libMesh::MeshBase::element_iterator 
const&, std::vector<unsigned int, std::allocator<unsigned int> >&) const 
(mesh_communication_global_indices.C:657)
==3691==    by 0x6300EBD: 
libMesh::Partitioner::partition_unpartitioned_elements(libMesh::MeshBase&, 
unsigned int) (partitioner.C:233)
==3691==    by 0x6307DB1: libMesh::Partitioner::partition(libMesh::MeshBase&, 
unsigned int) (partitioner.C:81)
==3691==    by 0x60C4BC9: libMesh::MeshBase::prepare_for_use(bool) 
(mesh_base.h:494)
==3691==    by 0x6116F2F: 
libMesh::MeshTools::Generation::build_cube(libMesh::UnstructuredMesh&, unsigned 
int, unsigned int, unsigned int, double, double, double, double, double, 
double, libMeshEnums::ElemType, bool) (mesh_generation.C:1455)
==3691==    by 0x40E38C: main (hilbertProblem.cpp:21)
==3691== 

 Mesh Information:
  mesh_dimension()=3
  spatial_dimension()=3
  n_nodes()=140491
    n_local_nodes()=21716
  n_elem()=648000
    n_local_elem()=92574
    n_active_elem()=648000
  n_subdomains()=1
  n_partitions()=7
  n_processors()=7
  n_threads()=1
  processor_id()=0

 Mesh Information:
  mesh_dimension()=3
  spatial_dimension()=3
  n_nodes()=939781
    n_local_nodes()=140725
  n_elem()=648000
    n_local_elem()=92574
    n_active_elem()=648000
  n_subdomains()=1
  n_partitions()=7
  n_processors()=7
  n_threads()=1
  processor_id()=0

==3685== 
==3685== HEAP SUMMARY:
==3685==     in use at exit: 410,530 bytes in 2,838 blocks
==3685==   total heap usage: 39,081,491 allocs, 39,078,653 frees, 3,725,814,672 
bytes allocated
==3685== 
==3682== 
==3694== 
==3682== HEAP SUMMARY:
==3682==     in use at exit: 410,530 bytes in 2,838 blocks
==3682==   total heap usage: 28,947,500 allocs, 28,944,662 frees, 2,847,866,190 
bytes allocated
==3682== 
==3694== HEAP SUMMARY:
==3694==     in use at exit: 410,530 bytes in 2,838 blocks
==3694==   total heap usage: 39,081,194 allocs, 39,078,356 frees, 3,725,276,073 
bytes allocated
==3694== 
==3684== 
==3685== LEAK SUMMARY:
==3685==    definitely lost: 14,372 bytes in 89 blocks
==3685==    indirectly lost: 25,245 bytes in 73 blocks
==3685==      possibly lost: 0 bytes in 0 blocks
==3685==    still reachable: 370,913 bytes in 2,676 blocks
==3685==         suppressed: 0 bytes in 0 blocks
==3685== Rerun with --leak-check=full to see details of leaked memory
==3685== 
==3685== For counts of detected and suppressed errors, rerun with: -v
==3685== Use --track-origins=yes to see where uninitialised values come from
==3685== ERROR SUMMARY: 618 errors from 3 contexts (suppressed: 2 from 2)
==3691== 
==3684== HEAP SUMMARY:
==3684==     in use at exit: 410,530 bytes in 2,838 blocks
==3684==   total heap usage: 39,081,198 allocs, 39,078,360 frees, 3,726,017,713 
bytes allocated
==3684== 
==3683== 
==3691== HEAP SUMMARY:
==3691==     in use at exit: 410,530 bytes in 2,838 blocks
==3691==   total heap usage: 39,081,234 allocs, 39,078,396 frees, 3,726,244,269 
bytes allocated
==3691== 
==3683== HEAP SUMMARY:
==3683==     in use at exit: 410,530 bytes in 2,838 blocks
==3683==   total heap usage: 39,081,251 allocs, 39,078,413 frees, 3,725,992,505 
bytes allocated
==3683== 
==3687== 
==3687== HEAP SUMMARY:
==3687==     in use at exit: 410,530 bytes in 2,838 blocks
==3687==   total heap usage: 39,081,178 allocs, 39,078,340 frees, 3,725,994,869 
bytes allocated
==3687== 
==3694== LEAK SUMMARY:
==3694==    definitely lost: 14,372 bytes in 89 blocks
==3694==    indirectly lost: 25,245 bytes in 73 blocks
==3694==      possibly lost: 0 bytes in 0 blocks
==3694==    still reachable: 370,913 bytes in 2,676 blocks
==3694==         suppressed: 0 bytes in 0 blocks
==3694== Rerun with --leak-check=full to see details of leaked memory
==3694== 
==3694== For counts of detected and suppressed errors, rerun with: -v
==3694== Use --track-origins=yes to see where uninitialised values come from
==3694== ERROR SUMMARY: 657 errors from 4 contexts (suppressed: 2 from 2)
==3682== LEAK SUMMARY:
==3682==    definitely lost: 14,452 bytes in 90 blocks
==3682==    indirectly lost: 25,165 bytes in 72 blocks
==3682==      possibly lost: 0 bytes in 0 blocks
==3682==    still reachable: 370,913 bytes in 2,676 blocks
==3682==         suppressed: 0 bytes in 0 blocks
==3682== Rerun with --leak-check=full to see details of leaked memory
==3682== 
==3682== For counts of detected and suppressed errors, rerun with: -v
==3682== Use --track-origins=yes to see where uninitialised values come from
==3682== ERROR SUMMARY: 618 errors from 3 contexts (suppressed: 2 from 2)
==3684== LEAK SUMMARY:
==3684==    definitely lost: 14,372 bytes in 89 blocks
==3684==    indirectly lost: 25,245 bytes in 73 blocks
==3684==      possibly lost: 0 bytes in 0 blocks
==3684==    still reachable: 370,913 bytes in 2,676 blocks
==3684==         suppressed: 0 bytes in 0 blocks
==3684== Rerun with --leak-check=full to see details of leaked memory
==3684== 
==3684== For counts of detected and suppressed errors, rerun with: -v
==3684== Use --track-origins=yes to see where uninitialised values come from
==3684== ERROR SUMMARY: 618 errors from 3 contexts (suppressed: 2 from 2)
==3691== LEAK SUMMARY:
==3691==    definitely lost: 14,372 bytes in 89 blocks
==3691==    indirectly lost: 25,245 bytes in 73 blocks
==3691==      possibly lost: 0 bytes in 0 blocks
==3691==    still reachable: 370,913 bytes in 2,676 blocks
==3691==         suppressed: 0 bytes in 0 blocks
==3691== Rerun with --leak-check=full to see details of leaked memory
==3691== 
==3691== For counts of detected and suppressed errors, rerun with: -v
==3691== Use --track-origins=yes to see where uninitialised values come from
==3691== ERROR SUMMARY: 657 errors from 4 contexts (suppressed: 2 from 2)
==3683== LEAK SUMMARY:
==3683==    definitely lost: 14,372 bytes in 89 blocks
==3683==    indirectly lost: 25,245 bytes in 73 blocks
==3683==      possibly lost: 0 bytes in 0 blocks
==3683==    still reachable: 370,913 bytes in 2,676 blocks
==3683==         suppressed: 0 bytes in 0 blocks
==3683== Rerun with --leak-check=full to see details of leaked memory
==3683== 
==3683== For counts of detected and suppressed errors, rerun with: -v
==3683== Use --track-origins=yes to see where uninitialised values come from
==3683== ERROR SUMMARY: 618 errors from 3 contexts (suppressed: 2 from 2)
==3687== LEAK SUMMARY:
==3687==    definitely lost: 14,372 bytes in 89 blocks
==3687==    indirectly lost: 25,245 bytes in 73 blocks
==3687==      possibly lost: 0 bytes in 0 blocks
==3687==    still reachable: 370,913 bytes in 2,676 blocks
==3687==         suppressed: 0 bytes in 0 blocks
==3687== Rerun with --leak-check=full to see details of leaked memory
==3687== 
==3687== For counts of detected and suppressed errors, rerun with: -v
==3687== Use --track-origins=yes to see where uninitialised values come from
==3687== ERROR SUMMARY: 657 errors from 4 contexts (suppressed: 2 from 2)

#include <libmesh/mesh.h> //provides Mesh
#include <libmesh/mesh_generation.h> //provides MeshTools
#include <libmesh/mesh_communication.h>
#include <libmesh/linear_partitioner.h>


int main(int argc, char* argv[]) {

   LibMeshInit init(argc, argv);

   //create a mesh
   const int dim=3;
   Mesh mesh(dim);
   MeshTools::Generation::build_cube(mesh,
   30, 30, 30, //increase this if needed to generate the bug.
   -1,1,
   -1,1,
   -1,1,
   TET4
   );

   MeshCommunication().broadcast(mesh);
   mesh.allow_renumbering(false);
   mesh.partitioner() = AutoPtr<Partitioner>(new LinearPartitioner);
   mesh.prepare_for_use();
   //not sure if this is strictly necessary, but we need 2nd order
   //elements for our application. Included for completeness.
   mesh.print_info();
   mesh.all_second_order(true); 
   mesh.print_info();

   using libMesh::MeshTools::Private::globally_renumber_nodes_and_elements;
   globally_renumber_nodes_and_elements(mesh);

   mesh.fix_broken_node_and_element_numbering();

   return 0;
}
------------------------------------------------------------------------------
_______________________________________________
Libmesh-devel mailing list
Libmesh-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/libmesh-devel

Reply via email to