The problem is likely this part:
download: |
mumps, scalapack, blacs, ml
If you remove this, then it should find the correct ParMETIS. If you need
to build PETSc with any of these packages, then we will have to fix them in
HashStack.
Manipulating PATH and LD_LIBRARY_PATH is probably not the way to go.
Johannes
On Mon, Oct 5, 2015 at 4:54 PM Johannes Neumann <[email protected]>
wrote:
> I think you are right about the warning. The paramters are the one from
> the default linux yaml in the fenics dev tools, i.e.:
>
> petsc:
> version: '3.6.1'
> build_with: |
> openblas, parmetis, scotch, suitesparse, superlu_dist, hypre
>
> download: |
> mumps, scalapack, blacs, ml
> coptflags: -O2
> link: shared
> debug: false
>
> Removing all references form $PATH and $LD_LIBRARY_PATH seems to help but
> now I will have to recompile everything as dolfin comlains about missing
> MPI refs (libmpi_mpifh.so.2: cannot open shared object file: No such file
> or directory). It seems that ipython also uses the wrong MPI but the
> ipython exectuable is the one from the hashdist profile. BTW. this also
> extends to scotch etc. as ldd reveals. I will come back with informations
> once that build is complete.
>
>
> Am 05.10.2015 um 16:36 schrieb Johannes Ring:
>
> How do you specify how to build PETSc in your HashDist profile? Are you
> using 'download' to allow PETSc to download and build any packages or are
> you only using 'build_with'?
>
>
> I do not think the warning about MPI_DIR has anything to do with this..
>
> Johannes
>
>
> On Mon, Oct 5, 2015 at 3:19 PM Johannes Neumann <[email protected]>
> wrote:
>
>>
>>
>>
>>
>>
>> Hello,
>>
>>
>>
>> I have an update here as I ran into the problem again, where the
>> wrong parmetis is used by petsc. I tried a couple of different
>> things (static build, tell petsc to download parmetis on its own etc
>> pp.) but this did not help. I think the problem with this is
>> connected to the following warning that I get during the petsc buid
>> in the log:
>>
>>
>>
>> 2015/10/05 15:09:41 - INFO: [package:run_job]
>>
>> ===============================================================================
>>
>> 2015/10/05 15:09:41 - INFO: [package:run_job]
>> ***** WARNING: MPI_DIR (set to
>> /Home/flow/neumann/.hashdist/bld/mpi/gmkx3ftk3tz2) found in
>> environment variables - ignoring
>>
>> 2015/10/05 15:09:41 - INFO: [package:run_job]
>> use ./configure MPI_DIR=$MPI_DIR if you really want to use that
>> value ******
>>
>> 2015/10/05 15:09:41 - INFO: [package:run_job]
>>
>> ===============================================================================
>>
>>
>>
>> I couldn't find anything related to that variable in the petsc.yaml
>> or the accompanying .py script. So I gues that is why Petsc uses the
>> system parmetis and not the locally build one. As I understand, I
>> will have to add this in the py. script. I will try to do it myself
>> locally but help would be appreciated.
>>
>>
>>
>> Best,
>>
>> Jo
>>
>>
>>
>>
>> Am 25.08.2015 um 18:49 schrieb Johannes
>> Ring:
>>
>>
>>
>> Yes, mumps will not be available, but we can try to add it
>> back later.
>>
>>
>>
>>
>> Johannes
>>
>>
>> On Tue, Aug 25, 2015 at 6:41 PM Johannes
>> Neumann <[email protected]>
>> wrote:
>>
>>
>>
>>> ok i will try it now. thanks for your help. this
>>> means that mumps for example will not be available? I
>>> don't need it at the moment so just, that I know.
>>>
>>>
>>>
>>>
>>>
>>> Am 25.08.2015 um 18:35 schrieb Johannes Ring:
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Try to remove this part from the petsc
>>> build:
>>>
>>>
>>>
>>> download: |
>>>
>>> mumps, scalapack, blacs, ml
>>>
>>>
>>>
>>>
>>> I think this could be the problem.
>>>
>>>
>>>
>>>
>>> Johannes
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Tue, Aug 25, 2015 at
>>> 6:26 PM Johannes Neumann <[email protected]>
>>>
>>> wrote:
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>>
>>>> oh, all right ;) here it is.
>>>>
>>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>>
>>>>
>>>>
>>>>
>>>> Am 25.08.2015 um 18:24 schrieb
>>>> Johannes Ring:
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>>
>>>>
>>>> [Please keep
>>>> fenics-support in Cc]
>>>>
>>>>
>>>>
>>>> Thanks Jo, but I was thinking
>>>> about the yaml file, where you
>>>> specify the packages you want to
>>>> build.
>>>>
>>>>
>>>>
>>>>
>>>> Johannes
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Tue, Aug 25,
>>>> 2015 at 6:13 PM Johannes
>>>> Neumann <[email protected]>
>>>>
>>>>
>>>> wrote:
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> I compressed the profile
>>>>> without most of the libs and
>>>>> shared
>>>>>
>>>>> data/header files. If you
>>>>> need any of these, please
>>>>> let me know. You
>>>>>
>>>>> can download it here:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> https://drive.google.com/file/d/0BybZyFhAQBNMa3pYdGk5SURKWUk/view?usp=sharing
>>>>> <https://drive.google..com/file/d/0BybZyFhAQBNMa3pYdGk5SURKWUk/view?usp=sharing>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> The custom profile is the
>>>>> default linux profile plus
>>>>> scipy and
>>>>>
>>>>> joblib.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Best,
>>>>>
>>>>>
>>>>>
>>>>> Jo
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Am 25.08.2015 um 14:06
>>>>> schrieb Johannes
>>>>>
>>>>> Ring:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Can you send us
>>>>> your hashdist profile?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Johannes
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Tue,
>>>>> Aug 25, 2015 at 1:27
>>>>> PM Johannes Neumann
>>>>>
>>>>> <[email protected]>
>>>>>
>>>>>
>>>>>
>>>>> wrote:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> we use a compute
>>>>>> cluster with shared
>>>>>> home dir but
>>>>>> different
>>>>>>
>>>>>> software
>>>>>>
>>>>>>
>>>>>>
>>>>>> running on some of
>>>>>> the nodes. so I am
>>>>>> using the
>>>>>>
>>>>>> hashdist/hashstack
>>>>>>
>>>>>>
>>>>>>
>>>>>> version of fenics
>>>>>> with works great
>>>>>> except for when I
>>>>>> use
>>>>>>
>>>>>> other nodes
>>>>>>
>>>>>>
>>>>>>
>>>>>> with different
>>>>>> library versions.
>>>>>> The output it
>>>>>> generates is
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>>>> from dolfin import
>>>>>> *
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> File
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> "$HOME/.hashdist/bld/profile/7tifyrwdwqtd/lib/python2.7/site-packages/dolfin/__init__.py",
>>>>>>
>>>>>>
>>>>>>
>>>>>> line 16, in
>>>>>> <module>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> from . import
>>>>>> cpp
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> File
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> "$HOME/.hashdist/bld/profile/7tifyrwdwqtd/lib/python2.7/site-packages/dolfin/cpp/__init__.py",
>>>>>>
>>>>>>
>>>>>>
>>>>>> line 42, in
>>>>>> <module>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> exec("from ..
>>>>>> import %s" %
>>>>>> module_name)
>>>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> File
>>>>>> "<string>",
>>>>>> line 1, in
>>>>>> <module>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> File
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> "$HOME/.hashdist/bld/profile/7tifyrwdwqtd/lib/python2.7/site-packages/dolfin/cpp/common.py",
>>>>>>
>>>>>>
>>>>>>
>>>>>> line 32, in
>>>>>> <module>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> _common =
>>>>>> swig_import_helper()
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> File
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> "$HOME/.hashdist/bld/profile/7tifyrwdwqtd/lib/python2.7/site-packages/dolfin/cpp/common.py",
>>>>>>
>>>>>>
>>>>>>
>>>>>> line 28, in
>>>>>> swig_import_helper
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> _mod =
>>>>>> imp.load_module('_common',
>>>>>> fp, pathname,
>>>>>>
>>>>>> description)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ImportError:
>>>>>>
>>>>>>
>>>>>>
>>>>>> $HOME/.hashdist/bld/petsc/cgj5xhkahatd/lib/libpetsc.so.3.5:
>>>>>>
>>>>>>
>>>>>>
>>>>>> undefined symbol:
>>>>>> ParMETIS_V3_Mesh2Dual
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> And
>>>>>>
>>>>>> indeed, when I ldd
>>>>>>
>>>>>>
>>>>>>
>>>>>> the petsc
>>>>>> library I get
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> neumann@erhard-17:~/fenics/tests>
>>>>>>
>>>>>> ldd
>>>>>>
>>>>>>
>>>>>>
>>>>>> $HOME/.hashdist/bld/petsc/cgj5xhkahatd/lib/libpetsc.so.3.5
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> linux-vdso.so.1
>>>>>> (0x00007ffc359b3000)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> libHYPRE-2.9.1a.so
>>>>>>
>>>>>> =>
>>>>>>
>>>>>>
>>>>>>
>>>>>> $HOME/.hashdist/bld/hypre/kh4vd5dptalu/lib/libHYPRE-2.9.1a.so
>>>>>>
>>>>>>
>>>>>>
>>>>>> (0x00002acfabca6000)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> libmpichcxx.so.10
>>>>>> =>
>>>>>>
>>>>>>
>>>>>>
>>>>>> $HOME/.hashdist/bld/mpi/gmkx3ftk3tz2/lib/libmpichcxx.so.10
>>>>>>
>>>>>>
>>>>>>
>>>>>> (0x00002acfac13c000)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> libstdc++.so.6
>>>>>> =>
>>>>>> /usr/lib64/libstdc++.so.6
>>>>>>
>>>>>>
>>>>>>
>>>>>> (0x00002acfac35f000)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> libopenblas.so.0
>>>>>> =>
>>>>>>
>>>>>>
>>>>>>
>>>>>> $HOME/.hashdist/bld/lapack/vwdn72phzwm3/lib/libopenblas.so.0
>>>>>>
>>>>>>
>>>>>>
>>>>>> (0x00002acfac667000)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> libparmetis.so
>>>>>> =>
>>>>>>
>>>>>>
>>>>>>
>>>>>> /usr/lib64/mpi/gcc/openmpi/lib64/libparmetis.so
>>>>>>
>>>>>>
>>>>>>
>>>>>> (0x00002acfad417000)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>> libmetis..so =>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> $HOME/.hashdist/bld/parmetis/hbifmb4gf2ps/lib/libmetis.so
>>>>>>
>>>>>>
>>>>>>
>>>>>> (0x00002acfad61b000)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> libX11.so.6 =>
>>>>>> /usr/lib64/libX11.so.6
>>>>>>
>>>>>>
>>>>>>
>>>>>> (0x00002acfad887000)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> libpthread.so.0
>>>>>> =>
>>>>>> /lib64/libpthread.so.0
>>>>>>
>>>>>>
>>>>>>
>>>>>> (0x00002acfadbc5000)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> libm.so.6
>>>>>> =>
>>>>>> /lib64/libm.so.6
>>>>>>
>>>>>> (0x00002acfadde2000)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> libmpichf90.so.10
>>>>>> =>
>>>>>>
>>>>>>
>>>>>>
>>>>>> $HOME/.hashdist/bld/mpi/gmkx3ftk3tz2/lib/libmpichf90.so.10
>>>>>>
>>>>>>
>>>>>>
>>>>>> (0x00002acfae0e3000)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> libgfortran.so.3
>>>>>> =>
>>>>>> /usr/lib64/libgfortran.so.3
>>>>>>
>>>>>>
>>>>>>
>>>>>> (0x00002acfae2e6000)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> libquadmath.so.0
>>>>>> =>
>>>>>> /usr/lib64/libquadmath.so.0
>>>>>>
>>>>>>
>>>>>>
>>>>>> (0x00002acfae600000)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>> libdl.so..2 =>
>>>>>> /lib64/libdl.so.2
>>>>>>
>>>>>> (0x00002acfae83c000)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> libmpich..so.10
>>>>>> =>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>>
>>>>>
>>>>>
>>>>>
_______________________________________________
fenics-support mailing list
[email protected]
http://fenicsproject.org/mailman/listinfo/fenics-support
