Looks like some issue with your MPI. You might want to talk with your sysadmin about it.
Also send us some compile logs - so we know whats hapenning. For eg: cd src/ksp/ksp/examples/tutorials/ make ex2 mpiexec -n 2 ./ex2 [or however you are supporsed to run MPI binaries on this cluster] BTW: If you are currently doing development - don't bother with a cluster MPI - and just use --download-mpich=1 Satish On Sun, 5 Jul 2009, Yixun Liu wrote: > Hi, > After making test, I received a lot of warnings and lib load failures. > > Running test examples to verify correct installation > Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI > process > See http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html > DAT: library load failure: /usr/lib64/libdaplcma.so.1: undefined symbol: > dat_registry_add_provider > -------------------------------------------------------------------------- > > WARNING: Failed to open "OpenIB-cma" > [DAT_PROVIDER_NOT_FOUND:DAT_NAME_NOT_REGISTERED]. > This may be a real error or it may be an invalid entry in the uDAPL > Registry which is contained in the dat.conf file. Contact your local > System Administrator to confirm the availability of the interfaces in > the dat.conf file. > -------------------------------------------------------------------------- > DAT: library load failure: /usr/lib64/libdaplcma.so.1: undefined symbol: > dat_registry_add_provider > -------------------------------------------------------------------------- > > WARNING: Failed to open "OpenIB-cma-1" > [DAT_PROVIDER_NOT_FOUND:DAT_NAME_NOT_REGISTERED]. > This may be a real error or it may be an invalid entry in the uDAPL > Registry which is contained in the dat.conf file. Contact your local > System Administrator to confirm the availability of the interfaces in > the dat.conf file. > -------------------------------------------------------------------------- > DAT: library load failure: /usr/lib64/libdaplscm.so.1: undefined symbol: > dat_registry_add_provider > -------------------------------------------------------------------------- > > WARNING: Failed to open "OpenIB-mthca0-1" > [DAT_PROVIDER_NOT_FOUND:DAT_NAME_NOT_REGISTERED]. > This may be a real error or it may be an invalid entry in the uDAPL > Registry which is contained in the dat.conf file. Contact your local > System Administrator to confirm the availability of the interfaces in > the dat.conf file. > -------------------------------------------------------------------------- > DAT: library load failure: /usr/lib64/libdaplscm.so.1: undefined symbol: > dat_registry_add_provider > -------------------------------------------------------------------------- > > WARNING: Failed to open "OpenIB-mthca0-2" > [DAT_PROVIDER_NOT_FOUND:DAT_NAME_NOT_REGISTERED]. > This may be a real error or it may be an invalid entry in the uDAPL > Registry which is contained in the dat.conf file. Contact your local > System Administrator to confirm the availability of the interfaces in > the dat.conf file. > -------------------------------------------------------------------------- > DAT: library load failure: /usr/lib64/libdaplscm.so.1: undefined symbol: > dat_registry_add_provider > -------------------------------------------------------------------------- > > WARNING: Failed to open "OpenIB-mlx4_0-1" > [DAT_PROVIDER_NOT_FOUND:DAT_NAME_NOT_REGISTERED]. > This may be a real error or it may be an invalid entry in the uDAPL > Registry which is contained in the dat.conf file. Contact your local > System Administrator to confirm the availability of the interfaces in > the dat.conf file. > -------------------------------------------------------------------------- > DAT: library load failure: /usr/lib64/libdaplscm.so.1: undefined symbol: > dat_registry_add_provider > -------------------------------------------------------------------------- > > WARNING: Failed to open "OpenIB-mlx4_0-2" > [DAT_PROVIDER_NOT_FOUND:DAT_NAME_NOT_REGISTERED]. > This may be a real error or it may be an invalid entry in the uDAPL > Registry which is contained in the dat.conf file. Contact your local > System Administrator to confirm the availability of the interfaces in > the dat.conf file. > -------------------------------------------------------------------------- > DAT: library load failure: /usr/lib64/libdaplcma.so.1: undefined symbol: > dat_registry_add_provider > -------------------------------------------------------------------------- > > WARNING: Failed to open "OpenIB-iwarp" > [DAT_PROVIDER_NOT_FOUND:DAT_NAME_NOT_REGISTERED]. > This may be a real error or it may be an invalid entry in the uDAPL > Registry which is contained in the dat.conf file. Contact your local > System Administrator to confirm the availability of the interfaces in > the dat.conf file. > -------------------------------------------------------------------------- > -------------------------------------------------------------------------- > [0,1,0]: uDAPL on host md was unable to find any NICs. > Another transport will be used instead, although this may result in > lower performance. > -------------------------------------------------------------------------- > lid velocity = 0.0016, prandtl # = 1, grashof # = 1 > Number of Newton iterations = 2 > lid velocity = 0.0016, prandtl # = 1, grashof # = 1 > Number of Newton iterations = 2 > Possible error running C/C++ src/snes/examples/tutorials/ex19 with 2 MPI > processes > See http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html > DAT: library load failure: /usr/lib64/libdaplcma.so.1: undefined symbol: > dat_registry_add_provider > -------------------------------------------------------------------------- > > WARNING: Failed to open "OpenIB-cma" > [DAT_PROVIDER_NOT_FOUND:DAT_NAME_NOT_REGISTERED]. > This may be a real error or it may be an invalid entry in the uDAPL > Registry which is contained in the dat.conf file. Contact your local > System Administrator to confirm the availability of the interfaces in > the dat.conf file. > -------------------------------------------------------------------------- > DAT: library load failure: /usr/lib64/libdaplcma.so.1: undefined symbol: > dat_registry_add_provider > -------------------------------------------------------------------------- > > WARNING: Failed to open "OpenIB-cma-1" > [DAT_PROVIDER_NOT_FOUND:DAT_NAME_NOT_REGISTERED]. > This may be a real error or it may be an invalid entry in the uDAPL > Registry which is contained in the dat.conf file. Contact your local > System Administrator to confirm the availability of the interfaces in > the dat.conf file. > -------------------------------------------------------------------------- > DAT: library load failure: /usr/lib64/libdaplscm.so.1: undefined symbol: > dat_registry_add_provider > -------------------------------------------------------------------------- > > WARNING: Failed to open "OpenIB-mthca0-1" > [DAT_PROVIDER_NOT_FOUND:DAT_NAME_NOT_REGISTERED]. > This may be a real error or it may be an invalid entry in the uDAPL > Registry which is contained in the dat.conf file. Contact your local > System Administrator to confirm the availability of the interfaces in > the dat.conf file. > -------------------------------------------------------------------------- > DAT: library load failure: /usr/lib64/libdaplscm.so.1: undefined symbol: > dat_registry_add_provider > -------------------------------------------------------------------------- > > WARNING: Failed to open "OpenIB-mthca0-2" > [DAT_PROVIDER_NOT_FOUND:DAT_NAME_NOT_REGISTERED]. > This may be a real error or it may be an invalid entry in the uDAPL > Registry which is contained in the dat.conf file. Contact your local > System Administrator to confirm the availability of the interfaces in > the dat.conf file. > -------------------------------------------------------------------------- > DAT: library load failure: /usr/lib64/libdaplscm.so.1: undefined symbol: > dat_registry_add_provider > -------------------------------------------------------------------------- > > WARNING: Failed to open "OpenIB-mlx4_0-1" > [DAT_PROVIDER_NOT_FOUND:DAT_NAME_NOT_REGISTERED]. > This may be a real error or it may be an invalid entry in the uDAPL > Registry which is contained in the dat.conf file. Contact your local > System Administrator to confirm the availability of the interfaces in > the dat.conf file. > -------------------------------------------------------------------------- > DAT: library load failure: /usr/lib64/libdaplscm.so.1: undefined symbol: > dat_registry_add_provider > -------------------------------------------------------------------------- > > WARNING: Failed to open "OpenIB-mlx4_0-2" > [DAT_PROVIDER_NOT_FOUND:DAT_NAME_NOT_REGISTERED]. > This may be a real error or it may be an invalid entry in the uDAPL > Registry which is contained in the dat.conf file. Contact your local > System Administrator to confirm the availability of the interfaces in > the dat.conf file. > -------------------------------------------------------------------------- > DAT: library load failure: /usr/lib64/libdaplcma.so.1: undefined symbol: > dat_registry_add_provider > -------------------------------------------------------------------------- > > WARNING: Failed to open "OpenIB-iwarp" > [DAT_PROVIDER_NOT_FOUND:DAT_NAME_NOT_REGISTERED]. > This may be a real error or it may be an invalid entry in the uDAPL > Registry which is contained in the dat.conf file. Contact your local > System Administrator to confirm the availability of the interfaces in > the dat.conf file. > -------------------------------------------------------------------------- > -------------------------------------------------------------------------- > [0,1,1]: uDAPL on host md was unable to find any NICs. > Another transport will be used instead, although this may result in > lower performance. > -------------------------------------------------------------------------- > DAT: library load failure: /usr/lib64/libdaplcma.so.1: undefined symbol: > dat_registry_add_provider > -------------------------------------------------------------------------- > > WARNING: Failed to open "OpenIB-cma" > [DAT_PROVIDER_NOT_FOUND:DAT_NAME_NOT_REGISTERED]. > This may be a real error or it may be an invalid entry in the uDAPL > Registry which is contained in the dat.conf file. Contact your local > System Administrator to confirm the availability of the interfaces in > the dat.conf file. > -------------------------------------------------------------------------- > DAT: library load failure: /usr/lib64/libdaplcma.so.1: undefined symbol: > dat_registry_add_provider > -------------------------------------------------------------------------- > > WARNING: Failed to open "OpenIB-cma-1" > [DAT_PROVIDER_NOT_FOUND:DAT_NAME_NOT_REGISTERED]. > This may be a real error or it may be an invalid entry in the uDAPL > Registry which is contained in the dat.conf file. Contact your local > System Administrator to confirm the availability of the interfaces in > the dat.conf file. > -------------------------------------------------------------------------- > DAT: library load failure: /usr/lib64/libdaplscm.so.1: undefined symbol: > dat_registry_add_provider > -------------------------------------------------------------------------- > > WARNING: Failed to open "OpenIB-mthca0-1" > [DAT_PROVIDER_NOT_FOUND:DAT_NAME_NOT_REGISTERED]. > This may be a real error or it may be an invalid entry in the uDAPL > Registry which is contained in the dat.conf file. Contact your local > System Administrator to confirm the availability of the interfaces in > the dat.conf file. > -------------------------------------------------------------------------- > DAT: library load failure: /usr/lib64/libdaplscm.so.1: undefined symbol: > dat_registry_add_provider > -------------------------------------------------------------------------- > > WARNING: Failed to open "OpenIB-mthca0-2" > [DAT_PROVIDER_NOT_FOUND:DAT_NAME_NOT_REGISTERED]. > This may be a real error or it may be an invalid entry in the uDAPL > Registry which is contained in the dat.conf file. Contact your local > System Administrator to confirm the availability of the interfaces in > the dat.conf file. > -------------------------------------------------------------------------- > DAT: library load failure: /usr/lib64/libdaplscm.so.1: undefined symbol: > dat_registry_add_provider > -------------------------------------------------------------------------- > > WARNING: Failed to open "OpenIB-mlx4_0-1" > [DAT_PROVIDER_NOT_FOUND:DAT_NAME_NOT_REGISTERED]. > This may be a real error or it may be an invalid entry in the uDAPL > Registry which is contained in the dat.conf file. Contact your local > System Administrator to confirm the availability of the interfaces in > the dat.conf file. > -------------------------------------------------------------------------- > DAT: library load failure: /usr/lib64/libdaplscm.so.1: undefined symbol: > dat_registry_add_provider > -------------------------------------------------------------------------- > > WARNING: Failed to open "OpenIB-mlx4_0-2" > [DAT_PROVIDER_NOT_FOUND:DAT_NAME_NOT_REGISTERED]. > This may be a real error or it may be an invalid entry in the uDAPL > Registry which is contained in the dat.conf file. Contact your local > System Administrator to confirm the availability of the interfaces in > the dat.conf file. > -------------------------------------------------------------------------- > DAT: library load failure: /usr/lib64/libdaplcma.so.1: undefined symbol: > dat_registry_add_provider > -------------------------------------------------------------------------- > > WARNING: Failed to open "OpenIB-iwarp" > [DAT_PROVIDER_NOT_FOUND:DAT_NAME_NOT_REGISTERED]. > This may be a real error or it may be an invalid entry in the uDAPL > Registry which is contained in the dat.conf file. Contact your local > System Administrator to confirm the availability of the interfaces in > the dat.conf file. > -------------------------------------------------------------------------- > -------------------------------------------------------------------------- > [0,1,0]: uDAPL on host md was unable to find any NICs. > Another transport will be used instead, although this may result in > lower performance. > -------------------------------------------------------------------------- > lid velocity = 0.0016, prandtl # = 1, grashof # = 1 > Number of Newton iterations = 2 > lid velocity = 0.0016, prandtl # = 1, grashof # = 1 > Number of Newton iterations = 2 > Possible error running Graphics examples > src/snes/examples/tutorials/ex19 1 MPI process > See http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html > DAT: library load failure: /usr/lib64/libdaplcma.so.1: undefined symbol: > dat_registry_add_provider > -------------------------------------------------------------------------- > > WARNING: Failed to open "OpenIB-cma" > [DAT_PROVIDER_NOT_FOUND:DAT_NAME_NOT_REGISTERED]. > This may be a real error or it may be an invalid entry in the uDAPL > Registry which is contained in the dat.conf file. Contact your local > System Administrator to confirm the availability of the interfaces in > the dat.conf file. > -------------------------------------------------------------------------- > DAT: library load failure: /usr/lib64/libdaplcma.so.1: undefined symbol: > dat_registry_add_provider > DAT: library load failure: /usr/lib64/libdaplscm.so.1: undefined symbol: > dat_registry_add_provider > DAT: library load failure: /usr/lib64/libdaplscm.so.1: undefined symbol: > dat_registry_add_provider > DAT: library load failure: /usr/lib64/libdaplscm.so.1: undefined symbol: > dat_registry_add_provider > DAT: library load failure: /usr/lib64/libdaplscm.so.1: undefined symbol: > dat_registry_add_provider > DAT: library load failure: /usr/lib64/libdaplcma.so.1: undefined symbol: > dat_registry_add_provider > -------------------------------------------------------------------------- > > WARNING: Failed to open "OpenIB-cma-1" > [DAT_PROVIDER_NOT_FOUND:DAT_NAME_NOT_REGISTERED]. > This may be a real error or it may be an invalid entry in the uDAPL > Registry which is contained in the dat.conf file. Contact your local > System Administrator to confirm the availability of the interfaces in > the dat.conf file. > -------------------------------------------------------------------------- > -------------------------------------------------------------------------- > > WARNING: Failed to open "OpenIB-mthca0-1" > [DAT_PROVIDER_NOT_FOUND:DAT_NAME_NOT_REGISTERED]. > This may be a real error or it may be an invalid entry in the uDAPL > Registry which is contained in the dat.conf file. Contact your local > System Administrator to confirm the availability of the interfaces in > the dat.conf file. > -------------------------------------------------------------------------- > -------------------------------------------------------------------------- > > WARNING: Failed to open "OpenIB-mthca0-2" > [DAT_PROVIDER_NOT_FOUND:DAT_NAME_NOT_REGISTERED]. > This may be a real error or it may be an invalid entry in the uDAPL > Registry which is contained in the dat.conf file. Contact your local > System Administrator to confirm the availability of the interfaces in > the dat.conf file. > -------------------------------------------------------------------------- > -------------------------------------------------------------------------- > > WARNING: Failed to open "OpenIB-mlx4_0-1" > [DAT_PROVIDER_NOT_FOUND:DAT_NAME_NOT_REGISTERED]. > This may be a real error or it may be an invalid entry in the uDAPL > Registry which is contained in the dat.conf file. Contact your local > System Administrator to confirm the availability of the interfaces in > the dat.conf file. > -------------------------------------------------------------------------- > -------------------------------------------------------------------------- > > WARNING: Failed to open "OpenIB-mlx4_0-2" > [DAT_PROVIDER_NOT_FOUND:DAT_NAME_NOT_REGISTERED]. > This may be a real error or it may be an invalid entry in the uDAPL > Registry which is contained in the dat.conf file. Contact your local > System Administrator to confirm the availability of the interfaces in > the dat.conf file. > -------------------------------------------------------------------------- > -------------------------------------------------------------------------- > > WARNING: Failed to open "OpenIB-iwarp" > [DAT_PROVIDER_NOT_FOUND:DAT_NAME_NOT_REGISTERED]. > This may be a real error or it may be an invalid entry in the uDAPL > Registry which is contained in the dat.conf file. Contact your local > System Administrator to confirm the availability of the interfaces in > the dat.conf file. > -------------------------------------------------------------------------- > -------------------------------------------------------------------------- > [0,1,0]: uDAPL on host md was unable to find any NICs. > Another transport will be used instead, although this may result in > lower performance. > -------------------------------------------------------------------------- > lid velocity = 0.0016, prandtl # = 1, grashof # = 1 > Number of Newton iterations = 2 > lid velocity = 0.0016, prandtl # = 1, grashof # = 1 > Number of Newton iterations = 2 > Error running Fortran example src/snes/examples/tutorials/ex5f with 1 > MPI process > See http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html > DAT: library load failure: /usr/lib64/libdaplcma.so.1: undefined symbol: > dat_registry_add_provider > -------------------------------------------------------------------------- > > WARNING: Failed to open "OpenIB-cma" > [DAT_PROVIDER_NOT_FOUND:DAT_NAME_NOT_REGISTERED]. > This may be a real error or it may be an invalid entry in the uDAPL > Registry which is contained in the dat.conf file. Contact your local > System Administrator to confirm the availability of the interfaces in > the dat.conf file. > -------------------------------------------------------------------------- > DAT: library load failure: /usr/lib64/libdaplcma.so.1: undefined symbol: > dat_registry_add_provider > -------------------------------------------------------------------------- > > WARNING: Failed to open "OpenIB-cma-1" > [DAT_PROVIDER_NOT_FOUND:DAT_NAME_NOT_REGISTERED]. > This may be a real error or it may be an invalid entry in the uDAPL > Registry which is contained in the dat.conf file. Contact your local > System Administrator to confirm the availability of the interfaces in > the dat.conf file. > -------------------------------------------------------------------------- > DAT: library load failure: /usr/lib64/libdaplscm.so.1: undefined symbol: > dat_registry_add_provider > -------------------------------------------------------------------------- > > WARNING: Failed to open "OpenIB-mthca0-1" > [DAT_PROVIDER_NOT_FOUND:DAT_NAME_NOT_REGISTERED]. > This may be a real error or it may be an invalid entry in the uDAPL > Registry which is contained in the dat.conf file. Contact your local > System Administrator to confirm the availability of the interfaces in > the dat.conf file. > -------------------------------------------------------------------------- > DAT: library load failure: /usr/lib64/libdaplscm.so.1: undefined symbol: > dat_registry_add_provider > -------------------------------------------------------------------------- > > WARNING: Failed to open "OpenIB-mthca0-2" > [DAT_PROVIDER_NOT_FOUND:DAT_NAME_NOT_REGISTERED]. > This may be a real error or it may be an invalid entry in the uDAPL > Registry which is contained in the dat.conf file. Contact your local > System Administrator to confirm the availability of the interfaces in > the dat.conf file. > -------------------------------------------------------------------------- > DAT: library load failure: /usr/lib64/libdaplscm.so.1: undefined symbol: > dat_registry_add_provider > -------------------------------------------------------------------------- > > WARNING: Failed to open "OpenIB-mlx4_0-1" > [DAT_PROVIDER_NOT_FOUND:DAT_NAME_NOT_REGISTERED]. > This may be a real error or it may be an invalid entry in the uDAPL > Registry which is contained in the dat.conf file. Contact your local > System Administrator to confirm the availability of the interfaces in > the dat.conf file. > -------------------------------------------------------------------------- > DAT: library load failure: /usr/lib64/libdaplscm.so.1: undefined symbol: > dat_registry_add_provider > -------------------------------------------------------------------------- > > WARNING: Failed to open "OpenIB-mlx4_0-2" > [DAT_PROVIDER_NOT_FOUND:DAT_NAME_NOT_REGISTERED]. > This may be a real error or it may be an invalid entry in the uDAPL > Registry which is contained in the dat.conf file. Contact your local > System Administrator to confirm the availability of the interfaces in > the dat.conf file. > -------------------------------------------------------------------------- > DAT: library load failure: /usr/lib64/libdaplcma.so.1: undefined symbol: > dat_registry_add_provider > -------------------------------------------------------------------------- > > WARNING: Failed to open "OpenIB-iwarp" > [DAT_PROVIDER_NOT_FOUND:DAT_NAME_NOT_REGISTERED]. > This may be a real error or it may be an invalid entry in the uDAPL > Registry which is contained in the dat.conf file. Contact your local > System Administrator to confirm the availability of the interfaces in > the dat.conf file. > -------------------------------------------------------------------------- > -------------------------------------------------------------------------- > [0,1,0]: uDAPL on host md was unable to find any NICs. > Another transport will be used instead, although this may result in > lower performance. > -------------------------------------------------------------------------- > Number of Newton iterations = 4 > Completed test examples > > > Thanks. > > Yixun >
