Re: [OMPI users] psec warning when launching with srun
Hello all, yes, I have noticed that with the individual development libraries for a very long time, because I rather compile the entire products by HAND to my needs myself and do not want to rely on the libraries of the individual Linux - variants. I have done very well with this at the moment. What I would like to do now is to test and implement this on a Rocky Linux system. I can gladly report the results here Best regards from Berlin Z. Matthias > On Samstag, Mai 20, 2023 at 1:29 PM, christof.koehler--- via users > mailto:users@lists.open-mpi.org)> wrote: > Hello Gilles, > > thank you very much for the prompt patch. > > I can confirm that configure now prefers the external PMIx. I can > confirm that the munge warnings and PMIx errors we observed are gone. An > mpi hello world runs successfully with srun --mpi=pmix and --mpi=pmi2. > > I noticed that configure complained loudly about a missing external > libevent (i.e. libevent-devel package), but did not complain at all > that an external hwloc-devel was also missing. > > Best Regards > > Christof > > > > On Sat, May 20, 2023 at 06:54:54PM +0900, Gilles Gouaillardet wrote: > > Christof, > > > > Open MPI switching to the internal PMIx is a bug I addressed in > > https://github.com/open-mpi/ompi/pull/11704 > > > > Feel free to manually download and apply the patch, you will then need > > recent autotools and run > > ./autogen.pl --force > > > > An other option is to manually edit the configure file > > > > Look for the following snippet > > > > # Final check - if they didn't point us explicitly at an > > external version > > > > # but we found one anyway, use the internal version if it is > > higher > > > > if test "$opal_external_pmix_version" != "internal" && (test -z > > "$with_pmix" || test "$with_pmix" = "yes") > > > > then : > > > > if test "$opal_external_pmix_version" != "3x" > > > > > > and replace the last line with > > > > if test $opal_external_pmix_version_major -lt 3 > > > > > > Cheers, > > > > Gilles > > > > On Sat, May 20, 2023 at 6:13 PM christof.koehler--- via users < > > users@lists.open-mpi.org> wrote: > > > > > Hello Z. Matthias Krawutschke, > > > > > > On Fri, May 19, 2023 at 09:08:08PM +0200, Zhéxué M. Krawutschke wrote: > > > > Hello Christoph, > > > > what exactly is your problem with OpenMPI and Slurm? > > > > Do you compile the products yourself? Which LINUX distribution and > > > version are you using? > > > > > > > > If you compile the software yourself, could you please tell me what the > > > "configure" command looks like and which MUNGE version is in use? From the > > > distribution or compiled by yourself? > > > > > > > > I would be very happy to take on this topic and help you. You can also > > > reach me at +49 176 67270992. > > > > Best regards from Berlin > > > > > > please refer to (especially the end) of my first mail in this thread > > > which is available here > > > https://www.mail-archive.com/users@lists.open-mpi.org/msg35141.html > > > > > > I believe this contains the relevant information you are requesting. The > > > second mail which you are replying to was just additional information. > > > My apologies if this led to confusion. > > > > > > Please let me know if any relevant information is missing from my first > > > email. At the bottom of this email I include the ompi_info output as > > > further addendum. > > > > > > To summarize: I would like to understand where the munge warning > > > and PMIx error described in the first email (and the github link > > > included) come from. The explanation in the github issue > > > does not appear to be correct as all munge libraries are > > > available everywhere. To me, it appears at the moment that OpenMPIs > > > configure decides erroneously to build and use the internal pmix > > > instead of using the (presumably) newer externally available PMIx, > > > leading to launcher problems with srun. > > > > > > > > > Best Regards > > > > > > Christof > > > > > > Package: Open MPI root@admin.service Distribution > > > Open MPI: 4.1.5 > > > Open MPI repo revision: v4.1.5 > > > Open MPI release date: Feb 23, 2023 > > > Open RTE: 4.1.5 > > > Open RTE repo revision: v4.1.5 > > > Open RTE release date: Feb 23, 2023 > > > OPAL: 4.1.5 > > > OPAL repo revision: v4.1.5 > > > OPAL release date: Feb 23, 2023 > > > MPI API: 3.1.0 > > > Ident string: 4.1.5 > > > Prefix: /cluster/mpi/openmpi/4.1.5/gcc-11.3.1 > > > Configured architecture: x86_64-pc-linux-gnu > > > Configure host: admin.service > > > Configured by: root > > > Configured on: Wed May 17 18:45:42 UTC 2023 > > > Configure host: admin.service > > > Configure command line: '--enable-mpi1-compatibility' > > > '--enable-orterun-prefix-by-default' > > > '--with-ofi=/cluster/libraries/libfabric/1.18.0/' '--with-slurm' > > > '--with-pmix' '--with-pmix-libdir=/usr/lib64' '--with-pmi' > > > '--with-pmi-libdir=/usr/lib64' > > > '--prefix=/cluster/mpi/openmpi/4.1.5/gcc-11.3.1' > > > Built by: root > > > Built on: Wed May
Re: [OMPI users] psec warning when launching with srun
Hello Gilles, thank you very much for the prompt patch. I can confirm that configure now prefers the external PMIx. I can confirm that the munge warnings and PMIx errors we observed are gone. An mpi hello world runs successfully with srun --mpi=pmix and --mpi=pmi2. I noticed that configure complained loudly about a missing external libevent (i.e. libevent-devel package), but did not complain at all that an external hwloc-devel was also missing. Best Regards Christof On Sat, May 20, 2023 at 06:54:54PM +0900, Gilles Gouaillardet wrote: > Christof, > > Open MPI switching to the internal PMIx is a bug I addressed in > https://github.com/open-mpi/ompi/pull/11704 > > Feel free to manually download and apply the patch, you will then need > recent autotools and run > ./autogen.pl --force > > An other option is to manually edit the configure file > > Look for the following snippet > ># Final check - if they didn't point us explicitly at an > external version > ># but we found one anyway, use the internal version if it is > higher > >if test "$opal_external_pmix_version" != "internal" && (test -z > "$with_pmix" || test "$with_pmix" = "yes") > > then : > > if test "$opal_external_pmix_version" != "3x" > > > and replace the last line with > > if test $opal_external_pmix_version_major -lt 3 > > > Cheers, > > Gilles > > On Sat, May 20, 2023 at 6:13 PM christof.koehler--- via users < > users@lists.open-mpi.org> wrote: > > > Hello Z. Matthias Krawutschke, > > > > On Fri, May 19, 2023 at 09:08:08PM +0200, Zhéxué M. Krawutschke wrote: > > > Hello Christoph, > > > what exactly is your problem with OpenMPI and Slurm? > > > Do you compile the products yourself? Which LINUX distribution and > > version are you using? > > > > > > If you compile the software yourself, could you please tell me what the > > "configure" command looks like and which MUNGE version is in use? From the > > distribution or compiled by yourself? > > > > > > I would be very happy to take on this topic and help you. You can also > > reach me at +49 176 67270992. > > > Best regards from Berlin > > > > please refer to (especially the end) of my first mail in this thread > > which is available here > > https://www.mail-archive.com/users@lists.open-mpi.org/msg35141.html > > > > I believe this contains the relevant information you are requesting. The > > second mail which you are replying to was just additional information. > > My apologies if this led to confusion. > > > > Please let me know if any relevant information is missing from my first > > email. At the bottom of this email I include the ompi_info output as > > further addendum. > > > > To summarize: I would like to understand where the munge warning > > and PMIx error described in the first email (and the github link > > included) come from. The explanation in the github issue > > does not appear to be correct as all munge libraries are > > available everywhere. To me, it appears at the moment that OpenMPIs > > configure decides erroneously to build and use the internal pmix > > instead of using the (presumably) newer externally available PMIx, > > leading to launcher problems with srun. > > > > > > Best Regards > > > > Christof > > > > Package: Open MPI root@admin.service Distribution > > Open MPI: 4.1.5 > > Open MPI repo revision: v4.1.5 > >Open MPI release date: Feb 23, 2023 > > Open RTE: 4.1.5 > > Open RTE repo revision: v4.1.5 > >Open RTE release date: Feb 23, 2023 > > OPAL: 4.1.5 > > OPAL repo revision: v4.1.5 > >OPAL release date: Feb 23, 2023 > > MPI API: 3.1.0 > > Ident string: 4.1.5 > > Prefix: /cluster/mpi/openmpi/4.1.5/gcc-11.3.1 > > Configured architecture: x86_64-pc-linux-gnu > > Configure host: admin.service > >Configured by: root > >Configured on: Wed May 17 18:45:42 UTC 2023 > > Configure host: admin.service > > Configure command line: '--enable-mpi1-compatibility' > > '--enable-orterun-prefix-by-default' > > '--with-ofi=/cluster/libraries/libfabric/1.18.0/' '--with-slurm' > > '--with-pmix' '--with-pmix-libdir=/usr/lib64' '--with-pmi' > > '--with-pmi-libdir=/usr/lib64' > > '--prefix=/cluster/mpi/openmpi/4.1.5/gcc-11.3.1' > > Built by: root > > Built on: Wed May 17 06:48:36 PM UTC 2023 > > Built host: admin.service > > C bindings: yes > > C++ bindings: no > > Fort mpif.h: yes (all) > > Fort use mpi: yes (full: ignore TKR) > >Fort use mpi size: deprecated-ompi-info-value > > Fort use mpi_f08: yes > > Fort mpi_f08 compliance: The mpi_f08 module is available, but due to > > limitations in the gfortran compiler and/or Open MPI, does not support > > the following: array subsections, direct passthru (where possible) to > >
Re: [OMPI users] psec warning when launching with srun
Christof, Open MPI switching to the internal PMIx is a bug I addressed in https://github.com/open-mpi/ompi/pull/11704 Feel free to manually download and apply the patch, you will then need recent autotools and run ./autogen.pl --force An other option is to manually edit the configure file Look for the following snippet # Final check - if they didn't point us explicitly at an external version # but we found one anyway, use the internal version if it is higher if test "$opal_external_pmix_version" != "internal" && (test -z "$with_pmix" || test "$with_pmix" = "yes") then : if test "$opal_external_pmix_version" != "3x" and replace the last line with if test $opal_external_pmix_version_major -lt 3 Cheers, Gilles On Sat, May 20, 2023 at 6:13 PM christof.koehler--- via users < users@lists.open-mpi.org> wrote: > Hello Z. Matthias Krawutschke, > > On Fri, May 19, 2023 at 09:08:08PM +0200, Zhéxué M. Krawutschke wrote: > > Hello Christoph, > > what exactly is your problem with OpenMPI and Slurm? > > Do you compile the products yourself? Which LINUX distribution and > version are you using? > > > > If you compile the software yourself, could you please tell me what the > "configure" command looks like and which MUNGE version is in use? From the > distribution or compiled by yourself? > > > > I would be very happy to take on this topic and help you. You can also > reach me at +49 176 67270992. > > Best regards from Berlin > > please refer to (especially the end) of my first mail in this thread > which is available here > https://www.mail-archive.com/users@lists.open-mpi.org/msg35141.html > > I believe this contains the relevant information you are requesting. The > second mail which you are replying to was just additional information. > My apologies if this led to confusion. > > Please let me know if any relevant information is missing from my first > email. At the bottom of this email I include the ompi_info output as > further addendum. > > To summarize: I would like to understand where the munge warning > and PMIx error described in the first email (and the github link > included) come from. The explanation in the github issue > does not appear to be correct as all munge libraries are > available everywhere. To me, it appears at the moment that OpenMPIs > configure decides erroneously to build and use the internal pmix > instead of using the (presumably) newer externally available PMIx, > leading to launcher problems with srun. > > > Best Regards > > Christof > > Package: Open MPI root@admin.service Distribution > Open MPI: 4.1.5 > Open MPI repo revision: v4.1.5 >Open MPI release date: Feb 23, 2023 > Open RTE: 4.1.5 > Open RTE repo revision: v4.1.5 >Open RTE release date: Feb 23, 2023 > OPAL: 4.1.5 > OPAL repo revision: v4.1.5 >OPAL release date: Feb 23, 2023 > MPI API: 3.1.0 > Ident string: 4.1.5 > Prefix: /cluster/mpi/openmpi/4.1.5/gcc-11.3.1 > Configured architecture: x86_64-pc-linux-gnu > Configure host: admin.service >Configured by: root >Configured on: Wed May 17 18:45:42 UTC 2023 > Configure host: admin.service > Configure command line: '--enable-mpi1-compatibility' > '--enable-orterun-prefix-by-default' > '--with-ofi=/cluster/libraries/libfabric/1.18.0/' '--with-slurm' > '--with-pmix' '--with-pmix-libdir=/usr/lib64' '--with-pmi' > '--with-pmi-libdir=/usr/lib64' > '--prefix=/cluster/mpi/openmpi/4.1.5/gcc-11.3.1' > Built by: root > Built on: Wed May 17 06:48:36 PM UTC 2023 > Built host: admin.service > C bindings: yes > C++ bindings: no > Fort mpif.h: yes (all) > Fort use mpi: yes (full: ignore TKR) >Fort use mpi size: deprecated-ompi-info-value > Fort use mpi_f08: yes > Fort mpi_f08 compliance: The mpi_f08 module is available, but due to > limitations in the gfortran compiler and/or Open MPI, does not support > the following: array subsections, direct passthru (where possible) to > underlying Open MPI's C functionality > Fort mpi_f08 subarrays: no >Java bindings: no > Wrapper compiler rpath: runpath > C compiler: gcc > C compiler absolute: /usr/bin/gcc > C compiler family name: GNU > C compiler version: 11.3.1 > C++ compiler: g++ >C++ compiler absolute: /usr/bin/g++ >Fort compiler: gfortran >Fort compiler abs: /usr/bin/gfortran > Fort ignore TKR: yes (!GCC$ ATTRIBUTES NO_ARG_CHECK ::) >Fort 08 assumed shape: yes > Fort optional args: yes > Fort INTERFACE: yes > Fort ISO_FORTRAN_ENV: yes >Fort STORAGE_SIZE: yes > Fort BIND(C) (all): yes > Fort ISO_C_BINDING: yes > Fort SUBROUTINE BIND(C): yes >Fort TYPE,BIND(C): yes >
Re: [OMPI users] psec warning when launching with srun
Hello Z. Matthias Krawutschke, On Fri, May 19, 2023 at 09:08:08PM +0200, Zhéxué M. Krawutschke wrote: > Hello Christoph, > what exactly is your problem with OpenMPI and Slurm? > Do you compile the products yourself? Which LINUX distribution and version > are you using? > > If you compile the software yourself, could you please tell me what the > "configure" command looks like and which MUNGE version is in use? From the > distribution or compiled by yourself? > > I would be very happy to take on this topic and help you. You can also reach > me at +49 176 67270992. > Best regards from Berlin please refer to (especially the end) of my first mail in this thread which is available here https://www.mail-archive.com/users@lists.open-mpi.org/msg35141.html I believe this contains the relevant information you are requesting. The second mail which you are replying to was just additional information. My apologies if this led to confusion. Please let me know if any relevant information is missing from my first email. At the bottom of this email I include the ompi_info output as further addendum. To summarize: I would like to understand where the munge warning and PMIx error described in the first email (and the github link included) come from. The explanation in the github issue does not appear to be correct as all munge libraries are available everywhere. To me, it appears at the moment that OpenMPIs configure decides erroneously to build and use the internal pmix instead of using the (presumably) newer externally available PMIx, leading to launcher problems with srun. Best Regards Christof Package: Open MPI root@admin.service Distribution Open MPI: 4.1.5 Open MPI repo revision: v4.1.5 Open MPI release date: Feb 23, 2023 Open RTE: 4.1.5 Open RTE repo revision: v4.1.5 Open RTE release date: Feb 23, 2023 OPAL: 4.1.5 OPAL repo revision: v4.1.5 OPAL release date: Feb 23, 2023 MPI API: 3.1.0 Ident string: 4.1.5 Prefix: /cluster/mpi/openmpi/4.1.5/gcc-11.3.1 Configured architecture: x86_64-pc-linux-gnu Configure host: admin.service Configured by: root Configured on: Wed May 17 18:45:42 UTC 2023 Configure host: admin.service Configure command line: '--enable-mpi1-compatibility' '--enable-orterun-prefix-by-default' '--with-ofi=/cluster/libraries/libfabric/1.18.0/' '--with-slurm' '--with-pmix' '--with-pmix-libdir=/usr/lib64' '--with-pmi' '--with-pmi-libdir=/usr/lib64' '--prefix=/cluster/mpi/openmpi/4.1.5/gcc-11.3.1' Built by: root Built on: Wed May 17 06:48:36 PM UTC 2023 Built host: admin.service C bindings: yes C++ bindings: no Fort mpif.h: yes (all) Fort use mpi: yes (full: ignore TKR) Fort use mpi size: deprecated-ompi-info-value Fort use mpi_f08: yes Fort mpi_f08 compliance: The mpi_f08 module is available, but due to limitations in the gfortran compiler and/or Open MPI, does not support the following: array subsections, direct passthru (where possible) to underlying Open MPI's C functionality Fort mpi_f08 subarrays: no Java bindings: no Wrapper compiler rpath: runpath C compiler: gcc C compiler absolute: /usr/bin/gcc C compiler family name: GNU C compiler version: 11.3.1 C++ compiler: g++ C++ compiler absolute: /usr/bin/g++ Fort compiler: gfortran Fort compiler abs: /usr/bin/gfortran Fort ignore TKR: yes (!GCC$ ATTRIBUTES NO_ARG_CHECK ::) Fort 08 assumed shape: yes Fort optional args: yes Fort INTERFACE: yes Fort ISO_FORTRAN_ENV: yes Fort STORAGE_SIZE: yes Fort BIND(C) (all): yes Fort ISO_C_BINDING: yes Fort SUBROUTINE BIND(C): yes Fort TYPE,BIND(C): yes Fort T,BIND(C,name="a"): yes Fort PRIVATE: yes Fort PROTECTED: yes Fort ABSTRACT: yes Fort ASYNCHRONOUS: yes Fort PROCEDURE: yes Fort USE...ONLY: yes Fort C_FUNLOC: yes Fort f08 using wrappers: yes Fort MPI_SIZEOF: yes C profiling: yes C++ profiling: no Fort mpif.h profiling: yes Fort use mpi profiling: yes Fort use mpi_f08 prof: yes C++ exceptions: no Thread support: posix (MPI_THREAD_MULTIPLE: yes, OPAL support: yes, OMPI progress: no, ORTE progress: yes, Event lib: yes) Sparse Groups: no Internal debug support: no MPI interface warnings: yes MPI parameter check: runtime Memory profiling support: no Memory debugging support: no dl support: yes Heterogeneous support: no mpirun default --prefix: yes MPI_WTIME support: native Symbol vis. support: yes Host topology support: yes IPv6 support: no MPI1 compatibility: yes
Re: [OMPI users] psec warning when launching with srun
Hello Christoph, what exactly is your problem with OpenMPI and Slurm? Do you compile the products yourself? Which LINUX distribution and version are you using? If you compile the software yourself, could you please tell me what the "configure" command looks like and which MUNGE version is in use? From the distribution or compiled by yourself? I would be very happy to take on this topic and help you. You can also reach me at +49 176 67270992. Best regards from Berlin Z. Matthias Krawutschke > On Donnerstag, Mai 18, 2023 at 5:47 PM, christof.koehler--- via users > mailto:users@lists.open-mpi.org)> wrote: > Hello again, > > I should add that the openmpi configure decides to use the internal pmix > > configure: WARNING: discovered external PMIx version is less than internal > version 3.x > configure: WARNING: using internal PMIx > ... > ... > checking if user requested PMI support... yes > checking for pmi.h in /usr/include... not found > checking for pmi.h in /usr/include/slurm... found > checking pmi.h usability... yes > checking pmi.h presence... yes > checking for pmi.h... yes > checking for libpmi in /usr/lib64... found > checking for PMI_Init in -lpmi... yes > checking for pmi2.h in /usr/include... not found > checking for pmi2.h in /usr/include/slurm... found > checking pmi2.h usability... yes > checking pmi2.h presence... yes > checking for pmi2.h... yes > checking for libpmi2 in /usr/lib64... found > checking for PMI2_Init in -lpmi2... yes > checking for pmix.h in ... not found > checking for pmix.h in /include... not found > checking can PMI support be built... yes > checking if user requested internal PMIx support(yes)... no > checking for pmix.h in /usr... not found > checking for pmix.h in /usr/include... found > checking libpmix.* in /usr/lib64... found > checking PMIx version... version file found > checking version 4x... found > checking PMIx version to be used... internal > > I am not sure how it decides that, the external one is already a quite > new version. > > # srun --mpi=list > MPI plugin types are... > pmix > cray_shasta > none > pmi2 > specific pmix plugin versions available: pmix_v4 > > > Best Regards > > Christof > > -- > Dr. rer. nat. Christof Köhler email: c.koeh...@uni-bremen.de > Universitaet Bremen/FB1/BCCMS phone: +49-(0)421-218-62334 > Am Fallturm 1/ TAB/ Raum 3.06 fax: +49-(0)421-218-62770 > 28359 Bremen >
Re: [OMPI users] psec warning when launching with srun
Hello again, I should add that the openmpi configure decides to use the internal pmix configure: WARNING: discovered external PMIx version is less than internal version 3.x configure: WARNING: using internal PMIx ... ... checking if user requested PMI support... yes checking for pmi.h in /usr/include... not found checking for pmi.h in /usr/include/slurm... found checking pmi.h usability... yes checking pmi.h presence... yes checking for pmi.h... yes checking for libpmi in /usr/lib64... found checking for PMI_Init in -lpmi... yes checking for pmi2.h in /usr/include... not found checking for pmi2.h in /usr/include/slurm... found checking pmi2.h usability... yes checking pmi2.h presence... yes checking for pmi2.h... yes checking for libpmi2 in /usr/lib64... found checking for PMI2_Init in -lpmi2... yes checking for pmix.h in ... not found checking for pmix.h in /include... not found checking can PMI support be built... yes checking if user requested internal PMIx support(yes)... no checking for pmix.h in /usr... not found checking for pmix.h in /usr/include... found checking libpmix.* in /usr/lib64... found checking PMIx version... version file found checking version 4x... found checking PMIx version to be used... internal I am not sure how it decides that, the external one is already a quite new version. # srun --mpi=list MPI plugin types are... pmix cray_shasta none pmi2 specific pmix plugin versions available: pmix_v4 Best Regards Christof -- Dr. rer. nat. Christof Köhler email: c.koeh...@uni-bremen.de Universitaet Bremen/FB1/BCCMS phone: +49-(0)421-218-62334 Am Fallturm 1/ TAB/ Raum 3.06 fax: +49-(0)421-218-62770 28359 Bremen
[OMPI users] psec warning when launching with srun
Hello everybody, we are seeing the sypmptoms described in https://github.com/open-mpi/ompi/issues/11557 However, according to the systems package manager (dnf) all munge related packages on the build node and the execution node are identical, see details at the bottom. So, the explanation given by Ralph Castain in the git issue, which I read to refer to a missing munge library, does not appear to explain the warning. I would like to note that a PMIX_MCA_psec=native is sufficient to make the warning go away, it is not necessary to disable munge completely. Launching with srun --mpi=pmi2 instead works fine, anyway. Related to that: pmix has a configure switch to disable munge support. Would it be possible and/or adviseable to disable munge in the pmix build ? I have to note that we are seeing also PMIX ERROR: ERROR in file gds_ds12_lock_pthread.c at line 168 for each rank launched. This might be a separate issue or related. Setting PMIX_MCA_gds=^ds12 makes this go away. With or without setting PMIX_MCA_psec and PMIX_MCA_gds the job launches and is executed. Still, I would like to understand this better in case we have a broken slurm or mpi setup which would need to be corrected. Best Regards Christof System Details: Rocky Linux 9.1, slurm 23.02.2, open pmix 4.2.4rc1, openmpi 4.1.5. slurm and pmix were compiled in the same step to rpms in a mock build environment using the spec files included with them. I can provide the build logs of the rpms. Both are not from distribution repositories. openmpi was configured with ./configure --enable-mpi1-compatibility --enable-orterun-prefix-by-default --with-ofi=/cluster/libraries/libfabric/1.18.0/ --with-slurm --with-pmix --with-pmix-libdir=/usr/lib64 --with-pmi --with-pmi-libdir=/usr/lib64 Installed munge packages from distribution according to dnf on build node munge.x86_64 0.5.13-13.el9 munge-devel.x86_64 0.5.13-13.el9 munge-libs.x86_64 0.5.13-13.el9 on execution node munge.x86_640.5.13-13.el9 munge-devel.x86_64 0.5.13-13.el9 munge-libs.x86_64 0.5.13-13.el9 -- Dr. rer. nat. Christof Köhler email: c.koeh...@uni-bremen.de Universitaet Bremen/FB1/BCCMS phone: +49-(0)421-218-62334 Am Fallturm 1/ TAB/ Raum 3.06 fax: +49-(0)421-218-62770 28359 Bremen
