Re: [OMPI devel] Dynamic languages, dlopen() issues, and symbol visibility of libtool ltdl API in current trunk

2009-09-22 Thread Lisandro Dalcin 
On Mon, Sep 21, 2009 at 9:45 AM, Jeff Squyres  wrote:
> Ick; I appreciate Lisandro's quandry, but don't quite know what to do.
>

I'm just asking the library "libopen-pal.so" exposing ltdl calls
wrapped with an "opal_" prefix. This way, the original ltdl calls hare
hidden (no chance to collide with user code using an incompatible
libtool version), but Open MPI provides a portable way to dlopen()
shared libs/dynamic modules. In simple terms, I'm asking
"libopen-pal.so" to contain ltdl wrapper calls like this one:

OMPI_DECLSPEC lt_dlhandle opal_lt_dlopenadvise(const char *filename,
lt_dladvise advise) /* note opal_ prefix! */
{
   return lt_dlopenadvise(filename,advise); /* original ltdl call*/
}


Then, third-party code (like mpi4py or any other dynamic MPI module
for any other dynamic language) can do this:

#include 
#if defined(OPEN_MPI)
typedef void *lt_dlhandle;
typedef void *lt_dladvise;
OMPI_DECLSPEC extern lt_dlhandle opal_lt_dlopenadvise(const char *, lt_dladvise)
#endif
...
#if defined(OPEN_MPI)
/* init advice, not shown ... */
opal_lt_dlopenadvise("mpi", advice);
/* destroy advice, not shown ... */
#endif
MPI_Init(0,0);

>
> How about keeping libltdl fvisibility=hidden inside mpi4py?
>

Not sure if I was clear enough in my comments above, but mpi4py does
not bundles/link libtool. Just abuses on libtool availability in
"libopen-pal.so" for the sake of portability.

>
> On Sep 17, 2009, at 11:16 AM, Josh Hursey wrote:
>
>> So I started down this road a couple months ago. I was using the
>> lt_dlopen() and friends in the OPAL CRS self module. The visibility
>> changes broke that functionality. The one solution that I started
>> implementing was precisely what you suggested, wrapping a subset the
>> libtool calls and prefixing them with opal_*. The email thread is below:
>>   http://www.open-mpi.org/community/lists/devel/2009/07/6531.php
>>
>> The problem that I hit was that libtool's build system did not play
>> well with the visibility symbols. This caused dlopen to be disabled
>> incorrectly. The libtool folks have a patch and, I believe, they are
>> planning on incorporating in the next release. The email thread is
>> below:
>>   http://thread.gmane.org/gmane.comp.gnu.libtool.patches/9446
>>
>> So we would (others can speak up if not) certainly consider such a
>> wrapper, but I think we need to wait for the next libtool release
>> (unless there is other magic we can do) before it would be usable.
>>
>> Do others have any other ideas on how we might get around this in the
>> mean time?
>>
>> -- Josh
>>
>>
>> On Sep 16, 2009, at 5:59 PM, Lisandro Dalcin wrote:
>>
>> > Hi all.. I have to contact you again about the issues related to
>> > dlopen()ing libmpi with RTLD_LOCAL, as many dynamic languages (Python
>> > in my case) do.
>> >
>> > So far, I've been able to manage the issues (despite the "do nothing"
>> > policy from Open MPI devs, which I understand) in a more or less
>> > portable manner by taking advantage of the availability of libtool
>> > ltdl symbols in the Open MPI libraries (specifically, in libopen-pal).
>> > For reference, all this hackery is here:
>> > http://code.google.com/p/mpi4py/source/browse/trunk/src/compat/openmpi.h
>> >
>> > However, I noticed that in current trunk (v1.4, IIUC) things have
>> > changed and libtool symbols are not externally available. Again, I
>> > understand the reason and acknowledge that such change is a really
>> > good thing. However, this change has broken all my hackery for
>> > dlopen()ing libmpi before the call to MPI_Init().
>> >
>> > Is there any chance that libopen-pal could provide some properly
>> > prefixed (let say, using "opal_" as a prefix) wrapper calls to a small
>> > subset of the libtool ltdl API? The following set of wrapper calls
>> > would is the minimum required to properly load libmpi in a portable
>> > manner and cleanup resources (let me abuse of my previous suggestion
>> > and add the opal_ prefix):
>> >
>> > opal_lt_dlinit()
>> > opal_lt_dlexit()
>> >
>> > opal_lt_dladvise_init(a)
>> > opal_lt_dladvise_destroy(a)
>> > opal_lt_dladvise_global(a)
>> > opal_lt_dladvise_ext(a)
>> >
>> > opal_lt_dlopenadvise(n,a)
>> > opal_lt_dlclose(h)
>> >
>> > Any chance this request could be considered? I would really like to
>> > have this before any Open MPI tarball get released without libtool
>> > symbols exposed...
>> >
>> >
>> > --
>> > Lisandro Dalcín
>> > ---
>> > Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC)
>> > Instituto de Desarrollo Tecnológico para la Industria Química (INTEC)
>> > Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET)
>> > PTLC - Güemes 3450, (3000) Santa Fe, Argentina
>> > Tel/Fax: +54-(0)342-451.1594
>> >
>> > ___
>> > devel mailing list
>> > de...@open-mpi.org
>> > http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>
>>
>> ___
>> devel mailing list
>>

Re: [OMPI devel] Dynamic languages, dlopen() issues, and symbol visibility of libtool ltdl API in current trunk

2009-09-21 Thread Ralf Wildenhues 
As a workaround, Lisandro could just pre-seed the cache variables of the
respective configure tests that come out wrong.

  ./configure lt_cv_dlopen_self=yes lt_cv_dlopen_self_static=yes

HTH.

Cheers,
Ralf

* Jeff Squyres wrote on Mon, Sep 21, 2009 at 02:45:28PM CEST:
> Ick; I appreciate Lisandro's quandry, but don't quite know what to do.
> 
> How about keeping libltdl fvisibility=hidden inside mpi4py?
> 
> 
> On Sep 17, 2009, at 11:16 AM, Josh Hursey wrote:
> 
> >So I started down this road a couple months ago. I was using the
> >lt_dlopen() and friends in the OPAL CRS self module. The visibility
> >changes broke that functionality. The one solution that I started
> >implementing was precisely what you suggested, wrapping a subset the
> >libtool calls and prefixing them with opal_*. The email thread is
> >below:
> >   http://www.open-mpi.org/community/lists/devel/2009/07/6531.php
> >
> >The problem that I hit was that libtool's build system did not play
> >well with the visibility symbols. This caused dlopen to be disabled
> >incorrectly. The libtool folks have a patch and, I believe, they are
> >planning on incorporating in the next release. The email thread is
> >below:
> >   http://thread.gmane.org/gmane.comp.gnu.libtool.patches/9446
> >
> >So we would (others can speak up if not) certainly consider such a
> >wrapper, but I think we need to wait for the next libtool release
> >(unless there is other magic we can do) before it would be usable.
> >
> >Do others have any other ideas on how we might get around this in the
> >mean time?
> >
> >-- Josh
> >
> >
> >On Sep 16, 2009, at 5:59 PM, Lisandro Dalcin wrote:
> >
> >> Hi all.. I have to contact you again about the issues related to
> >> dlopen()ing libmpi with RTLD_LOCAL, as many dynamic languages
> >(Python
> >> in my case) do.
> >>
> >> So far, I've been able to manage the issues (despite the "do
> >nothing"
> >> policy from Open MPI devs, which I understand) in a more or less
> >> portable manner by taking advantage of the availability of libtool
> >> ltdl symbols in the Open MPI libraries (specifically, in
> >libopen-pal).
> >> For reference, all this hackery is here:
> >> http://code.google.com/p/mpi4py/source/browse/trunk/src/compat/openmpi.h
> >>
> >> However, I noticed that in current trunk (v1.4, IIUC) things have
> >> changed and libtool symbols are not externally available. Again, I
> >> understand the reason and acknowledge that such change is a really
> >> good thing. However, this change has broken all my hackery for
> >> dlopen()ing libmpi before the call to MPI_Init().
> >>
> >> Is there any chance that libopen-pal could provide some properly
> >> prefixed (let say, using "opal_" as a prefix) wrapper calls to a
> >small
> >> subset of the libtool ltdl API? The following set of wrapper calls
> >> would is the minimum required to properly load libmpi in a portable
> >> manner and cleanup resources (let me abuse of my previous suggestion
> >> and add the opal_ prefix):
> >>
> >> opal_lt_dlinit()
> >> opal_lt_dlexit()
> >>
> >> opal_lt_dladvise_init(a)
> >> opal_lt_dladvise_destroy(a)
> >> opal_lt_dladvise_global(a)
> >> opal_lt_dladvise_ext(a)
> >>
> >> opal_lt_dlopenadvise(n,a)
> >> opal_lt_dlclose(h)
> >>
> >> Any chance this request could be considered? I would really like to
> >> have this before any Open MPI tarball get released without libtool
> >> symbols exposed...

Re: [OMPI devel] Dynamic languages, dlopen() issues, and symbol visibility of libtool ltdl API in current trunk

2009-09-21 Thread Jeff Squyres 

Ick; I appreciate Lisandro's quandry, but don't quite know what to do.

How about keeping libltdl fvisibility=hidden inside mpi4py?


On Sep 17, 2009, at 11:16 AM, Josh Hursey wrote:


So I started down this road a couple months ago. I was using the
lt_dlopen() and friends in the OPAL CRS self module. The visibility
changes broke that functionality. The one solution that I started
implementing was precisely what you suggested, wrapping a subset the
libtool calls and prefixing them with opal_*. The email thread is  
below:

   http://www.open-mpi.org/community/lists/devel/2009/07/6531.php

The problem that I hit was that libtool's build system did not play
well with the visibility symbols. This caused dlopen to be disabled
incorrectly. The libtool folks have a patch and, I believe, they are
planning on incorporating in the next release. The email thread is
below:
   http://thread.gmane.org/gmane.comp.gnu.libtool.patches/9446

So we would (others can speak up if not) certainly consider such a
wrapper, but I think we need to wait for the next libtool release
(unless there is other magic we can do) before it would be usable.

Do others have any other ideas on how we might get around this in the
mean time?

-- Josh


On Sep 16, 2009, at 5:59 PM, Lisandro Dalcin wrote:

> Hi all.. I have to contact you again about the issues related to
> dlopen()ing libmpi with RTLD_LOCAL, as many dynamic languages  
(Python

> in my case) do.
>
> So far, I've been able to manage the issues (despite the "do  
nothing"

> policy from Open MPI devs, which I understand) in a more or less
> portable manner by taking advantage of the availability of libtool
> ltdl symbols in the Open MPI libraries (specifically, in libopen- 
pal).

> For reference, all this hackery is here:
> http://code.google.com/p/mpi4py/source/browse/trunk/src/compat/openmpi.h
>
> However, I noticed that in current trunk (v1.4, IIUC) things have
> changed and libtool symbols are not externally available. Again, I
> understand the reason and acknowledge that such change is a really
> good thing. However, this change has broken all my hackery for
> dlopen()ing libmpi before the call to MPI_Init().
>
> Is there any chance that libopen-pal could provide some properly
> prefixed (let say, using "opal_" as a prefix) wrapper calls to a  
small

> subset of the libtool ltdl API? The following set of wrapper calls
> would is the minimum required to properly load libmpi in a portable
> manner and cleanup resources (let me abuse of my previous suggestion
> and add the opal_ prefix):
>
> opal_lt_dlinit()
> opal_lt_dlexit()
>
> opal_lt_dladvise_init(a)
> opal_lt_dladvise_destroy(a)
> opal_lt_dladvise_global(a)
> opal_lt_dladvise_ext(a)
>
> opal_lt_dlopenadvise(n,a)
> opal_lt_dlclose(h)
>
> Any chance this request could be considered? I would really like to
> have this before any Open MPI tarball get released without libtool
> symbols exposed...
>
>
> --
> Lisandro Dalcín
> ---
> Centro Internacional de Métodos Computacionales en Ingeniería  
(CIMEC)
> Instituto de Desarrollo Tecnológico para la Industria Química  
(INTEC)

> Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET)
> PTLC - Güemes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
>
> ___
> devel mailing list
> de...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/devel


___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel




--
Jeff Squyres
jsquy...@cisco.com

Re: [OMPI devel] Dynamic languages, dlopen() issues, and symbol visibility of libtool ltdl API in current trunk

2009-09-17 Thread Josh Hursey 
So I started down this road a couple months ago. I was using the  
lt_dlopen() and friends in the OPAL CRS self module. The visibility  
changes broke that functionality. The one solution that I started  
implementing was precisely what you suggested, wrapping a subset the  
libtool calls and prefixing them with opal_*. The email thread is below:

  http://www.open-mpi.org/community/lists/devel/2009/07/6531.php

The problem that I hit was that libtool's build system did not play  
well with the visibility symbols. This caused dlopen to be disabled  
incorrectly. The libtool folks have a patch and, I believe, they are  
planning on incorporating in the next release. The email thread is  
below:

  http://thread.gmane.org/gmane.comp.gnu.libtool.patches/9446

So we would (others can speak up if not) certainly consider such a  
wrapper, but I think we need to wait for the next libtool release  
(unless there is other magic we can do) before it would be usable.


Do others have any other ideas on how we might get around this in the  
mean time?


-- Josh


On Sep 16, 2009, at 5:59 PM, Lisandro Dalcin wrote:


Hi all.. I have to contact you again about the issues related to
dlopen()ing libmpi with RTLD_LOCAL, as many dynamic languages (Python
in my case) do.

So far, I've been able to manage the issues (despite the "do nothing"
policy from Open MPI devs, which I understand) in a more or less
portable manner by taking advantage of the availability of libtool
ltdl symbols in the Open MPI libraries (specifically, in libopen-pal).
For reference, all this hackery is here:
http://code.google.com/p/mpi4py/source/browse/trunk/src/compat/openmpi.h

However, I noticed that in current trunk (v1.4, IIUC) things have
changed and libtool symbols are not externally available. Again, I
understand the reason and acknowledge that such change is a really
good thing. However, this change has broken all my hackery for
dlopen()ing libmpi before the call to MPI_Init().

Is there any chance that libopen-pal could provide some properly
prefixed (let say, using "opal_" as a prefix) wrapper calls to a small
subset of the libtool ltdl API? The following set of wrapper calls
would is the minimum required to properly load libmpi in a portable
manner and cleanup resources (let me abuse of my previous suggestion
and add the opal_ prefix):

opal_lt_dlinit()
opal_lt_dlexit()

opal_lt_dladvise_init(a)
opal_lt_dladvise_destroy(a)
opal_lt_dladvise_global(a)
opal_lt_dladvise_ext(a)

opal_lt_dlopenadvise(n,a)
opal_lt_dlclose(h)

Any chance this request could be considered? I would really like to
have this before any Open MPI tarball get released without libtool
symbols exposed...


--
Lisandro Dalcín
---
Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC)
Instituto de Desarrollo Tecnológico para la Industria Química (INTEC)
Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET)
PTLC - Güemes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel