On Sep 26, 2011, at 2:42 PM, IOhannes m zmölnig wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 09/26/2011 06:52 PM, Hans-Christoph Steiner wrote:
IOhannes and I were discussing how to load shared code for a libdir
on
IRC. The sticking point is that Mac OS X seems to require dylibs to
have a hard-coded path. IOhannes posted a test lib to try to work
out
how to do it using DYLD_LIBRARY_PATH:
http://iem.at/~zmoelnig/OSX/test.tgz
So here are my results from Mac OS X 10.5.8:
creating a [test] object:
-------------------------
/private/tmp/test/tmp/test.pd_darwin:
dlopen(/private/tmp/test/tmp/test.pd_darwin, 10): no suitable image
found. Did find:
/private/tmp/test/tmp/test.pd_darwin: unknown required load command
0x80000022
test
... couldn't create
hans@dhcp-10 tmp $ ~/code/pd-extended.git/src/pd -noprefs -lib test
-------------------------------------------------------------------
./test.pd_darwin: dlopen(./test.pd_darwin, 10): no suitable image
found. Did find:
./test.pd_darwin: unknown required load command 0x80000022
test: can't load library
just to clarify a bit:
the binaries posted under the link above were compiled under OSX-10.6,
without any precautions for backward compatibility.
they don't work on 10.5, nor 10.4
i also put the same set of binaries online, but this time compiled
under
10.4, and they work and function as expected under 10.4 and 10.6
(and i
expect them to work on 10.5 as well)
on w32, the search paths for dll's should be modifiable via the %PATH%
environment, whereas on linux (and most i think BSD and hurd) the
envvar
in question is LD_LIBRARY_PATH
after the technical details, let's step back and have a look at what
this is all about:
the main motivation is to be able to bundle shared code and 3rd party
libraries with externals.
the dynamic linkers of the various OSs, will automatically search for
dylinked libraries your external depends on, in various places (e.g.
/usr/lib).
this is great if you have an easy way to install the needed library
into
this place (e.g. a dependy-informed installer system like debian's
"apt").
it's not so great, if you don't have such a system, and the user is
expected to find and download the given dependencies and put them into
your system's search path.
even if an external ships with all dependencies, they still have to be
installed in the "correct" places, which makes installation more
complicated than just dragging a folder onto your desktop.
why?
because your OS's dynamic linker doesn't look for 3rd party
libraries in
weird places like a folder on your desktop.
now the idea, hans and me came up with, was to make Pd tell the OS to
look into those places.
e.g. if library "foo.pd_darwin" gets installed into ~/Library/Pd/foo/,
then it would be nice, if the dynamic linker would search for any
depending libraries in the ~/Library/Pd/foo/ directory.
likewise "bar.pd_linux" (installed in ~/my-externals/pub/) might have
it's dependencies searched for in ~/my-externals/pub/.
if both "foo" and "bar" depend on "libdudelsack", and both provide a
"libdudelsack.dll" in their homedirectories, then of course we want
both
libraries to choose "their" correct one.
so the idea is, to modify the automatic search path of the dynamic
linker whenever a library is loaded.
example:
the user installs the "frotzel" library, by downloading it, and
unzipping it into /usr/local/pd-externals/
now, if the user requests a new object [frotzel], Pd will search for a
matching binary and eventually find frotzel.l_ia64 in
/usr/local/pd-externals/frotzel/
before dlopen()ing this binary, it will read the LD_LIBRARY_PATH
variable, push it to some stack, and then add
/usr/local/pd-externals/frotzel/ at the beginning of the
LD_LIBRARY_PATH.
it will then dlopen() the frotzel.l_ia64 binary, and the linker
discovers, that it has to load the libknurbel.so library, which it
will
(magically) find in /usr/local/pd-externals/frotzel/
because all dependencies are fullfilled, [frotzel] will successfully
be
loaded and will work.
after fortzel.l_ia64 has been successfully loaded, we no longer need
to
search for libraries in /usr/local/pd-externals/frotzel/, and
therefore
the original content of LD_LIBRARY_PATH is restored (popped from the
stack).
I got this building and working on Mac OS X, since that's where the
discussion started. It was really just a matter of getting the build
flags right. I am using tkwidgets as my test bed, then I'll integrate
the changes into the library template. You can see my work here in
the 'newentry' branch:
https://github.com/pd-projects/tkwidgets
the exact name of "LD_LIBRARY_PATH" is system dependent, but it seems
that LD_LIBRARY_PATH, DYLD_LIBRARY_PATH and PATH cover most systems.
fasdmrt
IOhannes
My guess is that it'll be something like this:
#ifdef _WIN32
PATH
#elif defined(__APPLE__)
DYLD_LIBRARY_PATH
#else
LD_LIBRARY_PATH
#endif
And the IRC transcript for the record:
i don't exactly know how to do LD_LIBRARY_PATH on w32 yet
_hc
IRC
1:54:17
IOhannes: the annoyances don't end there...
1:54:30
if libtest10.dylib links to something esle....
IOhannes
IRC
1:54:31
i _thought_ it's a matter of %PATH% or so, but i never checked
_hc
IRC
1:54:52
there seems to be some vars for dyld to handle this:
IOhannes
IRC
1:54:58
i'm pretty sure there is aa mechanism on w32 that allows us to do that
_hc
IRC
1:54:59
DYLD_FALLBACK_LIBRARY_PATH
This is a colon separated list of directories
that contain
libraries. It is used as the default location for
libraries not
found in their install path. By default, it
is set to
$(HOME)/lib:/usr/local/lib:/lib:/usr/lib.
DYLD_ROOT_PATH
This is a colon separated list of directories. The
dynamic linker
will prepend each of this directory paths to every
image access
until a file is found.
IOhannes
IRC
1:55:20
would you mind posting to pastie?
1:55:37
it's hard to read on irssie
_hc
IRC
1:55:41
http://developer.apple.com/library/mac/#documentation/Darwin/Reference/
ManPages/man1/dyld.1.html
IOhannes
IRC
1:55:46
irsee, that is
1:56:11
excalibas left the room (quit: Remote host closed the connection).
IOhannes
IRC
1:56:50
irssi that is
_hc
IRC
1:56:50
so you are proposing that each time Pd loads a lib, it sets the
*_PATH, then unsets it?
IOhannes
IRC
1:57:13
i fail to see the difference between DYLD_LIBRARY_PATH and
DYLD_FALLBACK_PATH
1:57:36
and yes, this is what my proposal is about
_hc
IRC
1:57:47
I don't quite understand it, but it seems to be related to the
"install name" thing
IOhannes
IRC
1:58:39
the big pro (imho), is, that it allows devs to work as they are used
to work with dylibs, and not invent some ramba zama magic dlopen()
hack for a standard problem
1:59:14
but don't my DYLD_LIBRARY_PATH scripts work for you?
_hc
IRC
2:00:05
some dlopen hack like externals?
IOhannes
IRC
2:00:54
no, an external is a plugin, and dlopen() is designed for plugins
2:01:52
in contrast, the automatic dynamic linker is designed for dynamic
sharing of code
2:02:07
which is what we want when we talk about "code-sharing"
2:02:16
and not: plugins
2:02:45
bot are closely related on a technical level, but this doesn't make
them interchangeable
2:03:27
s/bot/both/
2:03:36
is on a bad connection
2:05:12
according to http://msdn.microsoft.com/en-us/library/7d83bc18%28v=VS.100%29.aspx
, %PATH% is the envvar to use to trick the w32 linker
2:15:21
pob [[email protected]] entered the room.
_hc
IRC
3:07:09
IOhannes: what about when the libdir's .dylib is linked to other libs?
3:07:22
say /sw/bin/libspeex.dylib
3:09:59
r33p left the room (quit: Quit: ronflette mode).
IOhannes
IRC
3:14:56
what about it?
_hc
IRC
3:15:17
I feel like that's where there will be problems
IOhannes
IRC
3:15:26
why?
3:15:37
olsen [[email protected]] entered the room.
IOhannes
IRC
3:15:51
maybe this is unclear: i'm talking about _adding_ to LD_LIBRARY_PATH,
not about replacing it
3:16:12
the dynamic linker will then additionally search in yet another
directory
_hc
IRC
3:16:14
did you run your tests on 10.4?
3:16:17
or just build them?
IOhannes
IRC
3:16:25
both
_hc
IRC
3:16:47
hmm, my decision on this was informed by some experience a long time
ago that i can't remember
3:17:00
so I feel like something is not going to work with this
IOhannes
IRC
3:18:08
maybe it was something trivial back then
_hc
IRC
3:18:48
with luck...
3:20:18
IOhannes: this project was already setup with this idea in mind, so
I'm trying it: github.com/pd-projects/tkwidgets
3:20:25
it has shared code in tkwidgets.c
IOhannes
IRC
3:22:16
i cannot checkout with my uplink right now...
_hc
IRC
3:22:41
wow, that's slow
IOhannes
IRC
3:24:26
what? my connection or your download speed?
_hc
IRC
3:25:06
your uplink
IOhannes
IRC
3:28:27
i know; officially i'm without internet at home
_hc
IRC
3:32:09
IOhannes: the test thing doesn't build for me on 10.5: http://pastebin.com/
3:32:24
http://pastebin.com/reTPU8MR
IOhannes
IRC
3:33:32
try "make distclean & autoreconf -fiv && ./configure && make"
3:33:42
(and get the ampersands rights)
_hc
IRC
3:34:57
oy, there were go: make distclean; /sw/bin/autoreconf-2.63 -fiv && ./
configure && make
3:36:18
IOhannes: so you link it normally, without DYLD_LIBRARY_PATH?
3:36:26
then set DLYD_ later?
3:36:50
lorenzosu [[email protected]] entered the room.
IOhannes
IRC
3:36:58
DYLD_LIBRARY_PATH is an environment variable that allows you to tweak
the dynamic linker at runtime
_hc
IRC
3:37:19
I know
IOhannes
IRC
3:37:38
"dynamic linker at runtime" means: the piece of software that resolves
dynamic libraries when you run your executable
3:37:59
so DYLD_LIBRARY_PATH is never set in the compilation/linking stage
_hc
IRC
3:38:11
does it affect the linkign is my question
IOhannes
IRC
3:38:14
only when you run the actual program
3:38:21
which linking?
3:39:27
(there's the gcc linking the object files into a binary and adding
information on where to find external references)
3:39:58
(and theres the dynamic linker that resolves the external references
when you run the binary)
3:40:11
the first thing happens on the dev machine
3:40:19
the second thing happens on the deployment machine
3:40:31
DYLIB_LIBRARY_PATH has nothing to do with the first stop
3:40:44
it's _only_ about deployment
_hc
IRC
3:41:08
can you make libtool show you its command line?
3:41:22
I can't get this workign in the Makefile yet
IOhannes
IRC
3:41:28
usually it print's out the cmdline
_hc
IRC
3:42:25
this seems incomplete to me: libtool: link: gcc -g -O2 -o .libs/test
test.o libtest/.libs/libtest.dylib
3:42:42
oh wait, wrong one
3:43:02
that's the one: /bin/sh ./libtool --tag=CC --mode=link gcc -g -O2 -
module -avoid-version -shared -shrext .pd_darwin libtest/libtest.la -
o test.la -rpath /usr/local/lib test_la-test.lo
IOhannes
IRC
3:43:40
the line following that, is the actual gcc call
3:43:54
> gcc -Wl,-undefined -Wl,dynamic_lookup -o .libs/test.pd_darwin -
bundle .libs/test_la-test.o libtest/.libs/libtest.0.0.0.dylib
_hc
IRC
3:43:55
libtool: link: gcc -dynamiclib -Wl,-undefined -Wl,dynamic_lookup -
o .libs/libtest.0.dylib .libs/libtest.o -O2 -install_name /usr/
local/lib/libtest.0.dylib -compatibility_version 1 -current_version
1.0 -Wl,-single_module
3:43:56
yup
3:44:15
it has to have an install_name...
IOhannes
IRC
3:44:20
so it sets the install_name
_hc
IRC
3:44:22
and version
3:44:23
yup
IOhannes
IRC
3:44:39
but you can override that with DYLIB_LIBRARY_PATH
_hc
IRC
3:44:53
and a .dylib won't work without a install_name
3:44:55
that's the pisser
IOhannes
IRC
3:44:59
the libtool libraries are supposed to be as standard as can be
_hc
IRC
3:45:00
even tho you can override it
IOhannes
IRC
3:45:10
so what's the problem then?
_hc
IRC
3:45:19
that's probably what got me in the past
IOhannes
IRC
3:45:55
because our "overriding" is really only telling the linker to search
for all libraries in an additional place
3:46:28
it's not forcing anything (apart from a certain search _order_)
3:46:40
i have no clue what apple want the install_name to do
_hc
IRC
3:49:37
got it!
3:49:43
well, it built
3:50:11
doh!
3:50:23
the install_name is what is used in the linking
3:51:09
http://pastebin.com/Anf0BnUf
IOhannes
IRC
3:51:10
in which linking?
_hc
IRC
3:51:24
where would you do the DYLD_LIBRARY_PATH linking?
IOhannes
IRC
3:52:07
in the pd-loader
3:52:44
oiata [[email protected]] entered the room.
_hc
IRC
3:52:52
right, where in that process?
IOhannes
IRC
3:54:55
s_loader.c:172 (push (DY)LD_LIBRARY_PATH)
3:55:13
s_loader.c:173 pop (DY)LD_LIBRARY_PATH
3:55:25
s_loader.c:183: push PATH
3:55:33
s_loader.c:184: pop PATH
3:55:54
oops, s_loader.c:185: pop PATH
3:57:35
and actually it should be done for the scheduler-loader as well
(s_loader.c:246)
3:57:42
and for pd~
3:57:58
i wish miller had created a dlopen() wrapper
_hc
IRC
3:59:23
just read your email on this, I didn't think about including random
other libs, that's true it should work also
3:59:30
that would be very helpful on Windows and Mac OS X
_hc
IRC
4:06:23
IOhannes: strange, it seems to work without the DYLD_LIBRARY_PATH set
IOhannes
IRC
4:07:10
what are you doing?
4:07:40
maybe you have the library sitting around somewhere
_hc
IRC
4:07:56
to link the dylib:
4:07:57
-dynamiclib -undefined dynamic_lookup -install_name $(SHARED_LIB) -
compatibility_version 1 -current_version 1.0
4:08:05
to link the objects:
4:08:15
./$(SHARED_LI
4:08:27
cc -arch ppc -arch i386 -arch x86_64 -mmacosx-version-min=10.4 -
bundle -bundle_loader /Applications/Pd-extended.app/Contents/Resources/
bin/pd -undefined dynamic_lookup -o "text.pd_darwin" "text.o" -lc -L/
sw/lib ./libtkwidgets.dylib
4:11:19
here's the actual dylib link:
4:11:19
cc -arch ppc -arch i386 -arch x86_64 -mmacosx-version-min=10.4 -
dynamiclib -undefined dynamic_lookup -install_name libtkwidgets.dylib -
compatibility_version 1 -current_version 1.0 -o libtkwidgets.dylib
tkwidgets.o -lc -L/sw/lib
4:12:29
IOhannes: I figured it out, I was starting pd in the tkwidgets/
folder, that's how it found it
IOhannes
IRC
4:15:39
the only problem i have right now, is that it is crashing on linux
_hc
IRC
4:16:34
IOhannes: does the lib need a versioned name in Linux? ie. can we just
have libtkwidgets.so or must it be libtkwidgets.so.1.
IOhannes
IRC
4:18:52
i _think_ the latter (though i'm not 100% sure)
4:19:27
ah, i think if the library has no so_version, you might get away
with .sl
4:19:31
.so, i meant
_______________________________________________
Pd-dev mailing list
[email protected]
http://lists.puredata.info/listinfo/pd-dev
