Re: [Numpy-discussion] C Extensions, CTypes and external code libraries
--- David Cournapeau [EMAIL PROTECTED] wrote: On Wed, 2008-02-13 at 08:20 -0800, Lou Pecora wrote: Yes, a good question. Two reasons I started off with the static library. One is that Gnu instructions claimed the dynamic library did not always build properly on the Mac OS X. If true, that's a good argument. I don't know the state of libtool of mac os X (the part of autotools which deals with building libraries in a cross platform way). Given the history of apple with open source, I would not be surprised if the general support was subpar compared to other unices. I ran the GSL install again and allowed the dynamic lib to be built. It worked fine so maybe Apple has done better lately. I should have tried it right from the beginning. Anyway, I have a make script that seems to work find and I can link my shared lib to the GSL library or, I would guess, any other library I have. Thanks, again for your helpful suggestions. I will put up my code (it's small) for others to see on this list in a separate message. That might help others not to make the same mistakes I did. I don't know what kind of applications you are developing, but taking care of the time to load the application because of the huge number of symbols seems like really premature optimization to me. That's the kind of problems you don't see if your applications are not huge (or developed with C++, which put a huge pressure on the linker/loader tools by its very nature). Also, note that all modern OS (this includes even windows since NT) do not load the whole shared library in memory, and that two applications needing the GSL will share the same version in memory. The same physical page of a shared library can be mapped into different address spaces (for different processes). I use , because that's a huge over-simplification, and that's where it reaches my own understanding of the thing. This sharing cannot happen for static libraries. Good advice. You are right. My code is not large and it loads fast. -- Lou Pecora, my views are my own. Looking for last minute shopping deals? Find them fast with Yahoo! Search. http://tools.search.yahoo.com/newsearch/category.php?category=shopping ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] C Extensions, CTypes and external code libraries
On Feb 14, 2008 12:14 AM, Lou Pecora [EMAIL PROTECTED] wrote: --- David Cournapeau [EMAIL PROTECTED] wrote: Oh, I may have misunderstood what you are trying to do then. You just want to call a shared library from another shared library ? This is possible on any platform supporting shared library (including but not limited to mac os x, windows, linux, most not ancient unices). [cut] David, First, thanks very much for all the information. I am still digesting it, but you gave a clear explanation about the difference between shared and dynamic libraries on the Mac. I tried some of your compile/like commands, but the Mac gcc did not understand some things like -Bstatic and -shared. It seems to want to make bundles. I guess your code was a Linux version which the Mac doesn't like. Yes, I forgot that -shared does not work on mac os X. -Bstatic, being a linker option as I said, had little chance to work on mac os X. But encouraged by your help, I got the # Library make --- mysharedlib.so: mysharedlib.o mysharedlib.mak gcc -bundle -flat_namespace -undefined suppress -o mysharedlib.so mysharedlib.o \ fcnlib.a # gcc C compile -- mysharedlib.o: mysharedlib.c mysharedlib.h mysharedlib.mak gcc -c mysharedlib.c -o mysharedlib.o In the above fcnlib.a is a simple static library I made before using the above make. This created the shared library mysharedlib.so which I imported and handled with CTypes. Calling a function in fcnlib.a from python worked. A possible downside is that the shared library contains *all* of the fcnlib. I examined it using nm mysharedlib.so. That showed that all the functions of fcnlib.a were present in mysharedlib.so even though the function in mysharedlib.c only called one function of fcnlib.a. I don't know how much of a burden this will impose at run time if do this with GSL. It would be nice to only pick up the stuff I need. But at least I have workable approach. This is logical, or more exactly, the tools did what it thinks you are asking: putting the archive (the .a library) in your executable. If instead of libfnclib.a, you just put -lfcnlib in the command line, it will pick up only the necessary code for the functions you are calling (at least it does on linux if I remember correctly). But the real question is : if you are concerned with code bload, why using static lib at all ? Why not using shared library, which is exactly designed to solve what you are trying to do ? cheers, David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] C Extensions, CTypes and external code libraries
--- David Cournapeau [EMAIL PROTECTED] wrote: But the real question is : if you are concerned with code bload, why using static lib at all ? Why not using shared library, which is exactly designed to solve what you are trying to do ? cheers, David Yes, a good question. Two reasons I started off with the static library. One is that Gnu instructions claimed the dynamic library did not always build properly on the Mac OS X. So I just built the static GSL and figured if I got that to link up to my code, I could then spend some time trying the dynamic build. The other reason is that I am just learning this and I am probably backing into the right way to do this rather than starting right off with the right way. Maybe my worries about bloat and (even more) time to load are not important for the GSL and the code will load fast enough and not take up too much in resources to matter. Later today I will try to build the dynamic version of GSL and see what that yields. If I get it I will link to that as you suggest. Thanks, again. Your suggestions have moved me along nicely. -- Lou Pecora, my views are my own. Never miss a thing. Make Yahoo your home page. http://www.yahoo.com/r/hs ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] C Extensions, CTypes and external code libraries
--- David Cournapeau [EMAIL PROTECTED] wrote: Oh, I may have misunderstood what you are trying to do then. You just want to call a shared library from another shared library ? This is possible on any platform supporting shared library (including but not limited to mac os x, windows, linux, most not ancient unices). [cut] David, First, thanks very much for all the information. I am still digesting it, but you gave a clear explanation about the difference between shared and dynamic libraries on the Mac. I tried some of your compile/like commands, but the Mac gcc did not understand some things like -Bstatic and -shared. It seems to want to make bundles. I guess your code was a Linux version which the Mac doesn't like. But encouraged by your help, I got the following make file to work: # Library make --- mysharedlib.so: mysharedlib.o mysharedlib.mak gcc -bundle -flat_namespace -undefined suppress -o mysharedlib.so mysharedlib.o \ fcnlib.a # gcc C compile -- mysharedlib.o: mysharedlib.c mysharedlib.h mysharedlib.mak gcc -c mysharedlib.c -o mysharedlib.o In the above fcnlib.a is a simple static library I made before using the above make. This created the shared library mysharedlib.so which I imported and handled with CTypes. Calling a function in fcnlib.a from python worked. A possible downside is that the shared library contains *all* of the fcnlib. I examined it using nm mysharedlib.so. That showed that all the functions of fcnlib.a were present in mysharedlib.so even though the function in mysharedlib.c only called one function of fcnlib.a. I don't know how much of a burden this will impose at run time if do this with GSL. It would be nice to only pick up the stuff I need. But at least I have workable approach. Thanks for your help. Comments welcome. -- Lou Pecora, my views are my own. Never miss a thing. Make Yahoo your home page. http://www.yahoo.com/r/hs ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] C Extensions, CTypes and external code libraries
--- Jon Wright [EMAIL PROTECTED] wrote:
Lou Pecora wrote:
... This appears to be the way
static and shared libraries work, especially on
Mac OS
X, maybe elsewhere.
Have you tried linking against a GSL static library?
I don't have a mac,
but most linkers only pull in the routines you need.
For example, using
windows and mingw:
#include stdio.h
#include gsl/gsl_sf_bessel.h
int main (void)
{ double x = 5.0;
double y = gsl_sf_bessel_J0 (x);
printf (J0(%g) = %.18e\n, x, y);
return 0; }
...compiles to a.exe which outputs:
J0(5) = -1.775967713143382900e-001
Yes, I know about this approach if I am making an
executable. But I want to make my code into a shared
library (my code will not have a main, just the
functions I write) and, if possible, let my code call
the GSL code it needs from the C function I write
(i.e. no python interface). If what you did can be
done for a shared library, then that would be great.
However, I am ignorant of how to do this. I will try
to make my shared library using gcc and then add the
GSL library using the -l option as someone else
suggested. Maybe that will work. I'll report back.
I have been searching for info on the right approach
to this on the Mac, since, as I understand, Mac OS X
does make a distinction between shared libraries and
dynamic libraries (which I don't understand fully).
Thanks.
-- Lou Pecora, my views are my own.
Be a better friend, newshound, and
know-it-all with Yahoo! Mobile. Try it now.
http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] C Extensions, CTypes and external code libraries
Lou Pecora wrote:
--- Jon Wright [EMAIL PROTECTED] wrote:
Lou Pecora wrote:
... This appears to be the way
static and shared libraries work, especially on
Mac OS
X, maybe elsewhere.
Have you tried linking against a GSL static library?
I don't have a mac,
but most linkers only pull in the routines you need.
For example, using
windows and mingw:
#include stdio.h
#include gsl/gsl_sf_bessel.h
int main (void)
{ double x = 5.0;
double y = gsl_sf_bessel_J0 (x);
printf (J0(%g) = %.18e\n, x, y);
return 0; }
...compiles to a.exe which outputs:
J0(5) = -1.775967713143382900e-001
Yes, I know about this approach if I am making an
executable. But I want to make my code into a shared
library (my code will not have a main, just the
functions I write) and, if possible, let my code call
the GSL code it needs from the C function I write
(i.e. no python interface). If what you did can be
done for a shared library, then that would be great.
However, I am ignorant of how to do this. I will try
to make my shared library using gcc and then add the
GSL library using the -l option as someone else
suggested. Maybe that will work.
Oh, I may have misunderstood what you are trying to do then. You just
want to call a shared library from another shared library ? This is
possible on any platform supporting shared library (including but not
limited to mac os x, windows, linux, most not ancient unices).
As Albert said, just do (with gcc):
gcc -shared -o mysharedlib mysharedlib.c -lgsl
This works on mac os X as well as linux (and even windows with mingw).
If you want to link the gsl statically (so that your own lib does not
depend on the gsl anymore), you have use a trick to tell gcc to link the
gsl:
gcc -shared -o mysharedlib mysharedlib.c -Wl,-Bstatic -lgsl -Wl,-Bdynamic
-Wl is used by gcc to pass option to the linker directly. -Bstatic says
that all link options after will be static. You have to use -Bdynamic
after, to avoid linking everything static (like gcc runtime, the C lib,
which are automatically linked by default by gcc: it is almost always a
bad idea to statically link those).
In the first case, mysharedlib.so will need libgsl.so:
ldd mysharedlib.so - linux-gate.so.1 = (0xe000)
libgsl.so.0 = /usr/lib/libgsl.so.0 (0xb7ddb000)
libc.so.6 = /lib/tls/i686/cmov/libc.so.6 (0xb7c91000)
libm.so.6 = /lib/tls/i686/cmov/libm.so.6 (0xb7c6b000)
/lib/ld-linux.so.2 (0x8000)
In the second case:
linux-gate.so.1 = (0xe000)
libc.so.6 = /lib/tls/i686/cmov/libc.so.6 (0xb7e6)
/lib/ld-linux.so.2 (0x8000)
I don't know if the second method works on mac os X: since it bypass gcc
and goes directly to the linker, which is notably different on mac os X,
you may have to do it differently.
I'll report back.
I have been searching for info on the right approach
to this on the Mac, since, as I understand, Mac OS X
does make a distinction between shared libraries and
dynamic libraries (which I don't understand fully).
To be over-simplistic: shared libraries are linked into the executable,
and all its symbols (function, variables) are solved when you launch the
executable. A dynamic library is not linked into the executable, and can
be loaded at anytime during the execution of the executable. Shared
library are just a way to avoid duplicate code, but are totally
transparent to the code user:
int foo()
{
return bar();
}
If bar is in a shared lib (libbar.so) or in another object code (bar.o),
it does not make a difference for you. With a dynamic lib, on most
unices, you do
hdl = dlopen(libbar.so)
((int)(*bar)()) = dlsym(hdl, bar);
That is you explicitly load the functions you want to use. Without this
scheme, you would have to link your extension to the python executable
when python is built, which is totally impractical of course. IOW,
dynamic libraries are used for plugins, things which can be added to
an executable *after* the executable is built.
On linux (and other unices using the elf binary format), both types of
libraries are built exactly the same. On mac os X (and windows as well),
they are not. Again, this is oversimplification, but you don't need to
know much more in almost all the cases.
cheers,
David
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] C Extensions, CTypes and external code libraries
Lou Pecora wrote:
... This appears to be the way
static and shared libraries work, especially on Mac OS
X, maybe elsewhere.
Have you tried linking against a GSL static library? I don't have a mac,
but most linkers only pull in the routines you need. For example, using
windows and mingw:
#include stdio.h
#include gsl/gsl_sf_bessel.h
int main (void)
{ double x = 5.0;
double y = gsl_sf_bessel_J0 (x);
printf (J0(%g) = %.18e\n, x, y);
return 0; }
...compiles to a.exe which outputs:
J0(5) = -1.775967713143382900e-001
The stripped executable is about 92 kB in comparison to the 2 mega byte
libgsl.a. Unstripped there are about 150 symbols containing gsl,
compared to 5351 symbols in the library libgsl.a. I just needed to put
-lgsl on the command line and rename $LIB/libgsl.dll.def to
something else so the shared version wasn't found.
In this case the linker has not pulled in all of the library. Presumably
just the parts it needed, including various things like error reporting,
sin, cos, exp etc. Older platforms, including vax and various unix'es
also seemed to behave in the same way in the past.
Are you saying the mac is somehow different? Perhaps they're trying to
hold people to open source ransom, where they have to give away to
source so it can be recompiled when the next OSX escapes ;-)
Cheers,
Jon
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] C Extensions, CTypes and external code libraries
Albert Strasheim [EMAIL PROTECTED] wrote: Hello, I only quickly read through the previous thread, but I get that idea that what you want to do is to link your shared library against the the GSL shared library and then access your own library using ctypes. If done like this, you don't need to worry about wrapping GSL or pulling GSL code into your own library. As far as I know, this works exactly like it does when you link an executable against a shared library. If distutils doesn't allow you to do this easily, you could try using SCons's SharedLibrary builder instead. Regards, Albert ___ Albert, Yes, I think you got the idea right. I want to call my own C code using CTypes interface, then from within my C code call GSL C code, i.e. a C function calling another C function directly. I do *not* want to go back out through the Python interface. So you are right, I do not want to wrap GSL. It sounds like I can just add something like -lnameofGSLdylib (where I put in the real name of the GSL library after the -l) in my gcc command to make my shared lib. Is that right? Thanks for your help. -- Lou Pecora, my views are my own. - Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it now.___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] C Extensions, CTypes and external code libraries
First, thanks to all who answered my questions about trying to use a large library with CTypes and my own shared library. The bottom line seems to be this: There is no way to incorporate code external to your own shared library. You have to either pull out the code you want from the static library's source (painful) or you must just include the whole library (huge!) and make it all one big shared library. Did I get that right? If so, it's a sad statement that makes shared libraries harder to write and works against the reuse of older established code bases. I am not criticizing CTypes. This appears to be the way static and shared libraries work, especially on Mac OS X, maybe elsewhere. I'd really like to be wrong about this and I will follow up on some of the suggested reading you all gave me. Thanks, again. -- Lou Pecora, my views are my own. Looking for last minute shopping deals? Find them fast with Yahoo! Search. http://tools.search.yahoo.com/newsearch/category.php?category=shopping ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
