Re: Compiling to shared library with static dependencies
On Wednesday, 14 December 2016 at 12:33:04 UTC, Tanel L wrote: Any ideas what to do or how to proceed? Thanks a bunch! This kind of questions is better adressed to the "Learn" forum.
Re: Compiling to shared library with static dependencies
On Wednesday, 14 December 2016 at 22:17:25 UTC, Tanel L wrote: First of all thanks Johan for urging me to compile it with only with that flag. It worked! Previously I tried to compile it with only BUILD_SHARED or both - and that failed. But I got it working. Thanks! More details below. Awesome! Thanks, I will use this as a template in the future. Please don't :) - consider other options (D is full of them, I'm sure you'll find much more elegant solution), I pulled this just as a quick test! Thanks, I don't need ffmpeg and I will surely use your advice. I am not worried about the size much. Is there any way to include many/most of them into the binary? Or should I just ship the deps alongside the shared lib? You could build ffmpeg for static linking, but I'm not sure about ggplotd's deps - I suppose there's a way I just haven't done it before. Success! Thank you both so much. With this I can successfully move forward with using D in our company :) Congrats! - if it's possible keep us posted of your endeavors for which you're using Mir and DCV! And of course, feel free to make a question, bug report, feature request etc at our repositories: https://github.com/libmir/mir https://github.com/libmir/dcv All of the best, Relja
Re: Compiling to shared library with static dependencies
First of all thanks Johan for urging me to compile it with only
with that flag. It worked! Previously I tried to compile it with
only BUILD_SHARED or both - and that failed. But I got it
working. Thanks! More details below.
On Wednesday, 14 December 2016 at 21:41:53 UTC, Relja
Ljubobratovic wrote:
On Wednesday, 14 December 2016 at 20:46:15 UTC, Tanel L wrote:
Hi, thanks for the answer.
I had tried disabling all imports, but now I created a clean
new project to test this - it worked.
Awesome. So LDC compile-link is ok, and also python runtime
linking is good.
After that I moved the compiling and linking over to DUB, with
dependencies:
dependency "dcv" version="0.1.7"
dependency "mir" version="0.22.0"
targetPath "output"
targetType "dynamicLibrary"
dflags "-defaultlib=phobos2"
Why are you defining defaultlib explicitly here? - wouldn't
this work fine without it? Also, like I said before, try
removing mir from dependencies since its included with dcv.
Yes it probably would, call it inexperience :) I'm now much
smarter b/c I got it working :D
The library source:
import std.stdio;
import dcv;
import core.runtime:Runtime;
extern (C) int doit(int a, int b) {
return a*b + dcv.imread("1.jpg").width.to!int;
}
Did you actually call Runtime.initialize before calling doit?
Also, did you try the same code in ordinary D app? Anyways,
this works for me:
No, I will use your example as an example for the future, but It
seems it is not required. Although I really did it with
Initialize, I just forgot it from the sample I pasted here - it
didn't make a difference.
D source
import std.stdio;
import core.runtime;
extern (C) void initd()
{
Runtime.initialize;
}
extern (C) void terminated()
{
Runtime.terminate();
}
extern (C) void showim()
{
import dcv;
auto im = imread("img.jpg");
imshow(im);
waitKey();
}
Thanks, I will use this as a template in the future.
Python source
from ctypes import *
d = CDLL("libctypestest.so")
d.initd()
d.showim()
d.terminated()
This created an .so with huge amount of all kinds of external
so deps, but it still failed with the same error:
OSError: libdynlibtest.so: undefined symbol: _d_eh_personality
All kinds of external deps are expected since dcv is linking to
ggplotd and ffmpeg, which also link to whole lotta libraries.
I'm not sure about the binary size though. If you can, use only
dcv:core which has no other dependencies other than Mir. Also,
dcv:io links to ffmpeg at the moment, but if you need only
image i/o and would like to avoid linking to ffmpeg, you could
use just dcv:core with imageformats[1] (dcv wraps it's io
methods in imread). If you need help with this let me know.
Thanks, I don't need ffmpeg and I will surely use your advice. I
am not worried about the size much. Is there any way to include
many/most of them into the binary? Or should I just ship the deps
alongside the shared lib?
Otherwise:
Linux Mint 18 (basically Ubuntu 16.04)
LDC - the LLVM D compiler (1.1.0-beta6):
based on DMD v2.071.2 and LLVM 3.9.0
built with LDC - the LLVM D compiler (1.0.0)
gcc (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609
DUB version 1.1.1, built on Nov 30 2016
Thank you for helping! :)
Hey, no sweat - its a pleasure! I'm really glad you'd like to
use D (and Mir and DCV for that matter). I'd really like to
help you out as much as I can, so this can work out for you! :)
Cheers,
Relja
So I got it working.
Going to leave a mark behind about what I did incase someone else
runs into this.
First of all, I successfully compiled ldc2 with:
cmake .. -DBUILD_SHARED_LIBS=ON
make -j 7
I then removed the installed "dmd" DEB package because it would
always link with its libphobos2.so. I installed only "dub" from
the apt repo instead.
Added the "lflags" param to dub, to have a final dub.sdl:
targetPath "output"
targetType "dynamicLibrary"
--dflags "-defaultlib=phobos2-ldc"
lflags "-L/home/tanel/thirdparty/ldc-dynamic-working/build/lib/"
Mucked around figuring out why dub enforces a
"-defaultlib=phobos2" argument to ldc2 - didn't figure it out.
LDC won't compile because it doesn't exists and if I add my own
to dub.sdl LDC whines that the flag is declared twice.
So i created a symlink in the dynamic ldc lib dir and removed the
"dflags" from my .sdl:
ln -s libphobos2-ldc.so libphobos2.so
Then it still couldn't find it, so I discovered that I had used
LD_LIBRARY_PATH, instead of LIBRARY_PATH - doh! Fixed it.
In .bashrc:
export
LIBRARY_PATH=/home/tanel/thirdparty/ldc-dynamic-working/build/lib/
$ dub run --config=library --compiler=ldc2
$ python -c "from ctypes import *; print
CDLL('libdynlibtest.so').doit(10,10)"
2020
Success!
Thank you both so much. With this I can successfully move forward
with using D in our company :)
Re: Compiling to shared library with static dependencies
On Wednesday, 14 December 2016 at 21:41:53 UTC, Relja Ljubobratovic wrote: use just dcv:core with imageformats[1] http://code.dlang.org/packages/imageformats
Re: Compiling to shared library with static dependencies
On Wednesday, 14 December 2016 at 20:46:15 UTC, Tanel L wrote:
Hi, thanks for the answer.
I had tried disabling all imports, but now I created a clean
new project to test this - it worked.
Awesome. So LDC compile-link is ok, and also python runtime
linking is good.
After that I moved the compiling and linking over to DUB, with
dependencies:
dependency "dcv" version="0.1.7"
dependency "mir" version="0.22.0"
targetPath "output"
targetType "dynamicLibrary"
dflags "-defaultlib=phobos2"
Why are you defining defaultlib explicitly here? - wouldn't this
work fine without it? Also, like I said before, try removing mir
from dependencies since its included with dcv.
The library source:
import std.stdio;
import dcv;
import core.runtime:Runtime;
extern (C) int doit(int a, int b) {
return a*b + dcv.imread("1.jpg").width.to!int;
}
Did you actually call Runtime.initialize before calling doit?
Also, did you try the same code in ordinary D app? Anyways, this
works for me:
D source
import std.stdio;
import core.runtime;
extern (C) void initd()
{
Runtime.initialize;
}
extern (C) void terminated()
{
Runtime.terminate();
}
extern (C) void showim()
{
import dcv;
auto im = imread("img.jpg");
imshow(im);
waitKey();
}
Python source
from ctypes import *
d = CDLL("libctypestest.so")
d.initd()
d.showim()
d.terminated()
This created an .so with huge amount of all kinds of external
so deps, but it still failed with the same error:
OSError: libdynlibtest.so: undefined symbol: _d_eh_personality
All kinds of external deps are expected since dcv is linking to
ggplotd and ffmpeg, which also link to whole lotta libraries. I'm
not sure about the binary size though. If you can, use only
dcv:core which has no other dependencies other than Mir. Also,
dcv:io links to ffmpeg at the moment, but if you need only image
i/o and would like to avoid linking to ffmpeg, you could use just
dcv:core with imageformats[1] (dcv wraps it's io methods in
imread). If you need help with this let me know.
Otherwise:
Linux Mint 18 (basically Ubuntu 16.04)
LDC - the LLVM D compiler (1.1.0-beta6):
based on DMD v2.071.2 and LLVM 3.9.0
built with LDC - the LLVM D compiler (1.0.0)
gcc (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609
DUB version 1.1.1, built on Nov 30 2016
Thank you for helping! :)
Hey, no sweat - its a pleasure! I'm really glad you'd like to use
D (and Mir and DCV for that matter). I'd really like to help you
out as much as I can, so this can work out for you! :)
Cheers,
Relja
Re: Compiling to shared library with static dependencies
On Wednesday, 14 December 2016 at 12:33:04 UTC, Tanel L wrote: So I tried to compile LDC2 dynamically: cmake .. -DBUILD_SHARED=ON Have you tried `cmake .. -DBUILD_SHARED_LIBS=ON` ? -Johan
Re: Compiling to shared library with static dependencies
Hi, thanks for the answer.
I had tried disabling all imports, but now I created a clean new
project to test this - it worked.
After that I moved the compiling and linking over to DUB, with
dependencies:
dependency "dcv" version="0.1.7"
dependency "mir" version="0.22.0"
targetPath "output"
targetType "dynamicLibrary"
dflags "-defaultlib=phobos2"
The library source:
import std.stdio;
import dcv;
import core.runtime:Runtime;
extern (C) int doit(int a, int b) {
return a*b + dcv.imread("1.jpg").width.to!int;
}
This created an .so with huge amount of all kinds of external so
deps, but it still failed with the same error:
OSError: libdynlibtest.so: undefined symbol: _d_eh_personality
Otherwise:
Linux Mint 18 (basically Ubuntu 16.04)
LDC - the LLVM D compiler (1.1.0-beta6):
based on DMD v2.071.2 and LLVM 3.9.0
built with LDC - the LLVM D compiler (1.0.0)
gcc (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609
DUB version 1.1.1, built on Nov 30 2016
Thank you for helping! :)
Re: Compiling to shared library with static dependencies
On Wednesday, 14 December 2016 at 12:33:04 UTC, Tanel L wrote:
Hello,
Hi Tanel,
I have just tried replicating the task you describe, and it
worked flawlessly for me.
I am very new to the D world - but serious in moving over to it
after I have seen what a cool language it really is.
I am trying to incorporate D into our main product as an
externally loadable module.
Basically I have a minimal project:
./app.d
./helpers/image.d (mir and dcv libraries imported)
./helpers/utils.d
I am using the latest (statically compiled) ldc2
In my app file(besides the void main) I have a test function
which I am trying to call from python using ctypes:
extern (C) int doit()
Probably a dumb question - have you tried compiling your library
without dependencies, and using it through ctypes? Does this work?
Currently I have gotten as far as getting a loading error in
python:
library.so: undefined symbol: _d_eh_personality
Googling disclosed that this is satisfied by libdruntime.so,
but I don't have a shared libdruntime anywhere. I guess I have
to compile ldc2 in dynamic mode and use the libdruntime.so from
there?
My build steps until now:
dub.sdl (pseudocode):
project_description "nudesc"
dependency "mir"
dependency "dcv"
targetPath "lib"
dflags "-relocation-mode=pic"
dub build --compiler ldc2
cp .dub/obj/*.o lib/
cd lib
ldc2 -of library.so helpers.image.o helpers.utils.o nudesc.o
-shared -defaultlib=phobos2
Why are you linking manually? Why not let dub do the job? Also, I
believe if you're adding dcv as dependency, you don't need to add
mir, since mir is the dependency of dcv. (it has worked for me
this way so far) but this has nothing to do with errors you
describe.
strace shows that ldc2 loads "phobos2" from DMD:
"/usr/lib/gcc/x86_64-linux-gnu/5/../../../x86_64-linux-gnu/libphobos2.so"
The LDC2 folder only seems to contain libphobos2-ldc.a, no
.so's.
Compilation succeeds, but trying to load the so from python:
python -c "from ctypes import *; CDLL('library.so',
mode=RTLD_GLOBAL)"
OSError: library.so: undefined symbol: _d_eh_personality
I figured that I don't have druntime.so available because
druntime is incorporated into LDC2, but mine is statically
compiled (downloaded precompiled from github i think).
Again, dumb one (and I'm quite sure this is not the issue) - have
you initialized druntime in the doit function?
So I tried to compile LDC2 dynamically:
cmake .. -DBUILD_SHARED=ON
But I get a bunch of these errors before the LDC2 build fails:
/home/tanel/thirdparty/ldc-dynamic/build/bin/ldc2: error while
loading shared libraries: libldc.so: cannot open shared object
file: No such file or directory
runtime/CMakeFiles/druntime-ldc.dir/build.make:93: recipe for
target 'runtime/src/core/bitop.o' failed
make[2]: *** [runtime/src/core/bitop.o] Error 127
Basically I am cornered... I really need to be able to compile
dcv and mir into my dynamic libraries. Probably LDC2 is a must
because DMD and GDC have given me compilation errors on the
project even in regular mode. Also LDC2 seems to be the best.
Any ideas what to do or how to proceed?
Thanks a bunch!
What is your configuration - OS, ldc and gcc version etc.?
Anyhow, I'm surely not the best person to help here - I hope some
of more experienced guys will drop a word to help you out...
And for the end, have you tried creating the python module with
pyd[1]?
Cheers,
Relja
[1] https://github.com/ariovistus/pyd
