On 12 December 2011 17:18, Jeremy O'Donoghue <jeremy.odonog...@gmail.com> wrote: > Hi Dave, all, > > This is fantastic news - especially the bit about GHCi. I have put a couple > of general comments inline, but it looks like you have found most of the > issues, at least on Linux. > > I think that at least some of the issues would go away if, as you say, > > "we could have an entirely separate cabal project (perhaps "wxc") for this > shared library, and then have wxcore depend on it?" > > In this case, we would be able to ensure that libwxc.so.x.y.z (or wxc.dll or > libwxc.dylib or whatever) was in a 'normal' place before wxcore gets built. > > On 9 December 2011 14:13, Dave Tapley <duked...@gmail.com> wrote: >> >> On 8 December 2011 22:34, Dave Tapley <duked...@gmail.com> wrote: >> > Hello everyone, >> > >> > I wrote a story, and I invite you all to comment and help me make it >> > better. >> > As an experiment I've decided to put it on hpaste instead of having >> > the mail thread get out of hand (I'm also cross posting it to the >> > cabal list and have mentioned it in #haskell, and I want to keep all >> > correspondence in one place): >> > http://hpaste.org/55027 >> >> Ha, well that was good timing for hpaste to go down! >> Here's it is: >> >> I've been trying to resurrect the idea of building a shared library >> for the C++ component of the wxhaskell library. >> >> Jeremy (to my knowledge) first put this forward, here: >> >> http://wewantarock.wordpress.com/2010/11/01/working-around-the-static-libstdc-restriction/ >> >> In addition to the advantages he lists there, I have the following reason: >> When building wx-core (in its current incarnation) cabal will *always* >> rebuild all the C++ code, which takes a majority of the build time. >> This becomes very frustrating when you change one line of Haskell and >> have to wait seven minutes to rebuild. >> I complained about it here: >> http://sourceforge.net/mailarchive/message.php?msg_id=28099997 >> >> So, I decided to try and stop this complete C++ rebuild, I partially >> succeed, but I eventually got stuck for reasons I complained here: >> http://www.haskell.org/pipermail/cabal-devel/2011-October/007816.html >> >> Side note: I did eventually discover why (or, where) the c-sources >> (cSources) list is used to compile and link, and it is here: >> >> http://hackage.haskell.org/packages/archive/Cabal/latest/doc/html/src/Distribution-Simple-GHC.html#buildLib >> You can see: >> "| filename <- cSources libBi]" under "-- build any C sources", and >> you can see: >> "cSharedObjs = map (`replaceExtension` ("dyn_" ++ objExtension)) >> (cSources libBi)" >> under "-- link:". >> Is this assumption correct? > > > This is correct - it's the standard way Cabal builds C/C++ code. To be > honest, this is really a consequence of the fact that Cabal is more of a > tool for simple distribution than a Make replacement, and the developers > probably didn't expect Cabal would be used to build large bodies of C++ > code. > > My approach is far from perfect, too - in fact it is incorrect because I > don't do dependency tracking on header files, so if you edit a header, it > (incorrectly) fails to rebuild. This is because I wasn't keen on rewriting > 'make' in the wxHaskell build system - it seems to me like something Cabal > could usefully support. I did play with some very hacky code which > automatically rebuilt all of the C++ code if any header is newer than any > C++ source. This is aesthetically dreadful, but it somewhat captures the > reality, since in practice, pretty much all of the C++ files depend on all > of the headers (and it's easy to write). > > On reflection, we should probably do this as what exists at present could > lead to unexpectedly incorrect code being generated if someone is in full > flow of development. I'll see if I can dig up some code - I think I have it > somewhere.
You know, I'm surprised this didn't occur to me until now: Does the C++ code actually need to use cabal at all? If it's a separate project, perhaps it would be more sensible to use a C++ oriented build system? Then, unless I'm missing something, it can install the headers somewhere, and wxcore can run wxdirect on the headers are currently. The big disadvantage I can see with moving the C++ out of the wxcore project is lots of pain for users when they 'cabal update' and suddenly wxcore fails because they need to update the C++ headers / library. Another disadvantage is that this will make development a little more painful (although the time saved not having to rebuild all the C++ is a huge bonus) because you wouldn't (without some hackery) be able to use cabal-dev as described here: http://haskell.org/haskellwiki/WxHaskell/Development/Environment#Building_and_testing_wxHaskell I happened upon this (dead) project, which I seems to have the goal I am outlining: http://wxc.sourceforge.net/ > >> At this point I thought about either: >> 1. Getting the cabal source and starting to write code to give >> BuildInfo a "cObjs" in addition to "cSources". >> 2. Picking up Jeremy's shared library code, so wxhaskell would have >> its own code to build the library, in which I could do sensible >> re-compilation. >> >> Given that there were other advantages to 2, I went with that. >> >> Jeremy had written one blog post on building a such a shared library, >> here: >> >> http://wewantarock.wordpress.com/2010/11/03/building-a-shared-library-in-cabal/ >> In response to an email on the wxhaskell-devel list he also kindly put >> up this gist, with the code he'd been working on: >> https://gist.github.com/1301115 >> >> I dutifully took this gist, and have now attempted to integrate in to >> my wxhaskell-dev branch, which you can find here: >> http://darcsden.com/DukeDave/wxhaskell-dev >> (note: I haven't pushed any of the shared library code to darcsden >> yet, for reasons I'm about to explain) >> >> This is where things got interesting, after a few hours of hacking I >> have built a shared library, but in this very back-handed way: >> >> Firstly, in Jeremy's code the to-be-compiled shared library is added >> to the wxcore BuildInfo through a custom hook. >> We can see this because: >> > let all_dlls = parseDLLs ["x-dll-name", "x-dll-extra-libraries"] >> > custom_bi >> (the modified wxcore.cabal contains the line: "x-dll-name: wxc") >> >> However on a clean build of wxcore we get the following error: >> >> setup: Missing dependency on a foreign library: >> * Missing C library: wxc >> This problem can usually be solved by installing the system package that >> provides this library (you may need the "-dev" version). If the library is >> already installed but in a non-standard location then you can use the >> flags >> --extra-include-dirs= and --extra-lib-dirs= to specify where it is. > > > My code worked for Windows - but this is a better cross-platform solution. Interesting. I found this fixed feature on cabal: http://hackage.haskell.org/trac/hackage/ticket/262 It suggests that cabal uses Autoconf to resolve dependencies, specifically the AC_CHECK_LIB macro. I have zero knowledge of build systems in Windows, but is it reasonable to suggest that Autoconf could be more relaxed? I really should get a Windows build environment set up.. > >> >> The error is dumped *before* cabal calls myBuildHook, and since this >> actually builds the library the error makes sense; cabal is looking >> for the shared library before we've built it. >> >> To get around this I modified the offending line, thus: >> > let all_dlls = parseDLLs ["x-dll-extra-libraries"] custom_bi >> >> With that modification cabal would now get to myBuildHook, but then >> another curious error arose: >> We see that the actual linking is called here: >> > runProgram verbose gcc (opts' ++ objs' ++ lib_dirs' ++ libs') >> >> But on hitting that line the following error is spat out: >> /usr/bin/ld: cannot open output file dist/build/libwxc.so.0.13.1: No >> such file or directory > > > Windows doesn't behave this way - this part worked for me as well. > >> >> I checked all my permissions and couldn't see anything wrong, I could >> touch the file. >> Conveniently I noticed that, if the verbosity is set high enough, >> runProgram will call printRawCommandAndArgs: >> >> http://hackage.haskell.org/packages/archive/Cabal/latest/doc/html/src/Distribution-Simple-Utils.html#printRawCommandAndArgs >> >> The output (i.e. the linker invocation) looks like this: >> /usr/bin/gcc -fno-stack-protector -shared -Wl,-soname,libwxc.so.0 -o >> dist/build/libwxc.so.0.13.1 dist/build/src/cpp/apppath.o >> dist/build/src/cpp/dragimage.o dist/build/src/cpp/eljaccelerator.o >> [snip-rest-of-.o-files] -L/usr/local/lib -lstdc++ -lwx_baseu-2.9 >> [snip-rest-of-wx-libs] >> >> Now if I cd into the wxcore and paste the command *verbatim* then gcc >> works and generates libwxc.so.0.13.1 as expected. >> You can see in Jeremy's code the linkShareLib function contains: >> > cwd <- getCurrentDirectory >> >> I used this to confirm that the we were in ./wxcore and we are, even >> making the path for -o absolute didn't sovle the issue. >> I ended up replacing runProgram line with this, less satisfactory line: >> > system $ (unwords ([show . locationPath . programLocation $ gcc] ++ >> > opts' ++ objs' ++ lib_dirs' ++ libs')) >> >> Which works, but still doesn't explain why runProgram doesn't. >> Any suggestions? > > > I've tried this on Linux. You're absolutely right and I'm stumped as to why. > I think we should pull this specific part of the thread out and post to the > Haskell list. Maybe someone who knows Cabal and/or GHC would have an idea > why this is the case. Ah, good to know it works on Windows. I'll try and replicate it in isolation and send a mail to the cabal list. > >> >> So with that complete I was able to link a shared library to: >> wxcore/dist/build/libwxc.so.0.13.1 >> >> Of course, at this point we've generated the shared library, but >> wxcore still isn't aware of it, so the build completes successfully, >> but any attempt to compile against it results (understandably) in >> hundreds of linker errors. >> To resolve this I had to add "x-dll-name" (read in from wxcore.cabal >> as "wxc") back to parseDLLs: >> > let all_dlls = parseDLLs ["x-dll-name", "x-dll-extra-libraries"] >> > custom_bi >> >> Well, not quite, attempt to build wxcore again and we're back to this >> error: >> * Missing C library: wxc >> >> Of course, the .so isn't in a 'normal' place, so add its location as >> an extra lib directory: >> > { extraLibDirs = extraLibDirs libbi ++ extraLibDirs wx ++ >> > ["/full/path/to/wxcore/dist/build"] >> (note: I tried using just "dist/build", but cabal said: "library-dirs: >> dist/build is a relative path") > > > I think we need a more robust solution in the longer run, as > <wxhaskell>/wcore/dirt/build is really a temporary location. I guess the 'correct' solution would be to 'install' the library to somewhere like LD_LIBRARY_PATH in Linux. Again I don't know how this fits in with Windows. It's also a pain for people developing wxhaskell, because as I mentioned before, you'd have to 'install' a potentially unstable, development version of the shared library in order to be able to reference it. > >> >> This still isn't quite enough, it took one more kludge: >> /full/path/to/wxcore/dist/build/$ ln -s >> /full/path/to/wxcore/dist/build/libwxc.so.0.13.1 libwxc.so > > > Earlier you mentioned that, at least on Linux, the build command is: > > > /usr/bin/gcc -fno-stack-protector -shared -Wl,-soname,libwxc.so.0 -o > dist/build/libwxc.so.0.13.1 dist/build/src/cpp/apppath.o > dist/build/src/cpp/dragimage.o dist/build/src/cpp/eljaccelerator.o > [snip-rest-of-.o-files] -L/usr/local/lib -lstdc++ -lwx_baseu-2.9 > [snip-rest-of-wx-libs] > > I know GCC generated this and not you, but it looks like the root of the > issue. I am slightly confused though. ld.so is supposed to *know* that > libwxc.so.0.13.1 is a valid instance of libwxc.so.0 Firstly, I'm a little confused by your statement "GCC generated this and not you", that build command is generated by linkCxxOpts, not GCC? This is curious: The name of the symlink has to be "libwxc.so", not even "libwxc.so.0". I think this makes sense though, the only "information" the simple build system gets about the dependency is the parsing of this line in wxcore.cabal by parseDLLs: x-dll-name: wxc I assume that the "wxc" is some how turned in to "libwxc.so", I'd like to know how, and if it is possible to specify the major revision of a shared library passed; this would have to be some convention in the naming, because extraLibs is just [String]: http://hackage.haskell.org/packages/archive/Cabal/latest/doc/html/Distribution-PackageDescription.html#v:extraLibs > > In normal Linux-land, you run /sbin/ldconfig after installing a new library, > and it creates these symlinks automatically You could do this by executing > something like (not tested) > > /sbin/ldconfig -n <path-to-directory> Thanks for the tip, that makes more sense. I can now see that the symlink which "ldconfig -n" generates is extracted from the -soname given at link time. So, by modifying the -soname to *not* specify the major revision, i.e: This: > "-Wl,-soname,lib" ++ basename ++ ".so", Instead of this: > "-Wl,-soname,lib" ++ basename ++ ".so" ++ maj_ver, I can now use "ldconfig -n" in dist/build/ to create the symlink (to libwxc.so), instead of doing it by hand, I suppose this is a little less broken. > >> I'd like to think there's a more elegant solution than this and I'm >> open to suggestions. >> I should note that at this point I became aware that we could have an >> entirely separate cabal project (perhaps "wxc") for this shared >> library, and then have wxcore depend on it? >> >> Running one more time I *thought* I was finally there, until I noticed >> this line, hidden in cabal's output: >> /full/path/to/wxcore/dist/build/libwxc.so: file not recognized: File >> truncated >> >> As far as I can tell, this occurs because we now load(?) >> "libwxc.so.0.13.1" in myConfHook, but then in myBuildHook we want to >> use it as the destination for the linker. Worse still it seems that >> 'truncated' actually means 'deleted'. To get around this I had to one >> again remove "x-dll-name" from the parseDLLs call (so I could get to >> myBuildHook and create "libwxc.so.0.13.1" again), build wxcore, then >> add "x-dll-name" back again, but this time also comment out the call >> to linkSharedLib (and so stop cabal attempting to open >> "libwxc.so.0.13.1") before building wxcore one last time. >> >> Finally everything seemed to work. >> >> Now, I should note that I have been using cabal-dev, and so I now have >> a package-conf I need to reference when building with my new shared >> library wxhaskell. To test it I built the wxhaskell HelloWorld sample >> thus: >> /full/path/to/samples/wxcore$ ghc --make -package-conf >> ../../cabal-dev/packages-7.0.3.conf HelloWorld.hs >> >> It worked, but then when I try to run HelloWorld I get: >> ./HelloWorld: error while loading shared libraries: libwxc.so.0: >> cannot open shared object file: No such file or directory >> >> Okay, again, libwxc.so.0 isn't in a 'normal' place, so I kludged it >> with another symlink: >> /full/path/to/samples/wxcore$ ln -s >> /full/path/to/wxcore/dist/build/libwxc.so libwxc.so.0 >> >> And now, I can report that HelloWorld runs. >> [imagine a screen shot of the HelloWorld sample application running here] >> Rejoice. >> >> But wait, there's more: >> One of the promises of the shared library approach is that we'd all be >> able to use wxhaskell with ghci again, and to my utter disbelief, it's >> true: >> /full/path/to/samples/wxcore$ ghci -package-conf >> ../../cabal-dev/packages-7.0.3.conf HelloWorld.hs >> Ok, modules loaded: Main. >> Prelude Main> main >> >> [imagine a screen shot of the HelloWorld sample application running here] >> Double rejoice. >> >> >> >> > >> > Dave, >> >> >> ------------------------------------------------------------------------------ >> Cloud Services Checklist: Pricing and Packaging Optimization >> This white paper is intended to serve as a reference, checklist and point >> of >> discussion for anyone considering optimizing the pricing and packaging >> model >> of a cloud services business. Read Now! >> http://www.accelacomm.com/jaw/sfnl/114/51491232/ >> _______________________________________________ >> wxhaskell-devel mailing list >> wxhaskell-devel@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/wxhaskell-devel > > ------------------------------------------------------------------------------ Learn Windows Azure Live! Tuesday, Dec 13, 2011 Microsoft is holding a special Learn Windows Azure training event for developers. It will provide a great way to learn Windows Azure and what it provides. You can attend the event by watching it streamed LIVE online. Learn more at http://p.sf.net/sfu/ms-windowsazure _______________________________________________ wxhaskell-devel mailing list wxhaskell-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/wxhaskell-devel
