Lots of issues. Firstly, GMP 5 is currently a drop-in replacement for MPIR for us. We prefer to use MPIR since I am the maintainer of the latter, and significant work has gone into making various features faster.
For flint we definitely require the latest bleeding edge git version. Even the last official release from us, flint-2.4.4, will not work with Nemo. The CP text file is absolutely enormous (nearly a megabyte). Encoding it as a string is certainly out of the question (we strictly adhere to the ANSI standard, which limits strings to something like 750 bytes). We've discussed many other ways of handling it, and numerous people have tried various things. It's a major pain in the backside, but totally necessary to have it (mathematically speaking). Currently it is not too bad in that once flint is installed, so is the file, in a location with a fixed offset from the library (which will have been specified by --prefix). If the user chooses to build flint in the source tree, it is located in a fixed location within the source tree which flint can also find it. This is all fine, so long as nothing gets moved after compilation and installation, which is the usual case, it seems. The problem with linking against Julia's GMP and MPFR was as follows. We can tell Julia where to find flint by pushing to DL_LOAD_PATH. But this only tells Julia where to find flint. The system linker is the one that tries to subsequently link flint with MPIR and MPFR, and it hasn't got a clue where to find them. What we push on DL_LOAD_PATH is irrelevant, as only Julia knows about that. This actually causes Julia to report that it can't find flint. I spent many, many hours figuring that one out. Now it occurs to me that perhaps if we build flint against Julia's GMP and MPFR (assuming GMP is version 5 and MPFR version 3.1), then because Julia's libs are placed in the path by Julia, the system linker might just find them. That hadn't occurred to me as a possibility. The obvious problem is GMP is not MPIR and we are limited to using whatever version of GMP Julia chooses. Actually, maybe the git version of flint already supports GMP 6 nowadays too. Also, flint uses libgcc_s and pthreads (and libm). We've currently set it up to statically build libgcc_s into flint, but we can't do that with pthreads. Initially at least we could disable pthreads when building flint (there is an option for it) but I guess in the long run we won't want to do that. I guess we can for now, but it seems like a compromise. Regarding licenses, I would guess the GPL v3 and LGPL v3 license texts need to be distributed with Julia binaries, along with a written offer to provide the source code (or the actual source code). Certainly the LGPL allows Julia to dynamically link against LGPL libraries without becoming a derivative work. I don't want to make absolute statements about that issue, as IANAL, but I'd be interested in more details of how Julia is satisfying the conditions summarised here http://en.wikipedia.org/wiki/GNU_General_Public_License#Terms_and_conditions Aside from those issues, I don't seem to easily be able to find out what version of GMP or MPFR Julia is currently using in v0.3. That could be helpful. Regarding the .jl extension, I did read that, but am not entirely sure I know how to change it on github. I will take a look and see if I can change it. I didn't realise @unix_only was the opposite of @windows_only. I thought unix meant Free BSD, Net BSD, Unix, etc. And I couldn't see a way to combine Unix, Linux and OSX using the macros. Given that OSX is probably going to need something special, I'm not sure what the best combination of macros will be. Popcnt is processor dependent. TLS is more OS dependent and I think available on Mingw. We can just disable hardware popcnt. It's not vitally important for more than about one noncritical function. Thanks for all your comments and your assistance. Unfortunately flint is slightly complicated by various issues. For now I'm happy to go with GMP on Windows (ironic given that MPIR was originally a Windows-friendly fork of GMP). But for the simple reason that GMP is already there in Julia it shouldn't be a problem for now. In the long term we definitely want to go with MPIR. I will probably add some additional files to assure there is no question of compliance with licenses for Nemo at least. Bill. On 13 September 2014 06:22, Tony Kelman <[email protected]> wrote: > The library cross-compiles okay (at least at the last released version, I > can try git master if you've made changes that are necessary), see > https://build.opensuse.org/package/show/home:kelman/mingw32-flint and > https://build.opensuse.org/package/show/home:kelman/mingw64-flint > > It was made a little more challenging by the fact that flint uses a > hand-rolled configure script that errors on unrecognized arguments. That > configure script uses --build for what is conventionally meant by --host, > the two differ in a cross compile. --host is where the compiled code is > intended to run, --build is (in standard autotools) the machine that is > doing the compiling. > > There was an existing openSUSE Linux package for flint here > https://build.opensuse.org/package/show/science/flint with a patch to use > real autotools, though it's not completely usable for a cross compile at > the moment since it doesn't handle using GMP instead of MPIR. MPIR is not > available as a cross-compiled package yet, though making one would probably > be easy. The existing cross-compiled version of MPFR already links to GMP > so I think it would be redundant. > > Flint has configure-time checks for popcnt and tls that run executables, > neither of which work properly in a cross-compile. Locally on Windows they > both succeed, but could those potentially be processor-dependent in any > way? I could patch configure to override the results for the purposes of > cross compiling, if you can recommend one way or the other. > > On the text file, it's really bad for library relocatability to hard-code > the compile-time location of required data. Is the content of the file ever > expected to change? If not, I would just hard-code its contents into a > string or more appropriate data structure. If so, then its location should > really be made a runtime setting, either through a library initialization > API or from an environment variable or something. > > I could be wrong, but I don't think rpath does anything on Windows. > Standard practice for MinGW libraries is to put dll files in bin/ rather > than lib/ for this reason. If you were using libtool, it would create a > .dll.a import library stub in lib/ to assist in linking to the dll if you > wanted to compile a C/C++ application or library with -lflint. Luckily > Julia doesn't need that stub to just ccall into a dll. > > -Tony > > > On Friday, September 12, 2014 6:11:49 PM UTC-7, Tony Kelman wrote: >> >> > I'm worried about the legality of downloading binaries for users, since >> the packages involved, GMP/MPIR, MPFR and FLINT are (L)GPL. >> >> Quite a few packages rely on (L)GPL code. The GPL is a long confusing >> document, and limits commercial application of some code, that's for sure, >> but to the best of my (IANAL) understanding it's not too onerous to comply >> with for open-source projects. Mostly including the license in the >> documentation and being careful about bundling/derived works. The spec >> files on the build service have to clearly state the license, plenty of >> libraries there are GPL and/or have GPL dependencies. >> >> > I see now that Julia adds things to the system PATH, which is why it is >> picking up things in the GitBash bundled in Julia. If I can figure out how >> to get PATH back to its original state temporarily, I think I can get the >> build to succeed (supposing the user has MSYS2 and Mingw64 correctly >> installed -- not easy, admittedly). >> >> You can modify ENV["PATH"] as you like, but I recommend against this >> approach. Compare the relative difficulty of building Julia from source to >> downloading and using the binary installer. On a slightly smaller scale, >> having the package rely on MSYS2 and MinGW vs just downloading binaries >> will lead to a similar usability comparison for your package. On Linux >> there's much more variety in compiler and library versions so the best way >> to download binaries is going through the system package manager, but if >> your library isn't available in the package manager then building from >> source is easy because compilers are really easy to install. On Mac it's a >> little more annoying to install compilers, so a decent number of packages >> use Homebrew.jl to distribute binaries. (Just have to set up a short >> "formula" file with instructions on how to build, and bug Elliot, >> @staticfloat on github, to build binaries for you). >> >> > I agree binary distribution is the normal method on Windows. But I've >> already been forced to go through one FSF "license compliance review" >> because a GNU project reported us for being in breach of the license. I'm >> not doing it again. >> > >> > If there is a way to do things in strict accordance with the LGPL and >> GPL I can go down that route. In that case, you can see the invocations >> here: >> >> I can imagine that experience would give you pause here. If you can link >> directly to the same version of GMP and MPFR as Julia is using (they are >> currently required dependencies, so this should be fairly safe - just have >> to figure out the best way to do it that would also work if someone is >> using their system's version of GMP and/or MPFR), then you don't need to >> worry about distributing anything except Flint itself. Would GMP be a >> drop-in replacement for MPIR for you? If we use WinRPM, I believe it would >> automatically detect GMP and MPFR (both are already available there) as >> linked DLL's and automatically download them, we just have to declare the >> dependency in the spec file and make sure configure picks them up or is >> told explicitly where they are located. They would be in a standard >> predictable location on the cross-compiling build machine. >> >> > https://github.com/wbhart/Nemo/blob/master/deps/build.jl >> > >> > Is that sufficient? >> >> I can see from that script how you're configuring, yes. Downloading and >> building your own copy of MPIR and MPFR seems slightly redundant and >> possibly problematic to load them from within the same process as Julia's >> copies of GMP and MPFR. >> >> > I should mention that there are some complications. Flint needs to be >> built in situ because it needs a large .txt file at runtime which contains >> data it uses (Conway polynomials). The only way to tell flint where it will >> be is through the build system, and the only way to know where it will be >> is to actually build flint. >> >> Can that text file be included along with the binary? >> >> > Moreover, the system linker cannot find the dependencies MPIR and MPFR >> of flint unless the -rpath is set by flint during build. And again, the >> only way -rpath can be set is if flint knows where it is being built. >> >> You can't just provide a full -L path? >> >> A few minor notes, standard convention for Julia packages is to name the >> Git repository include the ".jl" extension in its name. This helps visually >> identify Julia package repositories easily, and is nice to keep consistent. >> >> You're also frequently doing "if on_windows", there's a pre-defined macro >> for that: @windows_only, or the opposite @unix_only. >> >> -Tony >> >> >>
