RE: GHCI/FFI/GMP/Me madness
On 13 August 2004 14:04, Remi Turk wrote: How do you arrange to free a GMP integer when it is no longer referenced from the heap? You'd need finalizers, and that way lies madness. The memory allocation tricks we play with GMP are all to support GC of Integers. Errr, I guess I'll have to start looking for a straitjacket then ;) Finalisers where exactly what I was thinking about, and though below implementation without doubt contains a bunch of bugs, perhaps even show-stoppers, the following code _does_ print True: import Mpz main= print $ show (z::Integer) == show (z::Mpz) where -- some random calculations x, y, z :: Blub a = a x = 2^64 - sum (take 100 $ iterate (63*) 3) y = 3^200 * 234233432 - (34 `pow` 38) + sum [1,87..2] z = fac 5000 * (x + y) `div` (2^100) Or are finalisers simply going to be too slow to take seriously for this? You can't use mallocForeignPtr (because GMP needs to allocate its own memory), so you have to use full-blown ForeignPtrs with malloc/free and finalizers. Yes, these are very slow compared to using GC'd memory. Cheers, Simon ___ Glasgow-haskell-users mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
RE: GHCI/FFI/GMP/Me madness
On 11 August 2004 20:45, Remi Turk wrote: But as long as GMP doesn't mind about being abused the way my most recent util.c does, I can get away with the mp_set_memory_functions-trick, can't I? Not really. You can't let GMP realloc() a memory block that was allocated using one method, with a different method. Also, GMP must not try to free() a block that was allocated using a different method. (*Be sure to call `mp_set_memory_functions' only when there are no active GMP objects allocated using the previous memory functions! Usually that means calling it before any other GMP function.*, and using undocumented features) And with this trick and a ffi GMP-binding implement a working Mpz datatype. And when (if?) this is done, drop in a type Mpz = Integer, rip out all Integer-primops, remove the mp_set_memory_functions-trick and start benchmarking? (Conveniently forgetting that fromInteger :: Integer - Integer most certainly has to stay a primop anyway...) How do you arrange to free a GMP integer when it is no longer referenced from the heap? You'd need finalizers, and that way lies madness. The memory allocation tricks we play with GMP are all to support GC of Integers. Or is the rts using Integers in such a way that any (standard malloc) allocations are forbidden while e.g. (*) :: Integer - Integer - Integer is running? Not sure what you mean here - malloc() can always be used. Cheers, Simon ___ Glasgow-haskell-users mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
RE: GHCI/FFI/GMP/Me madness
On 10 August 2004 16:04, Remi Turk wrote:
- reset GMP's memory allocators before calling it from your code,
and set them back to the RTS allocators afterward. Slow, but it
should work. It doesn't solve the problem properly though:
external libraries which use GMP are still broken.
It does indeed seem to work, after a quick test. (new util.c attached)
And it does solve _my_ immediate problem: I can surround every
gmp-operation with a gmp_begin()/gmp_end() and pretend to be
happy. (and anyway, I'm just messing around, failure merely means
I've got yet another unfinished project ;))
Part of the reason for all this messy FFIing is your post:
http://www.haskell.org/pipermail/glasgow-haskell-users/2004-June/006767.
html
Hmm yes, I now realise that it's not quite as easy as I implied in that
message. The problem is the memory allocation. If a GMP function
allocates some memory, we have to swizzle the pointer that comes back
(where swizzle(p) { return p-sizeof(StgArrWords) }). Unfortunately you
have to do this without giving the GC a chance to run, and there's no
way to get this atomicity in plain Haskell+FFI, which is why the primops
are still necessary.
Perhaps one way to do it would be to define generic Integer primop
wrappers - i.e. one wrapper for an mpz function that takes two arguments
and returns one, etc. The primop implementations already work like
this, except that the wrappers are CPP macros. If the wrapper were
lifted to the level of a primop itself, then you could easily use
different mpz functions by calling the appropriate primop passing the
address of the mpz function.
Cheers,
Simon
___
Glasgow-haskell-users mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: GHCI/FFI/GMP/Me madness
On Wed, Aug 11, 2004 at 02:27:19PM +0100, Simon Marlow wrote:
On 10 August 2004 16:04, Remi Turk wrote:
http://www.haskell.org/pipermail/glasgow-haskell-users/2004-June/006767.html
Hmm yes, I now realise that it's not quite as easy as I implied in that
message. The problem is the memory allocation. If a GMP function
allocates some memory, we have to swizzle the pointer that comes back
(where swizzle(p) { return p-sizeof(StgArrWords) }). Unfortunately you
have to do this without giving the GC a chance to run, and there's no
way to get this atomicity in plain Haskell+FFI, which is why the primops
are still necessary.
Perhaps one way to do it would be to define generic Integer primop
wrappers - i.e. one wrapper for an mpz function that takes two arguments
and returns one, etc. The primop implementations already work like
this, except that the wrappers are CPP macros. If the wrapper were
lifted to the level of a primop itself, then you could easily use
different mpz functions by calling the appropriate primop passing the
address of the mpz function.
Ah, the shockingly inefficient family of GMP_TAKEx_RETy macros ;)
(I understand the swizzle-talk only partly, so I'll ignore it and
hope my words won't turn out to be utter nonsense..)
But as long as GMP doesn't mind about being abused the way my
most recent util.c does, I can get away with the
mp_set_memory_functions-trick, can't I?
(*Be sure to call `mp_set_memory_functions' only when there are no
active GMP objects allocated using the previous memory functions!
Usually that means calling it before any other GMP function.*,
and using undocumented features)
And with this trick and a ffi GMP-binding implement a working Mpz
datatype.
And when (if?) this is done, drop in a type Mpz = Integer, rip
out all Integer-primops, remove the mp_set_memory_functions-trick
and start benchmarking?
(Conveniently forgetting that fromInteger :: Integer - Integer
most certainly has to stay a primop anyway...)
Or is the rts using Integers in such a way that any (standard
malloc) allocations are forbidden while e.g. (*) :: Integer -
Integer - Integer is running?
Cheers,
Simon
Greetings,
Remi
--
Nobody can be exactly like me. Even I have trouble doing it.
___
Glasgow-haskell-users mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
RE: GHCI/FFI/GMP/Me madness
On 09 August 2004 16:57, Remi Turk wrote: On Sun, Aug 08, 2004 at 07:34:04AM -0700, Sigbjorn Finne wrote: Hi, please be aware that the RTS uses GMP as well, and upon initialisation it sets GMP's 'memory functions' to allocate memory from the RTS' heap. So, in the code below, the global variable 'p' will end up having components pointing into the heap. Which is fine, until a GC occurs and the pointed-to GMP allocated value is eventually stomped on by the storage manager for some other purpose. I'm _guessing_ that's the reason for the behaviour you're seeing. Hm, I _was_ aware of mp_set_memory_functions being used by the RTS. I've seen it often enough in ltrace's ;) It does indeed sound rather plausible (and making big allocations and such does indeed cause it to happen earlier). At which point my next question is: what now? I don't feel really confident about my GHC-hacking skills (huh? skills? where? ;) so does that mean I'm out of luck? *looks* Am I correct that I'd have to copy any GMP-allocated memory to my own memory before returning from C and vice-versa? I hope not :( GHC's use of GMP does cause problems if you want to use GMP for your own purposes, or if you link with external code that wants to use GMP. The real problem is that GMP has internal state, which means it can't be used in a modular way. But there's not much we can do about that. Possibilities: - Rename all the symbols in our GMP to be unique. (sounds hard) - Replace GMP with something else (someone is working on this, I believe). - try to get two copies of GMP into your program by pre-linking the RTS with one copy, then linking the rest of the program with the other copy. I'm being intentionally vague here - I feel that this ought to be possible, but it's an ugly hack at best. - reset GMP's memory allocators before calling it from your code, and set them back to the RTS allocators afterward. Slow, but it should work. It doesn't solve the problem properly though: external libraries which use GMP are still broken. Cheers, Simon ___ Glasgow-haskell-users mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
RE: GHCI/FFI/GMP/Me madness
On 10 August 2004 13:03, MR K P SCHUPKE wrote: Re GMP, Why not provide more GMP functions as primitives on the Integer type, and avoid the need to call out to GMP via the FFI? Show us the code! :-p Cheers, Simon ___ Glasgow-haskell-users mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: GHCI/FFI/GMP/Me madness
Hi! On Tue, Aug 10, 2004 at 12:59:46PM +0100, Simon Marlow wrote: On 09 August 2004 16:57, Remi Turk wrote: Hm, I _was_ aware of mp_set_memory_functions being used by the RTS. I've seen it often enough in ltrace's ;) It does indeed sound rather plausible (and making big allocations and such does indeed cause it to happen earlier). At which point my next question is: what now? I don't feel really confident about my GHC-hacking skills (huh? skills? where? ;) so does that mean I'm out of luck? *looks* Am I correct that I'd have to copy any GMP-allocated memory to my own memory before returning from C and vice-versa? I hope not :( GHC's use of GMP does cause problems if you want to use GMP for your own purposes, or if you link with external code that wants to use GMP. The real problem is that GMP has internal state, which means it can't be used in a modular way. But there's not much we can do about that. Possibilities: [...] - reset GMP's memory allocators before calling it from your code, and set them back to the RTS allocators afterward. Slow, but it should work. It doesn't solve the problem properly though: external libraries which use GMP are still broken. I would be careful though: http://www.swox.com/gmp/manual/Custom-Allocation.html Be sure to call mp_set_memory_functions only when there are no active GMP objects allocated using the previous memory functions! Usually that means calling it before any other GMP function. Greetings, Carsten -- Carsten Schultz (2:38, 33:47), FB Mathematik, FU Berlin http://carsten.codimi.de/ PGP/GPG key on the pgp.net key servers, fingerprint on my home page. pgpHRM263ZIUM.pgp Description: PGP signature ___ Glasgow-haskell-users mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: GHCI/FFI/GMP/Me madness
- Rename all the symbols in our GMP to be unique. (sounds hard) Nah, easy. First get a list of symbols: nm libgmp.a | grep gmp [Supply a few flags to nm to get a better list.] Then turn it into a list of #defines: #define gmp_blah ghc_gmp_blah ... #include your list of #defines into some header file that everyone eats. make clean; make You can generate the list by hand or have it rebuilt by make every time. -- Alastair Reid ___ Glasgow-haskell-users mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
RE: GHCI/FFI/GMP/Me madness
On 10 August 2004 14:08, Alastair Reid wrote: - Rename all the symbols in our GMP to be unique. (sounds hard) Nah, easy. First get a list of symbols: nm libgmp.a | grep gmp [Supply a few flags to nm to get a better list.] Then turn it into a list of #defines: #define gmp_blah ghc_gmp_blah ... #include your list of #defines into some header file that everyone eats. make clean; make You can generate the list by hand or have it rebuilt by make every time. Hmmm... well maybe. We'd have to be sure to always use the version of GMP that comes with GHC - currently we use an installed dynamic version if one is available. Statically linking GMP runs into licensing problems, so I'd have to figure out how to build it dynamically. The native codegen has to use the revised names, too. (the #defines won't work there because it mentions the names inside strings). I'd bet there are almost certainly some other gotchas. Cheers, SImon ___ Glasgow-haskell-users mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: GHCI/FFI/GMP/Me madness
On Tue, Aug 10, 2004 at 12:59:46PM +0100, Simon Marlow wrote: GHC's use of GMP does cause problems if you want to use GMP for your own purposes, or if you link with external code that wants to use GMP. The real problem is that GMP has internal state, which means it can't be used in a modular way. But there's not much we can do about that. Possibilities: - Rename all the symbols in our GMP to be unique. (sounds hard) and ugly - Replace GMP with something else (someone is working on this, I believe). Do you have a pointer? It sounds interesting. *see below* - try to get two copies of GMP into your program by pre-linking the RTS with one copy, then linking the rest of the program with the other copy. I'm being intentionally vague here - I feel that this ought to be possible, but it's an ugly hack at best. I'm not sure I share your feelings about that ;) It sounds like symbol-clash-hell. But quite possibly I'm just being ignorant. - reset GMP's memory allocators before calling it from your code, and set them back to the RTS allocators afterward. Slow, but it should work. It doesn't solve the problem properly though: external libraries which use GMP are still broken. It does indeed seem to work, after a quick test. (new util.c attached) And it does solve _my_ immediate problem: I can surround every gmp-operation with a gmp_begin()/gmp_end() and pretend to be happy. (and anyway, I'm just messing around, failure merely means I've got yet another unfinished project ;)) Part of the reason for all this messy FFIing is your post: http://www.haskell.org/pipermail/glasgow-haskell-users/2004-June/006767.html If Integers where implemented via the FFI that would make it quite a bit easier to special-case e.g. (^) and Show for Integer. (IIRC, GMP's mpz-to-string recently got a huge speedup, it would be nice if GHC would automagically profit of that..) Cheers, Simon Happy hacking keep up the good work ;) Remi -- Nobody can be exactly like me. Even I have trouble doing it. ___ Glasgow-haskell-users mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: GHCI/FFI/GMP/Me madness
On Tue, Aug 10, 2004 at 01:09:03PM +0100, Simon Marlow wrote: On 10 August 2004 13:03, MR K P SCHUPKE wrote: Re GMP, Why not provide more GMP functions as primitives on the Integer type, and avoid the need to call out to GMP via the FFI? Show us the code! :-p Or implement Integers via the FFI, and make it much easier to provide more GMP functions as primitives (that is: simple foreign imports) Which is what I was trying, until I bumped into those weird memory problems I had almost forgotten existed ;) -- Nobody can be exactly like me. Even I have trouble doing it. ___ Glasgow-haskell-users mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: GHCI/FFI/GMP/Me madness
On Sun, Aug 08, 2004 at 07:34:04AM -0700, Sigbjorn Finne wrote: Hi, please be aware that the RTS uses GMP as well, and upon initialisation it sets GMP's 'memory functions' to allocate memory from the RTS' heap. So, in the code below, the global variable 'p' will end up having components pointing into the heap. Which is fine, until a GC occurs and the pointed-to GMP allocated value is eventually stomped on by the storage manager for some other purpose. I'm _guessing_ that's the reason for the behaviour you're seeing. Hm, I _was_ aware of mp_set_memory_functions being used by the RTS. I've seen it often enough in ltrace's ;) It does indeed sound rather plausible (and making big allocations and such does indeed cause it to happen earlier). At which point my next question is: what now? I don't feel really confident about my GHC-hacking skills (huh? skills? where? ;) so does that mean I'm out of luck? *looks* Am I correct that I'd have to copy any GMP-allocated memory to my own memory before returning from C and vice-versa? I hope not :( Happy hacking, Remi 3212th unfinished project Turk -- Nobody can be exactly like me. Even I have trouble doing it. ___ Glasgow-haskell-users mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: GHCI/FFI/GMP/Me madness
On Sun, Aug 08, 2004 at 07:34:04AM -0700, Sigbjorn Finne wrote: Hi, please be aware that the RTS uses GMP as well, and upon initialisation it sets GMP's 'memory functions' to allocate memory from the RTS' heap. What about linking Haskell programs with C libraries that use GMP internally? Can it cause similar problems? Best regards, Tom -- .signature: Too many levels of symbolic links ___ Glasgow-haskell-users mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: GHCI/FFI/GMP/Me madness
FWIW, I couldn't reproduce this problem on my system (i.e. str_test always printed 1). GHC 6.2.1, libgmp 4.1.3, debian unstable Abe On Mon, 9 Aug 2004 17:57:14 +0200, Remi Turk [EMAIL PROTECTED] wrote: On Sun, Aug 08, 2004 at 07:34:04AM -0700, Sigbjorn Finne wrote: Hi, please be aware that the RTS uses GMP as well, and upon initialisation it sets GMP's 'memory functions' to allocate memory from the RTS' heap. So, in the code below, the global variable 'p' will end up having components pointing into the heap. Which is fine, until a GC occurs and the pointed-to GMP allocated value is eventually stomped on by the storage manager for some other purpose. I'm _guessing_ that's the reason for the behaviour you're seeing. Hm, I _was_ aware of mp_set_memory_functions being used by the RTS. I've seen it often enough in ltrace's ;) It does indeed sound rather plausible (and making big allocations and such does indeed cause it to happen earlier). At which point my next question is: what now? I don't feel really confident about my GHC-hacking skills (huh? skills? where? ;) so does that mean I'm out of luck? *looks* Am I correct that I'd have to copy any GMP-allocated memory to my own memory before returning from C and vice-versa? I hope not :( Happy hacking, Remi 3212th unfinished project Turk -- Nobody can be exactly like me. Even I have trouble doing it. ___ Glasgow-haskell-users mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/glasgow-haskell-users ___ Glasgow-haskell-users mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: GHCI/FFI/GMP/Me madness
On Mon, Aug 09, 2004 at 01:09:40PM -0400, Abraham Egnor wrote: FWIW, I couldn't reproduce this problem on my system (i.e. str_test always printed 1). GHC 6.2.1, libgmp 4.1.3, debian unstable Abe Same versions here, on an old heavily-patched/FUBAR rock linux 1.4 system. Does the following make any difference? (trying to cause GCing) Haskell/Mpz/weird% make ghci util.o -#include util.h PrimMpz.hs ___ ___ _ / _ \ /\ /\/ __(_) / /_\// /_/ / / | | GHC Interactive, version 6.2.1, for Haskell 98. / /_\\/ __ / /___| | http://www.haskell.org/ghc/ \/\/ /_/\/|_| Type :? for help. Loading package base ... linking ... done. Loading object (static) util.o ... done final link ... done Compiling Main ( PrimMpz.hs, interpreted ) Ok, modules loaded: Main. *Main mpz_new *Main sum (replicate (200*1000) 0) 0 *Main str_test 1076535944 *Main Groeten, Remi -- Nobody can be exactly like me. Even I have trouble doing it. ___ Glasgow-haskell-users mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: GHCI/FFI/GMP/Me madness
Hi,
please be aware that the RTS uses GMP as well, and upon
initialisation it sets GMP's 'memory functions' to allocate memory
from the RTS' heap. So, in the code below, the global variable
'p' will end up having components pointing into the heap.
Which is fine, until a GC occurs and the pointed-to
GMP allocated value is eventually stomped on by the storage
manager for some other purpose.
I'm _guessing_ that's the reason for the behaviour you're seeing.
--sigbjorn
- Original Message -
From: Remi Turk [EMAIL PROTECTED]
To: [EMAIL PROTECTED]
Sent: Sunday, August 08, 2004 05:52
Subject: GHCI/FFI/GMP/Me madness
[second attempt, this time from my bulk mailinglist address
instead of the normal one.]
Hi all,
I recently tried to create a ffi-binding to gmp in ghc, and
failed miserably. After a few days of debugging, simplifying the
code and tearing my hear out, I'm slightly completely stumped,
and crying for help ;)
In short: calling gmp-functions from GHCI *with a prompt between*
them seems to do Really Bad Things. (read: memory corruption)
The long story:
---
mpz_t p;
str_test()
{
gmp_printf(%Zd\n, p);
}
void mpz_new()
{
mpz_init_set_si(p, 1);
}
foreign import ccall mpz_new:: IO ()
foreign import ccall str_test :: IO ()
Prelude Main mpz_new
Prelude Main str_test
1
Prelude Main str_test
1
Prelude Main str_test
1
Prelude Main str_test
1
Prelude Main str_test
1
Prelude Main str_test
1
Prelude Main str_test
1
Prelude Main str_test
1
Prelude Main str_test
1
Prelude Main str_test
1
Prelude Main str_test
1
Prelude Main str_test
1
Prelude Main str_test
1
Prelude Main str_test
142833060
Prelude Main str_test
142833060
Using other flags, importing extra modules, using CVS 6.3 (a few
weeks old) or not compiling it before loading it in GHCI slightly
changes the symptoms (other wrong numbers or make it happen
later/earlier) but copypasting the code from main some 10 to 20
times seems to be a sure way to reproduce it.
Simply running main doesn't seem to expose the problem.
Now of course, GHCI uses Integer-ops during it's REPL, which I
suspect is exactly what causes/exposes the problem.
Am I doing (Un)Officially Forbidden Things? Is it time for a
bug-report? Do I finally have to learn drinking coffee? ;)
I'd be delighted to know.
The full code is attached.
TIA,
Remi
--
Nobody can be exactly like me. Even I have trouble doing it.
___
Glasgow-haskell-users mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
___
Glasgow-haskell-users mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
