Claus Reinke ha scritto:
I split the string in lines, then map some functions on each line to
parse the data, and finally calling toU, for converting to an UArr.
Just to make sure (code fragments or, better, reduced examples
would make it easier to see what the discussion is about): are you
forcing the UArr to be constructed before putting it into the Map?
parse handle =
contents <- S.hGetContents handle
let v = map singleton' $ ratings contents
let m = foldl1' (unionWith appendU) v
v `seq` return $! m
where
-- Build a Map with a single movie rating
singleton' :: (Word32, Word8) -> MovieRatings
singleton' (id, rate) =
singleton (fromIntegral $ id) (singletonU $ pairS (id, rate))
That helps to make things clearer, I think. One issue is
the nature of Maps (strict in keys, non-strict in values).
- neither singleton nor unionWith are strict in the Map values, so
nothing here forces the evaluation of rate or construction of UArr
But, as I have written, in one of my tests I also tried rnf to force
evaluation:
rnf v `seq` rnf m `seq` return m
where:
instance NFData a => NFData (Data.IntMap.IntMap a) where
rnf = rnf . Data.IntMap.toList
Isn't this sufficient?
[...]
- seq on a list only forces the first node of the list ((:),[],_|_),
so (v `seq`) isn't likely to help much. Also, you probably
do not want to force the whole list of singletons before
builing the Map, you want the singletons to be constructed and
consumed incrementally.
Right, thanks.
type Rating = Word32 :*: Word8
type MovieRatings = IntMap (UArr Rating) -- UArr from uvector
A standard trick to keep Map values evaluated by construction
is to make the availability of keys dependent on their values, eg
(singleton key) $! value. That won't help with unionWith and
the appendUs, but it should allow the source string references
to be dropped early, as the singletons are constructed.
Tried; but, even using union instead of unionWith, the memory grows fast
as before.
Just to check if the culprit is IntMap, isn't possible to write a test
program that build a big IntMap (UArr Int) ?
Right now I don't have the time.
Manlio
_______________________________________________
Haskell-Cafe mailing list
[email protected]
http://www.haskell.org/mailman/listinfo/haskell-cafe
