Send Beginners mailing list submissions to
[email protected]
To subscribe or unsubscribe via the World Wide Web, visit
http://www.haskell.org/mailman/listinfo/beginners
or, via email, send a message with subject or body 'help' to
[email protected]
You can reach the person managing the list at
[email protected]
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Beginners digest..."
Today's Topics:
1. Diagnosing : Large memory usage + low CPU (Hugo Ferreira)
2. Re: Diagnosing : Large memory usage + low CPU (Stephen Tetley)
----------------------------------------------------------------------
Message: 1
Date: Tue, 29 Nov 2011 11:20:19 +0000
From: Hugo Ferreira <[email protected]>
Subject: [Haskell-beginners] Diagnosing : Large memory usage + low CPU
To: [email protected]
Message-ID: <[email protected]>
Content-Type: text/plain; charset=ISO-8859-15; format=flowed
Hello,
I am testing a simple algorithm and find that during part of the
execution this is fast and uses acceptable memory. However, when
it gets to a certain point the memory climbs to 75-80 % of the
OS's memory and CPU plummets to a mere 5% at most.
In the part of the application that executes ok I have calls as
follows:
let r9' = evalTagger'' (tagFun suffixCapsFreq) $ test
let a9' = tagginAccuracy r9'
putStrLn ("Suffix(3) + Caps + Freq tag result done = " ++ (show a9'))
The slow one has:
let r12' = evalTagger'' ruleSuffixCapsFreq $ test
let a12' = tagginAccuracy r12'
putStrLn ("Rules + Suffix(3) + Caps + Freq tag result done = " ++
(show a12'))
The difference lies in the following functions:
ruleApplication :: TransformationRule -> POSTags -> Maybe Tag
ruleApplication (NextTagRule (Replacement old new) next) z = do
(_, _, proposed) <- Z.safeCursor z
(_, _, nextProposed) <- rightCursor z
if proposed == old && nextProposed == next then Just new else Nothing
....
updateState :: (TransformationRule,Int) -> POSTags -> POSTags
updateState r = Z.fromList . reverse . Z.foldlz' (update r) []
where update (r,_) !xs z =
case ruleApplication r z of
Just tag -> (token, correct, tag):xs
Nothing -> e:xs
where e@(token, correct, _proposed) = Z.cursor z
rulesT :: [(TransformationRule, Int)] -> POSTags -> POSTags
rulesT rs state = L.foldl' tag state rs
where
tag !s rule = updateState rule s
(.>) :: (POSTags -> Tag) -> (POSTags -> Tag) -> POSTags -> Tag
f .> g = \ x ->
case (g x) of
"" -> f x
t -> t
(|>) :: (POSTags -> POSTags) -> (POSTags -> Tag) -> POSTags -> POSTags
f |> g = \ x -> f $ tagFun g x
let ruleT = rulesT top10Rules
let suffixCapsFreq = suffixT .> capitalizeT .> freqT
let ruleSuffixCapsFreq = ruleT |> suffixCapsFreq
where top10Rules is a list. Note that suffixCapsFreq and tagFun
are common to both calls.
I have placed bangs in many places and used eager evaluation in the
folds. However nothing seems to help. Can anyone tell me how I should
diagnose this problem. Any suggestions are welcome.
TIA,
Hugo F.
------------------------------
Message: 2
Date: Tue, 29 Nov 2011 22:57:33 +0000
From: Stephen Tetley <[email protected]>
Subject: Re: [Haskell-beginners] Diagnosing : Large memory usage + low
CPU
Cc: [email protected]
Message-ID:
<cab2tprab6xpj5wfw-qrhpz06ots20zxscuu+ejepnetj6y4...@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1
Hi Hugo
What is a POSTags and how big do you expect it to be?
Generally I'd recommend you first try to calculate the size of your
data rather than try to strictify things, see Johan Tibell's very
useful posts:
http://blog.johantibell.com/2011/06/memory-footprints-of-some-common-data.html
http://blog.johantibell.com/2011/06/computing-size-of-hashmap.html
Once you know the size of your data - you can decide if it is too big
to comfortably work with in memory. If it is too big you need to make
sure you're are streaming[*] it rather than forcing it into memory.
If POSTags is large, I'd be very concerned about the top line of
updateState - reversing lists (or sorting them) simply doesn't play
well with streaming.
[*] Even in a lazy language like Haskell, streaming data isn't
necessarily automatic.
------------------------------
_______________________________________________
Beginners mailing list
[email protected]
http://www.haskell.org/mailman/listinfo/beginners
End of Beginners Digest, Vol 41, Issue 41
*****************************************
