Just a quick status report, and to note a couple of lessons learned:
Things work adequately, as far as I can tell. I can now process heaps
of data, without blowing up anything. Appears to be faster than
spam-stat.el, at least, although I haven't measured.
I'm back to using readFile for file
One question though, about hFlush. I print out the status by
repeatedly putStr'ing blah blah \r. With NoBuffering set, it works,
but when following the putStr with 'hFlush stdout', it doesn't (only
outputs very sporadically. I guess I'm misunderstanding the function
of hFlush, anybody care
Just a quick status report, and to note a couple of lessons learned:
Things work adequately, as far as I can tell. I can now process heaps
of data, without blowing up anything. Appears to be faster than
spam-stat.el, at least, although I haven't measured.
I'm back to using readFile for file
You should do the counting strictly:
Just n - case n+1 of n1 - addToFM f w n1
Careful - that case expression doesn't actually force n to be evaluated.
It's exactly equivalent to
let n1 = n+1 in addToFM f w n1
You need to use seq to force evaluation.
Cheers,
Dean Herington [EMAIL PROTECTED] writes:
Yes, getting the right amount of strictness--and in the right places--can be
tricky.
Tell me about it!
You should do the counting strictly:
Just n - case n+1 of n1 - addToFM f w n1
Thanks for the tip. Just performing this change didn't
You should do the counting strictly:
Just n - case n+1 of n1 - addToFM f w n1
Careful - that case expression doesn't actually force n to be evaluated.
It's exactly equivalent to
let n1 = n+1 in addToFM f w n1
You need to use seq to force evaluation.
Cheers,
File reading is not a pure operation: running out
of file descriptors is a good counter-example.
How does this differ from running out of memory whilst trying
to evaluate something?
You're absolutely right, of course.
I think the point is that by using implicit memory allocation we've
Simon Marlow [EMAIL PROTECTED] writes:
To fix these
problems you have to think carefully about strictness and demand in your
program. For memory we have heap profilers to help out, but we don't
have I/O descriptor profilers for lazy I/O!
What is the limit on open files, and why? I think it'd be nice to
just schedule a huge amount of IO operations, and have them all be
performed when required (i.e. when the FM is first accessed).
Apparently, my addDir took the trouble to open the files, but not
generate the FMs -- any idea
Yes, getting the right amount of strictness--and in the right places--can be
tricky. For your program, it seems you should process each file completely
(reflecting its contents strictly in your histogram so the contents can be dropped
and the file descriptor closed) before moving on to the next
What is the limit on open files, and why? I think it'd be nice to
just schedule a huge amount of IO operations, and have them all be
performed when required (i.e. when the FM is first accessed).
Apparently, my addDir took the trouble to open the files, but not
generate the FMs -- any idea
Yes, getting the right amount of strictness--and in the right places--can be
tricky. For your program, it seems you should process each file completely
(reflecting its contents strictly in your histogram so the contents can be dropped
and the file descriptor closed) before moving on to the next
Dean Herington [EMAIL PROTECTED] writes:
Ketil Z. Malde wrote:
-- | add data from a file to the histogram
addFile :: FiniteMap String Int - String - IO (FiniteMap String Int)
addFile fm name = do
x - readFile name
return (addHist fm x)
I changed this to read x
[EMAIL PROTECTED] (Ketil Z. Malde) writes:
[EMAIL PROTECTED] (Ketil Z. Malde) writes:
Prelude :r
phase `Literate pre-processor' failed (exitcode = 1)
I've no idea what trigged this, perhaps running out of file handles?
I forgot: GHC version 5.04.1. It seems that this is definitely
Simon Marlow [EMAIL PROTECTED] writes:
-- | add data from a file to the histogram
addFile :: FiniteMap String Int - String - IO (FiniteMap
String Int)
addFile fm name = do
x - readFile name
return (addHist fm x)
-- | add data from all files in a
Ketil Z. Malde wrote:
Simon Marlow [EMAIL PROTECTED] writes:
-- | add data from a file to the histogram
addFile :: FiniteMap String Int - String - IO (FiniteMap
String Int)
addFile fm name = do
x - readFile name
return (addHist fm x)
-- |
[EMAIL PROTECTED] (Ketil Z. Malde) writes:
[EMAIL PROTECTED] (Ketil Z. Malde) writes:
Prelude :r
phase `Literate pre-processor' failed (exitcode = 1)
I've no idea what trigged this, perhaps running out of file handles?
I forgot: GHC version 5.04.1. It seems that this is definitely
The following code runs out of file handles, so I am seeking
enlightenment as to why.
-- | add data from a file to the histogram
addFile :: FiniteMap String Int - String - IO (FiniteMap
String Int)
addFile fm name = do
x - readFile name
return (addHist
Simon Marlow [EMAIL PROTECTED] writes:
-- | add data from a file to the histogram
addFile :: FiniteMap String Int - String - IO (FiniteMap
String Int)
addFile fm name = do
x - readFile name
return (addHist fm x)
-- | add data from all files in a
On Thu, 13 Feb 2003 10:45:10 -, Simon Marlow [EMAIL PROTECTED]
wrote:
File reading is not a pure operation: running out
of file descriptors is a good counter-example.
How does this differ from running out of memory whilst trying to evaluate
something?
Cheers,
Ganesh
Hi Ketil,
I know your problem all too well. If you are sure that you need all of
the file's content, then you should read it all from the start. I've
found this little function handy and I'm using it all over the place:
readFileStrictly :: FilePath - IO String
readFileStrictly filePath =
do h
These things are always tricky to understand, which is why I recommend
not using lazy I/O. File reading is not a pure operation: running out
of file descriptors is a good counter-example.
Without saying wether I agree with lazy I/O or not, I suggest that
this particular problem may also be
22 matches
Mail list logo
