Re: How to make reading an array from disk more efficient

2003-12-25 Thread S. Alexander Jacobson
On Wed, 24 Dec 2003, Hal Daume III wrote:
 one other thing you might find useful is to read it imperatively and then
 use unsafeFreezeArray (i think that's the name) to get a pure array out of
 it.  since all you'll be doing is reading, this should work nicely for
 you.

I must say, this sort of reply is EXTREMELY DISTURBING.

It is ridiculous that one needs to get so intimate
with implementation issues just to read a file!

Is this inherent in Haskell or is it just that no
one has bothered to create a the Haskell
equivalent of BufferedReader?
http://java.sun.com/j2se/1.4.2/docs/api/java/io/BufferedReader.html

And if so, is there something about Haskell that
makes a Haskell implementation of such a lib
difficult?

-Alex-
_
S. Alexander Jacobson  mailto:[EMAIL PROTECTED]
tel:917-770-6565   http://alexjacobson.com


 On Wed, 24 Dec 2003, andrew cooke wrote:

 
  Thanks.  I should have added that I will only use the array for reading
  once it's created.  I don't mind whether creating is lazy or eager (it's
  currently eager because I was fighting a space leak, but I think that was
  down to some other error).
 
  I don't fully understand how either of the suggestions you make will speed
  up creation, though (I guess allocating less memory is faster).  I'm
  looking for a factor of tens of thousands improvement.  But I will try
  what you suggest.
 
  Cheers,
  Andrew
 
  PS Thanks for the very speedy reply and thanks to whoever maintains the
  list archive for updating the archive for each message (I'm pretty sure it
  used to be made daily, which was a nuisance if you wanted to forward URLs
  to interesting discussions).
 
  Hal Daume III said:
   (1) use unboxed arrays, otherwise you're wasting too much space with
   pointers.  that is, unless you need laziness on the elements, which i
   don't think you do based on your list
  
   (2) (maybe) use imperative arrays; this will help you ensure that
   everything is being evaluated quickly.
 
 

 --
  Hal Daume III   | [EMAIL PROTECTED]
  Arrest this man, he talks in maths.   | www.isi.edu/~hdaume

 ___
 Haskell mailing list
 [EMAIL PROTECTED]
 http://www.haskell.org/mailman/listinfo/haskell




___
Haskell mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/haskell


Re: How to make reading an array from disk more efficient

2003-12-25 Thread Tomasz Zielonka
On Thu, Dec 25, 2003 at 02:53:00PM -0500, S. Alexander Jacobson wrote:
 I must say, this sort of reply is EXTREMELY DISTURBING.

Why are you shouting?

I find your reply EXTREMELY UNHELPFUL.

 It is ridiculous that one needs to get so intimate
 with implementation issues just to read a file!

Its quite easy to read a file, but it can be difficult to handle big
files efficiently. Been there, done that, been satisfied with the
results.

 Is this inherent in Haskell or is it just that no
 one has bothered to create a the Haskell
 equivalent of BufferedReader?
 http://java.sun.com/j2se/1.4.2/docs/api/java/io/BufferedReader.html

Handles are buffered by default. It's not the case here. There are other
factors involved (eg. laziness, ineffective file format).

 And if so, is there something about Haskell that
 makes a Haskell implementation of such a lib
 difficult?

Not really.

Tom.

-- 
.signature: Too many levels of symbolic links
___
Haskell mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/haskell


How to make reading an array from disk more efficient

2003-12-24 Thread andrew cooke

Hi,

I have some code (http://www.acooke.org/andrew/ReadTest.hs) that reads
data from a file (an image in ppm format; example data (256*256 pixels) at
http://www.acooke.org/andrew/test.ppm) and stores it in an array of Word8
values.  The aim is to read a file that contains 5000 * 5000 * 3 Word8
values.  I think this should take under 100Mb of memory, if the Array
implementation is efficient.  However, when I run the code on a file of
that size it looks like it will need several days to complete.  This seems
rather slow - the GIMP can read the same file maybe 30 seconds).

How do I make the Haskell code faster while keeping it flexible?  My
requirements are:

- machine limited to 1Gb memory
- display status bar
- the possibility to filter the pixel stream so that the image is
subsampled (see everyN in the code)
- the possibility to filter the pixel strean so that a subsection of the
image is selected.

All that is possible in the code I have now, but it's slow.

Thanks,
Andrew

-- 
personal web site: http://www.acooke.org/andrew
personal mail list: http://www.acooke.org/andrew/compute.html
___
Haskell mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/haskell


Re: How to make reading an array from disk more efficient

2003-12-24 Thread Hal Daume III
(1) use unboxed arrays, otherwise you're wasting too much space with 
pointers.  that is, unless you need laziness on the elements, which i 
don't think you do based on your list

(2) (maybe) use imperative arrays; this will help you ensure that 
everything is being evaluated quickly.

On Wed, 24 Dec 2003, andrew cooke wrote:

 
 Hi,
 
 I have some code (http://www.acooke.org/andrew/ReadTest.hs) that reads
 data from a file (an image in ppm format; example data (256*256 pixels) at
 http://www.acooke.org/andrew/test.ppm) and stores it in an array of Word8
 values.  The aim is to read a file that contains 5000 * 5000 * 3 Word8
 values.  I think this should take under 100Mb of memory, if the Array
 implementation is efficient.  However, when I run the code on a file of
 that size it looks like it will need several days to complete.  This seems
 rather slow - the GIMP can read the same file maybe 30 seconds).
 
 How do I make the Haskell code faster while keeping it flexible?  My
 requirements are:
 
 - machine limited to 1Gb memory
 - display status bar
 - the possibility to filter the pixel stream so that the image is
 subsampled (see everyN in the code)
 - the possibility to filter the pixel strean so that a subsection of the
 image is selected.
 
 All that is possible in the code I have now, but it's slow.
 
 Thanks,
 Andrew
 
 

-- 
 Hal Daume III   | [EMAIL PROTECTED]
 Arrest this man, he talks in maths.   | www.isi.edu/~hdaume

___
Haskell mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/haskell


Re: How to make reading an array from disk more efficient

2003-12-24 Thread andrew cooke

Thanks.  I should have added that I will only use the array for reading
once it's created.  I don't mind whether creating is lazy or eager (it's
currently eager because I was fighting a space leak, but I think that was
down to some other error).

I don't fully understand how either of the suggestions you make will speed
up creation, though (I guess allocating less memory is faster).  I'm
looking for a factor of tens of thousands improvement.  But I will try
what you suggest.

Cheers,
Andrew

PS Thanks for the very speedy reply and thanks to whoever maintains the
list archive for updating the archive for each message (I'm pretty sure it
used to be made daily, which was a nuisance if you wanted to forward URLs
to interesting discussions).

Hal Daume III said:
 (1) use unboxed arrays, otherwise you're wasting too much space with
 pointers.  that is, unless you need laziness on the elements, which i
 don't think you do based on your list

 (2) (maybe) use imperative arrays; this will help you ensure that
 everything is being evaluated quickly.

-- 
personal web site: http://www.acooke.org/andrew
personal mail list: http://www.acooke.org/andrew/compute.html
___
Haskell mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/haskell


Re: How to make reading an array from disk more efficient

2003-12-24 Thread Hal Daume III
one other thing you might find useful is to read it imperatively and then 
use unsafeFreezeArray (i think that's the name) to get a pure array out of 
it.  since all you'll be doing is reading, this should work nicely for 
you.

On Wed, 24 Dec 2003, andrew cooke wrote:

 
 Thanks.  I should have added that I will only use the array for reading
 once it's created.  I don't mind whether creating is lazy or eager (it's
 currently eager because I was fighting a space leak, but I think that was
 down to some other error).
 
 I don't fully understand how either of the suggestions you make will speed
 up creation, though (I guess allocating less memory is faster).  I'm
 looking for a factor of tens of thousands improvement.  But I will try
 what you suggest.
 
 Cheers,
 Andrew
 
 PS Thanks for the very speedy reply and thanks to whoever maintains the
 list archive for updating the archive for each message (I'm pretty sure it
 used to be made daily, which was a nuisance if you wanted to forward URLs
 to interesting discussions).
 
 Hal Daume III said:
  (1) use unboxed arrays, otherwise you're wasting too much space with
  pointers.  that is, unless you need laziness on the elements, which i
  don't think you do based on your list
 
  (2) (maybe) use imperative arrays; this will help you ensure that
  everything is being evaluated quickly.
 
 

-- 
 Hal Daume III   | [EMAIL PROTECTED]
 Arrest this man, he talks in maths.   | www.isi.edu/~hdaume

___
Haskell mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/haskell