Re: How to make reading an array from disk more efficient

[Tomasz Zielonka]

On Thu, Dec 25, 2003 at 02:53:00PM -0500, S. Alexander Jacobson wrote:
> I must say, this sort of reply is EXTREMELY DISTURBING.

Why are you shouting?

I find your reply EXTREMELY UNHELPFUL.

> It is ridiculous that one needs to get so intimate
> with implementation issues just to read a file!

Its quite easy to read a file, but it can be difficult to handle big
files efficiently. Been there, done that, been satisfied with the
results.

> Is this inherent in Haskell or is it just that no
> one has bothered to create a the Haskell
> equivalent of BufferedReader?
> 

Handles are buffered by default. It's not the case here. There are other
factors involved (eg. laziness, ineffective file format).

> And if so, is there something about Haskell that
> makes a Haskell implementation of such a lib
> difficult?

Not really.

Tom.

-- 
.signature: Too many levels of symbolic links
___
Haskell mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/haskell

Re: How to make reading an array from disk more efficient

[S. Alexander Jacobson]

On Wed, 24 Dec 2003, Hal Daume III wrote:
> one other thing you might find useful is to read it imperatively and then
> use unsafeFreezeArray (i think that's the name) to get a pure array out of
> it.  since all you'll be doing is reading, this should work nicely for
> you.

I must say, this sort of reply is EXTREMELY DISTURBING.

It is ridiculous that one needs to get so intimate
with implementation issues just to read a file!

Is this inherent in Haskell or is it just that no
one has bothered to create a the Haskell
equivalent of BufferedReader?


And if so, is there something about Haskell that
makes a Haskell implementation of such a lib
difficult?

-Alex-
_
S. Alexander Jacobson  mailto:[EMAIL PROTECTED]
tel:917-770-6565   http://alexjacobson.com


> On Wed, 24 Dec 2003, andrew cooke wrote:
>
> >
> > Thanks.  I should have added that I will only use the array for reading
> > once it's created.  I don't mind whether creating is lazy or eager (it's
> > currently eager because I was fighting a space leak, but I think that was
> > down to some other error).
> >
> > I don't fully understand how either of the suggestions you make will speed
> > up creation, though (I guess allocating less memory is faster).  I'm
> > looking for a factor of tens of thousands improvement.  But I will try
> > what you suggest.
> >
> > Cheers,
> > Andrew
> >
> > PS Thanks for the very speedy reply and thanks to whoever maintains the
> > list archive for updating the archive for each message (I'm pretty sure it
> > used to be made daily, which was a nuisance if you wanted to forward URLs
> > to interesting discussions).
> >
> > Hal Daume III said:
> > > (1) use unboxed arrays, otherwise you're wasting too much space with
> > > pointers.  that is, unless you need laziness on the elements, which i
> > > don't think you do based on your list
> > >
> > > (2) (maybe) use imperative arrays; this will help you ensure that
> > > everything is being evaluated quickly.
> >
> >
>
> --
>  Hal Daume III   | [EMAIL PROTECTED]
>  "Arrest this man, he talks in maths."   | www.isi.edu/~hdaume
>
> ___
> Haskell mailing list
> [EMAIL PROTECTED]
> http://www.haskell.org/mailman/listinfo/haskell
>



___
Haskell mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/haskell

Re: How to make reading an array from disk more efficient

[Hal Daume III]

one other thing you might find useful is to read it imperatively and then 
use unsafeFreezeArray (i think that's the name) to get a pure array out of 
it.  since all you'll be doing is reading, this should work nicely for 
you.

On Wed, 24 Dec 2003, andrew cooke wrote:

> 
> Thanks.  I should have added that I will only use the array for reading
> once it's created.  I don't mind whether creating is lazy or eager (it's
> currently eager because I was fighting a space leak, but I think that was
> down to some other error).
> 
> I don't fully understand how either of the suggestions you make will speed
> up creation, though (I guess allocating less memory is faster).  I'm
> looking for a factor of tens of thousands improvement.  But I will try
> what you suggest.
> 
> Cheers,
> Andrew
> 
> PS Thanks for the very speedy reply and thanks to whoever maintains the
> list archive for updating the archive for each message (I'm pretty sure it
> used to be made daily, which was a nuisance if you wanted to forward URLs
> to interesting discussions).
> 
> Hal Daume III said:
> > (1) use unboxed arrays, otherwise you're wasting too much space with
> > pointers.  that is, unless you need laziness on the elements, which i
> > don't think you do based on your list
> >
> > (2) (maybe) use imperative arrays; this will help you ensure that
> > everything is being evaluated quickly.
> 
> 

-- 
 Hal Daume III   | [EMAIL PROTECTED]
 "Arrest this man, he talks in maths."   | www.isi.edu/~hdaume

___
Haskell mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/haskell

Re: How to make reading an array from disk more efficient

[andrew cooke]

Thanks.  I should have added that I will only use the array for reading
once it's created.  I don't mind whether creating is lazy or eager (it's
currently eager because I was fighting a space leak, but I think that was
down to some other error).

I don't fully understand how either of the suggestions you make will speed
up creation, though (I guess allocating less memory is faster).  I'm
looking for a factor of tens of thousands improvement.  But I will try
what you suggest.

Cheers,
Andrew

PS Thanks for the very speedy reply and thanks to whoever maintains the
list archive for updating the archive for each message (I'm pretty sure it
used to be made daily, which was a nuisance if you wanted to forward URLs
to interesting discussions).

Hal Daume III said:
> (1) use unboxed arrays, otherwise you're wasting too much space with
> pointers.  that is, unless you need laziness on the elements, which i
> don't think you do based on your list
>
> (2) (maybe) use imperative arrays; this will help you ensure that
> everything is being evaluated quickly.

-- 
personal web site: http://www.acooke.org/andrew
personal mail list: http://www.acooke.org/andrew/compute.html
___
Haskell mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/haskell

Re: How to make reading an array from disk more efficient

[Hal Daume III]

(1) use unboxed arrays, otherwise you're wasting too much space with 
pointers.  that is, unless you need laziness on the elements, which i 
don't think you do based on your list

(2) (maybe) use imperative arrays; this will help you ensure that 
everything is being evaluated quickly.

On Wed, 24 Dec 2003, andrew cooke wrote:

> 
> Hi,
> 
> I have some code (http://www.acooke.org/andrew/ReadTest.hs) that reads
> data from a file (an image in ppm format; example data (256*256 pixels) at
> http://www.acooke.org/andrew/test.ppm) and stores it in an array of Word8
> values.  The aim is to read a file that contains 5000 * 5000 * 3 Word8
> values.  I think this should take under 100Mb of memory, if the Array
> implementation is efficient.  However, when I run the code on a file of
> that size it looks like it will need several days to complete.  This seems
> rather slow - the GIMP can read the same file maybe 30 seconds).
> 
> How do I make the Haskell code faster while keeping it flexible?  My
> requirements are:
> 
> - machine limited to 1Gb memory
> - display "status bar"
> - the possibility to filter the pixel stream so that the image is
> subsampled (see "everyN" in the code)
> - the possibility to filter the pixel strean so that a subsection of the
> image is selected.
> 
> All that is possible in the code I have now, but it's slow.
> 
> Thanks,
> Andrew
> 
> 

-- 
 Hal Daume III   | [EMAIL PROTECTED]
 "Arrest this man, he talks in maths."   | www.isi.edu/~hdaume

___
Haskell mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/haskell

How to make reading an array from disk more efficient

[andrew cooke]

Hi,

I have some code (http://www.acooke.org/andrew/ReadTest.hs) that reads
data from a file (an image in ppm format; example data (256*256 pixels) at
http://www.acooke.org/andrew/test.ppm) and stores it in an array of Word8
values.  The aim is to read a file that contains 5000 * 5000 * 3 Word8
values.  I think this should take under 100Mb of memory, if the Array
implementation is efficient.  However, when I run the code on a file of
that size it looks like it will need several days to complete.  This seems
rather slow - the GIMP can read the same file maybe 30 seconds).

How do I make the Haskell code faster while keeping it flexible?  My
requirements are:

- machine limited to 1Gb memory
- display "status bar"
- the possibility to filter the pixel stream so that the image is
subsampled (see "everyN" in the code)
- the possibility to filter the pixel strean so that a subsection of the
image is selected.

All that is possible in the code I have now, but it's slow.

Thanks,
Andrew

-- 
personal web site: http://www.acooke.org/andrew
personal mail list: http://www.acooke.org/andrew/compute.html
___
Haskell mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/haskell