Re: [Haskell-cafe] ANN: stm-conduit-0.2.1
Message: 6 Date: Sun, 12 Feb 2012 01:47:40 -0500 From: wren ng thornton w...@freegeek.org Subject: Re: [Haskell-cafe] ANN: stm-conduit-0.2.1 To: Haskell Cafe haskell-cafe@haskell.org Message-ID: 4f37608c.3090...@freegeek.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed On 2/9/12 2:29 PM, Felipe Almeida Lessa wrote: Your package uses TMChans which AFAIK are unbounded. That means that if the writer is faster than the reader, then everything will be kept into memory. This means that using TMChans you may no longer say that your program uses a constant amount of memory. Actually, you lose a lot of your space reasoning since, being concurrent processes, you can't guarantee almost anything wrt. progress of the reader. Of course, you're free to use TBMChans instead, which are bounded :) This is what I did in iteratee-stm. The stm-chans package is very nice. (and the results with iteratee-stm were very good as well) John L. ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] ANN: stm-conduit-0.2.1
I added that after reading his feedback, and seeing the flaw in using TMChans. On Sun, Feb 12, 2012 at 1:47 AM, wren ng thornton w...@freegeek.org wrote: On 2/9/12 2:29 PM, Felipe Almeida Lessa wrote: Your package uses TMChans which AFAIK are unbounded. That means that if the writer is faster than the reader, then everything will be kept into memory. This means that using TMChans you may no longer say that your program uses a constant amount of memory. Actually, you lose a lot of your space reasoning since, being concurrent processes, you can't guarantee almost anything wrt. progress of the reader. Of course, you're free to use TBMChans instead, which are bounded :) -- Live well, ~wren __**_ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/**mailman/listinfo/haskell-cafehttp://www.haskell.org/mailman/listinfo/haskell-cafe ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] ANN: stm-conduit-0.2.1
On 2/9/12 2:29 PM, Felipe Almeida Lessa wrote: Your package uses TMChans which AFAIK are unbounded. That means that if the writer is faster than the reader, then everything will be kept into memory. This means that using TMChans you may no longer say that your program uses a constant amount of memory. Actually, you lose a lot of your space reasoning since, being concurrent processes, you can't guarantee almost anything wrt. progress of the reader. Of course, you're free to use TBMChans instead, which are bounded :) -- Live well, ~wren ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
[Haskell-cafe] ANN: stm-conduit-0.2.1
Hi all, I've just released stm-conduit [1] on Hackage. This package introduces conduits to the wonderful world of concurrency. My package solves the common problem of constant bottleneck switching loaders have. This is when, for example, we stream XML from the disk and then parse the XML in one conduit pipeline. While it streams a file from the disk, the process is clearly IO bound, and while it parses the XML, the process is CPU bound. By putting each task on its own thread, the disk IO doesn't need to wait for the CPU to parse a document before loading the next file. By using stm-based conduits, we have full resource utilization. The way it does this is by creating a source and sink for TChans, letting us stream data between conduits and channels. There are more examples in the docs. Check it out! Regards, - clark [1] http://hackage.haskell.org/package/stm-conduit ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] ANN: stm-conduit-0.2.1
Hello. Thu, Feb 09, 2012 at 01:32:41PM -0500, Clark Gaebel wrote Hi all, I've just released stm-conduit [1] on Hackage. This package introduces conduits to the wonderful world of concurrency. My package solves the common problem of constant bottleneck switching loaders have. This is when, for example, we stream XML from the disk and then parse the XML in one conduit pipeline. While it streams a file from the disk, the process is clearly IO bound, and while it parses the XML, the process is CPU bound. By putting each task on its own thread, the disk IO doesn't need to wait for the CPU to parse a document before loading the next file. By using stm-based conduits, we have full resource utilization. The way it does this is by creating a source and sink for TChans, letting us stream data between conduits and channels. There are more examples in the docs. Check it out! Regards, - clark [1] http://hackage.haskell.org/package/stm-conduit A day ago I've make analogical library in utils for my project, code was 90% same. Thanks for putting such a library on hackage -- Best regards, Alexander. signature.asc Description: Digital signature ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] ANN: stm-conduit-0.2.1
Happy to help! I'm new to this whole package on hackage thing, so any feedback would be great. - clark On Thu, Feb 9, 2012 at 2:19 PM, Alexander V Vershilov alexander.vershi...@gmail.com wrote: Hello. Thu, Feb 09, 2012 at 01:32:41PM -0500, Clark Gaebel wrote Hi all, I've just released stm-conduit [1] on Hackage. This package introduces conduits to the wonderful world of concurrency. My package solves the common problem of constant bottleneck switching loaders have. This is when, for example, we stream XML from the disk and then parse the XML in one conduit pipeline. While it streams a file from the disk, the process is clearly IO bound, and while it parses the XML, the process is CPU bound. By putting each task on its own thread, the disk IO doesn't need to wait for the CPU to parse a document before loading the next file. By using stm-based conduits, we have full resource utilization. The way it does this is by creating a source and sink for TChans, letting us stream data between conduits and channels. There are more examples in the docs. Check it out! Regards, - clark [1] http://hackage.haskell.org/package/stm-conduit A day ago I've make analogical library in utils for my project, code was 90% same. Thanks for putting such a library on hackage -- Best regards, Alexander. ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] ANN: stm-conduit-0.2.1
Your package uses TMChans which AFAIK are unbounded. That means that if the writer is faster than the reader, then everything will be kept into memory. This means that using TMChans you may no longer say that your program uses a constant amount of memory. Actually, you lose a lot of your space reasoning since, being concurrent processes, you can't guarantee almost anything wrt. progress of the reader. This doesn't mean that your package is broken, this means that it has a caveat that should be stated on the docs. Congrats on your Hackage debut! Cheers! =) -- Felipe. ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] ANN: stm-conduit-0.2.1
Actually, that is a moderately fatal flaw. I just uploaded 0.2.2 which addresses this by highly recommending using bounded channels, as well as adding sources/sinks for them. Thanks for catching that! - clark On Thu, Feb 9, 2012 at 2:29 PM, Felipe Almeida Lessa felipe.le...@gmail.com wrote: Your package uses TMChans which AFAIK are unbounded. That means that if the writer is faster than the reader, then everything will be kept into memory. This means that using TMChans you may no longer say that your program uses a constant amount of memory. Actually, you lose a lot of your space reasoning since, being concurrent processes, you can't guarantee almost anything wrt. progress of the reader. This doesn't mean that your package is broken, this means that it has a caveat that should be stated on the docs. Congrats on your Hackage debut! Cheers! =) -- Felipe. ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
