Hello,
This question is regarding streaming library. What is a good way to pass lazy bytestring from the result of a streaming function while keeping the memory bounded? I will like to pass lazy bytestring to Network websocket, but if I use `toLazy_ . fromChunks` to convert bytestring stream to lazy bytestring, it seems to take lot of memory, about 70MB residency for 100MB HTTP download. First, some imports: import qualified Data.ByteString.Streaming as SBS import qualified Data.ByteString.Lazy as LBS import Data.Conduit (($$+-)) import qualified Data.Conduit.List as CL (mapM_) import Streaming.Prelude as S import Streaming as S import qualified Data.ByteString.Streaming.HTTP as SP -- first type of HTTP download from AWS - we pass the HTTP body to SP.responseBody function ghci > :t (($$+- CL.mapM_ S.yield) . hoist lift ) (SP.responseBody undefined ) (($$+- CL.mapM_ S.yield) . hoist lift ) (SP.responseBody undefined) :: Monad m => Stream (Of a) m () The code below works fine and takes ~4MB memory for 100MB download. It just simulates an in-place sink of lazy bytestring - a smoke check to make sure memory can be bounded - rsp here is HTTP body S.foldM_ (\_ a -> liftIO $ LBS.appendFile "out" (LBS.fromStrict a)) (liftIO $ LBS.writeFile "out" LBS.empty) (return . id) $ (($$+- CL.mapM_ Streaming. Prelude.yield) . hoist lift ) (SP.responseBody rsp) Now, let us just take the lazy bytestring which we can then pass to Network websocket for outgoing message - writeFile here simulates write to network socket - this one takes ~70MB memory for 100MB download: liftIO . (LBS.writeFile "out") =<< (SBS.toLazy_ . SBS.fromChunks $ (($$+- CL .mapM_ S.yield) . hoist lift ) (SP.responseBody rsp)) So, the memory usage goes up substantially when returning lazy bytestring. If I am not mistaken, `LBS.writeFile` doesn't keep the whole bytestring in memory unlike `LBS.readFile`. What will be a good way to return lazy bytestring for write to network socket while keeping memory usage low? -- You received this message because you are subscribed to the Google Groups "Haskell Pipes" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected].
