On Friday, August 14, 2015 at 6:49:07 PM UTC-5, Erik Rantapaa wrote:
>
> I'm trying to figure out how to process a large number of files
> (ByteStrings) in groups of, say 20, with the results of each group going to
> a different output file.
>
>
Ok, I've come up with something I can live with which uses
Pipes.Parse.splitAt:
loop n = do
g <- zoom (PP.splitAt 3) PP.drawAll
case g of
[] -> return ()
_ -> do let out = "output-" ++ show n
liftIO $ withFile out WriteMode $ \h -> do
putStrLn $ "got a group, writing to " ++ out
forM_ g $ \(fname,content) -> putStrLn $ " -
processing " ++ fname
loop (n+1)
runExample = do
let input = P.each [ (c, "content for file " ++ {c]) | c <-
"abcdefghij" ]
evalStateT (loop 1) input
What I don't like about it is that it (apparently) has to construct the
entire list `g` in memory before processing its elements.
I also came up with this idea which (I think) addresses that concern:
loop = do
end <- PP.isEndOfInput
if end
then return ()
else do s <- S.get
let x = s ^. (PP.splitAt 3)
liftIO $ putStrLn "starting a new group"
y <- lift $ runEffect (x >-> P.print)
S.put y
loop
runLoop = do
let input = P.each [ ("a", "asds"), ("b", "qwe"), ("c", "asdas"), ("d",
"qwkjqkwe") ]
evalStateT loop input
Am I getting warmer? colder?
--
You received this message because you are subscribed to the Google Groups
"Haskell Pipes" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to haskell-pipes+unsubscr...@googlegroups.com.
To post to this group, send email to haskell-pipes@googlegroups.com.
