Wow, Daniel, that helped a lot with the counter, thank you!
But my "real" code continues eating memory like crazy even after updating
it to modify'. I assume it could be something similar?
Can you spot something bad in the following one?
data RefField a = RefField String (a -> ByteString)
ref :: [RefField a] -> Pipe a a (StateT (IndexTables ByteString) IO) ()
ref fields =
let updateTables x ts = F.foldl' (\s' (RefField n f) -> tableInsert n
(f x) s') ts fields
in forever $ await >>= (\x -> do { S.modify' (updateTables x); yield x
})
And IndexTable is just a wrapper for Map. I tried to use Data.Map.Strict
and it didn't help...
Cheers,
Alexey.
On Saturday, August 15, 2015 at 9:06:41 PM UTC+10, Daniel Díaz wrote:
>
> What happens if you use modify'
> <http://hackage.haskell.org/package/transformers-0.4.3.0/docs/Control-Monad-Trans-State-Strict.html#v:modify-39->
>
> instead of modify? A common gotcha with Writer and State is that even the
> "strict versions" of these monads aren't strict in the accumulators. You
> have to explicitly use modify' in the case of State.
>
> On Saturday, August 15, 2015 at 12:52:09 PM UTC+2, Alexey Raga wrote:
>>
>> Hi All,
>>
>> I have noticed that as soon as I introduce StateT into my pipeline the
>> memory consumption increases a lot, and it keeps increasing until the
>> program stops.
>>
>> Here is a simple counter example, but even this one reproduces the
>> problem:
>>
>> counter :: Pipe a a (StateT Integer IO) ()
>> counter = do
>> x <- await
>> S.modify(+1)
>> yield x
>> counter
>>
>> Piping a very large file (~40M rows) through a simple in/out pipeline
>> only consumes 3mb of memory, but once I introduce this "counter" step the
>> memory consumption keeps increasing up to ~700mb when the pipeline finishes.
>>
>> Am I doing it wrong? And what would be the correct way of implementing
>> such a "stateful" Pipe?
>>
>> Cheers,
>> Alexey.
>>
>
--
You received this message because you are subscribed to the Google Groups
"Haskell Pipes" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to haskell-pipes+unsubscr...@googlegroups.com.
To post to this group, send email to haskell-pipes@googlegroups.com.
