I talked with Merijn about this on IRC. The main issue he encountered
was that the type of `Parser` is not suitable for defining intermediate
transformations of the form:
Producer a m x -> Producer b m (r, Producer a m x)
The reason why is that the `Parser` type is defined as:
type Parser a m r = forall x . StateT (Producer a m x) m r
... and when you `runStateT` that you get:
Producer a m x -> m (r, Producer a m x)
This type doesn't work, because if you want to unify `m` with `Producer
b` then you will inadvertently introduce a `Producer b` layer underneath
your `Producer a`:
-- Bad:
Producer a m (Producer b m) x -> Producer b m (r, Producer a m x)
The solution is to bypass the `Parser` type synonym and use the
following `StateT` type:
type Transformation a b m r = forall x . StateT (Producer a m x)
(Producer b m) r
However, that's still not quite enough. The issue is that you can't use
a `Parser` within the above `StateT` type. An example would be:
decode :: Binary a => Parser ByteString m (Maybe a)
To convert between `Parser`s and the above state type, you have to
`runStateT` the parser any time you want to `yield` a value within a
`Transformation`, like this:
-- example :: Transformation a b m r
example :: StateT (Producer ByteString m x) (Producer b m) r
example = do
(x, p) <- lift $ runStateT decode (p :: Producer ByteString m x)
yield x
...
So that's what he's doing for now.
I was aware of this issue when I first wrote `pipes-parse`, but I'm not
aware of a type that works well for both `Parser`s and `Transformation`s
so I initially provided just `Parser` and required users to manually
pass leftovers for `Transformation`s. Perhaps I should provide a type
synonym for `Transformation`s and then provide a function that converts
a `Parser` to a `Transformation`:
??? :: Parser a m r -> Transformation a b m r
Of course, I don't really like the name `Transformation`, but that's the
basic idea.
What do you all think?
On 7/29/14, 4:43 PM, Merijn Verstraaten wrote:
Hey Gabriel,
Thanks for your response, but I'm not quite sure how to apply this in
my case? (I don't have any experience with conduit, so I'll ignore
those remarks for now). With your suggested type of:
handShake
:: SockAddr
-> Producer ByteString M r
-> Producer ByteString M ((Get Frame, Frame -> Put), Producer
ByteString M r)
I'm not quite sure how this will fit? The problem is that during
handshake I need to do things like "read/parse N bytes of the
ByteString coming from the socket, then based on that write M bytes to
the socket". I'm not quite sure how to do that with this type? Take
for example "zmtpVersion", this needs to read ByteString's from the
input, write output to the socket and return a Version. Does that mean
that, just like handShake, zmtpVersion's type becomes "Producer
ByteString M r -> Producer ByteString M (Version, Producer ByteString
M r)"? How do I get the ByteString's to write to the socket from there?
zmtpVersion
:: Producer ByteString M r
-> Producer ByteString M (Version, Producer ByteString M r)
zmtpVersion prod = do
??
I can't use any of the pipes-binary parsing utilities here, as they
all producer Parser's and you just said that using a Producer as a
base monad for Parser won't work (which is the only way I could run a
Parser and end up with a Producer like this type)?
I really don't know how to incrementally parse a value from the input
producer here, except: 1) manually thread leftovers everywhere like my
initial design or 2) manually step through the input producer using
"next".
Both of these seem really painful and cumbersome, and the annoyance of
the former was the entire reason I was hoping pipes-parse would make
things better. But so far it seems that it can't?
Cheers,
Merijn
On Tuesday, July 29, 2014 1:57:06 AM UTC+2, Gabriel Gonzalez wrote:
The `Parser` type is intended to be used with a base monad that is
not a `Producer` or any other pipe type. The intuition you should
have for `pipes-parse` is that they are almost exactly analogous
to `conduit` abstractions (minus the finalization handling of
`conduit`):
* `Producer a m ()` is analogous to a `Source m a`
* `Parser a m r` is analogous to a `conduit` `Sink a m r`
* `Lens' (Producer a m r) (Producer b m r)` is analogous to a
`Conduit a m b`that can propagate leftovers further upstream
* `Getter' (Producer a m r) (Producer b m r)` (or just: `Producer
a m r -> Producer b m r`) is analogous to a `Conduit a m b` that
cannot propagate leftovers further upstream
`conduit` does not make the distinction between transformations
which can or cannot propagate leftovers further upstream, which
leads to the leftover propagation bug described here:
http://www.haskellforall.com/2014/02/pipes-parse-30-lens-based-parsing.html
<http://www.haskellforall.com/2014/02/pipes-parse-30-lens-based-parsing.html>
So any time you find yourself writing a `Parser` that is trying to
`yield` things further downstream, that's your cue that you should
switch it to become a `Lens'` or a `Getter` (i.e. function) instead.
Back to your specific code: I believe the the type you want for
`handShake` is either a leftover-passing function equivalent to
the `Parser` you originally had:
type M = ReaderT SockConfig (SafeT IO)
handShake
:: SockAddr
-> Producer ByteString M r
-> Producer ByteString M ((Get Frame, Frame -> Put),
Producer ByteString M r)
... or a lens if you included enough information in the return
value to recover the original stream:
type M = ReaderT SockConfig (SafeT IO)
handShake
:: SockAddr
-> Lens' (Producer ByteString M r)
(Producer ByteString M ((ExtraInformation, Get Frame, Frame ->
Put), Producer ByteString M r)
If you did that, then you'd be able to write:
zoom (handShake . decoder):: Parser Frame m r -> Parser
ByteString m r
... and that would automatically handle all initial authentication
and provide you with a stream of decoded frames. Or you could use
it to transform a `Producer`:
view (handShake . decoder) :: Producer ByteString M r ->
Producer Frame M LargeReturnType
On 7/26/14, 10:55 AM, Merijn Verstraaten wrote:
Ok, so I have a network connection that has an interactive
handshake to setup a connection (i.e. I need to write things to
the network while parsing the handshake). Several months ago I
wrote some code that manually plumbed a ByteString state
throughout all my pipes to handle leftovers. Today I wanted to
try and rewrite this code to use the new pipes-parse stuff,
thinking the result would be much nicer to work with. However, I
get stuck with no clue how to continue. My original code was as
follows:
readSocket :: MonadIO m => Handle -> Producer ByteString m r
writeSocket :: MonadIO m => Handle -> Consumer ByteString m r
handShake :: SockAddr ->
StateT ByteString
(Pipe ByteString
ByteString
(ReaderT SockConfig (SafeT IO)))
( Pipe ByteString Frame (SafeT IO) r,
Pipe Frame ByteString (SafeT IO) r )
handShake addr = do
version <- zmtpVersion
let (decoder, encoder) = case version of
ZMTP1 {} -> (zmtp1Decoder, zmtp1Encoder)
:: MonadSafe m => (Pipe ByteString Frame m r, Pipe Frame
ByteString m r)
ZMTP2 {} -> (zmtp2Decoder, zmtp2Encoder)
ZMTP3 {} -> (zmtp3Decoder, zmtp3Encoder)
-- authenticate version addr :: Pipe Frame Frame m r
lift $ decoder >-> authenticate version addr >-> encoder
return (decoder, encoder)
session :: SockConfig
-> Producer Frame (SafeT IO) ()
-> Consumer Frame (SafeT IO) ()
-> IO Handle
-> SockAddr
-> IO ()
session cfg fromApp toApp newHandle addr =
runSafeT . runEffect . bracket newHandle hClose $ \hnd -> do
let fromNetwork = readSocket hnd
toNetwork = writeSocket hnd
handshake = runReaderP cfg (runStateT (handShake addr) empty)
((decoder, encoder), rest) <- fromNetwork >-> handshake >-> toNetwork
myTid <- liftIO myThreadId
let appToNet = fromApp >-> encoder >-> toNetwork
netToApp = (yield rest >> fromNetwork) >-> decoder >-> toApp
-- code using appToNet/netToApp Effect's
In other words, my handshake is just a pipe, reading and writing
ByteStrings from and to the same socket. After version
negotiation I do authentication, this uses a "Pipe Frame Frame"
that I compose with the relevant decoders/encoders and lift into
the handshake's pipe. So far, so good, the only real nastiness is
manually threading the leftover state throughout the zmtpVersion,
decoder and encoder.
So on to my attempt to rewrite this using pipes-parse.
readSocket :: MonadIO m => Handle -> Producer ByteString m r
writeSocket :: MonadIO m => Handle -> Consumer ByteString m r
handShake :: SockAddr
-> Parser ByteString (Producer ByteString (ReaderT SockConfig
(SafeT IO))) (Get Frame, Frame -> Put)
handShake _addr = do
version <- zmtpVersion
let (getFrame, putFrame) = case version of
ZMTP1 {} -> (ZMTP1.getFrame, ZMTP1.putFrame)
ZMTP2 {} -> (ZMTP2.getFrame, ZMTP2.putFrame)
ZMTP3 {} -> (ZMTP3.getFrame, ZMTP3.putFrame)
--lift $ undefined >-> authenticate version addr >-> undefined
return (getFrame, putFrame)
session :: SockConfig
-> Producer Frame (SafeT IO) ()
-> Consumer Frame (SafeT IO) ()
-> IO Handle
-> SockAddr
-> IO ()
session cfg fromApp toApp newHandle addr =
runSafeT . runEffect . bracket newHandle hClose $ \hnd -> do
let fromNetwork = readSocket hnd
toNetwork = writeSocket hnd
handshake = runReaderP cfg . runStateT (handShake addr)
((decoder, encoder), rest) <- handshake fromNetwork >-> toNetwork
myTid <- liftIO myThreadId
let appToNet = for fromApp (encodePut . encoder) >-> toNetwork
I immediately run into several problems. First off, my
authentication code wants to deal with Frame's, not ByteString,
but my initial has a ByteString producer. I can see that lenses
are supposed to let me "zoom" "Parser a" to "Parser b", but
unfortunately pipes-binary only has the following lens:
decoded :: (Monad m, Binary a) => Lens' (ProducerByteString m r)
(Producer a m (Either (DecodingError, ProducerByteString m r) r))
This only works with Binary instances, unfortunate in my case
where I need to use explicit getters/putters, but writing my own
decoder that work using decodeGet and encodePut is not too hard.
A bigger issue is, how to handle "authenticate", since it wants
to deal with Frame's it would ideally have the type "Parser Frame
(Producer Frame (ReaderT SockConfig (SafeT IO)) r", but I have no
way to run this within my "Parser ByteString (Producer ByteString
(ReaderT SockConfig (SafeT IO))" as the types don't match.
Even worse, after the "runState (handShake addr)" in session the
remainder of fromNetwork's output is in the Producer returned as
"rest", unfortunately the trip through the Parser's StateT has
changed fromNetwork's type to "Producer ByteString (Producer
ByteString (ReaderT SockConfig (SafeT IO))", I can't compose this
with session "Consumer Frame (SafeT IO) ()", because the inner
Producer and ReaderT are in the way, even though these are now
useless as only the outer Producer will still do something. I
could get rid of the ReaderT with a "runReaderP undefined", but I
have no way to get rid of the inner Producer...
Any suggestions on how to do this in the new pipes-parse world?
Or should I just give up and stick with my old design of explicit
passing around of leftovers?
Cheers,
Merijn
--
You received this message because you are subscribed to the
Google Groups "Haskell Pipes" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to [email protected] <javascript:>.
To post to this group, send email to [email protected]
<javascript:>.
--
You received this message because you are subscribed to the Google
Groups "Haskell Pipes" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to [email protected]
<mailto:[email protected]>.
To post to this group, send email to [email protected]
<mailto:[email protected]>.
--
You received this message because you are subscribed to the Google Groups "Haskell
Pipes" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
