Thanks, Gabriel. Very helpful.
I am going to do a prototype (just to get familiar with pipes - I already
use turtle and io-streams) and come back if any questions.
On Sunday, May 29, 2016 at 12:36:26 AM UTC-4, Gabriel Gonzalez wrote:
>
> I believe you won’t be able to reuse the `pipes-aeson` library because it
> doesn't provide a way to stream over a nested field of a decoded JSON
> record nor is there any support for cursor-like navigation of the struct.
> That means that you will need to parse the skeleton of the JSON record by
> hand.
>
> Also, some work needs to be done to wrap the `base64-bytestring` in a
> `pipes`-like API with this type:
>
> -- Convert a base64-encoded stream to a raw byte stream
> decodeBase64
> :: Producer ByteString m r
> -- ^ Base64-encoded bytes
> -> Producer ByteString m (Either SomeException (Producer
> ByteString m r))
> -- ^ Raw bytes
>
> Note that the result returns a `Producer` for the remainder of the byte
> string (i.e. everything after the base64-encoded bytes) if the decoding
> completes successfully. This lets you resume parsing where the image bytes
> end.
>
> However, assuming that you have a `decodeBase64` function, then the rough
> outline of how the code would work is that you’d have three parts:
>
> * Parse the prefix of the record before the image bytes using a `binary`
> parser adapted to `pipes`
> * Use the `decodeBase64` function to stream the decoded image bytes
> * Parse the suffix of the record after the image bytes also using a
> `binary` parser adapted to `pipes`
>
> In other words, the types and implementation would look roughly like this:
>
> -- This would match the "{ 'id' : 'foo', 'image' : '" prefix of the
> JSON record
> skipPrefix :: Data.Binary.Get ()
>
> skipPrefix’ :: Monad m => Producer ByteString m r -> m (Either
> DecodingError (Producer ByteString m r))
> skipPrefix’ = execStateT (Pipes.Binary.decodeGet skipPrefix)
>
> — This would match the "' }" suffix of the JSON record
> skipSuffix :: Data.Binary.Get ()
>
> skipSuffix’ :: Monad m => Producer ByteString m r -> m (Either
> DecodingError (Producer ByteString m r))
> skipSuffix’ = execStateT (Pipes.Binary.decodeGet skipSuffix)
>
> streamImage
> :: Monad m
> => Producer ByteString m r
> -> Producer ByteString m (Either SomeException (Producer
> ByteString m r))
> streamImage p0 = do
> e0 <- lift (skipPrefix’ p0)
> case e0 of
> Left exc -> return (Left (toException exc))
> Right p1 -> do
> e1 <- decodeBase64 p1
> case e1 of
> Left exc -> return (Left exc)
> Right p2 -> do
> e2 <- lift (skipSuffix’ p2)
> case e2 of
> Left exc -> return (Left (toException exc))
> Right p3 -> return (Right p3)
>
> In other words, `streamImage` would take a `Producer` as input that begins
> at the first character of the JSON record, and it will stream the decoded
> image bytes extracted from that record. If decoding succeeds, then it will
> return the remainder of the byte stream immediately after the JSON record.
>
> On May 28, 2016, at 5:20 AM, Sal <sanket....@gmail.com <javascript:>>
> wrote:
>
> Hello,
>
> I just posted this pipes related question at stackoverflow:
> http://stackoverflow.com/questions/37498848/decoding-json-stream-where-some-values-are-needed-before-others
>
> If any one has pointers on this, will very much appreciate it. I control
> the Javascript client that sends the JSON if this makes it easier to solve
> the problem.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Haskell Pipes" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to haskell-pipe...@googlegroups.com <javascript:>.
> To post to this group, send email to haskel...@googlegroups.com
> <javascript:>.
>
>
>
--
You received this message because you are subscribed to the Google Groups
"Haskell Pipes" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to haskell-pipes+unsubscr...@googlegroups.com.
To post to this group, send email to haskell-pipes@googlegroups.com.
