Re: [Haskell-cafe] ANN: conduit-network-stream, A base layer for network protocols with Conduits
Hi Alexander, Am 25.02.2013 06:52, schrieb Alexander V Vershilov: Can you describe if there any difference with latest conduit API (yield, await) that can be used to write functions in a very similar style, but without using exeternal packages: I have indeed written this library for the 0.5.6 API, so things might have changed a bit in the 1.0 API, but I'd be surprised if the fundamental flaws of this approach would have been fixed. runTCPServer settings app = appSource ap $$ go =$= CL.mapM_ encode =$ appSink app where go = forever $ do bp - decode $ await `await` works on a conduit of strict `ByteString` chunks. The size/length of each `ByteString` is dependent on your network connection, so if you have a fast internet connection and your bytestrings are sufficiently separated because you're waiting for a response between each message your program might actually work as you expect it to do (with a little bit of luck). But consider a simple server/client application where the messages are not seperated by a small delay: server = runTCPServer (..) $ \ad - appSource ad $$ Data.Conduit.List.mapM_ (liftIO . print) client = runTCPClient (..) $ \ad - (yield msg1 yield msg2) $$ appSink ad The server will simply print out msg1msg2 as one message, not as two separate messages. Even worse, if your network connection is bad or your chunks are getting too big for buffering, you might end up with something like: msg1ms g2 `await` is not reliable in that regard because the network source is not consistent and non-deterministic. My libraray makes sure that every yield from the client corresponds to exactly one (not more or less) await at the server. There are more benefits when using my library. For example consider a client which first sends an authorization message, then a couple of hashes from different files and then maybe some timestamps on some other files. Writing that server is straight forward: client = runTCPClient (..) $ \ad - send1ad $$ yield authenticationMsg sendList ad $$ mapM_ yield [file1hash, file2hash, file3hash] sendList ad $$ mapM_ yield [timestamp1, timestamp2] server = runTCPServer (..) $ \ad - (next,[authmsg]) - receive ad $$ Data.Conduit.List.consume (next,hashes) - receive next $$ Data.Conduit.List.consume (next,timestamps) - receive next $$ Data.Conduit.List.consume close next Each `receive` corresponds to exactly one `send`. Without this library you have to manually check/verify/associate each message by hand in one big loop, whereas this library allows you to split your application into logical conduit groups which are straight forward to use. - Nils ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] ANN: conduit-network-stream, A base layer for network protocols with Conduits
Thanks for explanation. However I'm still confused as if I'll use one of protocols (protobuf, json, sereal message, binary message) all of them has iteration interface i.e. function of type (ByteString - Result a) that returns data Result = Error Text ByteString | NeedMore (ByteString - Result a) | HaveResult a all of them can be used in conduits using sinkParser or same approach and so there will be no problem in sending and receiving messages even if they are split, and that will be done in a logical block. I see that there will be problems if I'll use raw data and I see that your library solves it, am I right? On 25 February 2013 15:53, Nils m...@nils.cc wrote: Hi Alexander, Am 25.02.2013 06:52, schrieb Alexander V Vershilov: Can you describe if there any difference with latest conduit API (yield, await) that can be used to write functions in a very similar style, but without using exeternal packages: I have indeed written this library for the 0.5.6 API, so things might have changed a bit in the 1.0 API, but I'd be surprised if the fundamental flaws of this approach would have been fixed. runTCPServer settings app = appSource ap $$ go =$= CL.mapM_ encode =$ appSink app where go = forever $ do bp - decode $ await `await` works on a conduit of strict `ByteString` chunks. The size/length of each `ByteString` is dependent on your network connection, so if you have a fast internet connection and your bytestrings are sufficiently separated because you're waiting for a response between each message your program might actually work as you expect it to do (with a little bit of luck). But consider a simple server/client application where the messages are not seperated by a small delay: server = runTCPServer (..) $ \ad - appSource ad $$ Data.Conduit.List.mapM_ (liftIO . print) client = runTCPClient (..) $ \ad - (yield msg1 yield msg2) $$ appSink ad The server will simply print out msg1msg2 as one message, not as two separate messages. Even worse, if your network connection is bad or your chunks are getting too big for buffering, you might end up with something like: msg1ms g2 `await` is not reliable in that regard because the network source is not consistent and non-deterministic. My libraray makes sure that every yield from the client corresponds to exactly one (not more or less) await at the server. There are more benefits when using my library. For example consider a client which first sends an authorization message, then a couple of hashes from different files and then maybe some timestamps on some other files. Writing that server is straight forward: client = runTCPClient (..) $ \ad - send1ad $$ yield authenticationMsg sendList ad $$ mapM_ yield [file1hash, file2hash, file3hash] sendList ad $$ mapM_ yield [timestamp1, timestamp2] server = runTCPServer (..) $ \ad - (next,[authmsg]) - receive ad $$ Data.Conduit.List.consume (next,hashes) - receive next $$ Data.Conduit.List.consume (next,timestamps) - receive next $$ Data.Conduit.List.consume close next Each `receive` corresponds to exactly one `send`. Without this library you have to manually check/verify/associate each message by hand in one big loop, whereas this library allows you to split your application into logical conduit groups which are straight forward to use. - Nils -- Alexander ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] ANN: conduit-network-stream, A base layer for network protocols with Conduits
Of course you don't have to use my library if your library already solves the same problem. Am 25.02.2013 13:10, schrieb Alexander V Vershilov: data Result = Error Text ByteString | NeedMore (ByteString - Result a) | HaveResult a I still see an issue with this if you're receiving multiple messages in one `ByteString` chunk. A type like `HaveResult a ByteString` which also returns the leftover `ByteString` after the first result has been found could solve this, but you'd still have to re-yield this bytestring by hand and you'll possible end up with more calls to decode than necessary. Of course, with json you could avoid this problem by wrapping all messages into a json-list and sending this list as one big message. But that way you lose almost all advantages of a conduit - you cannot map over those list iems, you cannot process each item one by one, you cannot fold or filter them. When using this approach I don't really see the advantage over using the plain old connectTo/hGetContents method? Anyway, my library is independent of the bytestring encoding you want to use with it. I think it's the most general solution and avoids having to re-implement the same solution for each and every protocol design over and over again. ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] ANN: conduit-network-stream, A base layer for network protocols with Conduits
One question - and sorry that I didn't get a chance to try some examples myself - but can bi-directional conversations be carried between server/client in interleaving fashion using your library? Something like (simplified): myClient = receive = send . compute I have previously accomplished this using TBMChan and cereal-conduit to achieve the message delineation, but your library would certainly remove a layer of complexity if interleaved bi-directional talk is possible. - Ozgun On Monday, February 25, 2013 at 8:09 AM, Nils wrote: Of course you don't have to use my library if your library already solves the same problem. Am 25.02.2013 13:10, schrieb Alexander V Vershilov: data Result = Error Text ByteString | NeedMore (ByteString - Result a) | HaveResult a I still see an issue with this if you're receiving multiple messages in one `ByteString` chunk. A type like `HaveResult a ByteString` which also returns the leftover `ByteString` after the first result has been found could solve this, but you'd still have to re-yield this bytestring by hand and you'll possible end up with more calls to decode than necessary. Of course, with json you could avoid this problem by wrapping all messages into a json-list and sending this list as one big message. But that way you lose almost all advantages of a conduit - you cannot map over those list iems, you cannot process each item one by one, you cannot fold or filter them. When using this approach I don't really see the advantage over using the plain old connectTo/hGetContents method? Anyway, my library is independent of the bytestring encoding you want to use with it. I think it's the most general solution and avoids having to re-implement the same solution for each and every protocol design over and over again. ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org (mailto:Haskell-Cafe@haskell.org) http://www.haskell.org/mailman/listinfo/haskell-cafe ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] ANN: conduit-network-stream, A base layer for network protocols with Conduits
Hello, Nils.
Can you describe if there any difference with latest conduit API (yield,
await) that can be
used to write functions in a very similar style, but without using
exeternal packages:
runTCPServer settings app = appSource ap $$ go =$= CL.mapM_ encode =$
appSink app
where
go = forever $ do
bp - decode $ await {- decode is inlined here assuming it
can return different types -}
{- logic here-}
yield $ result
mapM (yeild) [result1,result2...]
more over you don't need to write appSource $$ ... appSink on the top.
Here is an example of a client that authorizes and then reads commands
from TBMChan and requests server using json format.
runTCPClient settings ad go
where
go = do
source $$ await
is - authorize
when is loop'
authorize = do
yield u yield (S8.pack \n) $$ sink -- send authorization
token
mx - source $$ sinkParser json -- get responce
...
loop' = do
minfo - atomically $ readTBMChan ch
case minfo of
Nothing - return ()
Just (message, respBox) - do
yield message $$ C.concatMap (SL.toChunks . encode) =$ sink
resp - source $$ sinkParser json
atomically $ putTMVar respBox resp
loop'
source = ppSource ad
sink = appSink ad
On 24 February 2013 22:23, Nils m...@nils.cc wrote:
Hello everyone,
I just uploaded my conduit-network-stream package to hackage. It's a
base layer for network protocols based on the Conduit package by Michael
Snoyman which makes it possible to send multiple messages over a continuous
network connection in a convenient way.
I wrote more about it at:
https://github.com/mcmaniac/**conduit-network-stream/blob/**
master/README.mdhttps://github.com/mcmaniac/conduit-network-stream/blob/master/README.md
It is available on hackage and github:
http://hackage.haskell.org/**package/conduit-network-streamhttp://hackage.haskell.org/package/conduit-network-stream
https://github.com/mcmaniac/**conduit-network-streamhttps://github.com/mcmaniac/conduit-network-stream
Until the documentation on hackage is generated, I also host the haddock
documentation at:
http://hs.nils.cc/conduit-**network-stream/index.htmlhttp://hs.nils.cc/conduit-network-stream/index.html
For any questions/problems/requests, please ask!
- Nils
__**_
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/**mailman/listinfo/haskell-cafehttp://www.haskell.org/mailman/listinfo/haskell-cafe
--
Alexander
___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
