[Haskell-cafe] binary serialization
Is there a way to do binary serialization of Haskell values (in GHC, at least)? If you propose a method, what are its type safety and portability properties? ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] binary serialization
fft1976: Is there a way to do binary serialization of Haskell values (in GHC, at least)? If you propose a method, what are its type safety and portability properties? There are many ways. See Data.Binary (fast, portable). Most are type safe, or additional safety can be added. I would add to that: Data.Derive can be used to automatically generate Data.Binary.Binary instances for your custom data types. Tim Newsham http://www.thenewsh.com/~newsham/ ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: Re[2]: [Haskell-cafe] Binary serialization, was Re: Abstraction leak
On Thu, 2007-07-05 at 18:08 +0100, Duncan Coutts wrote: - Found that on hackage, downloaded and built OK. Lots of scary warnings about happy, greencard etc, not being found during configure, but let's go on. I've complained about these before, although I don't think anyone considered doing anything about it. We know what needs to change, but it's not a trivial change. The problem is that currently .cabal files to not specify what build tools they need, A possible stop-gap might be to only report the build tools that were actually found? The way it is now, it looks too much like an error message. -k ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: Re[2]: [Haskell-cafe] Binary serialization, was Re: Abstraction leak
On Fri, 2007-07-06 at 08:26 +0200, Ketil Malde wrote: On Thu, 2007-07-05 at 18:08 +0100, Duncan Coutts wrote: - Found that on hackage, downloaded and built OK. Lots of scary warnings about happy, greencard etc, not being found during configure, but let's go on. I've complained about these before, although I don't think anyone considered doing anything about it. We know what needs to change, but it's not a trivial change. The problem is that currently .cabal files to not specify what build tools they need, A possible stop-gap might be to only report the build tools that were actually found? The way it is now, it looks too much like an error message. Mmm, good point. Perhaps we could just change the verbosity of that message for now, so that it doesn't appear at the default verbosity level. Duncan ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Binary serialization, was Re: Abstraction leak
On Thu, Jul 05, 2007 at 08:50:42AM +1000, Donald Bruce Stewart wrote: [useful stuff] So, in fact pretty much everything I was looking for exists, in some form or other! It's just a bit hard to find at the moment, perhaps because none of this stuff is regarded as 'core Haskell' by any of the tutorials, books etc etc. I'll have a play with some of the libraries mentioned upthread anyway. Thanks everyone. Phil -- http://www.kantaka.co.uk/ .oOo. public key: http://www.kantaka.co.uk/gpg.txt ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Binary serialization, was Re: Abstraction leak
I was explaining Haskell to a perl/python hacking friend recently and characterized things thus: Perl is a horrible language with fantastic libraries. Haskell is a fantastic language with horrible libraries. Actually, many of the libraries that exist for Haskell *are* fantastic, it's just that Haskell lacks the *coverage* that Perl or Python have. cheers, T. -- Dr Thomas Conway [EMAIL PROTECTED] Silence is the perfectest herald of joy: I were but little happy, if I could say how much. ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Binary serialization, was Re: Abstraction leak
On Thursday 05 July 2007, Thomas Conway wrote: I was explaining Haskell to a perl/python hacking friend recently and characterized things thus: Perl is a horrible language with fantastic libraries. Haskell is a fantastic language with horrible libraries. Actually, many of the libraries that exist for Haskell *are* fantastic, it's just that Haskell lacks the *coverage* that Perl or Python have. Can't say I agree. I've been learning Python, and have been very un-impressed so far with its library coverage, which I would rate no better than (in terms of the POSIX bindings, worse than) Haskell. The one thing off the top of my head that Python had was Base64, but that's 20 lines of Haskell tops. Aside from that, nothing. Jonathan Cast http://sourceforge.net/projects/fid-core http://sourceforge.net/projects/fid-emacs ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Binary serialization, was Re: Abstraction leak
On 05/07/07, Jonathan Cast [EMAIL PROTECTED] wrote: Can't say I agree. I've been learning Python, and have been very un-impressed so far with its library coverage, which I would rate no better than (in terms of the POSIX bindings, worse than) Haskell. It probably depends on your perspective. I've found lots of tasks that would be a simple library call in Python, but which require me to write the code myself in Haskell. Examples: * Send an email * Parse an ini file * Gzip compress a data stream * Calculate the MD5 checksum of a file (Of course, I may just not have found the relevant library - that says something about discoverability rather than coverage, I guess). For bindings, Python's Windows bindings (pywin32) are superb, where Haskell's are minimal and unmaintained. Of course, that won't matter to you if you use POSIX... The one thing off the top of my head that Python had was Base64, but that's 20 lines of Haskell tops. Aside from that, nothing. But that's 20 lines of code I don't want to write, and more, I don't know how to write (without looking up the definition of Base64). Having lots of these seemingly trivial helpers available out of the box is what library coverage means to me. (And Python does have lots of these - I don't know how Haskell fares in practice). I'm not trying to start (or fan) a flamewar, but it's interesting how different people's perspectives on libraries can be... Paul. ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re[2]: [Haskell-cafe] Binary serialization, was Re: Abstraction leak
Hello Paul, Thursday, July 5, 2007, 7:00:46 PM, you wrote: * Gzip compress a data stream zlib * Send an email * Parse an ini file The one thing off the top of my head that Python had was Base64, but that's 20 MissingH * Calculate the MD5 checksum of a file crypto -- Best regards, Bulatmailto:[EMAIL PROTECTED] ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: Re[2]: [Haskell-cafe] Binary serialization, was Re: Abstraction leak
On 05/07/07, Bulat Ziganshin [EMAIL PROTECTED] wrote: * Gzip compress a data stream zlib * Send an email * Parse an ini file The one thing off the top of my head that Python had was Base64, but that's MissingH * Calculate the MD5 checksum of a file crypto Thanks. The need I had for these is no longer current, but sometime I'll try an experiment and see how easy it is, on a relatively clean Windows box with just GHC installed, to grab and use these libraries. (Side note: with Python, I'm used to 3rd party modules being available as Windows installer packages - does the concept of an installable binary for something like MissingH, which I can just install and use, make sense for a compiled language like Haskell? The build, find I'm missing a C library, get it, compile it, fix bugs, try again cycle I used to hit before Python Windows installers became common isn't something I'd like to repeat...) As I mentioned in passing, it may well be that the library issue with Haskell is more of a perception (or maybe organisational) issue than a technical one... Paul. ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Binary serialization, was Re: Abstraction leak
On Thu, Jul 05, 2007 at 09:41:23AM -0700, Dave Bayer wrote: There are people who claim with a straight face that they migrated to OS X primarily to use TextMate http://www.textmate.com Presumably you mean http://macromates.com/ ? Phil -- http://www.kantaka.co.uk/ .oOo. public key: http://www.kantaka.co.uk/gpg.txt ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: Re[2]: [Haskell-cafe] Binary serialization, was Re: Abstraction leak
Hi It's not a great experience now, but hopefully things are moving in the right direction. - Found crypto 3.0.3 on hackage. - Tried to build, it depends on NewBinary Cabal-install is intended to remove this problem, so that you can say i want crypto and it gets everything that requires. - Found that on hackage, downloaded and built OK. Lots of scary warnings about happy, greencard etc, not being found during configure, but let's go on. I've complained about these before, although I don't think anyone considered doing anything about it. - Installed NewBinary as I didn't know how to make crypto find it without installing it. I'm a bit nervous, as I don't know how to *un*install it after I've finished. And it installed to my C drive, where I'd really rather it went somewhere else. There is probably documentation on how to do this, but remember, all I really want to do is to write a tiny program to get the MD5 checksum of a file. --prefix should put it where you want. What you really want is a Windows user interface, which is what I want too. - crypto builds and installs OK. But where are the docs? Not installed anywhere obvious, not on hackage. Try google. You can install them with runhaskell Setup haddock, but I have no idea where they end up. They will be on hackage at some point, with cross-indexing etc. - But no simple examples, and the haddoc docs show APIs, but not usage examples! Complain to the author. I always try and include a manual with at least one short example of how to use the library to do something interesting. Unfortunately all these things take time, something that not everyone has. I see you've already responded, and we're in broad agreement. So I won't labour the point. It's an infrastructure issue rather than a technical one, and it *will* improve. What will be interesting is how much the generally lousy Windows experience can be improved Part of the problem is that the number of Windows Haskell developers is low. Another part of the problem is that some people have scorn and contempt for Windows. Alas, those things are hard to change. Thanks Neil ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Binary serialization, was Re: Abstraction leak
On Jul 5, 2007, at 8:00 AM, Paul Moore wrote: It probably depends on your perspective. I've found lots of tasks that would be a simple library call in Python, but which require me to write the code myself in Haskell. Examples: * Calculate the MD5 checksum of a file How's this, only one line is specific to your problem: import System.Process import IO doShell :: String - IO String doShell cmd = do (_,out,_,_) - runInteractiveCommand cmd hGetContents out main :: IO () main = do md5 - doShell md5 -q md5.hs putStrLn md5 It's not like you'll be kicked out of the tree house for leaving the Haskell world to get things done. For example, ghostscript and pdf2ps are well-supported open source tools for converting PS to PDF, that can be called from most languages. What's the deal with everyone rewriting PDF handling in their pet language, when it's so much easier to generate Postscript? I'd call that Balkanization; if I were managing a software group, I'd never let that happen. The true problem isn't adequate libraries in each language, it's interoperability so great open-source tools can get written once and then be supported by a cast of thousands. There are people who claim with a straight face that they migrated to OS X primarily to use TextMate http://www.textmate.com which is a GUI editor getting Emacs-like buzz, making Emacs seem by comparison like your grandfather's razor. It's as much a text-based operating system as an editor, and the whole thing is glued together with hundreds of snippets of code one can hack, written in every scripting language imaginable. Polyglots feel right at home... ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: Re[2]: [Haskell-cafe] Binary serialization, was Re: Abstraction leak
On 05/07/07, Paul Moore [EMAIL PROTECTED] wrote: The need I had for these is no longer current, but sometime I'll try an experiment and see how easy it is, on a relatively clean Windows box with just GHC installed, to grab and use these libraries. Just for fun I had a go with crypto: - Found crypto 3.0.3 on hackage. - Tried to build, it depends on NewBinary - Found that on hackage, downloaded and built OK. Lots of scary warnings about happy, greencard etc, not being found during configure, but let's go on. - Installed NewBinary as I didn't know how to make crypto find it without installing it. I'm a bit nervous, as I don't know how to *un*install it after I've finished. And it installed to my C drive, where I'd really rather it went somewhere else. There is probably documentation on how to do this, but remember, all I really want to do is to write a tiny program to get the MD5 checksum of a file. Ah, well. Carry on. - crypto builds and installs OK. But where are the docs? Not installed anywhere obvious, not on hackage. Try google. - Found the crypto website, but aargh! It looks like 3.0.3 is out of date and there's a 4.x available. Never mind, that's not likely to have changed. - But no simple examples, and the haddoc docs show APIs, but not usage examples! Going for the obvious approach: import System.IO import Data.Digest.MD5 main = do h - openBinaryFile md5.hs ReadMode s - hGetContents h hClose h md5 - hash s No surprise, this doesn't work. After all, hash wants [Octet], not String. OK, I know this is now getting beyond library availability. But it doesn't compare well to Python's I want an md5 checksum of a file - check the docs, there's a library function built in, use it, no problems. I see you've already responded, and we're in broad agreement. So I won't labour the point. It's an infrastructure issue rather than a technical one, and it *will* improve. What will be interesting is how much the generally lousy Windows experience can be improved - as Duncan points out, installing development libraries on Windows, whether Haskell or C, is a hugely irritating pain. If no-one has found a good answer for C in all these years, it would be great if Haskell could even do slightly better :-) Regards, Paul. ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: Re[2]: [Haskell-cafe] Binary serialization, was Re: Abstraction leak
On Thu, 2007-07-05 at 17:07 +0100, Paul Moore wrote: On 05/07/07, Bulat Ziganshin [EMAIL PROTECTED] wrote: * Gzip compress a data stream zlib * Send an email * Parse an ini file The one thing off the top of my head that Python had was Base64, but that's MissingH * Calculate the MD5 checksum of a file crypto Thanks. The need I had for these is no longer current, but sometime I'll try an experiment and see how easy it is, on a relatively clean Windows box with just GHC installed, to grab and use these libraries. Just to warn you, a lot of haskell packages that bind to C libs are much harder to get working on Windows, the zlib package for example. This is because on all other platforms zlib comes with the system and is installed in a location where any application can link to it. On Windows there is no equivalent of /usr/lib you cannot easily install a C lib somewhere that it can be used by any .exe on the system. To make things easier yuo could avoid using -fvia-C and then at least the zlib header files would not need to be installed, but to run a program that uses the zlib package you'd still have to copy the zlib.dll into the same dir as your .exe. There is a mechanism in newer versions of Windows that allows installing .dlls systemwide where any .exe can use them, however ghc and the gcc toolchain do not support them yet. It requires embeding xml manifests into the .dll and .exe files and you have to be the admin user to install one of these systemwide .dll things. It's all a bit of a pain. Duncan ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re[4]: [Haskell-cafe] Binary serialization, was Re: Abstraction leak
Hello Paul, Thursday, July 5, 2007, 8:07:34 PM, you wrote: note: with Python, I'm used to 3rd party modules being available as Windows installer packages - does the concept of an installable binary for something like MissingH, which I can just install and use, make sense for a compiled language like Haskell? The build, find I'm all cabalized libraries are installed using the same commands: runghc setup.hs configure runghc setup.hs build runghc setup.hs install after this, you just import modules you need in your program and when you build your program, ghc automatically links in all the library functions you used As I mentioned in passing, it may well be that the library issue with Haskell is more of a perception (or maybe organisational) issue than a technical one... the problem exists and for me, it's most important Haskell insfrastructure problem. but things you are mentioned are already implemented i personally implemented several libs required for my own program; and still miss ease-to-use and maintained GUI library -- Best regards, Bulatmailto:[EMAIL PROTECTED] ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Binary serialization, was Re: Abstraction leak
On 05/07/07, Dave Bayer [EMAIL PROTECTED] wrote: How's this, only one line is specific to your problem: [...] md5 - doShell md5 -q md5.hs Doesn't work on my (Windows) PC, where I have no md5 command available. While I agree in theory with the idea of combining focused tools, it's a potential portability nightmare, particularly on Windows where decent command line tools are almost non-existent on a clean install. You're changing the problem from finding a Haskell library (which only needs to be installed on the development machine at compile time) to finding a 3rd party utility, which has to be installed at runtime on any machine using the compiled Haskell program. Not a good trade-off. And I'm not going to get into Windows/Linux arguments... :-) Paul. ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: Re[2]: [Haskell-cafe] Binary serialization, was Re: Abstraction leak
On 05/07/07, Neil Mitchell [EMAIL PROTECTED] wrote: - But no simple examples, and the haddoc docs show APIs, but not usage examples! Complain to the author. Yes, that's completely unrelated to library availability issues. I got off the topic, in all my ranting. Sorry. Part of the problem is that the number of Windows Haskell developers is low. Another part of the problem is that some people have scorn and contempt for Windows. Alas, those things are hard to change. Agreed. I'm extremely grateful for all the contributions people *do* make, even where they don't fit my environment as well as I might like. Anyway, I'm running out of time for today, so I'll leave this thread now. Thanks to all for some interesting comments and pointers! Paul. ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: Re[2]: [Haskell-cafe] Binary serialization, was Re: Abstraction leak
On Thu, 2007-07-05 at 17:39 +0100, Paul Moore wrote: I see you've already responded, and we're in broad agreement. So I won't labour the point. It's an infrastructure issue rather than a technical one, and it *will* improve. What will be interesting is how much the generally lousy Windows experience can be improved - We have this slightly odd problem where half our user base use Windows (according to the GHC user survey) but almost every active developer uses Linux or OSX (or a few other BSD/Unix OSs). So we could enormously improve the Windows user experience (and a few heroes work hard on doing just that) but basically there just aren't enough developers who use windows to give it a satisfactory level of support. This is of course a slightly circular problem, since using/developing Haskell on Windows is a pain, developers avoids it and so there are not enough developers irritated by how difficult it is to motivate developers to fix it! Duncan ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: Re[2]: [Haskell-cafe] Binary serialization, was Re: Abstraction leak
On Thu, Jul 05, 2007 at 06:08:45PM +0100, Duncan Coutts wrote: On Thu, 2007-07-05 at 17:51 +0100, Neil Mitchell wrote: - Found that on hackage, downloaded and built OK. Lots of scary warnings about happy, greencard etc, not being found during configure, but let's go on. I've complained about these before, although I don't think anyone considered doing anything about it. We know what needs to change, but it's not a trivial change. If anyone's interested, this is the Cabal bug for it: http://hackage.haskell.org/trac/hackage/ticket/132 Thanks Ian ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Binary serialization, was Re: Abstraction leak
On Jul 5, 2007, at 9:52 AM, Paul Moore wrote: You're changing the problem from finding a Haskell library (which only needs to be installed on the development machine at compile time) to finding a 3rd party utility, which has to be installed at runtime ... Not a good trade-off. The intersection of Linux and Mac OS X is a pretty amazing standard library, that beats any single scripting language. I'd forgotten how dismal Windows is, sorry. Still, if you stick to non-GPL'd licenses, there's no way to build single file deliverables? I'd think someone would have written Unix as a static library, the way e.g. many languages can be embedded in apps. Then only you would have to maintain the Unix tools you want to use, and you'd be done. If no one has, someone who cares about Windows should. Unix rocks. On Jul 5, 2007, at 9:54 AM, Philip Armstrong wrote: Presumably you mean http://macromates.com/ ? Yup. Sorry. ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
[Haskell-cafe] Binary serialization, was Re: Abstraction leak
(Bonus points for being able to parse ASN.1 and generate appropriate
Haskell datatypes serialization primitives automatically :-) )
I think there's at least an ASN.1 definition in the crypto library.
Dominic might be able to enlighten us on that.
No bonus points I'm afraid. There is an embryonic ASN.1 library here
http://hackage.haskell.org/cgi-bin/hackage-scripts/package/ASN1-0.0.1
You can take an ASN.1 definition such as
FooBar {1 2 0 0 6 1} DEFINITIONS ::=
BEGIN
Journey ::= SEQUENCE {
origin IA5String,
stop1 [0] IA5String OPTIONAL,
stop2 [1] IA5String OPTIONAL,
destination IA5String
}
Odyssey ::= SEQUENCE {
start Journey,
trip1 [0] Journey OPTIONAL,
trip2 [1] Journey OPTIONAL,
trip3 [2] Journey OPTIONAL,
end Journey
}
END
And then create abstract Haskell representations of the ASN.1 types
journey =
Journey ::=
AbsSeq Universal 16 Implicit [
Regular (Just origin::
(Nothing :@: absIA5String)),
Optional (Just stop1::
(Just 0 :@: absIA5String)),
Optional (Just stop2::
(Just 1 :@: absIA5String)),
Regular (Just destination ::
(Nothing :@: absIA5String))
]
odyssey =
Odyssey ::=
AbsSeq Universal 16 Implicit [
Regular (Just start:: (Nothing :@: journey)),
Optional (Just trip1 :: (Just 0 :@: journey)),
Optional (Just trip2 :: (Just 1 :@: journey)),
Optional (Just trip3 :: (Just 2 :@: journey)),
Regular (Just end :: (Nothing :@: journey))
]
The library then allows you to decode BER representations of these
types. It's good enough to decode X.509 identity and attribute certificates.
There's no encoding routines currently as I didn't need them.
I'll try and make some documentation more easily available if I get time
at the weekend.
I'm working on PER at the moment both encoding and decoding using GADTs.
I will extend it at some point for BER but that won't be for some time.
I thought I read that someone was working on parsing ASN.1 so I'll try
and follow that up (again it will have to be at the weekend).
Dominic.
___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Binary serialization, was Re: Abstraction leak
p.f.moore: On 05/07/07, Jonathan Cast [EMAIL PROTECTED] wrote: Can't say I agree. I've been learning Python, and have been very un-impressed so far with its library coverage, which I would rate no better than (in terms of the POSIX bindings, worse than) Haskell. It probably depends on your perspective. I've found lots of tasks that would be a simple library call in Python, but which require me to write the code myself in Haskell. Examples: * Send an email Sounds like a job for MissingH? * Parse an ini file Probably have to write your own Parsec-based parser here. * Gzip compress a data stream We have a wonderful library for this! http://hackage.haskell.org/cgi-bin/hackage-scripts/package/zlib-0.3 * Calculate the MD5 checksum of a file In the Crypto library, or use the openssl binding posted a couple of days ago: http://thread.gmane.org/gmane.comp.lang.haskell.cafe/24165/focus=24170 (Of course, I may just not have found the relevant library - that says something about discoverability rather than coverage, I guess). Find more libraries on hackage: http://hackage.haskell.org/packages/archive/pkg-list.html For bindings, Python's Windows bindings (pywin32) are superb, where Haskell's are minimal and unmaintained. Of course, that won't matter to you if you use POSIX... The one thing off the top of my head that Python had was Base64, but that's 20 lines of Haskell tops. Aside from that, nothing. But that's 20 lines of code I don't want to write, and more, I don't know how to write (without looking up the definition of Base64). Having lots of these seemingly trivial helpers available out of the box is what library coverage means to me. (And Python does have lots of these - I don't know how Haskell fares in practice). I'm not trying to start (or fan) a flamewar, but it's interesting how different people's perspectives on libraries can be... -- Don ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Binary serialization, was Re: Abstraction leak
On Wed, Jul 04, 2007 at 09:44:13PM +1000, Donald Bruce Stewart wrote: Binary instances are pretty easy to write. For a simple data type: instance Binary Exp where put (IntE i) = do put (0 :: Word8) put i put (OpE s e1 e2) = do put (1 :: Word8) put s put e1 put e2 get = do tag - getWord8 case tag of 0 - liftM IntE get 1 - liftM3 OpE get get get That's quite verbose! Plus I'm a bit concerned by the boxing implied by those IntE / OpE constructors in get. If you were using those values in a pattern match on the result of get, would the compiler be able to eliminate them and refer directly to the values in the source data? The Data.Binary comes with one tool to derive these. The DrIFT preprocessor also can, as can Stefan O'Rear's SYB deriver. I just write them by hand, or use the tool that comes with the lib. More docs here, http://hackage.haskell.org/packages/archive/binary/0.3/doc/html/Data-Binary.html This doesn't seem to deal with endianness. Am I missing something? world, you could operate on the packets in place in Haskell where possible and save the deserialization overhead... Data.ByteString.* for this. Ah, does Data.Binary fuse with ByteString.* then? Hack those bytes! Quickly! :-) :) It's a shame the layout definition is so verbose. Erlang's is quite compact. I wonder if something could be done with template haskell to translate an Erlang-style data layout definition to the Data.Binary form? (Bonus points for being able to parse ASN.1 and generate appropriate Haskell datatypes serialization primitives automatically :-) ) Phil -- http://www.kantaka.co.uk/ .oOo. public key: http://www.kantaka.co.uk/gpg.txt ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re[2]: [Haskell-cafe] Binary serialization, was Re: Abstraction leak
Hello Philip, Wednesday, July 4, 2007, 5:50:42 PM, you wrote: This doesn't seem to deal with endianness. Am I missing something? alternative: http://haskell.org/haskellwiki/Library/AltBinary http://haskell.org/haskellwiki/Library/Streams -- Best regards, Bulatmailto:[EMAIL PROTECTED] ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Binary serialization, was Re: Abstraction leak
On Wed, Jul 04, 2007 at 06:52:08PM +0400, Bulat Ziganshin wrote: Hello Philip, Wednesday, July 4, 2007, 5:50:42 PM, you wrote: This doesn't seem to deal with endianness. Am I missing something? alternative: http://haskell.org/haskellwiki/Library/AltBinary http://haskell.org/haskellwiki/Library/Streams Nice: bit aligning if you want it, little or big endian IO. Intermixed endianness in the same datastream even[1]. However: 3.2.10 Defining Binary instances for custom serialization formats (unwritten) Does that mean that the code is unwritten or that the documentation is unwritten. IAMFI :) There seems to be some overlap between Streams and ByteStrings: Could a Stream built on a ByteString backend benefit from all the fusion work that's been put into ByteStrings recently? Oh wait, I see you list that as 'future work' on the wiki page... Phil [1] Which sick application *needs* intermixed endianness? -- http://www.kantaka.co.uk/ .oOo. public key: http://www.kantaka.co.uk/gpg.txt ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Binary serialization, was Re: Abstraction leak
On Wed, Jul 04, 2007 at 02:50:42PM +0100, Philip Armstrong wrote: The Data.Binary comes with one tool to derive these. The DrIFT preprocessor also can, as can Stefan O'Rear's SYB deriver. I just write them by hand, or use the tool that comes with the lib. More docs here, http://hackage.haskell.org/packages/archive/binary/0.3/doc/html/Data-Binary.html This doesn't seem to deal with endianness. Am I missing something? The Data.Binary high level interface standardizes on 64-bit big endian. The low level interface allows you to choose it yourself. Stefan ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Binary serialization, was Re: Abstraction leak
On Wed, Jul 04, 2007 at 09:15:59PM +0400, Bulat Ziganshin wrote: Does that mean that the code is unwritten or that the documentation is unwritten. IAMFI :) of course all unwritten notes means unfinished docs. library contains more than 100 functions so it was not easy to document them all. you can browse sources, although probably it will not help too much OK. There seems to be some overlap between Streams and ByteStrings: Could a Stream built on a ByteString backend benefit from all the fusion work that's been put into ByteStrings recently? Oh wait, I see you list that as 'future work' on the wiki page... if you will write all popular words together, this probably will be just a set of popular words, not something working :) how fusion should work together with serialization? I'm thinking of the elimination of the boxing of values drawn out of the input stream where possible, eg if I was writing a stream processor that folded across the values in the input stream, it would (presumably) be more efficient if the compiler noticed that the function in question was (say) just reading Int values at offsets within the stream, and could pass those as unboxed references in the compiled code rather than freshly constructed values. Fusion might be the wrong term: I was thinking by analogy with loop fusion, with one of the loops was the 'data reading' loop. Does that make sense? [1] Which sick application *needs* intermixed endianness? i just tried to implement everything possible :) Completeness is always good! Thanks for the pointers, Phil -- http://www.kantaka.co.uk/ .oOo. public key: http://www.kantaka.co.uk/gpg.txt ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Binary serialization, was Re: Abstraction leak
Philip Armstrong wrote: [1] Which sick application *needs* intermixed endianness? *Clearly* you've never been to Singapore... ...er, I mean, Ever tried playing with networking protocol stacks? ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re[2]: [Haskell-cafe] Binary serialization, was Re: Abstraction leak
Hello Philip, Wednesday, July 4, 2007, 9:41:27 PM, you wrote: I'm thinking of the elimination of the boxing of values drawn out of the input stream where possible, eg if I was writing a stream processor that folded across the values in the input stream, it would (presumably) be more efficient if the compiler noticed that the function in question was (say) just reading Int values at offsets within the stream, and could pass those as unboxed references in the compiled code rather than freshly constructed values. it will depend on your code. the library doesn't make unnecessary boxing, but (unlike Data.Binary?) it supports only monadic (de)serialization. so there is no room for ByteString-like fusion which pass unboxed data through several transformations. with my lib, you can only read whole unboxed structure and then process it: data T = C !Int32 !Word16 do x - get h :: IO T processT $! x -- Best regards, Bulatmailto:[EMAIL PROTECTED] ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Binary serialization, was Re: Abstraction leak
On Wed, Jul 04, 2007 at 07:36:11PM +0100, Andrew Coppin wrote: Philip Armstrong wrote: [1] Which sick application *needs* intermixed endianness? *Clearly* you've never been to Singapore... ...er, I mean, Ever tried playing with networking protocol stacks? No (thankfully?). Phil -- http://www.kantaka.co.uk/ .oOo. public key: http://www.kantaka.co.uk/gpg.txt ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Binary serialization, was Re: Abstraction leak
phil: On Wed, Jul 04, 2007 at 09:44:13PM +1000, Donald Bruce Stewart wrote: Binary instances are pretty easy to write. For a simple data type: instance Binary Exp where put (IntE i) = do put (0 :: Word8) put i put (OpE s e1 e2) = do put (1 :: Word8) put s put e1 put e2 get = do tag - getWord8 case tag of 0 - liftM IntE get 1 - liftM3 OpE get get get That's quite verbose! Plus I'm a bit concerned by the boxing implied by those IntE / OpE constructors in get. If you were using those values in a pattern match on the result of get, would the compiler be able to eliminate them and refer directly to the values in the source data? Well, here's you're flattening a Haskell structure, so it has to get reboxed. If it was bytestring chunks, or Ints, then you can avoid any serious copying. The 'get' just tags a value. The Data.Binary comes with one tool to derive these. The DrIFT preprocessor also can, as can Stefan O'Rear's SYB deriver. I just write them by hand, or use the tool that comes with the lib. More docs here, http://hackage.haskell.org/packages/archive/binary/0.3/doc/html/Data-Binary.html This doesn't seem to deal with endianness. Am I missing something? That's the Haskell serialisation layer. Look at Data.Binary.Get/Put for endian-primitives, to be used instead of 'get'. i.e. getWord16be world, you could operate on the packets in place in Haskell where possible and save the deserialization overhead... Data.ByteString.* for this. Ah, does Data.Binary fuse with ByteString.* then? They know about each other, and Binary avoids copying if you're reading ByteStrings. Hack those bytes! Quickly! :-) :) It's a shame the layout definition is so verbose. Erlang's is quite compact. I wonder if something could be done with template haskell to translate an Erlang-style data layout definition to the Data.Binary form? Right, simple but a bit verbose. The Erlang bit syntax is a nice pattern matching/layout syntax for bit/byte data. There's a couple of ports of this to Haskell -- one using pattern guards, another using Template Haskell. Look on hackage.haskell.org for bitsyntax if you're interested. (Bonus points for being able to parse ASN.1 and generate appropriate Haskell datatypes serialization primitives automatically :-) ) I think there's at least an ASN.1 definition in the crypto library. Dominic might be able to enlighten us on that. -- Don ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
