Re: [Haskell-cafe] handling multiple versions of a data structure
Hello, I'm not sure off-hand.. I think it would involve some hackery. How long do you want to keep support for the old Show format around? If you just want to convert some data you have right now, then you could write some code that uses the old Show based code to read in the old data and write it out using Serialize in the new format. If you want to support Show based migration for a long time -- then that could be trickier. While serialize does give you complete control over how the actual data is converted to a byte stream and back, it also assumes that there is some other meta-data involved (the Version stuff). So the problem I see is what happens if you try to read data from the old format which does not have that meta data.. I think what you would need to do is have your own code which detects if the saved data is in the Show format or the Serialize format. If it is in the Serialize format, then you just call the normal deserialization code. If it is in the old Show format, then you call your code to read the old Show format. That will give you data that is in the 'old' type that was last supported by the Show code. You would then is migrate function from the Migrate class to migrate it to the latest type. Does that make sense? - jeremy On Sun, Dec 19, 2010 at 10:58 AM, Dmitry V'yal akam...@gmail.com wrote: On 17.12.2010 01:09, Jeremy Shaw wrote: Hello, You should use happstack-data for this (you do not need the other happstack components to use happstack-data)*. It was created to solve this exact problem. happstack-data builds on type of the 'binary' library and adds versioned data types and automatic version migration. Thanks! Looks like what I need. There is a one problem, though. I don't have a time machine. I mean right now I have A primitive format based on Show instance and I'd like to maintain the compatibility with it. Is it possible to force a particular on-disk format for a particular version of data type? I guess I should write a Serialize instance myself. Are there any pitfalls awaiting me? Best wishes, Dmitry ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] handling multiple versions of a data structure
On 17.12.2010 01:09, Jeremy Shaw wrote: Hello, You should use happstack-data for this (you do not need the other happstack components to use happstack-data)*. It was created to solve this exact problem. happstack-data builds on type of the 'binary' library and adds versioned data types and automatic version migration. Thanks! Looks like what I need. There is a one problem, though. I don't have a time machine. I mean right now I have A primitive format based on Show instance and I'd like to maintain the compatibility with it. Is it possible to force a particular on-disk format for a particular version of data type? I guess I should write a Serialize instance myself. Are there any pitfalls awaiting me? Best wishes, Dmitry ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] handling multiple versions of a data structure
Nice. Do you think there is any reason we would not be able to / want to use it with happstack ? I would love happstack-data to 'go away' and just use some library from hackage which does the same thing. - jeremy On Dec 17, 2010, at 3:57 AM, Erik Hesselink wrote: I've recently been playing with code for versioning data types. It's based on happstacks implementation, but uses type families to make it more modular. I've got some proof of concept code on github [1]. We're also writing a small library based on this at typLAB, which we'll probably release as well. Erik [1] https://gist.github.com/704109 On Thu, Dec 16, 2010 at 19:26, Dmitry V'yal akam...@gmail.com wrote: Greetings, while developing my neural net simulator I stumbled upon a problem. I have a data type NeuralNet and use Show and Read instances for saving and loading configurations. As time passed, I changed the data type, so the program can no longer load files saved in previous versions. I want fix it. My current idea looks as follows. I'm going to create a bunch of types NN1, NN2, NN3..NNn for different versions and write converters c12 :: N1 - N2, c23 :: N2 - N3 and so on. But how to organize the whole process of parsing String into NNn so it's easy to change formats? Something based on using a list of parsers [read, c43 . read, c43 . c23 . read, c43, c23 . c12 . read, c43 . c32 . c21 . read] looks rather verbose and grows quadratically with N. I'm sure there must be a more elegant way. Any ideas? Dmitry ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] handling multiple versions of a data structure
No, I don't think so. It uses some extensions, but happstack-data already does, so that shouldn't be a problem. We don't have a releasable library yet, but when we do, it will have a versioned binary implementation, just like happstack-data does now. Perhaps even binary compatible with it, though I'm not sure about that. Erik On Sat, Dec 18, 2010 at 19:11, Jeremy Shaw jer...@n-heptane.com wrote: Nice. Do you think there is any reason we would not be able to / want to use it with happstack ? I would love happstack-data to 'go away' and just use some library from hackage which does the same thing. - jeremy On Dec 17, 2010, at 3:57 AM, Erik Hesselink wrote: I've recently been playing with code for versioning data types. It's based on happstacks implementation, but uses type families to make it more modular. I've got some proof of concept code on github [1]. We're also writing a small library based on this at typLAB, which we'll probably release as well. Erik [1] https://gist.github.com/704109 On Thu, Dec 16, 2010 at 19:26, Dmitry V'yal akam...@gmail.com wrote: Greetings, while developing my neural net simulator I stumbled upon a problem. I have a data type NeuralNet and use Show and Read instances for saving and loading configurations. As time passed, I changed the data type, so the program can no longer load files saved in previous versions. I want fix it. My current idea looks as follows. I'm going to create a bunch of types NN1, NN2, NN3..NNn for different versions and write converters c12 :: N1 - N2, c23 :: N2 - N3 and so on. But how to organize the whole process of parsing String into NNn so it's easy to change formats? Something based on using a list of parsers [read, c43 . read, c43 . c23 . read, c43, c23 . c12 . read, c43 . c32 . c21 . read] looks rather verbose and grows quadratically with N. I'm sure there must be a more elegant way. Any ideas? Dmitry ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] handling multiple versions of a data structure
I've recently been playing with code for versioning data types. It's based on happstacks implementation, but uses type families to make it more modular. I've got some proof of concept code on github [1]. We're also writing a small library based on this at typLAB, which we'll probably release as well. Erik [1] https://gist.github.com/704109 On Thu, Dec 16, 2010 at 19:26, Dmitry V'yal akam...@gmail.com wrote: Greetings, while developing my neural net simulator I stumbled upon a problem. I have a data type NeuralNet and use Show and Read instances for saving and loading configurations. As time passed, I changed the data type, so the program can no longer load files saved in previous versions. I want fix it. My current idea looks as follows. I'm going to create a bunch of types NN1, NN2, NN3..NNn for different versions and write converters c12 :: N1 - N2, c23 :: N2 - N3 and so on. But how to organize the whole process of parsing String into NNn so it's easy to change formats? Something based on using a list of parsers [read, c43 . read, c43 . c23 . read, c43, c23 . c12 . read, c43 . c32 . c21 . read] looks rather verbose and grows quadratically with N. I'm sure there must be a more elegant way. Any ideas? Dmitry ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
[Haskell-cafe] handling multiple versions of a data structure
Greetings, while developing my neural net simulator I stumbled upon a problem. I have a data type NeuralNet and use Show and Read instances for saving and loading configurations. As time passed, I changed the data type, so the program can no longer load files saved in previous versions. I want fix it. My current idea looks as follows. I'm going to create a bunch of types NN1, NN2, NN3..NNn for different versions and write converters c12 :: N1 - N2, c23 :: N2 - N3 and so on. But how to organize the whole process of parsing String into NNn so it's easy to change formats? Something based on using a list of parsers [read, c43 . read, c43 . c23 . read, c43, c23 . c12 . read, c43 . c32 . c21 . read] looks rather verbose and grows quadratically with N. I'm sure there must be a more elegant way. Any ideas? Dmitry ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] handling multiple versions of a data structure
I haven't tested this idea, but it occurred to me that this might be a good place for data families: data Z = Z newtype S n = S n -- Some nastiness to avoid having to see into the future? type family Pred n :: * type instance Pred (S n) = n type instance Pred Z = Z class MyDataClass ver where data MyData ver :: * upgrade :: MyData (Pred ver) - MyData ver -- other methods that won't change from version to version class Less a b instance Less Z a instance Less a b = Less a (S b) convert :: Less a b = MyData a - MyData b -- you may need a method in Less to witness the less-ness, but then you'd iterate your upgrade until you reach the version you want. -- Then you might want to abstract over this class with an existential, so it doesn't infect other things: data MyDataGeneral = forall ver. MyDataClass ver = MyDataGeneral ver -- now make instances for versions you have, with data instances for your current version of the structure. This might not even compile as I just wrote it into my email client, but it seems like it could work. Any comments? Dan On Thu, Dec 16, 2010 at 1:26 PM, Dmitry V'yal akam...@gmail.com wrote: Greetings, while developing my neural net simulator I stumbled upon a problem. I have a data type NeuralNet and use Show and Read instances for saving and loading configurations. As time passed, I changed the data type, so the program can no longer load files saved in previous versions. I want fix it. My current idea looks as follows. I'm going to create a bunch of types NN1, NN2, NN3..NNn for different versions and write converters c12 :: N1 - N2, c23 :: N2 - N3 and so on. But how to organize the whole process of parsing String into NNn so it's easy to change formats? Something based on using a list of parsers [read, c43 . read, c43 . c23 . read, c43, c23 . c12 . read, c43 . c32 . c21 . read] looks rather verbose and grows quadratically with N. I'm sure there must be a more elegant way. Any ideas? Dmitry ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] handling multiple versions of a data structure
On Thu, Dec 16, 2010 at 12:56 PM, Daniel Peebles pumpkin...@gmail.com wrote: I haven't tested this idea, but it occurred to me that this might be a good place for data families: data Z = Z newtype S n = S n -- Some nastiness to avoid having to see into the future? type family Pred n :: * type instance Pred (S n) = n type instance Pred Z = Z class MyDataClass ver where data MyData ver :: * upgrade :: MyData (Pred ver) - MyData ver -- other methods that won't change from version to version class Less a b instance Less Z a instance Less a b = Less a (S b) convert :: Less a b = MyData a - MyData b -- you may need a method in Less to witness the less-ness, but then you'd iterate your upgrade until you reach the version you want. -- Then you might want to abstract over this class with an existential, so it doesn't infect other things: data MyDataGeneral = forall ver. MyDataClass ver = MyDataGeneral ver -- now make instances for versions you have, with data instances for your current version of the structure. This might not even compile as I just wrote it into my email client, but it seems like it could work. Any comments? Dan The Happstack project includes versioning in its serialization libraries - Here is the API reference: http://happstack.com/docs/0.5.0/happstack-data/Happstack-Data-Serialize.html Although I don't know if there are any good tutorials on using it separated from the Happstack persistence layer. Antoine On Thu, Dec 16, 2010 at 1:26 PM, Dmitry V'yal akam...@gmail.com wrote: Greetings, while developing my neural net simulator I stumbled upon a problem. I have a data type NeuralNet and use Show and Read instances for saving and loading configurations. As time passed, I changed the data type, so the program can no longer load files saved in previous versions. I want fix it. My current idea looks as follows. I'm going to create a bunch of types NN1, NN2, NN3..NNn for different versions and write converters c12 :: N1 - N2, c23 :: N2 - N3 and so on. But how to organize the whole process of parsing String into NNn so it's easy to change formats? Something based on using a list of parsers [read, c43 . read, c43 . c23 . read, c43, c23 . c12 . read, c43 . c32 . c21 . read] looks rather verbose and grows quadratically with N. I'm sure there must be a more elegant way. Any ideas? Dmitry ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] handling multiple versions of a data structure
Hello, You should use happstack-data for this (you do not need the other happstack components to use happstack-data)*. It was created to solve this exact problem. happstack-data builds on type of the 'binary' library and adds versioned data types and automatic version migration. You can get an idea as to how it works by reading this old blog post, http://nhlab.blogspot.com/2008/12/data-migration-with-happs-data.html The modules names have changed from HAppS.* to Happstack.*, but otherwise it is still pretty accurate. The upcoming happstack 7 release cycle will be focusing on this area of happstack. Especially improved documentation. But, it is quite usable right now. If you have questions about happstack-data, feel free to ask on the happstack mailing list or irc channel. (http://happstack.com/community) I am happy to answer any questions or concerns you may have. - jeremy * the version on hackage depends on happstack-util, but the darcs version does not. On Dec 16, 2010, at 12:26 PM, Dmitry V'yal wrote: Greetings, while developing my neural net simulator I stumbled upon a problem. I have a data type NeuralNet and use Show and Read instances for saving and loading configurations. As time passed, I changed the data type, so the program can no longer load files saved in previous versions. I want fix it. My current idea looks as follows. I'm going to create a bunch of types NN1, NN2, NN3..NNn for different versions and write converters c12 :: N1 - N2, c23 :: N2 - N3 and so on. But how to organize the whole process of parsing String into NNn so it's easy to change formats? Something based on using a list of parsers [read, c43 . read, c43 . c23 . read, c43, c23 . c12 . read, c43 . c32 . c21 . read] looks rather verbose and grows quadratically with N. I'm sure there must be a more elegant way. Any ideas? Dmitry ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] handling multiple versions of a data structure
Have you considered moving these packages that are unrelated to web development into a separate namespace? I know that I never considered looking under the happstack namespace simply because I never do webapps. On Thu, Dec 16, 2010 at 5:09 PM, Jeremy Shaw jer...@n-heptane.com wrote: Hello, You should use happstack-data for this (you do not need the other happstack components to use happstack-data)*. It was created to solve this exact problem. happstack-data builds on type of the 'binary' library and adds versioned data types and automatic version migration. You can get an idea as to how it works by reading this old blog post, http://nhlab.blogspot.com/2008/12/data-migration-with-happs-data.html The modules names have changed from HAppS.* to Happstack.*, but otherwise it is still pretty accurate. The upcoming happstack 7 release cycle will be focusing on this area of happstack. Especially improved documentation. But, it is quite usable right now. If you have questions about happstack-data, feel free to ask on the happstack mailing list or irc channel. (http://happstack.com/community) I am happy to answer any questions or concerns you may have. - jeremy * the version on hackage depends on happstack-util, but the darcs version does not. On Dec 16, 2010, at 12:26 PM, Dmitry V'yal wrote: Greetings, while developing my neural net simulator I stumbled upon a problem. I have a data type NeuralNet and use Show and Read instances for saving and loading configurations. As time passed, I changed the data type, so the program can no longer load files saved in previous versions. I want fix it. My current idea looks as follows. I'm going to create a bunch of types NN1, NN2, NN3..NNn for different versions and write converters c12 :: N1 - N2, c23 :: N2 - N3 and so on. But how to organize the whole process of parsing String into NNn so it's easy to change formats? Something based on using a list of parsers [read, c43 . read, c43 . c23 . read, c43, c23 . c12 . read, c43 . c32 . c21 . read] looks rather verbose and grows quadratically with N. I'm sure there must be a more elegant way. Any ideas? Dmitry ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] handling multiple versions of a data structure
On Dec 16, 2010, at 5:48 PM, Daniel Peebles wrote: Have you considered moving these packages that are unrelated to web development into a separate namespace? I know that I never considered looking under the happstack namespace simply because I never do webapps. On Thu, Dec 16, 2010 at 5:09 PM, Jeremy Shaw jer...@n-heptane.com wrote: Hello, You should use happstack-data for this (you do not need the other happstack components to use happstack-data)*. It was created to solve this exact problem. +1 -- This package looks like it solves this (common) problem in a pretty good way. I've off and on considered writing a package that was very similar, and despite using happstack frequently, have never touched any of the data layers, and assumed they were all tied to the persistence layer, which I've avoided. This could definitely use some juice and traction on its own. Cheers, Sterl.___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] handling multiple versions of a data structure
On Dec 16, 2010, at 4:48 PM, Daniel Peebles wrote: Have you considered moving these packages that are unrelated to web development into a separate namespace? I know that I never considered looking under the happstack namespace simply because I never do webapps. Yes. I have been wanting to do it for a while -- just have not had the time to do it myself. But I would certainly support an effort to do it. Something like, binary-versioned, maybe? - jeremy ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
