> > > We do not need to define our own serialization system, nor do we need to > > Choose The One True Serialization Encoding To Rule Them All (TM)-- we > > just need to allow for enough information to identify what the data is. > > > > Aaron > > > > I think that's what he's getting at. > > Hmm. I wonder... Instead of just one serialization flag, we could have > two. > > The first, existing, one would mean "native language". Ruby marshal, > perl Storable, PHP ... whatever the hell. > Second would be 'application serialized', which it'd then run a client > callback as Tomash suggested. The client callback can use entirely its > own scheme to serialize/deserialize the structures (run it through > thrift, json, whatever).
I guess Tomash' point is (to put words in his mouth perhaps against his will ;) ) that there is no actual need to do anything other than agree on the value of the 'serialisation' flag and not abuse it anywhere, then encourage the client authors to provide (de)serialisation hooks (for example the Spy.Memcached Java client provides a 'transcoder' api to do arbritary encoding and de-coding of data). The clients could come configured by default to use their current 'native' (de)serialisers, but could be overriden by developers using the memcached platform to do their own application specific (de)serialisation. As an extension to this I still think it would be sensible to agree on some 80% fit cross-platform (de)serialisation format so when working in a mixed platform (which I have the dubious fortune to be myself) it would be possible to swap in one of these 'cross platform (de)serialisers/transcoders' to 'guarantee' the data [or a minimal set thereof] will work between clients. If the compressed + serialised flags were agreed on uniformly (compression flag pretty much is! [see the spreadsheet on the wiki and the previous url] ) then it would be a relatively trivial matter of sorting out how the serialisation occurs, the problem at the minute is people use the flags to express more information about exactly what has been serialised, and these flag values conflict with other client's interpretations which [for me at least] causes a fair amount of pain! Since trying to re-ignite this discussion I've meandered across other problems with getting cross-platform clients working, the biggy for me is implementations of the consistent hashing algorithm differ between clients meaning the same key for different platforms can end up hashed against different memcached nodes :(. Still working on that one [I have now got Enyim.Memcached (C#) and Spy.memcached (Java) working together successfully for UTF8 strings, with a minor tweak to the C# code and some custom code plugged into the Java client] *sigh* > > Then all clients would "standardize" on some familiar API for this. > ... and potentially provide example snippits that would use JSON. JSON does seem to be a popular choice, but I'm not sure how well this would work in practice, i.e. how do you differentiate between a byte array and a character array, or do we just not care etc. etc. (perhaps thats easy to solve and my knowledge is lacking, quite probably in fact ;) ) > > It also means the current defaults will continue to work. In a bulk of > cases you'd _want_ to just use the native method, since it's likely > pretty fast and well enough tested. I think Tomash' solution allows for the current defaults to continue to work, and for an optional 'cross platformish' (de)serialiser to be agreed amongst the client authors, however as far as I can tell there is no official communication channel to support cross-talk between the client authors other than this list (which fairly obviously is server orientated :) ) > > > Which is very close to what ya'll seem to be talking about :) Maybe? > > -Dormando > - ciaran
