So there are two major problems with our KVP API: 1. Users are not able to serialize their own data types, and are instead forced to use built-in types like String and Vector. 2. If the user builds their code with a different version of GWT than we did, and that version has different serialization IDs for certain types, the RPC will fail (bug #788918 <https://bugs.launchpad.net/mugle/+bug/788918> ).
A minor problem is also the warnings you get when compiling student code, due to being unable to serialize certain types. It was well-intentioned, but I think our KVP API has failed, and we need a more robust one. Unfortunately this will require changes to student code. Note that issue #2 can potentially affect the other APIs as well, but probably won't, since the other APIs use only primitive types and String, which are unlikely to change serialization IDs. (The problem we had today was because java.util.Vector's serialization ID changed.) So there are a couple of approaches we can take here (not necessarily mutually exclusive). *The problem with KVP serialization* It is important to note that the current KVP serialization is quite unnecessarily roundabout, given that the server is really just taking whatever the client gives it, shoving it in the database, and getting it out again. Currently, this is what happens: 1. The client takes an arbitrary object and serializes it using com.google.gwt.user.client.rpc.IsSerializable. 2. The client sends the stream to the server. 3. The server deserializes the stream, turning it into a Java object in memory. 4. The server serializes the object using java.io.Serializable. 5. The server sends the stream to the database for storage as a blob. Retrieving the data does this same process in reverse. What should be apparent is that the server is deserializing and immediately reserializing, yet it doesn't need to ever inspect the data. It is effectively treating the data as a dumb blob, yet it is unpacking and repacking. Most (but not all) of our nightmares happen during steps 3 and 4 -- the fact that the client and server have different versions is only an issue because the server deserializes. A much better approach would be: 1. The client takes an arbitrary object and serializes it using *some serialization framework*. 2. The client sends the stream to the server. 3. The server sends the stream to the database for storage as a blob. The question is what serialization framework to use. Using GWT-RPC seems a bit silly since it is no longer a client-server communication, it is just storage. I thought the original plan was to use java.io.Serializable on the server. But I'm still not sure if that works or not. And then there's JSON. I'll get to that later. Finally, note that our current requirement that KVP objects be in java.io.Serializable is pretty misleading, since we are currently using GWT-RPC instead. *Break away from GWT-RPC* Rather than basing our API on GWT-RPC, we can use a simpler, more standard serialization format (preferably JSON). This is what I wanted to do in the first place, but we decided to use GWT-RPC to simplify the interface for students. However, in hindsight, the interface would actually not need to change (ignoring the KVP which I'll get to later). This would just mean that we change the code in the Async classes we give to students. They have the same Async interface, but internally, they produce JSON on the client side, and send it to the server, which reads JSON, and sends JSON back. The KVP service is a bit special, since it currently allows arbitrary data types, but is broken. Switching to JSON would not break it further, as we could still perform arbitrary serialization on the client side *before*putting it into JSON. In fact, that might improve things somewhat, and even potentially allow user data types to be serialised, and also fix bug #788918<https://bugs.launchpad.net/mugle/+bug/788918>, since there would be no deserialization on the server at all. I will address this later, since I still don't think that's good enough, but just a note that JSON shouldn't make this situation worse. The benefits of this would be: - The actual network service would be a proper RESTful API. We could use GET and POST appropriately, whereas right now we only do POSTs even when receiving data. - Easier to debug. You can manually put in requests and see the results in a human-readable way. - Possibility of allowing non-GWT games on the platform. GWT would no longer be necessary; it would just be a recommended framework. Essentially, MUGLE would become a generic JavaScript platform with a REST API, and we would supply a GWT binding to that API to allow you to write your JavaScript code in GWT. - Absolutely fixes bug #788918<https://bugs.launchpad.net/mugle/+bug/788918>-- no versioning errors, since we completely control the serialization format. - Source-compatible with existing code. We would still offer the same Async API. The downsides: - Not binary-compatible with existing code. All projects on MUGLE would need to recompile, because there is client code in the student bundle that talks GWT-RPC. Thus it would be better to do this before we put lots more games up. - Need to change the KVP serialization format to something else, since we won't be using GWT-RPC (but that's something we should do anyway). So I think this would be a good thing to do right away. *Fixing the KVP value serialization for arbitrary student classes* Now, the matter of serializing arbitrary student data. So firstly, I would like to have another go at using java.io.Serializable on the client. I think we tried to use it on the server before but not the client. But from what I have read, that may not work. I don't think GWT's client implements this properly. Still, we should see exactly how it fails. If it works (that is, we can take an arbitrary class which implements Serializable, and turn it into a binary blob and back, without causing warnings), then we can just use that, without any changes to the API. After applying the changes suggested above, KeyValueServiceAsync will no longer be a special GWT class; it will be our class and we will be able to put code in it. Therefore, we will put code in it to serialize on the client, then send that serialized blob in the form of a JSON string. Note that I am *not*(yet) talking about encoding student data as a JSON string (since that is not possible in the general case without reflection) -- I am talking about serializing to a binary blob and sending that blob wrapped in a JSON string as a transport format. Again, this is fine, because while the server *will*be decoding the JSON objects used by the API, we *do not* want it to be decoding the blobs of the object data (steps 3 and 4 above), we just want it to store it in the database, and retrieve it again. Another possibility is that we use some other reflection framework. A quick google shows that there is someone claiming to have browser-side reflection working in GWT (GWT Reflection <http://gwtreflection.sourceforge.net/>). If this works, then it means we can write *our own* custom serialization mechanism which takes any Object and converts it into a String or byte[] -- for debugging purposes, a JSON format would be handy (but confusing -- our requests would have a JSON object encoded as a string inside another JSON object!). We would have special cases for primitive types, arrays, and other important types such as String, Vector and HashMap, and for all other classes we would just recursively serialize all of the fields as a JSON object. One tricky point is reconstructing the objects -- we would either need to store a lot of dynamic type information in our serialized format, or ask the user to supply the Class object to convert it back to. But if it doesn't work, we will need to do something more drastic. We can change the put/get interface so that rather than taking a java.io.Serializable, we take a com.google.gwt.json.client.JSONValue<http://google-web-toolkit.googlecode.com/svn/javadoc/latest/index.html?com/google/gwt/json/client/JSONValue.html>. This type can store arbitrary data by recursive arrays and dictionaries, and primitive strings, booleans and numbers, and can be easily serialized and deserialized. However, it would require work on the part of the students to set it up. We should probably then supply some other useful helper methods. We could have putLong, putDouble, putBoolean, putString and equivalent getters, which just take ordinary Java values and turn them into JSON primitives. That would allow most students to fix up their code just by calling the appropriate put/get methods. The only hard cases would be where students are currently serializing Vectors; they would need to manually turn them into JSONArrays and back (we can't do this automatically, since we don't know what the types of the objects inside the Vector are, unless we had a putVectorLong, putVectorDouble, etc, which would be awful). So this would be a bit of work for students. Still, it may be worth it, but I'm hoping one of the other two methods works. The last option is we just change put and get to take a good ol' String. Students can handle serialization themselves. In most cases, this will be trivial: Integer.toString/parseInt and the other primitives will work fine. Strings will not need to change. Vectors, students can just do something simple like insert commas between them to put them in, and call split(",") to get them out again. But that is certainly not as rich an API as we originally promised. Thoughts on all of this?
-- Mailing list: https://launchpad.net/~mugle-dev Post to : [email protected] Unsubscribe : https://launchpad.net/~mugle-dev More help : https://help.launchpad.net/ListHelp
