Re: [Opensim-dev] Standardizing OpenSim data formats
Hurliman, John wrote: Thank you for taking a look. Looks like I all of the ExtraData for prims, including sculpt/particles/flexi/etc. I'll see what OpenSim is doing for saving script state and add everything in a second pass. I'm hoping that this particular format captures everything needed to save simulator state to disk for OpenSim and load it back up, while being easy to create and parse by third party tools. Additional formats could be made later for more specific tasks such as saving assets offline (where things like InventorySerial definitely wouldn't matter). There could be a couple of different approaches to distinguishing between different content types. OAR loading is already doing this for terrain with a switch statement on the file extension. That works until two separate formats both decide to use the .prim or .xml extensions. Another possibility would be to require that all object serializations use the same format so a MIME type could be read from metadata embedded in the file. Using the OAR index file to store a list of filenames and content types could also work. This breaks some of the nice things about OAR files though, such as opening one up in a file viewer and dropping some content in without using special tools. Yeah, being able to drop content in with existing tools or produce archives from scratch was one of the attempted aims with OAR, so inserting extra information in an index wouldn't be an enjoyable solution. I guess using the terrain extension for identification isn't too bad as long as the format is centered around OpenSim (I'm not sure how far beyond that one can go since it is a mongrel format). But I can definitely see the problem with different object serializations, if such a thing comes about (as opposed to extensions to a single object serialization). Inserting mime type metadata in files is a solution that sounds good to me. -- justincc Justin Clark-Casey http://justincc.wordpress.com ___ Opensim-dev mailing list Opensim-dev@lists.berlios.de https://lists.berlios.de/mailman/listinfo/opensim-dev
Re: [Opensim-dev] Standardizing OpenSim data formats
Hurliman, John wrote: Currently, OpenSim server-server communication is primarily made up of .NET XML serialization. As a side effect of this, I've seen the wire format between simulator and grid services or grid service to grid service change many times since I started working on backend services. The OAR file format has also changed at least once while retaining the same version number because the underlying prim XML format changed. I am working on a libomv OAR reader/writer at the moment, and trying to deal with hacks for supporting pre and post change OAR files. OAR was intended more as a packaging mechanism than a complete format. This is partly why I didn't bump the version number when the serialization format changed. In principle, I think it would be good if the serializations themselves identified themselves as one format or another (as I think you've already indicated in a previous post). Then one could have different serializations (e.g. mesh) without having to produce different OAR structures or different formats entirely. I'm not sure how pragmatic this is - perhaps some indication of the format of serialization could be placed in the OAR control file (though even then there might be regions containing multiple different serializations). The XML serialization 'XML2' format was used for objects in the OAR partly just because it was already there and it worked, even if it is pretty horrible. I feel that one nasty point about it for offline content transfer is that it doesn't contain any versioning information to identify it (necessitating heuristics I guess). I feel that there may also come a point where an offline format will diverge from an on-the-wire format (e.g. inventory serial seems unncessary for offline purposes - though many other things will remain for the backup use case as opposed to the transfer to some one else use case). To increase compatibility between versions of OpenSim and with third party services I'd like to propose a standardization of (eventually) all over-the-wire communication in OpenSim. I started with an attempt at an LLIDL definition for primitives: http://opensimulator.org/wiki/PrimitiveFormatProposal Is anything missing? Does this look like a reasonable start? On a quick look through, I don't see any attribute for sculpt information. At the moment this is stored inline in the serialization. Also, I suspect state information for scripts may have crept in recently (not sure). Also, at least for archiving purposes, the creator id will at some point become a string rather than a uuid in order to allow identification information to be retrieved on grids other than the one on which the archive was saved. -- justincc Justin Clark-Casey http://justincc.wordpress.com ___ Opensim-dev mailing list Opensim-dev@lists.berlios.de https://lists.berlios.de/mailman/listinfo/opensim-dev
Re: [Opensim-dev] Standardizing OpenSim data formats
Thank you for taking a look. Looks like I all of the ExtraData for prims, including sculpt/particles/flexi/etc. I'll see what OpenSim is doing for saving script state and add everything in a second pass. I'm hoping that this particular format captures everything needed to save simulator state to disk for OpenSim and load it back up, while being easy to create and parse by third party tools. Additional formats could be made later for more specific tasks such as saving assets offline (where things like InventorySerial definitely wouldn't matter). There could be a couple of different approaches to distinguishing between different content types. OAR loading is already doing this for terrain with a switch statement on the file extension. That works until two separate formats both decide to use the .prim or .xml extensions. Another possibility would be to require that all object serializations use the same format so a MIME type could be read from metadata embedded in the file. Using the OAR index file to store a list of filenames and content types could also work. This breaks some of the nice things about OAR files though, such as opening one up in a file viewer and dropping some content in without using special tools. John -Original Message- From: opensim-dev-boun...@lists.berlios.de [mailto:opensim-dev- boun...@lists.berlios.de] On Behalf Of Justin Clark-Casey Sent: Wednesday, April 29, 2009 10:44 AM To: opensim-dev@lists.berlios.de Subject: Re: [Opensim-dev] Standardizing OpenSim data formats Hurliman, John wrote: Currently, OpenSim server-server communication is primarily made up of .NET XML serialization. As a side effect of this, I've seen the wire format between simulator and grid services or grid service to grid service change many times since I started working on backend services. The OAR file format has also changed at least once while retaining the same version number because the underlying prim XML format changed. I am working on a libomv OAR reader/writer at the moment, and trying to deal with hacks for supporting pre and post change OAR files. OAR was intended more as a packaging mechanism than a complete format. This is partly why I didn't bump the version number when the serialization format changed. In principle, I think it would be good if the serializations themselves identified themselves as one format or another (as I think you've already indicated in a previous post). Then one could have different serializations (e.g. mesh) without having to produce different OAR structures or different formats entirely. I'm not sure how pragmatic this is - perhaps some indication of the format of serialization could be placed in the OAR control file (though even then there might be regions containing multiple different serializations). The XML serialization 'XML2' format was used for objects in the OAR partly just because it was already there and it worked, even if it is pretty horrible. I feel that one nasty point about it for offline content transfer is that it doesn't contain any versioning information to identify it (necessitating heuristics I guess). I feel that there may also come a point where an offline format will diverge from an on-the- wire format (e.g. inventory serial seems unncessary for offline purposes - though many other things will remain for the backup use case as opposed to the transfer to some one else use case). To increase compatibility between versions of OpenSim and with third party services I'd like to propose a standardization of (eventually) all over-the-wire communication in OpenSim. I started with an attempt at an LLIDL definition for primitives: http://opensimulator.org/wiki/PrimitiveFormatProposal Is anything missing? Does this look like a reasonable start? On a quick look through, I don't see any attribute for sculpt information. At the moment this is stored inline in the serialization. Also, I suspect state information for scripts may have crept in recently (not sure). Also, at least for archiving purposes, the creator id will at some point become a string rather than a uuid in order to allow identification information to be retrieved on grids other than the one on which the archive was saved. -- justincc Justin Clark-Casey http://justincc.wordpress.com ___ Opensim-dev mailing list Opensim-dev@lists.berlios.de https://lists.berlios.de/mailman/listinfo/opensim-dev ___ Opensim-dev mailing list Opensim-dev@lists.berlios.de https://lists.berlios.de/mailman/listinfo/opensim-dev
Re: [Opensim-dev] Standardizing OpenSim data formats
Sculpt/flexi/light are probably stored in the ExtraParams byte array. The routines for deserializing that are in PrimitiveBaseShape. Unfortunately this is one of those 'legacy' things that ended up that way.Someone, a long time ago, exposed these options as the ExtraParams byte array and persisted it directly. The two important methods to use for this are: void PrimitiveBaseShape.ReadInExtraPramsBytes (byte[]), and byte[] PrimitiveBaseShape.ExtraparamsToBytes() At the time, changing it would have broken all existing inventory. I think particles have their own byte array. Best Regards Teravus On 4/29/09, Hurliman, John john.hurli...@intel.com wrote: Thank you for taking a look. Looks like I all of the ExtraData for prims, including sculpt/particles/flexi/etc. I'll see what OpenSim is doing for saving script state and add everything in a second pass. I'm hoping that this particular format captures everything needed to save simulator state to disk for OpenSim and load it back up, while being easy to create and parse by third party tools. Additional formats could be made later for more specific tasks such as saving assets offline (where things like InventorySerial definitely wouldn't matter). There could be a couple of different approaches to distinguishing between different content types. OAR loading is already doing this for terrain with a switch statement on the file extension. That works until two separate formats both decide to use the .prim or .xml extensions. Another possibility would be to require that all object serializations use the same format so a MIME type could be read from metadata embedded in the file. Using the OAR index file to store a list of filenames and content types could also work. This breaks some of the nice things about OAR files though, such as opening one up in a file viewer and dropping some content in without using special tools. John -Original Message- From: opensim-dev-boun...@lists.berlios.de [mailto:opensim-dev- boun...@lists.berlios.de] On Behalf Of Justin Clark-Casey Sent: Wednesday, April 29, 2009 10:44 AM To: opensim-dev@lists.berlios.de Subject: Re: [Opensim-dev] Standardizing OpenSim data formats Hurliman, John wrote: Currently, OpenSim server-server communication is primarily made up of .NET XML serialization. As a side effect of this, I've seen the wire format between simulator and grid services or grid service to grid service change many times since I started working on backend services. The OAR file format has also changed at least once while retaining the same version number because the underlying prim XML format changed. I am working on a libomv OAR reader/writer at the moment, and trying to deal with hacks for supporting pre and post change OAR files. OAR was intended more as a packaging mechanism than a complete format. This is partly why I didn't bump the version number when the serialization format changed. In principle, I think it would be good if the serializations themselves identified themselves as one format or another (as I think you've already indicated in a previous post). Then one could have different serializations (e.g. mesh) without having to produce different OAR structures or different formats entirely. I'm not sure how pragmatic this is - perhaps some indication of the format of serialization could be placed in the OAR control file (though even then there might be regions containing multiple different serializations). The XML serialization 'XML2' format was used for objects in the OAR partly just because it was already there and it worked, even if it is pretty horrible. I feel that one nasty point about it for offline content transfer is that it doesn't contain any versioning information to identify it (necessitating heuristics I guess). I feel that there may also come a point where an offline format will diverge from an on-the- wire format (e.g. inventory serial seems unncessary for offline purposes - though many other things will remain for the backup use case as opposed to the transfer to some one else use case). To increase compatibility between versions of OpenSim and with third party services I'd like to propose a standardization of (eventually) all over-the-wire communication in OpenSim. I started with an attempt at an LLIDL definition for primitives: http://opensimulator.org/wiki/PrimitiveFormatProposal Is anything missing? Does this look like a reasonable start? On a quick look through, I don't see any attribute for sculpt information. At the moment this is stored inline in the serialization. Also, I suspect state information for scripts may have crept in recently (not sure). Also, at least for archiving purposes, the creator id will at some point become a string rather than a uuid in order to allow identification information to be retrieved on grids other than the one on which the archive
Re: [Opensim-dev] Standardizing OpenSim data formats
The parser needs to not barf if it sees an unknown element in there. Eg, ModRex mesh parameters, etc. Ideally the SaveToXYZ function should fire an event on the object called something like say 'OnSaveRequest(out List??? extraData)' which allows other modules to save additional data. On-load/On-construct should probably also fire an event which allows the data to be restored. Adam -Original Message- From: opensim-dev-boun...@lists.berlios.de [mailto:opensim-dev- boun...@lists.berlios.de] On Behalf Of Hurliman, John Sent: Tuesday, 28 April 2009 4:32 PM To: opensim-dev@lists.berlios.de Subject: [Opensim-dev] Standardizing OpenSim data formats Currently, OpenSim server-server communication is primarily made up of .NET XML serialization. As a side effect of this, I've seen the wire format between simulator and grid services or grid service to grid service change many times since I started working on backend services. The OAR file format has also changed at least once while retaining the same version number because the underlying prim XML format changed. I am working on a libomv OAR reader/writer at the moment, and trying to deal with hacks for supporting pre and post change OAR files. To increase compatibility between versions of OpenSim and with third party services I'd like to propose a standardization of (eventually) all over-the-wire communication in OpenSim. I started with an attempt at an LLIDL definition for primitives: http://opensimulator.org/wiki/PrimitiveFormatProposal Is anything missing? Does this look like a reasonable start? John ___ Opensim-dev mailing list Opensim-dev@lists.berlios.de https://lists.berlios.de/mailman/listinfo/opensim-dev ___ Opensim-dev mailing list Opensim-dev@lists.berlios.de https://lists.berlios.de/mailman/listinfo/opensim-dev
Re: [Opensim-dev] Standardizing OpenSim data formats
This is fine if your use case is to consider the prim format I posted as a canonical format for all virtual world objects, where everything will only be plus or minus a few fields from that format. I don't think that's going to be the case moving forward. I would prefer to see content-types attached to formats. A Second Life prim is recognized as application/vnd.ll.primitive, while a realXtend mesh might be application/x-realxtend-mesh (or something to that effect). That way you can add support in the future for objects that share very little in common with Second Life prims. It also fits nicely into larger asset framework where you can do this type of standardization for notecards, clothing assets, etc. John -Original Message- From: opensim-dev-boun...@lists.berlios.de [mailto:opensim-dev- boun...@lists.berlios.de] On Behalf Of Frisby, Adam Sent: Tuesday, April 28, 2009 4:56 PM To: opensim-dev@lists.berlios.de Subject: Re: [Opensim-dev] Standardizing OpenSim data formats The parser needs to not barf if it sees an unknown element in there. Eg, ModRex mesh parameters, etc. Ideally the SaveToXYZ function should fire an event on the object called something like say 'OnSaveRequest(out List??? extraData)' which allows other modules to save additional data. On-load/On- construct should probably also fire an event which allows the data to be restored. Adam -Original Message- From: opensim-dev-boun...@lists.berlios.de [mailto:opensim-dev- boun...@lists.berlios.de] On Behalf Of Hurliman, John Sent: Tuesday, 28 April 2009 4:32 PM To: opensim-dev@lists.berlios.de Subject: [Opensim-dev] Standardizing OpenSim data formats Currently, OpenSim server-server communication is primarily made up of .NET XML serialization. As a side effect of this, I've seen the wire format between simulator and grid services or grid service to grid service change many times since I started working on backend services. The OAR file format has also changed at least once while retaining the same version number because the underlying prim XML format changed. I am working on a libomv OAR reader/writer at the moment, and trying to deal with hacks for supporting pre and post change OAR files. To increase compatibility between versions of OpenSim and with third party services I'd like to propose a standardization of (eventually) all over-the-wire communication in OpenSim. I started with an attempt at an LLIDL definition for primitives: http://opensimulator.org/wiki/PrimitiveFormatProposal Is anything missing? Does this look like a reasonable start? John ___ Opensim-dev mailing list Opensim-dev@lists.berlios.de https://lists.berlios.de/mailman/listinfo/opensim-dev ___ Opensim-dev mailing list Opensim-dev@lists.berlios.de https://lists.berlios.de/mailman/listinfo/opensim-dev ___ Opensim-dev mailing list Opensim-dev@lists.berlios.de https://lists.berlios.de/mailman/listinfo/opensim-dev
Re: [Opensim-dev] Standardizing OpenSim data formats
I see merit of a prim type definition, but then again I believe the oar format may have evolved as a mirror of the schema for the databases used by OpenSim. It may complicate future code if the oar format deviates substantially from these databases. Perhaps the format should reflect the easiest way to transfer content from one OpenSim region to another, regardless of content? On Tue, Apr 28, 2009 at 5:24 PM, Hurliman, John john.hurli...@intel.comwrote: This is fine if your use case is to consider the prim format I posted as a canonical format for all virtual world objects, where everything will only be plus or minus a few fields from that format. I don't think that's going to be the case moving forward. I would prefer to see content-types attached to formats. A Second Life prim is recognized as application/vnd.ll.primitive, while a realXtend mesh might be application/x-realxtend-mesh (or something to that effect). That way you can add support in the future for objects that share very little in common with Second Life prims. It also fits nicely into larger asset framework where you can do this type of standardization for notecards, clothing assets, etc. John -Original Message- From: opensim-dev-boun...@lists.berlios.de [mailto:opensim-dev- boun...@lists.berlios.de] On Behalf Of Frisby, Adam Sent: Tuesday, April 28, 2009 4:56 PM To: opensim-dev@lists.berlios.de Subject: Re: [Opensim-dev] Standardizing OpenSim data formats The parser needs to not barf if it sees an unknown element in there. Eg, ModRex mesh parameters, etc. Ideally the SaveToXYZ function should fire an event on the object called something like say 'OnSaveRequest(out List??? extraData)' which allows other modules to save additional data. On-load/On- construct should probably also fire an event which allows the data to be restored. Adam -Original Message- From: opensim-dev-boun...@lists.berlios.de [mailto:opensim-dev- boun...@lists.berlios.de] On Behalf Of Hurliman, John Sent: Tuesday, 28 April 2009 4:32 PM To: opensim-dev@lists.berlios.de Subject: [Opensim-dev] Standardizing OpenSim data formats Currently, OpenSim server-server communication is primarily made up of .NET XML serialization. As a side effect of this, I've seen the wire format between simulator and grid services or grid service to grid service change many times since I started working on backend services. The OAR file format has also changed at least once while retaining the same version number because the underlying prim XML format changed. I am working on a libomv OAR reader/writer at the moment, and trying to deal with hacks for supporting pre and post change OAR files. To increase compatibility between versions of OpenSim and with third party services I'd like to propose a standardization of (eventually) all over-the-wire communication in OpenSim. I started with an attempt at an LLIDL definition for primitives: http://opensimulator.org/wiki/PrimitiveFormatProposal Is anything missing? Does this look like a reasonable start? John ___ Opensim-dev mailing list Opensim-dev@lists.berlios.de https://lists.berlios.de/mailman/listinfo/opensim-dev ___ Opensim-dev mailing list Opensim-dev@lists.berlios.de https://lists.berlios.de/mailman/listinfo/opensim-dev ___ Opensim-dev mailing list Opensim-dev@lists.berlios.de https://lists.berlios.de/mailman/listinfo/opensim-dev ___ Opensim-dev mailing list Opensim-dev@lists.berlios.de https://lists.berlios.de/mailman/listinfo/opensim-dev
