Re: [Opensim-dev] .net serialization of SOGs
On Mon, Oct 11, 2010 at 10:40 AM, Toni Alatalo ant...@kyperjokki.fi wrote:
On su, 2010-10-10 at 17:06 -0700, d...@metaverseink.com wrote:
Adam ping!
How's the SOG/SOP refactor coming along? This is coming very close to
the need for the refactoring...
Yes, as I think was even written in the refactor plan, a goal there was
to get Collada saveload support. So that the scene code is first
refactored to use components, and then a new serialization system made
that can do many formats. First .net serialization, but then Collada
too.
All I know is that Adam has put the very first steps to the soprefactor
branch, a little addition that allows to store arbitrary data in
components in a map attached to prims. The refactor is not done, the
hardcoded prim data is still there etc. like before. With the refactor
the idea is that all data would be in smaller components, for example
the prim shape data is in an (optional) prim component etc. It seems
that Adam is so be busy with other things, that if we want the refactor
implemented someone else should probably step up and do it.
About the formats, yes, something that is not an autogenerated dump of
the internal code structures that AFAIK every dev wishes that would be
changed (sog/sop) would be good. A year ago I almost started
implementing OAR readwrite support to Blender, but then when actually
looked at the format decided otherwise and looked at the Collada scene
spec which seemed quite sane. We've since ended up using Ogre .scene
files 'cause it was most straightforward for now (dead simple xml scene
tree, http://www.ogre3d.org/tikiwiki/DotScene).
One suggestion from the talk we had on irc on #opensim-dev now:
k\o\w are you using collada to wrap entire scenes or for individual
meshes?
antont entire scenes
k\o\w I agree that OAR encapsulation should be replaced with collada
k\o\w but that the contained prim data still needs a saner format
like HPA
09:49 antont k\o\w: agreed
We are currently using Collada only when testing WebGL rendering, by
exporting old scenes from Blender that we've previously exported to
opensim+modrex using Ogre .scene and .mesh etc, to Collada .dae that
GLGE can load. But that is giving some more experience with the format
at least, and so far seems good. Of course it is not suitable out of the
box for OAR like usage where we need all the opensim specific bits, like
permissions etc. So perhaps Collada is for importexport, not for backup
and internal things. Unless OAR would be an extension to it somehow, the
x is for extensible after all :) . So I guess your plan to do continue
with the .net xml format but with more control is good.
I'm now about to diverge from your original question and the exact task
at hand, but perhaps it is helpful to explain a bit what we've done now
with scene formats and storing data to opensim to know what kind of
requirements and solutions there are around:
The goal that we have had for which would need the sog/sop refactor,
allowing storage of arbitrary data in scenes, we've implemented in
ModRex as a temporary measure so that the client development has been
able to continue.
The first solution made in March reused an old rex thing that had been
there forever, RexFreeData, which allowed storing max 1024 chars of text
attached to an object (to give params to scripts and for them to store
vals etc). That first impl, used in Naali 0.2 0.3 series, just stores
custom scene entity data as xml in that field and that works also
against the old pre-naalitaiga realxtend servers 'cause that freedata
mechanism was there already. But it is limited to that 1024 bytes
('cause uses a single generic message for synching all data for an
entity, and in lludp messages are max 1024 long).
Now, out since last week, there is a new implemention for custom scene
data storage in modrex. It has tables for entities, components and
attributes and new messages for doing per-attribute synchronization etc.
This is out in Taiga 0.2.0 rc1, but not implemented in Naali yet (I
guess the guys do the client side soon now?). Because it does per-attr
sync, I think is limited at 1024 for the size of an individual attr, but
doesn't have a limit for the aggregated size of all attributes of all
components that are attached to an entity. So should work for pretty
much everything.
Small correction in here. We have new table only for component data, not for
entities or attributes. And we have messages for synchronization of
individual components, not attributes. As said, these have now been
implemented in ModreX only, but will be implemented in Naali soonish. The
size of the component is only limited by the maximum capacity of the udp
generic message.
Cheers,
Mikko
We hope that this kind of functionality becomes supported by Opensim
core so that also the data of other applications, and not only the data
specified by the Second Life (tm) application, gets stored in
Re: [Opensim-dev] .net serialization of SOGs
On su, 2010-10-10 at 17:06 -0700, d...@metaverseink.com wrote:
Adam ping!
How's the SOG/SOP refactor coming along? This is coming very close to
the need for the refactoring...
Yes, as I think was even written in the refactor plan, a goal there was
to get Collada saveload support. So that the scene code is first
refactored to use components, and then a new serialization system made
that can do many formats. First .net serialization, but then Collada
too.
All I know is that Adam has put the very first steps to the soprefactor
branch, a little addition that allows to store arbitrary data in
components in a map attached to prims. The refactor is not done, the
hardcoded prim data is still there etc. like before. With the refactor
the idea is that all data would be in smaller components, for example
the prim shape data is in an (optional) prim component etc. It seems
that Adam is so be busy with other things, that if we want the refactor
implemented someone else should probably step up and do it.
About the formats, yes, something that is not an autogenerated dump of
the internal code structures that AFAIK every dev wishes that would be
changed (sog/sop) would be good. A year ago I almost started
implementing OAR readwrite support to Blender, but then when actually
looked at the format decided otherwise and looked at the Collada scene
spec which seemed quite sane. We've since ended up using Ogre .scene
files 'cause it was most straightforward for now (dead simple xml scene
tree, http://www.ogre3d.org/tikiwiki/DotScene).
One suggestion from the talk we had on irc on #opensim-dev now:
k\o\w are you using collada to wrap entire scenes or for individual
meshes?
antont entire scenes
k\o\w I agree that OAR encapsulation should be replaced with collada
k\o\w but that the contained prim data still needs a saner format
like HPA
09:49 antont k\o\w: agreed
We are currently using Collada only when testing WebGL rendering, by
exporting old scenes from Blender that we've previously exported to
opensim+modrex using Ogre .scene and .mesh etc, to Collada .dae that
GLGE can load. But that is giving some more experience with the format
at least, and so far seems good. Of course it is not suitable out of the
box for OAR like usage where we need all the opensim specific bits, like
permissions etc. So perhaps Collada is for importexport, not for backup
and internal things. Unless OAR would be an extension to it somehow, the
x is for extensible after all :) . So I guess your plan to do continue
with the .net xml format but with more control is good.
I'm now about to diverge from your original question and the exact task
at hand, but perhaps it is helpful to explain a bit what we've done now
with scene formats and storing data to opensim to know what kind of
requirements and solutions there are around:
The goal that we have had for which would need the sog/sop refactor,
allowing storage of arbitrary data in scenes, we've implemented in
ModRex as a temporary measure so that the client development has been
able to continue.
The first solution made in March reused an old rex thing that had been
there forever, RexFreeData, which allowed storing max 1024 chars of text
attached to an object (to give params to scripts and for them to store
vals etc). That first impl, used in Naali 0.2 0.3 series, just stores
custom scene entity data as xml in that field and that works also
against the old pre-naalitaiga realxtend servers 'cause that freedata
mechanism was there already. But it is limited to that 1024 bytes
('cause uses a single generic message for synching all data for an
entity, and in lludp messages are max 1024 long).
Now, out since last week, there is a new implemention for custom scene
data storage in modrex. It has tables for entities, components and
attributes and new messages for doing per-attribute synchronization etc.
This is out in Taiga 0.2.0 rc1, but not implemented in Naali yet (I
guess the guys do the client side soon now?). Because it does per-attr
sync, I think is limited at 1024 for the size of an individual attr, but
doesn't have a limit for the aggregated size of all attributes of all
components that are attached to an entity. So should work for pretty
much everything.
We hope that this kind of functionality becomes supported by Opensim
core so that also the data of other applications, and not only the data
specified by the Second Life (tm) application, gets stored in OARs and
works when rezzing things from inventory etc. And here the application
does not mean some realXtend impl, but what *you* or any user defines as
the components attributes of your application. For example, I have a
minimal test component 'Door' with two boolean attributes: opened/closed
locked/unlocked. That works on the platform, but the platform doesn't
need to know anything about doors (in fact is not allowed to know, but
has to be generic and allow any kind of data).
In the spirit of what Melanie wrote about LLUDP
Re: [Opensim-dev] .net serialization of SOGs
On ma, 2010-10-11 at 10:40 +0300, Toni Alatalo wrote: Now, out since last week, there is a new implemention for custom scene data storage in modrex. It has tables for entities, components and attributes and new messages for doing per-attribute synchronization etc. This is out in Taiga 0.2.0 rc1, but not implemented in Naali yet (I Oh BTW a reason for doing that is that there was a need to store custom runtime information about avatars (scenepresences) in entity-components too, so neither the Adam's map in SceneObjectPart in the branch nor the old rexfreedata thing (also for prims only) worked. I guess OARs don't need to save runtime scenepresence info though :) ~Toni ___ Opensim-dev mailing list Opensim-dev@lists.berlios.de https://lists.berlios.de/mailman/listinfo/opensim-dev
Re: [Opensim-dev] .net serialization of SOGs
After thinking about it, I think I know the way out of our dependence on .net serialization. I'm planning to do this [today], so if anyone has concerns, please speak up. The main problem with what we have is that the serialization is being done automagically by .net via reflection, leaving us completely out of the loop and at the mercy of the nasty .net serialization restrictions. So the smooth transition out of this is to reproduce the serialization strings that .net is producing, but via our own code. That will be stage 1, and will ensure 100% compatibility with the data that already exists. But that will give us the ground to then do the right things with serialization, like adding a MIME type and a version number to whatever is being serialized. Luckily jhurliman has already done most of it in libomv, probably as a consequence of him pulling his hair when he encountered the SOG serialization in OpenSim... I'm undecided on whether to reuse that code as-is or copy-paste-change it, since I'm planning to add more fields, and context, to the SOG serializations. Without getting into flame wars between OpenSim and libomv, any thoughts on this, from a pragmatic point of view? On 10/9/2010 12:39 PM, Dahlia Trimble wrote: How about the various OSD serializations in libomv? They seem to be pretty robust these days. On Sat, Oct 9, 2010 at 11:05 AM, d...@metaverseink.com mailto:d...@metaverseink.com wrote: Dear devs, I'm pulling my hair here with the serialization of scene objects. This is horrible! Using .Net serialization for something as important as this is the worst idea ever. We're completely frozen. I'm trying to add an additional field for the creator info, and I'm stuck in all sorts of ways: I can't compute the value of that field at serialization-time, and then there are the issues of compatibility with previous versions of the SOP class. I'd like to understand all the implications of doing an entirely different serialization procedure, one that does not use reflection, and that allows for processing-during-serialization. What will break, and therefore needs fixing? I understand all archiving related to previous versions will break, so we need to keep supporting the existing serialization 'disease' for the foreseeable future. I also understand that TPs/crossings between sims that talk different SOG serializations will break, but that's not so bad. Anything else I should be aware of before I go off and redo this? Diva / Crista ___ Opensim-dev mailing list Opensim-dev@lists.berlios.de mailto:Opensim-dev@lists.berlios.de https://lists.berlios.de/mailman/listinfo/opensim-dev ___ Opensim-dev mailing list Opensim-dev@lists.berlios.de https://lists.berlios.de/mailman/listinfo/opensim-dev
Re: [Opensim-dev] .net serialization of SOGs
Copy/Paste/Change please. it's been a discussed and stated goal to reduce use of OMV classes not in the Types dll. Making ourselves dependent on OMV code for something as critical as this isn't right. There are lots of fields that will need to be added, from simple ones like sit data to complex ones like vehicle physics. Melanie d...@metaverseink.com wrote: After thinking about it, I think I know the way out of our dependence on .net serialization. I'm planning to do this [today], so if anyone has concerns, please speak up. The main problem with what we have is that the serialization is being done automagically by .net via reflection, leaving us completely out of the loop and at the mercy of the nasty .net serialization restrictions. So the smooth transition out of this is to reproduce the serialization strings that .net is producing, but via our own code. That will be stage 1, and will ensure 100% compatibility with the data that already exists. But that will give us the ground to then do the right things with serialization, like adding a MIME type and a version number to whatever is being serialized. Luckily jhurliman has already done most of it in libomv, probably as a consequence of him pulling his hair when he encountered the SOG serialization in OpenSim... I'm undecided on whether to reuse that code as-is or copy-paste-change it, since I'm planning to add more fields, and context, to the SOG serializations. Without getting into flame wars between OpenSim and libomv, any thoughts on this, from a pragmatic point of view? On 10/9/2010 12:39 PM, Dahlia Trimble wrote: How about the various OSD serializations in libomv? They seem to be pretty robust these days. On Sat, Oct 9, 2010 at 11:05 AM, d...@metaverseink.com mailto:d...@metaverseink.com wrote: Dear devs, I'm pulling my hair here with the serialization of scene objects. This is horrible! Using .Net serialization for something as important as this is the worst idea ever. We're completely frozen. I'm trying to add an additional field for the creator info, and I'm stuck in all sorts of ways: I can't compute the value of that field at serialization-time, and then there are the issues of compatibility with previous versions of the SOP class. I'd like to understand all the implications of doing an entirely different serialization procedure, one that does not use reflection, and that allows for processing-during-serialization. What will break, and therefore needs fixing? I understand all archiving related to previous versions will break, so we need to keep supporting the existing serialization 'disease' for the foreseeable future. I also understand that TPs/crossings between sims that talk different SOG serializations will break, but that's not so bad. Anything else I should be aware of before I go off and redo this? Diva / Crista ___ Opensim-dev mailing list Opensim-dev@lists.berlios.de mailto:Opensim-dev@lists.berlios.de https://lists.berlios.de/mailman/listinfo/opensim-dev ___ Opensim-dev mailing list Opensim-dev@lists.berlios.de https://lists.berlios.de/mailman/listinfo/opensim-dev ___ Opensim-dev mailing list Opensim-dev@lists.berlios.de https://lists.berlios.de/mailman/listinfo/opensim-dev
Re: [Opensim-dev] .net serialization of SOGs
I can see good arguments on both sides. On the one hand we (in OpenSim) need to have complete, and immediate control over this code, in order to be able to do things such as what I'm doing right now -- adding more information to those external data representations. On the other hand, control, in OpenSim, often comes with entanglement. SOG serialization, for example, is horribly tangled with Scenes at the moment, and that is a major block for fixing things like having the Library work as a service, etc. -- not to mention enabling 3rd party tools for manipulating content in a sane manner. So, in that sense, I really like the idea of having this code in a separate library, far away from OpenSim Scenes, which should merely be clients of such library. We have a place for it: OpenSim.Framework.Serialization. If we're to copy-paste-change libomv, I'll have to bring in the entire OpenMetaverse.Assets namespace, which is where all the serialization goodies are. Is there a middle ground? On 10/10/2010 8:30 AM, Melanie wrote: Copy/Paste/Change please. it's been a discussed and stated goal to reduce use of OMV classes not in the Types dll. Making ourselves dependent on OMV code for something as critical as this isn't right. There are lots of fields that will need to be added, from simple ones like sit data to complex ones like vehicle physics. Melanie d...@metaverseink.com wrote: After thinking about it, I think I know the way out of our dependence on .net serialization. I'm planning to do this [today], so if anyone has concerns, please speak up. The main problem with what we have is that the serialization is being done automagically by .net via reflection, leaving us completely out of the loop and at the mercy of the nasty .net serialization restrictions. So the smooth transition out of this is to reproduce the serialization strings that .net is producing, but via our own code. That will be stage 1, and will ensure 100% compatibility with the data that already exists. But that will give us the ground to then do the right things with serialization, like adding a MIME type and a version number to whatever is being serialized. Luckily jhurliman has already done most of it in libomv, probably as a consequence of him pulling his hair when he encountered the SOG serialization in OpenSim... I'm undecided on whether to reuse that code as-is or copy-paste-change it, since I'm planning to add more fields, and context, to the SOG serializations. Without getting into flame wars between OpenSim and libomv, any thoughts on this, from a pragmatic point of view? On 10/9/2010 12:39 PM, Dahlia Trimble wrote: How about the various OSD serializations in libomv? They seem to be pretty robust these days. On Sat, Oct 9, 2010 at 11:05 AM,d...@metaverseink.com mailto:d...@metaverseink.com wrote: Dear devs, I'm pulling my hair here with the serialization of scene objects. This is horrible! Using .Net serialization for something as important as this is the worst idea ever. We're completely frozen. I'm trying to add an additional field for the creator info, and I'm stuck in all sorts of ways: I can't compute the value of that field at serialization-time, and then there are the issues of compatibility with previous versions of the SOP class. I'd like to understand all the implications of doing an entirely different serialization procedure, one that does not use reflection, and that allows for processing-during-serialization. What will break, and therefore needs fixing? I understand all archiving related to previous versions will break, so we need to keep supporting the existing serialization 'disease' for the foreseeable future. I also understand that TPs/crossings between sims that talk different SOG serializations will break, but that's not so bad. Anything else I should be aware of before I go off and redo this? Diva / Crista ___ Opensim-dev mailing list Opensim-dev@lists.berlios.demailto:Opensim-dev@lists.berlios.de https://lists.berlios.de/mailman/listinfo/opensim-dev ___ Opensim-dev mailing list Opensim-dev@lists.berlios.de https://lists.berlios.de/mailman/listinfo/opensim-dev ___ Opensim-dev mailing list Opensim-dev@lists.berlios.de https://lists.berlios.de/mailman/listinfo/opensim-dev
Re: [Opensim-dev] .net serialization of SOGs
And I can see the back and forth that has been going on :) libomv got some code from OpenSim.Framework.Serialization... Would it be worth establishing a 3rd project for this? One where both the OpenSim devs and the libomv devs would have commit rights? So that we minimize this copy-paste-change backforth... On 10/10/2010 10:24 AM, d...@metaverseink.com wrote: I can see good arguments on both sides. On the one hand we (in OpenSim) need to have complete, and immediate control over this code, in order to be able to do things such as what I'm doing right now -- adding more information to those external data representations. On the other hand, control, in OpenSim, often comes with entanglement. SOG serialization, for example, is horribly tangled with Scenes at the moment, and that is a major block for fixing things like having the Library work as a service, etc. -- not to mention enabling 3rd party tools for manipulating content in a sane manner. So, in that sense, I really like the idea of having this code in a separate library, far away from OpenSim Scenes, which should merely be clients of such library. We have a place for it: OpenSim.Framework.Serialization. If we're to copy-paste-change libomv, I'll have to bring in the entire OpenMetaverse.Assets namespace, which is where all the serialization goodies are. Is there a middle ground? On 10/10/2010 8:30 AM, Melanie wrote: Copy/Paste/Change please. it's been a discussed and stated goal to reduce use of OMV classes not in the Types dll. Making ourselves dependent on OMV code for something as critical as this isn't right. There are lots of fields that will need to be added, from simple ones like sit data to complex ones like vehicle physics. Melanie d...@metaverseink.com wrote: After thinking about it, I think I know the way out of our dependence on .net serialization. I'm planning to do this [today], so if anyone has concerns, please speak up. The main problem with what we have is that the serialization is being done automagically by .net via reflection, leaving us completely out of the loop and at the mercy of the nasty .net serialization restrictions. So the smooth transition out of this is to reproduce the serialization strings that .net is producing, but via our own code. That will be stage 1, and will ensure 100% compatibility with the data that already exists. But that will give us the ground to then do the right things with serialization, like adding a MIME type and a version number to whatever is being serialized. Luckily jhurliman has already done most of it in libomv, probably as a consequence of him pulling his hair when he encountered the SOG serialization in OpenSim... I'm undecided on whether to reuse that code as-is or copy-paste-change it, since I'm planning to add more fields, and context, to the SOG serializations. Without getting into flame wars between OpenSim and libomv, any thoughts on this, from a pragmatic point of view? On 10/9/2010 12:39 PM, Dahlia Trimble wrote: How about the various OSD serializations in libomv? They seem to be pretty robust these days. On Sat, Oct 9, 2010 at 11:05 AM,d...@metaverseink.com mailto:d...@metaverseink.com wrote: Dear devs, I'm pulling my hair here with the serialization of scene objects. This is horrible! Using .Net serialization for something as important as this is the worst idea ever. We're completely frozen. I'm trying to add an additional field for the creator info, and I'm stuck in all sorts of ways: I can't compute the value of that field at serialization-time, and then there are the issues of compatibility with previous versions of the SOP class. I'd like to understand all the implications of doing an entirely different serialization procedure, one that does not use reflection, and that allows for processing-during-serialization. What will break, and therefore needs fixing? I understand all archiving related to previous versions will break, so we need to keep supporting the existing serialization 'disease' for the foreseeable future. I also understand that TPs/crossings between sims that talk different SOG serializations will break, but that's not so bad. Anything else I should be aware of before I go off and redo this? Diva / Crista ___ Opensim-dev mailing list Opensim-dev@lists.berlios.demailto:Opensim-dev@lists.berlios.de https://lists.berlios.de/mailman/listinfo/opensim-dev ___ Opensim-dev mailing list Opensim-dev@lists.berlios.de https://lists.berlios.de/mailman/listinfo/opensim-dev ___ Opensim-dev mailing list Opensim-dev@lists.berlios.de https://lists.berlios.de/mailman/listinfo/opensim-dev ___ Opensim-dev mailing list Opensim-dev@lists.berlios.de https://lists.berlios.de/mailman/listinfo/opensim-dev
Re: [Opensim-dev] .net serialization of SOGs
Well, we should just migrate it back. Not the whole shebang, only what we need. It was a stated goal, from way back, that OpenSim should be able to run without OpenMetaverse.dll if the Linden client code is removed. That is why OpenMetaverse.Types.dll exists. It is the one that we decided we would keep relying on, but OpenMetaverse.dll should, so it was decided, eventually only be referenced by the LL client stack. If we want to change that policy, we'd have to have a core vote, as it was also a core vote that established it, iirc. Melanie d...@metaverseink.com wrote: And I can see the back and forth that has been going on :) libomv got some code from OpenSim.Framework.Serialization... Would it be worth establishing a 3rd project for this? One where both the OpenSim devs and the libomv devs would have commit rights? So that we minimize this copy-paste-change backforth... On 10/10/2010 10:24 AM, d...@metaverseink.com wrote: I can see good arguments on both sides. On the one hand we (in OpenSim) need to have complete, and immediate control over this code, in order to be able to do things such as what I'm doing right now -- adding more information to those external data representations. On the other hand, control, in OpenSim, often comes with entanglement. SOG serialization, for example, is horribly tangled with Scenes at the moment, and that is a major block for fixing things like having the Library work as a service, etc. -- not to mention enabling 3rd party tools for manipulating content in a sane manner. So, in that sense, I really like the idea of having this code in a separate library, far away from OpenSim Scenes, which should merely be clients of such library. We have a place for it: OpenSim.Framework.Serialization. If we're to copy-paste-change libomv, I'll have to bring in the entire OpenMetaverse.Assets namespace, which is where all the serialization goodies are. Is there a middle ground? On 10/10/2010 8:30 AM, Melanie wrote: Copy/Paste/Change please. it's been a discussed and stated goal to reduce use of OMV classes not in the Types dll. Making ourselves dependent on OMV code for something as critical as this isn't right. There are lots of fields that will need to be added, from simple ones like sit data to complex ones like vehicle physics. Melanie d...@metaverseink.com wrote: After thinking about it, I think I know the way out of our dependence on .net serialization. I'm planning to do this [today], so if anyone has concerns, please speak up. The main problem with what we have is that the serialization is being done automagically by .net via reflection, leaving us completely out of the loop and at the mercy of the nasty .net serialization restrictions. So the smooth transition out of this is to reproduce the serialization strings that .net is producing, but via our own code. That will be stage 1, and will ensure 100% compatibility with the data that already exists. But that will give us the ground to then do the right things with serialization, like adding a MIME type and a version number to whatever is being serialized. Luckily jhurliman has already done most of it in libomv, probably as a consequence of him pulling his hair when he encountered the SOG serialization in OpenSim... I'm undecided on whether to reuse that code as-is or copy-paste-change it, since I'm planning to add more fields, and context, to the SOG serializations. Without getting into flame wars between OpenSim and libomv, any thoughts on this, from a pragmatic point of view? On 10/9/2010 12:39 PM, Dahlia Trimble wrote: How about the various OSD serializations in libomv? They seem to be pretty robust these days. On Sat, Oct 9, 2010 at 11:05 AM,d...@metaverseink.com mailto:d...@metaverseink.com wrote: Dear devs, I'm pulling my hair here with the serialization of scene objects. This is horrible! Using .Net serialization for something as important as this is the worst idea ever. We're completely frozen. I'm trying to add an additional field for the creator info, and I'm stuck in all sorts of ways: I can't compute the value of that field at serialization-time, and then there are the issues of compatibility with previous versions of the SOP class. I'd like to understand all the implications of doing an entirely different serialization procedure, one that does not use reflection, and that allows for processing-during-serialization. What will break, and therefore needs fixing? I understand all archiving related to previous versions will break, so we need to keep supporting the existing serialization 'disease' for the foreseeable future. I also understand that TPs/crossings between sims that talk different SOG serializations will break, but that's not so bad. Anything else I should be aware of before I go off and redo this? Diva / Crista ___ Opensim-dev mailing list
Re: [Opensim-dev] .net serialization of SOGs
Currently, OpenSim.Region.Framework depends very much on libomv's serialization facilities. The Uuid gatherer, for example, uses the decode() methods to sniff UUIDs. OK, well, I'm going to port some code from libomv into OpenSim so that I can get going on this, but honestly I think this is silly. We should consider some other alternative. I'm a strong +1 on a library that defines Asset types and their [various] serializations, independent of who uses such library. It's true that libomv is/was focused on the Linden asset types formats, but many/all of those asset types make sense in general, even if their external representations are completely different than the ones used by the Linden viewer (notecards, gestures, etc). On 10/10/2010 11:02 AM, Melanie wrote: Well, we should just migrate it back. Not the whole shebang, only what we need. It was a stated goal, from way back, that OpenSim should be able to run without OpenMetaverse.dll if the Linden client code is removed. That is why OpenMetaverse.Types.dll exists. It is the one that we decided we would keep relying on, but OpenMetaverse.dll should, so it was decided, eventually only be referenced by the LL client stack. If we want to change that policy, we'd have to have a core vote, as it was also a core vote that established it, iirc. Melanie d...@metaverseink.com wrote: And I can see the back and forth that has been going on :) libomv got some code from OpenSim.Framework.Serialization... Would it be worth establishing a 3rd project for this? One where both the OpenSim devs and the libomv devs would have commit rights? So that we minimize this copy-paste-change backforth... On 10/10/2010 10:24 AM, d...@metaverseink.com wrote: I can see good arguments on both sides. On the one hand we (in OpenSim) need to have complete, and immediate control over this code, in order to be able to do things such as what I'm doing right now -- adding more information to those external data representations. On the other hand, control, in OpenSim, often comes with entanglement. SOG serialization, for example, is horribly tangled with Scenes at the moment, and that is a major block for fixing things like having the Library work as a service, etc. -- not to mention enabling 3rd party tools for manipulating content in a sane manner. So, in that sense, I really like the idea of having this code in a separate library, far away from OpenSim Scenes, which should merely be clients of such library. We have a place for it: OpenSim.Framework.Serialization. If we're to copy-paste-change libomv, I'll have to bring in the entire OpenMetaverse.Assets namespace, which is where all the serialization goodies are. Is there a middle ground? On 10/10/2010 8:30 AM, Melanie wrote: Copy/Paste/Change please. it's been a discussed and stated goal to reduce use of OMV classes not in the Types dll. Making ourselves dependent on OMV code for something as critical as this isn't right. There are lots of fields that will need to be added, from simple ones like sit data to complex ones like vehicle physics. Melanie d...@metaverseink.com wrote: After thinking about it, I think I know the way out of our dependence on .net serialization. I'm planning to do this [today], so if anyone has concerns, please speak up. The main problem with what we have is that the serialization is being done automagically by .net via reflection, leaving us completely out of the loop and at the mercy of the nasty .net serialization restrictions. So the smooth transition out of this is to reproduce the serialization strings that .net is producing, but via our own code. That will be stage 1, and will ensure 100% compatibility with the data that already exists. But that will give us the ground to then do the right things with serialization, like adding a MIME type and a version number to whatever is being serialized. Luckily jhurliman has already done most of it in libomv, probably as a consequence of him pulling his hair when he encountered the SOG serialization in OpenSim... I'm undecided on whether to reuse that code as-is or copy-paste-change it, since I'm planning to add more fields, and context, to the SOG serializations. Without getting into flame wars between OpenSim and libomv, any thoughts on this, from a pragmatic point of view? On 10/9/2010 12:39 PM, Dahlia Trimble wrote: How about the various OSD serializations in libomv? They seem to be pretty robust these days. On Sat, Oct 9, 2010 at 11:05 AM,d...@metaverseink.com mailto:d...@metaverseink.com wrote: Dear devs, I'm pulling my hair here with the serialization of scene objects. This is horrible! Using .Net serialization for something as important as this is the worst idea ever. We're completely frozen. I'm trying to add an additional field for the creator info, and I'm stuck in all sorts of ways: I can't compute the value of that field at serialization-time, and then there are the issues of compatibility with previous versions of
Re: [Opensim-dev] .net serialization of SOGs
I believe the decision was made that way because we didn't want to keep lugging the packets and sl protocol handling around. Thee will be a day when SL and LLUDP are a faint memory and OpenSim should still be there. So the core should not depend on a lib that contains LLUDP and the packets. Maybe OMV needs to be broken up further. Maybe, instead of breaking out the thigns we DO want to keep, they should break out the LLUDP into a separate lib that can then be dropped eventually. I haven't studied recent libOMV versions so I can't say anything about the feasibility of that. I do know that we need to be able to modify our own stuff without delay, so copy/paste for this crucial part of opensim seems to be in order. Melanie d...@metaverseink.com wrote: Currently, OpenSim.Region.Framework depends very much on libomv's serialization facilities. The Uuid gatherer, for example, uses the decode() methods to sniff UUIDs. OK, well, I'm going to port some code from libomv into OpenSim so that I can get going on this, but honestly I think this is silly. We should consider some other alternative. I'm a strong +1 on a library that defines Asset types and their [various] serializations, independent of who uses such library. It's true that libomv is/was focused on the Linden asset types formats, but many/all of those asset types make sense in general, even if their external representations are completely different than the ones used by the Linden viewer (notecards, gestures, etc). On 10/10/2010 11:02 AM, Melanie wrote: Well, we should just migrate it back. Not the whole shebang, only what we need. It was a stated goal, from way back, that OpenSim should be able to run without OpenMetaverse.dll if the Linden client code is removed. That is why OpenMetaverse.Types.dll exists. It is the one that we decided we would keep relying on, but OpenMetaverse.dll should, so it was decided, eventually only be referenced by the LL client stack. If we want to change that policy, we'd have to have a core vote, as it was also a core vote that established it, iirc. Melanie d...@metaverseink.com wrote: And I can see the back and forth that has been going on :) libomv got some code from OpenSim.Framework.Serialization... Would it be worth establishing a 3rd project for this? One where both the OpenSim devs and the libomv devs would have commit rights? So that we minimize this copy-paste-change backforth... On 10/10/2010 10:24 AM, d...@metaverseink.com wrote: I can see good arguments on both sides. On the one hand we (in OpenSim) need to have complete, and immediate control over this code, in order to be able to do things such as what I'm doing right now -- adding more information to those external data representations. On the other hand, control, in OpenSim, often comes with entanglement. SOG serialization, for example, is horribly tangled with Scenes at the moment, and that is a major block for fixing things like having the Library work as a service, etc. -- not to mention enabling 3rd party tools for manipulating content in a sane manner. So, in that sense, I really like the idea of having this code in a separate library, far away from OpenSim Scenes, which should merely be clients of such library. We have a place for it: OpenSim.Framework.Serialization. If we're to copy-paste-change libomv, I'll have to bring in the entire OpenMetaverse.Assets namespace, which is where all the serialization goodies are. Is there a middle ground? On 10/10/2010 8:30 AM, Melanie wrote: Copy/Paste/Change please. it's been a discussed and stated goal to reduce use of OMV classes not in the Types dll. Making ourselves dependent on OMV code for something as critical as this isn't right. There are lots of fields that will need to be added, from simple ones like sit data to complex ones like vehicle physics. Melanie d...@metaverseink.com wrote: After thinking about it, I think I know the way out of our dependence on .net serialization. I'm planning to do this [today], so if anyone has concerns, please speak up. The main problem with what we have is that the serialization is being done automagically by .net via reflection, leaving us completely out of the loop and at the mercy of the nasty .net serialization restrictions. So the smooth transition out of this is to reproduce the serialization strings that .net is producing, but via our own code. That will be stage 1, and will ensure 100% compatibility with the data that already exists. But that will give us the ground to then do the right things with serialization, like adding a MIME type and a version number to whatever is being serialized. Luckily jhurliman has already done most of it in libomv, probably as a consequence of him pulling his hair when he encountered the SOG serialization in OpenSim... I'm undecided on whether to reuse that code as-is or copy-paste-change it, since I'm planning to add more
Re: [Opensim-dev] .net serialization of SOGs
On 10/10/2010 12:03 PM, Melanie wrote: Maybe OMV needs to be broken up further. Maybe, instead of breaking out the thigns we DO want to keep, they should break out the LLUDP into a separate lib that can then be dropped eventually. The namespace OpenMetaverse.Assets strikes me as the cut that everyone needs. It depends on OpenMetaverse.Primitives. There's some overlap with OpenSim.Framework. Basically, we all need a library that defines asset types and their many external representations, independent of everything else. ___ Opensim-dev mailing list Opensim-dev@lists.berlios.de https://lists.berlios.de/mailman/listinfo/opensim-dev
Re: [Opensim-dev] .net serialization of SOGs
On 09/10/10 19:05, d...@metaverseink.com wrote: Dear devs, I'm pulling my hair here with the serialization of scene objects. This is horrible! Using .Net serialization for something as important as this is the worst idea ever. We're completely frozen. I'm trying to add an additional field for the creator info, and I'm stuck in all sorts of ways: I can't compute the value of that field at serialization-time, and then there are the issues of compatibility with previous versions of the SOP class. I'd like to understand all the implications of doing an entirely different serialization procedure, one that does not use reflection, and that allows for processing-during-serialization. What will break, and therefore needs fixing? Grovelling back through the archives, I found the e-mail where I proposed this very thing back in June 2009! https://lists.berlios.de/pipermail/opensim-dev/2009-June/006805.html And here's the old patch which partially implemented write. https://lists.berlios.de/pipermail/opensim-dev/2009-July/006954.html As you wrote later, some equivalent code was later put in libOMV, though I would test it carefully (at the time I vaguely recall thinking that it didn't deal with some serialization cases, but this may well have been addressed). I'm not excited about whether this code remains in libOMV or is placed in OpenSim if it's going to act as a legacy format in some way. As far as I recall, there are no implications of doing this manually as long as the XML ends up being the same. I understand all archiving related to previous versions will break, so we need to keep supporting the existing serialization 'disease' for the foreseeable future. I also understand that TPs/crossings between sims that talk different SOG serializations will break, but that's not so bad. If the SOG serialization is going to change in an incompatible manner, I absolutely need to bump the OAR/IAR version numbers at the same time in order to maintain confidence in the archiving. I also need to write the code to stop old OpenSims trying to load these new versions. Please can we co-ordinate this in a branch? Anything else I should be aware of before I go off and redo this? Although .net auto-serialization is fatal if you want to make type changes to existing fields, it has been very convenient for adding new fields, since extraneous xml elements are simply ignored on deserialization. I think it's almost essential that a new format behave in the same way. Also, are you going to put version numbers in the new serializations so that any future incompatible can be dealt with in some manner? Is it worth considering whether other virtual environments might store completely different object serializations? For any new format, I agree with Dahlia in that something OSD-like may be worth looking at, perhaps just a simple map as I was considering for the 'dynamic attributes' in SOP. In fact, this could allow that to fit in nicely. -- Justin Clark-Casey (justincc) http://justincc.org http://twitter.com/justincc ___ Opensim-dev mailing list Opensim-dev@lists.berlios.de https://lists.berlios.de/mailman/listinfo/opensim-dev
Re: [Opensim-dev] .net serialization of SOGs
Adam ping! How's the SOG/SOP refactor coming along? This is coming very close to the need for the refactoring... On 10/10/2010 4:45 PM, Justin Clark-Casey wrote: On 09/10/10 19:05, d...@metaverseink.com wrote: Dear devs, I'm pulling my hair here with the serialization of scene objects. This is horrible! Using .Net serialization for something as important as this is the worst idea ever. We're completely frozen. I'm trying to add an additional field for the creator info, and I'm stuck in all sorts of ways: I can't compute the value of that field at serialization-time, and then there are the issues of compatibility with previous versions of the SOP class. I'd like to understand all the implications of doing an entirely different serialization procedure, one that does not use reflection, and that allows for processing-during-serialization. What will break, and therefore needs fixing? Grovelling back through the archives, I found the e-mail where I proposed this very thing back in June 2009! https://lists.berlios.de/pipermail/opensim-dev/2009-June/006805.html And here's the old patch which partially implemented write. https://lists.berlios.de/pipermail/opensim-dev/2009-July/006954.html As you wrote later, some equivalent code was later put in libOMV, though I would test it carefully (at the time I vaguely recall thinking that it didn't deal with some serialization cases, but this may well have been addressed). I'm not excited about whether this code remains in libOMV or is placed in OpenSim if it's going to act as a legacy format in some way. As far as I recall, there are no implications of doing this manually as long as the XML ends up being the same. I understand all archiving related to previous versions will break, so we need to keep supporting the existing serialization 'disease' for the foreseeable future. I also understand that TPs/crossings between sims that talk different SOG serializations will break, but that's not so bad. If the SOG serialization is going to change in an incompatible manner, I absolutely need to bump the OAR/IAR version numbers at the same time in order to maintain confidence in the archiving. I also need to write the code to stop old OpenSims trying to load these new versions. Please can we co-ordinate this in a branch? Anything else I should be aware of before I go off and redo this? Although .net auto-serialization is fatal if you want to make type changes to existing fields, it has been very convenient for adding new fields, since extraneous xml elements are simply ignored on deserialization. I think it's almost essential that a new format behave in the same way. Also, are you going to put version numbers in the new serializations so that any future incompatible can be dealt with in some manner? Is it worth considering whether other virtual environments might store completely different object serializations? For any new format, I agree with Dahlia in that something OSD-like may be worth looking at, perhaps just a simple map as I was considering for the 'dynamic attributes' in SOP. In fact, this could allow that to fit in nicely. ___ Opensim-dev mailing list Opensim-dev@lists.berlios.de https://lists.berlios.de/mailman/listinfo/opensim-dev
Re: [Opensim-dev] .net serialization of SOGs
Hi, for allowing the avatar packs, the new serialization must also wind up being a string, so it can ba wrapped in XML, if it, itself, isn't XML. Also, the script state serialization is brittle. It must happen in the same way and at the same time as it does now. We need only read support, not write, for the old format. I really don't think we need to have write old oar commands. You want to use recent oars, use a recent version. Easy. Melanie d...@metaverseink.com wrote: Dear devs, I'm pulling my hair here with the serialization of scene objects. This is horrible! Using .Net serialization for something as important as this is the worst idea ever. We're completely frozen. I'm trying to add an additional field for the creator info, and I'm stuck in all sorts of ways: I can't compute the value of that field at serialization-time, and then there are the issues of compatibility with previous versions of the SOP class. I'd like to understand all the implications of doing an entirely different serialization procedure, one that does not use reflection, and that allows for processing-during-serialization. What will break, and therefore needs fixing? I understand all archiving related to previous versions will break, so we need to keep supporting the existing serialization 'disease' for the foreseeable future. I also understand that TPs/crossings between sims that talk different SOG serializations will break, but that's not so bad. Anything else I should be aware of before I go off and redo this? Diva / Crista ___ Opensim-dev mailing list Opensim-dev@lists.berlios.de https://lists.berlios.de/mailman/listinfo/opensim-dev ___ Opensim-dev mailing list Opensim-dev@lists.berlios.de https://lists.berlios.de/mailman/listinfo/opensim-dev
Re: [Opensim-dev] .net serialization of SOGs
How about the various OSD serializations in libomv? They seem to be pretty robust these days. On Sat, Oct 9, 2010 at 11:05 AM, d...@metaverseink.com wrote: Dear devs, I'm pulling my hair here with the serialization of scene objects. This is horrible! Using .Net serialization for something as important as this is the worst idea ever. We're completely frozen. I'm trying to add an additional field for the creator info, and I'm stuck in all sorts of ways: I can't compute the value of that field at serialization-time, and then there are the issues of compatibility with previous versions of the SOP class. I'd like to understand all the implications of doing an entirely different serialization procedure, one that does not use reflection, and that allows for processing-during-serialization. What will break, and therefore needs fixing? I understand all archiving related to previous versions will break, so we need to keep supporting the existing serialization 'disease' for the foreseeable future. I also understand that TPs/crossings between sims that talk different SOG serializations will break, but that's not so bad. Anything else I should be aware of before I go off and redo this? Diva / Crista ___ Opensim-dev mailing list Opensim-dev@lists.berlios.de https://lists.berlios.de/mailman/listinfo/opensim-dev ___ Opensim-dev mailing list Opensim-dev@lists.berlios.de https://lists.berlios.de/mailman/listinfo/opensim-dev
