[Opensim-users] Out of memory exception
I'm encountering a strange issue with one of my regions. It seems that this one particular region is throwing an out of memory exception error. This is occurring once every 6 or 7 hours. Here is a snippet from the log file. It seems to be happening when the sim is attempting to generate the maptile for the region. We are currently using the warp 3d maptile module. I've looked at the region and their really isn't an excessive amount of content on it; nor are their an excessive amount of scripts so not sure how to troubleshoot this issue. Is the out of memory exception error common for the warp 3d maptile module? if so, is there a fix for it? 2014-12-01 08:05:08,217 DEBUG - OpenSim.Region.CoreModules.ServiceConnectorsOut.MapImage.MapImageServiceModule [MAP IMAGE SERVICE MODULE]: map refresh! 2014-12-01 08:05:08,248 DEBUG - OpenSim.Region.CoreModules.ServiceConnectorsOut.MapImage.MapImageServiceModule [MAP IMAGE SERVICE MODULE]: upload maptile for Serenity Isle II 2014-12-01 08:05:24,488 WARN - OpenSim.Region.CoreModules.World.Warp3DMap.Warp3DImageModule [WARP 3D IMAGE MODULE]: Error decoding JPEG2000 texture efb8c686-a809-4307-8333-e8444d12a2eb (0 bytes): EOF reached before finding Contiguous Codestream Box 2014-12-01 08:05:53,145 DEBUG - OpenSim.Region.CoreModules.World.Warp3DMap.Warp3DImageModule [WARP 3D IMAGE MODULE]: GC.Collect() 2014-12-01 08:05:55,017 DEBUG - OpenSim.Services.Connectors.MapImageServicesConnector [MAP IMAGE CONNECTOR]: map tile uploaded in 1872ms 2014-12-01 08:06:07,325 ERROR - OpenSim.Region.Framework.Scenes.Serialization.SceneObjectSerializer [SERIALIZER]: Deserialization of xml failed with System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown. at System.String.GetStringForStringBuilder(String value, Int32 startIndex, Int32 length, Int32 capacity) at System.Text.StringBuilder.GetNewString(String currentString, Int32 requiredLength) at System.Text.StringBuilder.Append(Char value) at System.IO.StringWriter.Write(Char value) at System.Xml.XmlTextWriter.InternalWriteEndElement(Boolean longFormat) at System.Xml.XmlTextWriter.WriteFullEndElement() at System.Xml.XmlElement.WriteTo(XmlWriter w) at System.Xml.XmlElement.WriteContentTo(XmlWriter w) at System.Xml.XmlElement.WriteTo(XmlWriter w) at System.Xml.XmlElement.WriteContentTo(XmlWriter w) at System.Xml.XmlElement.WriteTo(XmlWriter w) at System.Xml.XmlElement.WriteContentTo(XmlWriter w) at System.Xml.XmlNode.get_InnerXml() at System.Xml.XmlElement.get_InnerXml() at OpenSim.Region.Framework.Scenes.Serialization.SceneObjectSerializer.FromOriginalXmlFormat(String xmlData) in C:\IO-Stuff\opensim-0.7.6-source\OpenSim\Region\Framework\Scenes\Serialization\SceneObjectSerializer.cs:line 80. xml was SceneObjectGroupRootPartSceneObjectPart xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance; xmlns:xsd=http://www.w3.org/2001/XMLSchema;AllowedDropfalse/AllowedDropCreatorIDUUID0b0bc9cd-5b44-4878-9634-2d2321eb08af/UUID/CreatorIDFolderIDUUID7e283dfe-a8dc-47e2-af9b-755338bc46d5/UUID/FolderIDInventorySerial0/InventorySerialUUIDUUID7e283dfe-a8dc-47e2-af9b-755338bc46d5/UUID/UUIDLocalId2648979438/LocalIdName*~*Volatile*~*Diabla Boots Blue M/NameMaterial3/MaterialPassTouchesfalse/PassTouchesPassCollisionsfalse/PassCollisionsRegionHandle7649302396239616/RegionHandleScriptAccessPin0/ScriptAccessPinGroupPositionX210.4605/XY89.11281/YZ752.5/Z/GroupPositionOffsetPositionX0/XY0/YZ0/Z/OffsetPositionRotationOffsetX0/XY0/YZ0/ZW1/W/RotationOffsetVelocityX0/XY0/YZ0/Z/VelocityAngularVelocityX0/XY0/YZ0/Z/AngularVelocityAccelerationX0/XY0/YZ0/Z/AccelerationDescription(No Description)/DescriptionColorR0/RG0/GB0/BA255/A/ColorText /SitName /TouchName /LinkNum0/LinkNumClickAction0/ClickActionShapeProfileCurve1/ProfileCurveTextureEntry82EbJgidQ/iTMSaud6E7BgQNqQziT01Ko7NvZPhWaLr8CA2pDOJPTUqjs29k+FZouvwQDakM4k9NSqOzb2T4Vmi6/AAABEBAQACAPwAAAIA/AAA=/TextureEntryExtraParamsATAAEQAAAMKq4NXyOEK6k3hzeWJNg+AF/ExtraParamsPathBegin0/PathBeginPathCurve16/PathCurvePathEnd0/PathEndPathRadiusOffset0/PathRadiusOffsetPathRevolutions0/PathRevolutionsPathScaleX100/PathScaleXPathScaleY100/PathScaleYPathShearX0/PathShearXPathShearY0/PathShearYPathSkew0/PathSkewPathTaperX0/PathTaperXPathTaperY0/PathTaperYPathTwist0/PathTwistPathTwistBegin0/PathTwistBeginPCode9/PCodeProfileBegin12500/ProfileBeginProfileEnd0/ProfileEndProfileHollow27500/ProfileHollowState0/StateProfileShapeSquare/ProfileShapeHollowShapeSame/HollowShapeSculptTextureUUIDc2aae0d5-f238-42ba-9378-7379624d83e0/UUID/SculptTextur eSculptType5/SculptTypeSculptData Following this line in the log file, there are pages and pages of what appears to be a binary dump. Is it possible that the maptile module failed while
Re: [Opensim-users] Script engine issue [listeners]
OK, just to close the loop on this. After making the adjustments to the correct set of ini files(ugh); the issue appears to be resolved. it turns out, it wasn't the max_listens_per region.. but in fact Was the max_listens_per_script. The default is set to 64 (I believe). I set it to 0 so as to not impose any limitations and Voilà .. it's fixed! I want to thank everyone who took the time to respond... it is much appreciated! Kind Regards, Karl This dump is after the issue presents itself and it's not showing any scripts waiting for load. That's what is really strange. They all work perfectly for an underterminate amount of time then, a random subset of them just stop. I will point out that when I adjusted the max_listens_per_script max_listens_per_region, I adjusted the ini files for a different region (smacks myself in the head) so I just finished adjusting the correct files and am not restarting the simulator. We'll see if that takes care of it. Also, what's the word on increasing the MaxThreads setting from 100 to lets say 300 ... Could that setting be imposing some sort of limitation and would adjusting it up cause any unexpected negative side effects? Kind Regards, Karl Hi Karl, When this stops working, can you check the xengine status in the console? I see the results you have included here, but is this before or after the communication stops? We had a similar issue back in December on 3rd Rock Grid and when things stopped working, checking the xengine status would always show scripts waiting for load. Can you see if this is the case for you? When this stops working does xengine status show scripts waiting for load? If so, I can share the steps we took to correct our issues which might work for you as well. Additionally, which version of opensim are you running? ~Butch On 6/1/2015 11:35 AM, Karl Dreyer wrote: Hello everyone... I have a member that is experiencing a strange issue. He has server and client modules (in world prims using llRegionSay llRegionSayTo to communicate commands to client prims). When the simulator is restarted, all works properly as expected BUT then, after what seems to be a random length of time it seems that communcations on the sim cease to function. i.e. It's almost as if the llRegionSay llRegionSayTo cease to work properly. the client prims aren't receiving the messages from the servers. What makes this even more strange is that it doesn't happen with all of them and the prims being effected is random.. in other words, an hour after the sim is restarted (for example) a subset of the servers are failing to properly send messages but the subset of scripts effected seems to be different each time. I have had my resident go through and confirm that channel settings are set correctly and that there is no channel confusion or conflicts ... the scripts themselves seem to be OK. I also had him test this system on a blank sim or a sim with considerably less scripts and the system works flawlessly. My first thought was that there was an issue with the listeners in the receiving prims and we're hitting some sort of imposed limit to the number of listeners or listener events allowed on a region. I found in the OpenSimDefaults.ini file that there is in fact a max_listens_per_region setting and a max_listens_per_script setting. My first inclination was that he was hitting one of these limits so, I modified these settings to 0 so that the system would not impose any limits and unfortunately, this did not fix the issue. Is there somewhere else in the ini files or hardcoded in the simulator code itself where a limitation is being set? If not, what else could be causing this behavior? Below is a dump of some of the relevant statistics for the region in question. Any assistance would be greatly appreciated. CONNECTION STATISTICS Client logouts due to no data receive timeout: 0 SAMPLE FRAME STATISTICS Dilatn SimFPS PhyFPS AgntUp RootAg ChldAg Prims AtvPrm AtvScr ScrLPS 1.00 54 53.8 0.0 0 0 6623 0 1760 5 PktsIn PktOut PendDl PendUl UnackB TotlFt NetFt PhysFt OthrFt AgntFt ImgsFt 15 24 0 0 0 19.1 0.0 0.0 0.0 0.0 0.0 MEMORY STATISTICS Heap allocated to OpenSim : 645 MB Last heap allocation rate : 0.323 MB/s Average heap allocation rate: 0.202 MB/s Process memory : 1270 MB Status of XEngine instance Scripts loaded : 1760 Scripts waiting for load : 0 Max threads : 100 Min threads : 2 Allocated threads : 12 In use threads : 2 Work items waiting : 0 Events queued : 0 Events processed : 214185 Sensors : 8 Dataserver requests : 0 Timers : 46
Re: [Opensim-users] Script engine issue [listeners]
I don't believe that the setting is referencing the number of listeners in a script.. but rather, the number of listen events per script .. i.e. the number of times a script can execute the listen capture event ... based on the scenario it made sense because the scripts would work perfectly for a random period of time, then a random subset of scripts would just cease to function. I would imagine that is due to the fact that those scripts had reached their limit ... so if the communications between prims such as server / client prims (via llRegionSay) is very active; it could conceivably hit the 64 limit. I will point out that all of this is hypothesis based on the behavior I've seen and the reaction to the change I've made in the opensimdefaults.ini file. I wonder why the scripts need so many listens? I seldom have more than 1 or 2. On Thu, Jun 4, 2015 at 9:40 AM, Karl Dreyer paoli...@islandoasis.biz wrote: OK, just to close the loop on this. After making the adjustments to the correct set of ini files(ugh); the issue appears to be resolved. it turns out, it wasn't the max_listens_per region.. but in fact Was the max_listens_per_script. The default is set to 64 (I believe). I set it to 0 so as to not impose any limitations and Voilà .. it's fixed! I want to thank everyone who took the time to respond... it is much appreciated! Kind Regards, Karl This dump is after the issue presents itself and it's not showing any scripts waiting for load. That's what is really strange. They all work perfectly for an underterminate amount of time then, a random subset of them just stop. I will point out that when I adjusted the max_listens_per_script max_listens_per_region, I adjusted the ini files for a different region (smacks myself in the head) so I just finished adjusting the correct files and am not restarting the simulator. We'll see if that takes care of it. Also, what's the word on increasing the MaxThreads setting from 100 to lets say 300 ... Could that setting be imposing some sort of limitation and would adjusting it up cause any unexpected negative side effects? Kind Regards, Karl Hi Karl, When this stops working, can you check the xengine status in the console? I see the results you have included here, but is this before or after the communication stops? We had a similar issue back in December on 3rd Rock Grid and when things stopped working, checking the xengine status would always show scripts waiting for load. Can you see if this is the case for you? When this stops working does xengine status show scripts waiting for load? If so, I can share the steps we took to correct our issues which might work for you as well. Additionally, which version of opensim are you running? ~Butch On 6/1/2015 11:35 AM, Karl Dreyer wrote: Hello everyone... I have a member that is experiencing a strange issue. He has server and client modules (in world prims using llRegionSay llRegionSayTo to communicate commands to client prims). When the simulator is restarted, all works properly as expected BUT then, after what seems to be a random length of time it seems that communcations on the sim cease to function. i.e. It's almost as if the llRegionSay llRegionSayTo cease to work properly. the client prims aren't receiving the messages from the servers. What makes this even more strange is that it doesn't happen with all of them and the prims being effected is random.. in other words, an hour after the sim is restarted (for example) a subset of the servers are failing to properly send messages but the subset of scripts effected seems to be different each time. I have had my resident go through and confirm that channel settings are set correctly and that there is no channel confusion or conflicts ... the scripts themselves seem to be OK. I also had him test this system on a blank sim or a sim with considerably less scripts and the system works flawlessly. My first thought was that there was an issue with the listeners in the receiving prims and we're hitting some sort of imposed limit to the number of listeners or listener events allowed on a region. I found in the OpenSimDefaults.ini file that there is in fact a max_listens_per_region setting and a max_listens_per_script setting. My first inclination was that he was hitting one of these limits so, I modified these settings to 0 so that the system would not impose any limits and unfortunately, this did not fix the issue. Is there somewhere else in the ini files or hardcoded in the simulator
Re: [Opensim-users] Script engine issue [listeners]
This dump is after the issue presents itself and it's not showing any scripts waiting for load. That's what is really strange. They all work perfectly for an underterminate amount of time then, a random subset of them just stop. I will point out that when I adjusted the max_listens_per_script max_listens_per_region, I adjusted the ini files for a different region (smacks myself in the head) so I just finished adjusting the correct files and am not restarting the simulator. We'll see if that takes care of it. Also, what's the word on increasing the MaxThreads setting from 100 to lets say 300 ... Could that setting be imposing some sort of limitation and would adjusting it up cause any unexpected negative side effects? Kind Regards, Karl Hi Karl, When this stops working, can you check the xengine status in the console? I see the results you have included here, but is this before or after the communication stops? We had a similar issue back in December on 3rd Rock Grid and when things stopped working, checking the xengine status would always show scripts waiting for load. Can you see if this is the case for you? When this stops working does xengine status show scripts waiting for load? If so, I can share the steps we took to correct our issues which might work for you as well. Additionally, which version of opensim are you running? ~Butch On 6/1/2015 11:35 AM, Karl Dreyer wrote: Hello everyone... I have a member that is experiencing a strange issue. He has server and client modules (in world prims using llRegionSay llRegionSayTo to communicate commands to client prims). When the simulator is restarted, all works properly as expected BUT then, after what seems to be a random length of time it seems that communcations on the sim cease to function. i.e. It's almost as if the llRegionSay llRegionSayTo cease to work properly. the client prims aren't receiving the messages from the servers. What makes this even more strange is that it doesn't happen with all of them and the prims being effected is random.. in other words, an hour after the sim is restarted (for example) a subset of the servers are failing to properly send messages but the subset of scripts effected seems to be different each time. I have had my resident go through and confirm that channel settings are set correctly and that there is no channel confusion or conflicts ... the scripts themselves seem to be OK. I also had him test this system on a blank sim or a sim with considerably less scripts and the system works flawlessly. My first thought was that there was an issue with the listeners in the receiving prims and we're hitting some sort of imposed limit to the number of listeners or listener events allowed on a region. I found in the OpenSimDefaults.ini file that there is in fact a max_listens_per_region setting and a max_listens_per_script setting. My first inclination was that he was hitting one of these limits so, I modified these settings to 0 so that the system would not impose any limits and unfortunately, this did not fix the issue. Is there somewhere else in the ini files or hardcoded in the simulator code itself where a limitation is being set? If not, what else could be causing this behavior? Below is a dump of some of the relevant statistics for the region in question. Any assistance would be greatly appreciated. CONNECTION STATISTICS Client logouts due to no data receive timeout: 0 SAMPLE FRAME STATISTICS Dilatn SimFPS PhyFPS AgntUp RootAg ChldAg Prims AtvPrm AtvScr ScrLPS 1.00 54 53.8 0.0 0 0 6623 0 1760 5 PktsIn PktOut PendDl PendUl UnackB TotlFt NetFt PhysFt OthrFt AgntFt ImgsFt 15 24 0 0 0 19.1 0.0 0.0 0.0 0.0 0.0 MEMORY STATISTICS Heap allocated to OpenSim : 645 MB Last heap allocation rate : 0.323 MB/s Average heap allocation rate: 0.202 MB/s Process memory : 1270 MB Status of XEngine instance Scripts loaded : 1760 Scripts waiting for load : 0 Max threads : 100 Min threads : 2 Allocated threads : 12 In use threads : 2 Work items waiting : 0 Events queued : 0 Events processed : 214185 Sensors : 8 Dataserver requests : 0 Timers : 46 Listeners : 61 ___ Opensim-users mailing list Opensim-users@opensimulator.org http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users ___ Opensim-users mailing list Opensim-users@opensimulator.org http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users ___ Karl M. Dreyer Jr. In World Avi : Damean Paolino Island Oasis, LLC Co-Founder | CIO http://www.islandoasis.biz Notice: The contents of this email and any files transmitted with it are confidential and intended solely for the individual or entity to whom it is addressed. The content of this email message
