Re: [Opensim-users] Intermittent crashing - System.OutofMemoryException

Wed, 15 Feb 2012 12:24:53 -0800 

Generated errors will not consume memory.
The best way to track usage is to use a system monitoring tool like rrdtool to record process memory use (not sure what the Windows equivalent is). You can also extract statistics from OpenSim itself [1] and record them in the same tool. 


I'm not sure if there is any public code on how to glue these things together, though I remember that osgrid was 
recording stats at some point.



If one can record stats then you can see memory usage and starting twiddling some config things to try and pin it down. 
 For instance, I have reason to be believe that right now, running a large number of scripts might leak memory even if 
no avatars are present in the region.  I don't know the cause of this, whether it's a general script issue or some 
particular LSL operation.



Unfortunately, this is getting into fairly complex diagnostics but running a high uptime simulator with significant 
usage is not easy.  The alternative solution is simply to restart the simulator periodically (e.g. daily) in order to 
reset the memory.


[1] http://opensimulator.org/wiki/Monitoring

On 15/02/12 14:58, Rick Anderson wrote:

Since one machine is faster than another is there a chance a process
that leaks memory is happening faster? Can the memory usage be shown
to go to the 2 gig limit then fail? What's a good way to check?

Could there be an issue with permissions/settings from OAR import?
Like ownership and groups not matching. Then the generated errors
consume memory?

A couple of random thoughts,
-_Rick

On Tue, Feb 14, 2012 at 9:50 PM, Justin Clark-Casey
<[email protected]>  wrote:

It's a good question.  I'm not sure there any such conditions - it appears
that the time saved by reading this out from memory is dwarfed by other
sources of delay (chiefly network, I expect).  All of the 'serious'
simulator operators that I know never go near Flotsam's memory cache.

James' point about the 2MB limit is a great one - I had forgotten about this
on Windows.  This is the problem that you'll hit first no matter how much
memory you have (which is part of the reason why people run multiple
instances of OpenSimulator rather than a large number of regions on a single
simulator).

I believe that it's perfect possible to get Windows to use 64 bit libraries
when appropriate but this requires doing a bit of grunt work (possibly
compiling 64 bit versions of ode.dll and sqlite3.dll and then putting the
required detection code into OpenSim).


On 14/02/12 15:13, core wrote:


this is an interesting and informative thread. both fleep's and gary's
comments about flotsam cache settings have made me wonder - exactly
under what conditions should, if at all, flotsam memory caching be set
to TRUE. i've tried it both ways and have seen little difference, but
admittedly this was under moderate usage at best. since there are plenty
of people here running live-fire instances in a variety of environments
i was hoping to gain some insight.

thanks to hiro for the excellent breakdown of .net memory usage. and to
fleep, we all know cat hair is in fact magical and makes our gear run
faster and smarter ;)

- core

On 2/14/2012 7:09 AM, Fleep Tuque wrote:


Thanks all for the info and thoughts, sounds like scheduling daily
restarts is probably the best option.

I'm pretty sure I haven't messed with whatever the default settings
are for the flotsam cache, I never copy over old config files when I
upgrade, I go through and update by hand and usually just base
settings to make sure the upgrade was successful and then turn on
other options that we've tested.

I have to say, it's almost funny that FleepGrid, which is running
Windows XP on a P4 box that was gathering dust and cat hair in the
basement, 4GB of RAM, and at least 8 sims full of content and scripts
isn't having that problem, but the new poweredge server running
WinServer2008, tons of RAM, and only 3 or 4 sims of content is flaking
out. In fact, three of the regions on the UCSIM grid are identical to
the ones on FleepGrid since I ported them over as OAR files, so it
can't be much of a difference in content..

Think I'll try going through the log files again, I'd sure like to
know what's tipping the out of memory exception!

Thanks again.

- Chris/Fleep

Chris M. Collins (SL/OS: Fleep Tuque)
Center for Simulations&    Virtual Environments Research (UCSIM)
UCIT Instructional&    Research Computing

University of Cincinnati
406A Zimmer Hall
315 College Drive
PO BOX 210088
Cincinnati, OH 45221-0088
[email protected]<mailto:[email protected]>
(513) 556-3018

http://ucsim.uc.edu

On Tue, Feb 14, 2012 at 7:29 AM, James Stallings II
<[email protected]<mailto:[email protected]>>    wrote:

     Regardless the amount of RAM on your system, the .net runtime uses
     a memory model of a specific size (2 GB) and cannot address more
     than that except under special circumstances.

     There is a hack around, involving the use of a Visual Studio C++
     utility to make a .net binary 'large memory aware', which will
     allow it to address 4 GB, but then only on operating systems that
     can address>2 GB (e.g., not 32 bit operating systems).

     Note that such measures will only help if the memory exhaustion is
     being caused by the amount/complexity of content. If the problem
     is truly a memory leak, this workaround will only prolong the
     inevitable, as it will simply provide more memory for the leaking.

     I saw this problem *very* briefly in the 0.7.3 series about a
     month ago. I never did have any indication of what was causing it,
     and it occurred day over day, without significant changes to the
     content. It went away just as fast as it came.



     FYI this is on a 25 region megaregion with>50k prims and at least
     several rather complex scripts in play.


     Cheers
     James/Hiro


     On Mon, Feb 13, 2012 at 8:39 PM, Gary Banham<[email protected]
     <mailto:[email protected]>>    wrote:

         Fleep I’m running into exactly the same problem I check the
         flotsam cache to make sure I wasn’t using memory

         My server runs windows seven but has the same amount of memory
         I have about 8 regions on the server some with quite a few
scripts

         I rolled back the instance to 7.1.1 to see what would happen
         and it worked fine there must be something different in the
         way 7.2 release uses the memory

         I’m in the process of distributing the regions over a number
         of 7.2 instances I’m not sure if that’s going to cause other
         problems

         TC

         Gary

         *From:*[email protected]
         <mailto:[email protected]>
         [mailto:[email protected]
         <mailto:[email protected]>] *On Behalf Of
         *Fleep Tuque
         *Sent:* February-13-12 11:06 AM
         *To:* [email protected]
         <mailto:[email protected]>
         *Subject:* [Opensim-users] Intermittent crashing -
         System.OutofMemoryException

         Hi all,

         Our campus opensim system (running version 0.7.2 Release in
         grid mode) seems to intermittently be crashing the last week
         or so and I can't figure out why or how to reproduce it. An
         out of memory exception doesn't make sense to me, the server
         has 24GB RAM and has very light load since we're still in
         pilot/testing phase. No more than 5 or 6 users concurrent,
         only a few sims with much content..

         Does this error mean anything to anyone?

         012-02-11 18:18:33,738 ERROR - OpenSim.Application [APPLICATION]:

         APPLICATION EXCEPTION DETECTED:
System.UnhandledExceptionEventArgs

         Exception: System.OutOfMemoryException: Exception of type
         'System.OutOfMemoryException' was thrown.

         at System.Threading.Thread.StartInternal(IPrincipal principal,
         StackCrawlMark&    stackMark)

         at System.Threading.Thread.Start()

         at Amib.Threading.SmartThreadPool.StartThreads(Int32
threadsCount)

         at Amib.Threading.SmartThreadPool.Enqueue(WorkItem workItem,
         Boolean incrementWorkItems)

         at
         Amib.Threading.SmartThreadPool.QueueWorkItem(WorkItemCallback
         callback, Object state)

         at OpenSim.Framework.Util.FireAndForget(WaitCallback callback,
         Object obj)

         at

OpenSim.Region.ClientStack.LindenUDP.LLUDPServer.PacketReceived(UDPPacketBuffer
         buffer)

         at OpenMetaverse.OpenSimUDPBase.AsyncEndReceive(IAsyncResult iar)

         at System.Net.LazyAsyncResult.Complete(IntPtr userToken)

         at System.Net.ContextAwareResult.CompleteCallback(Object state)

         at System.Threading.ExecutionContext.runTryCode(Object userData)

         at

System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(TryCode
         code, CleanupCode backoutCode, Object userData)

         at
         System.Threading.ExecutionContext.RunInternal(ExecutionContext
         executionContext, ContextCallback callback, Object state)

         at System.Threading.ExecutionContext.Run(ExecutionContext
         executionContext, ContextCallback callback, Object state)

         at System.Net.ContextAwareResult.Complete(IntPtr userToken)

         at System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object
         result, IntPtr userToken)

         at

System.Net.Sockets.BaseOverlappedAsyncResult.CompletionPortCallback(UInt32
         errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)

         at

System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32
         errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)

         Application is terminating: True

         Any pointers or advice appreciated!

         - Chris/Fleep

         Chris M. Collins (SL/OS: Fleep Tuque)

         Center for Simulations&    Virtual Environments Research (UCSIM)

         UCIT Instructional&    Research Computing


         University of Cincinnati

         406A Zimmer Hall

         315 College Drive

         PO BOX 210088

         Cincinnati, OH 45221-0088

         [email protected]<mailto:[email protected]>

         (513) 556-3018<tel:%28513%29%20556-3018>

         http://ucsim.uc.edu


         _______________________________________________
         Opensim-users mailing list
         [email protected]
         <mailto:[email protected]>
         https://lists.berlios.de/mailman/listinfo/opensim-users




     --
     ===================================
     http://simhost.com
     http://twitter.com/jstallings2
     http://www.linkedin.com/pub/5/770/a49

     _______________________________________________
     Opensim-users mailing list
     [email protected]<mailto:[email protected]>
     https://lists.berlios.de/mailman/listinfo/opensim-users




_______________________________________________
Opensim-users mailing list
[email protected]
https://lists.berlios.de/mailman/listinfo/opensim-users







--
Justin Clark-Casey (justincc)
http://justincc.org/blog
http://twitter.com/justincc
_______________________________________________
Opensim-users mailing list
[email protected]
https://lists.berlios.de/mailman/listinfo/opensim-users







--
Justin Clark-Casey (justincc)
http://justincc.org/blog
http://twitter.com/justincc
_______________________________________________
Opensim-users mailing list
[email protected]
https://lists.berlios.de/mailman/listinfo/opensim-users






















					Reply via email to