Re: [Opensim-users] Intermittent crashing - System.OutofMemoryException

BlueWall Thu, 16 Feb 2012 07:47:08 -0800

The rrdtools is very good for storing and organizing the data forstatistics and it will also produce graphs from that data. It relies onexternal scripts to collect the data - that's where Cacti comes in. Ithas the scripts and templates to graph your system (and snmp devices) inthe distribution. I have done it on Linux and they claim to work onWindows too.

You could also write scripts and templates to graph OpenSim. I haven'ttried that, though. There is documentation included in the distro, andscattered across the Cacti user forums.


-BlueWall

On 02/16/2012 09:55 AM, Rick Anderson wrote:

One question about Cacti, and RDDTools. The sites describe them as
general purpose graphing tools. Is there any documentation or tutorial
for configuring them to monitor system status. Maybe, it's really
simple, and I'm over complicating it.

-_Rick

On Thu, Feb 16, 2012 at 6:53 AM, BlueWall<[email protected]>  wrote:


Cacti will work -
http://www.cacti.net/downloads/docs/html/install_windows.html


On 02/15/2012 03:24 PM, Justin Clark-Casey wrote:


Generated errors will not consume memory.

The best way to track usage is to use a system monitoring tool like
rrdtool to record process memory use (not sure what the Windows
equivalent is). You can also extract statistics from OpenSim itself [1]
and record them in the same tool.

I'm not sure if there is any public code on how to glue these things
together, though I remember that osgrid was recording stats at some point.

If one can record stats then you can see memory usage and starting
twiddling some config things to try and pin it down. For instance, I
have reason to be believe that right now, running a large number of
scripts might leak memory even if no avatars are present in the region.
I don't know the cause of this, whether it's a general script issue or
some particular LSL operation.

Unfortunately, this is getting into fairly complex diagnostics but
running a high uptime simulator with significant usage is not easy. The
alternative solution is simply to restart the simulator periodically
(e.g. daily) in order to reset the memory.

[1] http://opensimulator.org/wiki/Monitoring

On 15/02/12 14:58, Rick Anderson wrote:


Since one machine is faster than another is there a chance a process
that leaks memory is happening faster? Can the memory usage be shown
to go to the 2 gig limit then fail? What's a good way to check?

Could there be an issue with permissions/settings from OAR import?
Like ownership and groups not matching. Then the generated errors
consume memory?

A couple of random thoughts,
-_Rick

On Tue, Feb 14, 2012 at 9:50 PM, Justin Clark-Casey
<[email protected]>  wrote:


It's a good question. I'm not sure there any such conditions - it
appears
that the time saved by reading this out from memory is dwarfed by other
sources of delay (chiefly network, I expect). All of the 'serious'
simulator operators that I know never go near Flotsam's memory cache.

James' point about the 2MB limit is a great one - I had forgotten
about this
on Windows. This is the problem that you'll hit first no matter how much
memory you have (which is part of the reason why people run multiple
instances of OpenSimulator rather than a large number of regions on a
single
simulator).

I believe that it's perfect possible to get Windows to use 64 bit
libraries
when appropriate but this requires doing a bit of grunt work (possibly
compiling 64 bit versions of ode.dll and sqlite3.dll and then putting
the
required detection code into OpenSim).


On 14/02/12 15:13, core wrote:



this is an interesting and informative thread. both fleep's and gary's
comments about flotsam cache settings have made me wonder - exactly
under what conditions should, if at all, flotsam memory caching be set
to TRUE. i've tried it both ways and have seen little difference, but
admittedly this was under moderate usage at best. since there are
plenty
of people here running live-fire instances in a variety of environments
i was hoping to gain some insight.

thanks to hiro for the excellent breakdown of .net memory usage. and to
fleep, we all know cat hair is in fact magical and makes our gear run
faster and smarter ;)

- core

On 2/14/2012 7:09 AM, Fleep Tuque wrote:



Thanks all for the info and thoughts, sounds like scheduling daily
restarts is probably the best option.

I'm pretty sure I haven't messed with whatever the default settings
are for the flotsam cache, I never copy over old config files when I
upgrade, I go through and update by hand and usually just base
settings to make sure the upgrade was successful and then turn on
other options that we've tested.

I have to say, it's almost funny that FleepGrid, which is running
Windows XP on a P4 box that was gathering dust and cat hair in the
basement, 4GB of RAM, and at least 8 sims full of content and scripts
isn't having that problem, but the new poweredge server running
WinServer2008, tons of RAM, and only 3 or 4 sims of content is flaking
out. In fact, three of the regions on the UCSIM grid are identical to
the ones on FleepGrid since I ported them over as OAR files, so it
can't be much of a difference in content..

Think I'll try going through the log files again, I'd sure like to
know what's tipping the out of memory exception!

Thanks again.

- Chris/Fleep

Chris M. Collins (SL/OS: Fleep Tuque)
Center for Simulations&  Virtual Environments Research (UCSIM)
UCIT Instructional&  Research Computing

University of Cincinnati
406A Zimmer Hall
315 College Drive
PO BOX 210088
Cincinnati, OH 45221-0088
[email protected]<mailto:[email protected]>
(513) 556-3018

http://ucsim.uc.edu

On Tue, Feb 14, 2012 at 7:29 AM, James Stallings II
<[email protected]<mailto:[email protected]>>  wrote:

Regardless the amount of RAM on your system, the .net runtime uses
a memory model of a specific size (2 GB) and cannot address more
than that except under special circumstances.

There is a hack around, involving the use of a Visual Studio C++
utility to make a .net binary 'large memory aware', which will
allow it to address 4 GB, but then only on operating systems that
can address>2 GB (e.g., not 32 bit operating systems).

Note that such measures will only help if the memory exhaustion is
being caused by the amount/complexity of content. If the problem
is truly a memory leak, this workaround will only prolong the
inevitable, as it will simply provide more memory for the leaking.

I saw this problem *very* briefly in the 0.7.3 series about a
month ago. I never did have any indication of what was causing it,
and it occurred day over day, without significant changes to the
content. It went away just as fast as it came.



FYI this is on a 25 region megaregion with>50k prims and at least
several rather complex scripts in play.


Cheers
James/Hiro


On Mon, Feb 13, 2012 at 8:39 PM, Gary Banham<[email protected]
<mailto:[email protected]>>  wrote:

Fleep I’m running into exactly the same problem I check the
flotsam cache to make sure I wasn’t using memory

My server runs windows seven but has the same amount of memory
I have about 8 regions on the server some with quite a few
scripts

I rolled back the instance to 7.1.1 to see what would happen
and it worked fine there must be something different in the
way 7.2 release uses the memory

I’m in the process of distributing the regions over a number
of 7.2 instances I’m not sure if that’s going to cause other
problems

TC

Gary

*From:*[email protected]
<mailto:[email protected]>
[mailto:[email protected]
<mailto:[email protected]>] *On Behalf Of
*Fleep Tuque
*Sent:* February-13-12 11:06 AM
*To:* [email protected]
<mailto:[email protected]>
*Subject:* [Opensim-users] Intermittent crashing -
System.OutofMemoryException

Hi all,

Our campus opensim system (running version 0.7.2 Release in
grid mode) seems to intermittently be crashing the last week
or so and I can't figure out why or how to reproduce it. An
out of memory exception doesn't make sense to me, the server
has 24GB RAM and has very light load since we're still in
pilot/testing phase. No more than 5 or 6 users concurrent,
only a few sims with much content..

Does this error mean anything to anyone?

012-02-11 18:18:33,738 ERROR - OpenSim.Application [APPLICATION]:

APPLICATION EXCEPTION DETECTED:
System.UnhandledExceptionEventArgs

Exception: System.OutOfMemoryException: Exception of type
'System.OutOfMemoryException' was thrown.

at System.Threading.Thread.StartInternal(IPrincipal principal,
StackCrawlMark&  stackMark)

at System.Threading.Thread.Start()

at Amib.Threading.SmartThreadPool.StartThreads(Int32
threadsCount)

at Amib.Threading.SmartThreadPool.Enqueue(WorkItem workItem,
Boolean incrementWorkItems)

at
Amib.Threading.SmartThreadPool.QueueWorkItem(WorkItemCallback
callback, Object state)

at OpenSim.Framework.Util.FireAndForget(WaitCallback callback,
Object obj)

at


OpenSim.Region.ClientStack.LindenUDP.LLUDPServer.PacketReceived(UDPPacketBuffer

buffer)

at OpenMetaverse.OpenSimUDPBase.AsyncEndReceive(IAsyncResult iar)

at System.Net.LazyAsyncResult.Complete(IntPtr userToken)

at System.Net.ContextAwareResult.CompleteCallback(Object state)

at System.Threading.ExecutionContext.runTryCode(Object userData)

at


System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(TryCode

code, CleanupCode backoutCode, Object userData)

at
System.Threading.ExecutionContext.RunInternal(ExecutionContext
executionContext, ContextCallback callback, Object state)

at System.Threading.ExecutionContext.Run(ExecutionContext
executionContext, ContextCallback callback, Object state)

at System.Net.ContextAwareResult.Complete(IntPtr userToken)

at System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object
result, IntPtr userToken)

at


System.Net.Sockets.BaseOverlappedAsyncResult.CompletionPortCallback(UInt32

errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)

at


System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32

errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)

Application is terminating: True

Any pointers or advice appreciated!

- Chris/Fleep

Chris M. Collins (SL/OS: Fleep Tuque)

Center for Simulations&  Virtual Environments Research (UCSIM)

UCIT Instructional&  Research Computing


University of Cincinnati

406A Zimmer Hall

315 College Drive

PO BOX 210088

Cincinnati, OH 45221-0088

[email protected]<mailto:[email protected]>

(513) 556-3018<tel:%28513%29%20556-3018>

http://ucsim.uc.edu


_______________________________________________
Opensim-users mailing list
[email protected]
<mailto:[email protected]>
https://lists.berlios.de/mailman/listinfo/opensim-users




--
===================================
http://simhost.com
http://twitter.com/jstallings2
http://www.linkedin.com/pub/5/770/a49

_______________________________________________
Opensim-users mailing list
[email protected]<mailto:[email protected]>
https://lists.berlios.de/mailman/listinfo/opensim-users




_______________________________________________
Opensim-users mailing list
[email protected]
https://lists.berlios.de/mailman/listinfo/opensim-users



--
Justin Clark-Casey (justincc)
http://justincc.org/blog
http://twitter.com/justincc
_______________________________________________
Opensim-users mailing list
[email protected]
https://lists.berlios.de/mailman/listinfo/opensim-users


_______________________________________________
Opensim-users mailing list
[email protected]
https://lists.berlios.de/mailman/listinfo/opensim-users


_______________________________________________
Opensim-users mailing list
[email protected]
https://lists.berlios.de/mailman/listinfo/opensim-users

Re: [Opensim-users] Intermittent crashing - System.OutofMemoryException

Reply via email to