Some really helpful information, looks like we better get the incoming
packet queue done quickly!
--- Begin Message ---
John Hurliman wrote:
I've been trying to track down a bug that multiple people have
experienced where clients running multiple bots and doing a lot of
things at the same time will suddenly start using exponential amounts of
memory until all of the system memory is used up under Mono on Linux.
I have used a few heap profilers concerning this bug as well. At least
for me, the problem occurs when the thread pool is exhausted, but timers
are still firing and trying to create new threads. I believe the bug is
in Mono, either in the delegate, timer, or threadpool implementation. I
suspect delegates, but I have still not fixed the real bug; but my
workaround is to make sure that timer code does not hold a lock too long
because that is a sure-fire way to trigger this bug.
I spoke a bit with LLuis Sanchez, the author of Heap Shot, and he
confirmed that the allocation problem is happening in unmanaged code,
which you can confirm if you attach gdb to mono and do some tracebacks
in most of the threads -- they are all stopped in the libc malloc()
stuff, and further up the stack is the managed timer code. Therefore,
John, if you want to confirm that we are talking about the same bug, run
it under heap shot and notice whether you have tons of memory allocated,
but not accounted for (i.e. not listed as a C# object in the heap).
So in summary, a workaround is to never allow the threadpool to fill up,
which entails never allowing timer-triggered code to fall behind, which
requires at least never taking a lock and then doing something stupid
like querying over the network (that was what I had done).
Attached is the minimal possible C# program I wrote which exhibits the
memory exhaustion behavior. I hope it helps.
--
Jason Smith
Vidya Corporation
Bangkok, Thailand
http://www.vidya-corporation.com
/* ThreadTest.cs
*
* This program causes my manually-installed Mono 1.2.3.1 to allocate all system
* memory on Fedora Core 6. If you are lucky, you will get an OutOfMemoryException
* before swapping slows things down.
*
* To compile: gmcs ThreadTest.cs
*/
using System;
namespace ThreadTest {
public class ThreadTest {
static public int GetThreadId() {
return System.Threading.Thread.CurrentThread.GetHashCode();
}
static string TheLock = "This is the lock object";
static int RunningThreads = 0;
static void ThreadEntry() {
/* Entry point for the problem thread. Take a lock and then sleep without releasing it. */
int id = GetThreadId();
System.Console.WriteLine("{0} started up", id);
lock (TheLock) {
System.Console.WriteLine("{0} holding lock for a long time", id);
System.Threading.Thread.Sleep(1000 * 300);
}
System.Console.WriteLine("{0} exiting", id);
}
static void OnElapsedTimer(object timer, System.Timers.ElapsedEventArgs args) {
RunningThreads += 1;
int id = GetThreadId();
System.Console.WriteLine("{0} elapsed (running={1}) at {2}", id, RunningThreads,
System.DateTime.Now.ToString());
lock (TheLock) {
/* In here will never execute because the lock is held by the sleeping first thread.
* For me at least, memory is exhausted before the lock frees up.
*/
System.Console.WriteLine("{0} got the lock", id);
}
RunningThreads -= 1;
}
static void Main(string[] args) {
System.Console.WriteLine("Main thread: {0}", GetThreadId());
int workers, port;
System.Threading.ThreadPool.GetMaxThreads(out workers, out port);
System.Console.WriteLine("Maximum worker threads: {0}", workers);
System.Threading.Thread t1;
t1 = new System.Threading.Thread(new System.Threading.ThreadStart(ThreadEntry));
t1.Start();
int duration = 1000;
System.Timers.Timer tim = new System.Timers.Timer(duration);
tim.Elapsed += new System.Timers.ElapsedEventHandler(OnElapsedTimer);
System.Console.WriteLine("Timer will start now with duration={0}ms", duration);
tim.Start();
/* It looks like problems happen once 30 threads queue up, so leave the timer to last longer than that. */
System.Threading.Thread.Sleep(1000 * 300);
t1.Join();
}
}
}
--- End Message ---
_______________________________________________
libsl-dev mailing list
libsl-dev@opensecondlife.org
http://opensecondlife.org/cgi-bin/mailman/listinfo/libsl-dev
