Re: [Opensim-dev] Issues with the Simulator under high load

2012-04-24 Thread Akira Sonoda
:-) just to be clear I am talking about git hash 007711 ( opensim 0.7.4 recommended in the osgrid twitter at the first of march 2012 ). I've seen a newer recommended version just came out ... and I'll give that one a try. Thank you for supporting my thought about a deadlock! I'll have a look at

Re: [Opensim-dev] Issues with the Simulator under high load

2012-04-23 Thread Justin Clark-Casey
Those kind of traces (where lots of threads are waiting for WaitHandler.WaitOne() or similar) are usually indicative of deadlock somewhere in the system. When this happens, I find that the best thing to do is take a vm thread dump and inspect the code to find out where the deadlock is

Re: [Opensim-dev] Issues with the Simulator under high load

2012-04-22 Thread Akira Sonoda
Hi Justin, I am still investigating the Slow GetTexture problem and the resulting instability under high Load. What i did so far: 1. I'm still using opensim-0007711 ( i didn't have the time to upgrade, the first upgrade was not so good due an error on my side and since then i did not

Re: [Opensim-dev] Issues with the Simulator under high load

2012-04-22 Thread Dahlia Trimble
I was able to re-create the slow get error. In my case I had a region with many unique textures. As my viewer was downloading them I was apparently running short of memory and the computer started swapping and thrashing the disk and the viewer would momentarily freeze. During the freezes the

Re: [Opensim-dev] Issues with the Simulator under high load

2012-03-18 Thread Dahlia Trimble
I've done a bit of tracing through the code and I can't seem to find where the http server in OpenSimulator uses threadpool threads. I did find them used in the LLUDP server and in asyncronous requests from the asset service, but I have yet to find any other uses. is it possible that the http

Re: [Opensim-dev] Issues with the Simulator under high load

2012-03-18 Thread Justin Clark-Casey
It's quite possible the 3rd party HTTP server doesn't use the threadpool though I've never looked in detail. You could supply any other http address in the GetTexture cap (e.g. the asset service directly with a suitable handler). However, I'm not sure that asset serving is such a bottleneck

Re: [Opensim-dev] Issues with the Simulator under high load

2012-03-17 Thread Justin Clark-Casey
Hi Akira. I have now updated the show threads method to show threadpool statistics for the main threadpool. Please note that each XEngine script engine will also have it's own threadpool (which can be seen using the xengine status command). Might need to improve this further. show threads

Re: [Opensim-dev] Issues with the Simulator under high load

2012-03-17 Thread Akira Sonoda
Hi Justin, I'll have a look and will report as soon as i get some results .. and hopefully i get more insight. Interesting findings from Dahlia as well !! Here's my report from the last party: https://docs.google.com/open?id=0B301xueh1kxda0hfNEpfWERRTWFzcnRjVGNELXowZw Looking forward to the

Re: [Opensim-dev] Issues with the Simulator under high load

2012-03-16 Thread Justin Clark-Casey
You may also be interested to know that in the very latest git master code I have changed the way that Top Scripts information is generated (as teported through the region menus). Rather than counting and resetting LPS for each object, which doesn't give a very good indication of script running

Re: [Opensim-dev] Issues with the Simulator under high load

2012-03-12 Thread Justin Clark-Casey
On 11/03/12 19:15, Akira Sonoda wrote: Am 10. März 2012 03:25 schrieb Justin Clark-Casey jjusti...@googlemail.com mailto:jjusti...@googlemail.com: I'm sorry to say that you'll have to take the ThreadPool numbers with a very very very large pinch of salt. I believe they only refer to

Re: [Opensim-dev] Issues with the Simulator under high load

2012-03-11 Thread Akira Sonoda
Am 10. März 2012 03:25 schrieb Justin Clark-Casey jjusti...@googlemail.com : I'm sorry to say that you'll have to take the ThreadPool numbers with a very very very large pinch of salt. I believe they only refer to the built-in mono thread pool and not the SmartThreadPool which is the one

Re: [Opensim-dev] Issues with the Simulator under high load

2012-03-09 Thread Akira Sonoda
600 on a 8 core machine ( i guess the hyperthreaded cores are visible to mono as real cores, at least in nmon they are reported as such ) is quite a lot of threads. This morning I went big and configured 1000, but I'm not really sure which approach to go... seeing the stats: Region (Close

Re: [Opensim-dev] Issues with the Simulator under high load

2012-03-09 Thread Justin Clark-Casey
I'm sorry to say that you'll have to take the ThreadPool numbers with a very very very large pinch of salt. I believe they only refer to the built-in mono thread pool and not the SmartThreadPool which is the one actually used (and beyond that the core simulator and xengine use separate pools).

Re: [Opensim-dev] Issues with the Simulator under high load

2012-03-09 Thread Dahlia Trimble
wouldn't the thread pool in OpenSim be using mono threads? Also perhaps I wasn't clear, the application I found the benefit in from increasing mono threads was *not* Opensim, but rather it was a different application that uses the same http server as OpenSim uses. On Fri, Mar 9, 2012 at 6:25 PM,

Re: [Opensim-dev] Issues with the Simulator under high load

2012-03-08 Thread Akira Sonoda
Hi Dahlia, Am 5. März 2012 01:14 schrieb Dahlia Trimble dahliatrim...@gmail.com: A couple thoughts, not sure if it's your problem or not. I would probably check to make sure the cache is set up properly and the file system it's on has plenty of space. Also make sure the disk isnt being

Re: [Opensim-dev] Issues with the Simulator under high load

2012-03-08 Thread Dahlia Trimble
Are you using Mono? I've seen poor performance of the http server used in OpenSimulator when insufficient threads are available. Manipulating the environment variable MONO_THREADS_PER_CPU has worked for me when I've encountered this problem before. Take a look at

Re: [Opensim-dev] Issues with the Simulator under high load

2012-03-08 Thread Akira Sonoda
Ooopps... my MONO_THREADS_PER_CPU=150 are obviously not enough. 2000 as stated in the article is quite a lot ... what are your settings? do you go with the 2000? Am 9. März 2012 00:07 schrieb Dahlia Trimble dahliatrim...@gmail.com: Are you using Mono? I've seen poor performance of the http

Re: [Opensim-dev] Issues with the Simulator under high load

2012-03-08 Thread Dahlia Trimble
Sorry I don't really know much about it. In my case it was an application that used the http server dll from OpenSim and served probably 40-60 simultaneous requests. Mono was defaulting to 25 threads per cpu but I changed it to 75 and I stopped having download problems. This was on a 4-core

[Opensim-dev] Issues with the Simulator under high load

2012-03-04 Thread Akira Sonoda
Hello everybody, Dorothea Lundquist and JayMaze Yao are organizing a party in the OSgrid each Friday during the last three years. I am the tech staff of the which maintains the server and the Software. The parties always are visited by 10 to 20 avatars. The max was at around 30 avatars

Re: [Opensim-dev] Issues with the Simulator under high load

2012-03-04 Thread Dahlia Trimble
A couple thoughts, not sure if it's your problem or not. I would probably check to make sure the cache is set up properly and the file system it's on has plenty of space. Also make sure the disk isnt being thrashed by other processes and that the disk is healthy and not fragmented. There's