I tested the following 10 versions of opensim built for the OSgrid.
The times are from the monitor program "monit",and are the uptime.
The test process was;
Place a fresh copy of opensim as distributed, copy the needed .ini files
into the simulator,
Restart the region. (This causes "monit" to detect a new PID and resets
it's "uptime" to be zero)
7 minute tick, login with "coolvl" with standard unscripted avatar, record
metrics.
10 minute tick, log off, record metrics.
15 minute tick record metrics.
The 7 minute tick metrics represents the IDLE usage of a freshly started and
un-visited region.
(the reason for waiting 7 minutes was to give ample time to compile the 626
scripts as an initial load)
The 10 minute tick metrics represents the usage after an avatar has logged
in and been on the region for approximately 3 minutes.
The 15 minute tick represents the usage 5 minutes after an avatar has left
the region and no other usage of the region has taken place.
Region used: Sanctuary Prims= 7985 Scripts= 626
------------------------------------------------------------------- idle
1 avatar avatar leave
-------------------------------------------------------- Time mem kb CPU%
Time mem kb CPU% Time mem kb CPU%
osgrid-opensim-01162015.v0.8.1.97ac80d 7m 574952k 3.0
osgrid-opensim-01172015.v0.8.1.1f04e1b 7m 585076k 2.8 27m 513324k 2.9
30m 513364k 2.6
osgrid-opensim-02212015.v0.8.1.7b9ad11 7m 565480k 2.5 10m 560624k 2.9
15m 536468k 2.5
osgrid-opensim-03142015.v0.8.1.8b13e4e 7m 568560k 2.6 10m 576828k 2.9
15m 567436k 2.5
osgrid-opensim-03172015.v0.8.2.83e58eb 7m 580736k 2.4 10m 583992k 2.8
15m 572664k 2.5
osgrid-opensim-04052015.v0.8.2.8d66284 7m 564208k 2.5 10m 574464k 2.8
15m 574704k 2.5
osgrid-opensim-04122015.v0.8.2.d96d31b 7m 574392k 2.6 10m 585156k 3.0
15m 546868k 2.6
osgrid-opensim-05072015.v0.8.2.c74cef0 7m 583924k 2.6 10m 1139092k 2.7
15m 1141972k 2.5
osgrid-opensim-05092015.v0.8.2.adf0f49 7m 578632k 2.5 10m 1207972k 2.9
15m 1208696k 2.5
osgrid-opensim-05232015.v0.8.2.abb3bb6 7m 585384K 2.6 10m 584852k 5.5
15m 1165304k 2.5
Additional comments:
This problem appeared for me in the May 7, 2015 OSgrid release build. It is
present in the latest build which was May 23, 2015.
The initial IDLE memory appears to be nearly the same for all versions
tested.
Prior to the May 7th release memory usage fluctuated very little when
avatars came and went.
After the May 7 release memory usage goes up dramatically, for EACH avatar
that arrives and does not go back down.
There was a change on how memory is used between the May 9 and May 23
builds. The first builds used the memory much faster, and CPU usage would
rise quickly last for a short time and then drop back down.
In the May 23 release, CPU usage goes up much higher, holds at that high
usage level much longer but does drop back to the "idle" level. Memory is
slower to be increased, but still rises to over double that of the idle rate
and does not drop back to the idle level ever.
The value of AppDomainLoading has been set to "false" in the OSgrid builds
for all the builds distributed this year that I have, that is what all of my
tests were run with.
However reading that parameter it reads to me that having it set to FALSE
would cause this exact problem.
That does not explain why it never caused this problem prior to the May 7th
build.
I have "monit" configured to allow both memory and CPU usage to go over a
set limit but if it remains above that limit for longer than what I believe
to be a reasonable time, in my case 3 minutes, it will restart the region.
From 2011 until May 7th 2015 it was rare any of my regions were restarted by
"monit". After the May 9th release "monit" would restart my regions as soon
as any avatar visited it. I had set the memory limit to be 1.3GB prior to
May 7th. Now setting it at 5GB still results in some regions restarting a
few times every day.
Until the May 9th release none of my four servers had ever used the swap
file. Now the usage of the swap file is in the mid 20%. Lag has become a
problem on my regions, and system CPU usage is much higher than it ever was
prior to the May 7th release.
To suggest nothing is wrong with whatever was done to opensim between the
March release I tested and the May 7th release is very difficult to accept.
If the only problem was higher CPU usage I would accept it, but since
performance and reliability have also been diminished this is not something
that can be ignored.
Memory that gets used and not released is not a good thing. This is
happening across three released of Fedora linux. Fedora 19, 20, and 21, on
four different servers.
I do not know if I should report this in mantis . Usually when I report
something it is already being worked on under some other report using
language I would not think to search on.
Tom
-----Original Message-----
From: Luisillo Contepomi
Sent: Thursday, May 28, 2015 12:00 PM
To: [email protected]
Subject: Re: [Opensim-users] Memory usage
I have setup always false in "AppDomainLoading" in all my regions.
Is very important the difference.
Results with "AppDomainLoading" in OpenSim.ini
( 4467 scripts in this region, no avatars login.)
AppDomainLoading = "false"
--------------------------------------------
MEMORY STATISTICS
Heap allocated to OpenSim : 1456 MB
Last heap allocation rate : 20.647 MB/s
Average heap allocation rate: 9.178 MB/s
Process memory : 1868 MB
AppDomainLoading = "true"
----------------------------------------
MEMORY STATISTICS
Heap allocated to OpenSim : 918 MB
Last heap allocation rate : 20.23 MB/s
Average heap allocation rate: 12.022 MB/s
Process memory : 3570 MB
---------------------------------------
Status of XEngine instance for Continente
Scripts loaded : 4467
Scripts waiting for load : 0
Max threads : 100
Min threads : 2
Allocated threads : 100
In use threads : 100
Work items waiting : 104
Events queued : 418
Events processed : 165762
Sensors : 157
Dataserver requests : 0
Timers : 702
Listeners : 78
Regards,
Luisillo
2015-05-28 17:06 GMT+02:00 Thomas Ringate <[email protected]>:
Very interesting. My setup is quite different in that I do not run the
grid
but rather use the OSgrid. I only run the regions. That could be a huge
difference. However since OSgrid is officially designated as the
"opensim"
test grid, I would hope the developers are serious about the results
people
are having who use that grid.
I am in the process of trying to discover exactly when this problem
started
for me as was earlier requested.
The different versions I can test with are these releases from OSgrid:
osgrid-opensim-01162015.v0.8.1.97ac80d
osgrid-opensim-01172015.v0.8.1.1f04e1b
osgrid-opensim-02212015.v0.8.1.7b9ad11
osgrid-opensim-03142015.v0.8.1.8b13e4e
osgrid-opensim-03172015.v0.8.2.83e58eb
osgrid-opensim-04052015.v0.8.2.8d66284
osgrid-opensim-04122015.v0.8.2.d96d31b
osgrid-opensim-05072015.v0.8.2.c74cef0
osgrid-opensim-05092015.v0.8.2.adf0f49
osgrid-opensim-05232015.v0.8.2.abb3bb6
Later today I will cycle all of these releases on one of my regions and
run
a test using "monit" with my original parameters set to see which version
it
was that started using so much simulator memory. The change control on my
home network is not near as professional as what I lived with at IBM for
31
years. Now I depend on my own memory. That is not a very good choice in
my
case. Luisillo's testing was far more accurate with excellent metrics to
compare. Bravo.
Tom
-----Original Message----- From: Luisillo Contepomi
Sent: Thursday, May 28, 2015 5:07 AM
To: [email protected]
Subject: Re: [Opensim-users] Memory usage
I have not problems or issues with memory.
My memory test results:
Grid and sim in the same machine.
Windows7 - 64 - RAM 8GB
Intel Core i7-3770 at 3.40GHz
Users in this test is out in other machines connecting by a 100Mb network.
All users in this test are using Radegast. All have Inventory with
1200 Items aprox.
and worn scripts as AO, radar and resizers in hair and shoes.
Continente is a Var Region 1024x1024
Prims used 55.048
Scripts 4.499
Region (Continente) # xengine status
Status of XEngine instance for Continente
Scripts loaded : 4499
Scripts waiting for load : 0
Max threads : 100
Min threads : 2
Allocated threads : 100
In use threads : 42
Work items waiting : 0
Events queued : 636
Events processed : 1028273
Sensors : 157
Dataserver requests : 0
Timers : 715
Listeners : 85
From tast manager in K
CPU 1 from 8 at 70 - 75% all time during test.
-----------------
Memory OpenSim.exe / MySql server
Start region at. 9.39h
------------------------------------------------------------
1.613.652 no users 9.40h 652.976
2.140.920 no users 9.45h "
------------------------------------------------------------
finish scripts load 9.46h "
I order in console "fcache clear file" 9.47h Im not using memory cache. "
------------------------------------------------------------
2.142.964 no users 9.47h "
2.134.240 no users 9.48h 653.140
1.468.540 no users 9.49h
1.148.276 no users 9.50h (fcache clear finished.)
Test with 2 users
-------------------------------------------------------------
1.148.384 Login in one user 9.54H (RADEGAST)
1.185.828 with 1 user 9.56h
1.337.280 with 2 users 9.58h 734.548
1.389.148 2 users one is searching in his inventory 735.900
1.411.896 logout one user 10.01h 736.452
1.515.700 1 user 10.02H
1.271.528 1 user 10.03H 736.468
---------------------------------------------------
1.257.620 logout user No users in 10.04H 736.692
1.221.208 no users 10.05h 736.692
1.207.296 no users 10.07h 736.692
1.204.980 no users 10.07h 735.660
1.204.736 no users 10.09h 735.660
------------simultaneus 8 users login at 10.10h
---------------------------------------------------------------
1.228.172 10.10h.
1.427.396 10.11h 5 users login
1.699.788 10.12h 8 users login
1.937.552 10.13h 8 users in home
2.008.496 10.14h 8 users in home 745.468
2.010.358 10.15h 8 users in home 745.928
1.960.540 10.14h 8 users in home 745.944
1.887.942 10.15h 8 users in home "
1.946.372 10.16h 8 users in home "
1.902.660 10.17h 8 users in home location "
1.927.866 10.18h " "
1.872.444 10.19h " "
1.885.264 10.20h " "
1.838.972 10.23h " "
1.908.056 10.25h "
Now logout all users one by minute.
1 - 1.906.736 10.26h 746.528
2 - 1.907.524 10.27h 746.532
3 - 1.876.456 10.28h 746.612
4 - 1.884.836 10.29h "
5 - 1.788.868 10.30h 746.852
6 - 1.732.212 10.31h "
7 - 1.756.340 10.32h "
8 - 1.648.408 10.33h "
NO USERS
1.629.464 10.34h "
1.618.988 10.36h
1.434.444 10.37h
Robust.exe always near 136.570K
---------------------------------------------------------
10.17h When have 8 users STATS:
---------------------------------------------------------
Region (Continente) # show stats
CONNECTION STATISTICS
Client logouts due to no data receive timeout: 0
SAMPLE FRAME STATISTICS
Dilatn SimFPS PhyFPS AgntUp RootAg ChldAg Prims AtvPrm AtvScr
ScrLPS
1.00 55 55.6 0.0 8 0 56152 25 4501
10991070
PktsIn PktOut PendDl PendUl UnackB TotlFt NetFt PhysFt OthrFt
AgntFt
ImgsFt
20 123 0 0 1142 18.1 0.0 0.4 0.0 0.6
0.2
MEMORY STATISTICS
Heap allocated to OpenSim : 1647 MB
Last heap allocation rate : 8.061 MB/s
Average heap allocation rate: 9.906 MB/s
Process memory : 2007 MB
10.46H NO USERS STATS
------------------------------------------------------
Region (Continente) # show stats
CONNECTION STATISTICS
Client logouts due to no data receive timeout: 0
SAMPLE FRAME STATISTICS
Dilatn SimFPS PhyFPS AgntUp RootAg ChldAg Prims AtvPrm AtvScr
ScrLPS
1.00 55 55.3 0.0 0 0 54875 24 4451
13512840
PktsIn PktOut PendDl PendUl UnackB TotlFt NetFt PhysFt OthrFt
AgntFt
ImgsFt
13 78 0 0 0 18.3 0.0 0.6 0.0 0.0
0.0
MEMORY STATISTICS
Heap allocated to OpenSim : 768 MB
Last heap allocation rate : 1.732 MB/s
Average heap allocation rate: 2.493 MB/s
Process memory : 1493 MB
For me all is ok.
2015-05-28 1:53 GMT+02:00 Teravus Ovares <[email protected]>:
https://www.jetbrains.com/dotmemory/ dotmemory has been relatively
successful at identifying memory issues in the past.
That said, it's not free and there may be free alternatives..... but be
advised you want one that supports managed memory... and one that
works
well with Mono.Addins. Many free ones in the past have had issues
dynamically loading modules with Mono.addins. If the modules are not
loaded, you will see the server come up with no regions and it will be
pretty useless for testing. Testing physics is a bit harder because it
is
both managed and unmanaged... so you'll need to examine the pinned
memory.
Also, note that while running the simulator in a profiler such as
dotMemory,
it will run noticeably slower depending on how you do the profiling.
Regards
Teravus
On Wed, May 27, 2015 at 3:01 AM, M.E. Verhagen <[email protected]>
wrote:
The memory consumption is troubling.
It seems to be an cross platform issue, it happens on linux, mac and
also
windows.
The script engine can be responsible for the skyrocketing of the memory
usage when an avatar enters the region.
The inventory system is also called as a possible suspect.
Also the physics system can be un issuer.
This throws up a question, how can the memory usage be broken down, to
see
what part is to blame ?
_______________________________________________
Opensim-users mailing list
[email protected]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
_______________________________________________
Opensim-users mailing list
[email protected]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
_______________________________________________
Opensim-users mailing list
[email protected]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
_______________________________________________
Opensim-users mailing list
[email protected]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
_______________________________________________
Opensim-users mailing list
[email protected]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
_______________________________________________
Opensim-users mailing list
[email protected]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
