What kind of connection do you have between the asset server and the
simulators. We have some very large regions that are pack to the brim,
they rarely take 30 min from cold start, a warm start is in minutes
before the login are enabled. Out asset servers are on the same lan as
the simulator servers It takes about 60 min to soft restart the
server. And this will delay start up of simulators keeps simulators
from locking out the core code from each other. We figured .8 to .9
its taks about double the memory and maybe 25% longer start times.
Main due to physic engine
Was looking at a few of our big regions, they take maybe 5 min to load
assets at max, and maybe the same with scripts. This is with 0 cache
hits. and that at teh largest regions we have right now with 30k
objects/8k scripts
We use an older version of .9 that 2 months old, and been waiting for
the release version to upgrade, we have a bit of work to incorporate
our grid mods into the code. were in a production environment and
have to stay a step behind the bleeding edge or it will cut deep :)
even now we have to restart the region about 1 to 2 days because of
memory leak and the "creep". The memory leak is when the region grow
from 1.4gb used to 20gb use. The creep is something a little
different, it apears the region disconnects from the grid, and you
can TP to it or out of it. A soft restart and it working like new. All
I have come with is it being disconnected from the Asset server and
not reconnecting.
Wish you luck, I might be making a new OSGrid region soon and maybe
enable hypergrid who knows our little garden works for us.
David.
On Tue, Jun 27, 2017 at 9:01 PM, Chris <mewtwo0...@gmail.com
<mailto:mewtwo0...@gmail.com>> wrote:
I have been seeing similar issues myself between the past week and
a half - two weeks on master
Though the previous responses in this thread mention map and asset
cache population as being a possible issue I'm not so sure that
this is the case in this example because there is very little
rezzed out in my case; 2 of the regions are completely empty and
the other 2 are sparsely populated with random cube prims here and
there mostly for script testing.
This is being seen on a test install independent from my main grid
(not running both at the same time), clean database, 4 regions,
grid mode (Not HG, 1 OpenSim.exe instance), Windows 7 x64 under
.NET, and on ubODE. I'm also using the default map tile module
(That is, the old one, and not Warp3D)
Log File Examples:
17:05:47 - [REGION LOADER FILE SYSTEM]: Loaded config for region
Test Sandbox
17:05:47 - [LOAD REGIONS PLUGIN]: Loading specific shared modules...
17:05:47 - [LOAD REGIONS PLUGIN]: Done.
17:05:47 - [LOAD REGIONS PLUGIN]: Creating Region: Monte Cristo
(ThreadID: 1)
17:05:47 - [SCENE]: Default script engine XEngine
17:05:47 - [SCENE]: Region Monte Cristo, WORLD MAP refresh time
set to 0 seconds
17:05:47 - [SCENE]: Using the BestAvatarResponsiveness
prioritization scheme
17:05:47 - [HEIGHTMAP TERRAIN DATA]: VD2Gzip 296 bytes input
17:05:47 - [HEIGHTMAP TERRAIN DATA]: V2DGzip. Heightmap
size=<256,256>. Region s
ize=<256,256>
17:05:47 - [HEIGHTMAP TERRAIN DATA]: HeightmapTerrainData create
from Variable2D
Gzip serialization. Size=<256,256>
17:05:47 - [HEIGHTMAP TERRAIN DATA]: VD2Gzip 296 bytes input
17:05:47 - [HEIGHTMAP TERRAIN DATA]: V2DGzip. Heightmap
size=<256,256>. Region s
ize=<256,256>
17:05:47 - [HEIGHTMAP TERRAIN DATA]: HeightmapTerrainData create
from Variable2D
Gzip serialization. Size=<256,256>
17:05:47 - [MODULES]: Loading Region's modules (old style)
17:05:47 - [REGIONMODULES]: Loading Region's modules (new style)
17:05:47 - [CHAT]: Initialized for Monte Cristo w:10 s:20 S:100
17:05:47 - [MESSAGE TRANSFER]: Message transfer module active
17:05:47 - [REMOTE GRID USER CONNECTOR]: Enabled remote grid user
for region Mon
te Cristo
(A lot of times it gets stuck for a long time here)
Another log from a different OS session:
17:07:55 - [LOAD REGIONS PLUGIN]: Creating Region: Monte Cristo
(ThreadID: 1)
17:07:55 - [SCENE]: Default script engine XEngine
17:07:55 - [SCENE]: Region Monte Cristo, WORLD MAP refresh time
set to 0 seconds
17:07:55 - [SCENE]: Using the BestAvatarResponsiveness
prioritization scheme
17:07:55 - [HEIGHTMAP TERRAIN DATA]: VD2Gzip 296 bytes input
17:07:55 - [HEIGHTMAP TERRAIN DATA]: V2DGzip. Heightmap
size=<256,256>. Region s
ize=<256,256>
17:07:55 - [HEIGHTMAP TERRAIN DATA]: HeightmapTerrainData create
from Variable2D
Gzip serialization. Size=<256,256>
17:07:55 - [HEIGHTMAP TERRAIN DATA]: VD2Gzip 296 bytes input
17:07:55 - [HEIGHTMAP TERRAIN DATA]: V2DGzip. Heightmap
size=<256,256>. Region s
ize=<256,256>
17:07:55 - [HEIGHTMAP TERRAIN DATA]: HeightmapTerrainData create
from Variable2D
Gzip serialization. Size=<256,256>
17:07:55 - [MODULES]: Loading Region's modules (old style)
17:07:55 - [REGIONMODULES]: Loading Region's modules (new style)
(Sometimes it gets stuck for a long time here but not as often as
the first example)
On 6/25/2017 7:40 PM, AJLDuarte wrote:
Hi
Good to read things are not so bad after all
About reboots, do them as often as your use case allows (and
backups of course).
Regions memory use keeps piling up.. etc, etc
Ubit
*From:*opensim-users-boun...@opensimulator.org
<mailto:opensim-users-boun...@opensimulator.org>
[mailto:opensim-users-boun...@opensimulator.org
<mailto:opensim-users-boun...@opensimulator.org>] *On Behalf Of
*tring...@gmail.com <mailto:tring...@gmail.com>
*Sent:* Monday, June 26, 2017 01:28
*To:* opensim-users@opensimulator.org
<mailto:opensim-users@opensimulator.org>
*Subject:* Re: [Opensim-users] What is going on with the latest
release ofopensim for OSgrid
Ubit,
After checking everything I decided to do a “warm” start, one
where it is not a total upgrade with all the compiled stuff
deleted. This time the restart took 14 minutes which is what I
normally observe. I guess I never watched a cold restart before
so I guess maybe it always took that long. Usually I let the
updates happen at the middle of the night restart I have
scheduled. I always found everything working just fine in the
mornings.
Sorry for the alarm, but the 70 minute restart time took me by
surprise.
One good thing I did learn from all of this was that it may not
be necessary to do a reboot every night to prevent lag. One of
my servers ran 12 days with no reboot because I had forgot to
turn the nightly reboot back on during a routine manual
maintenance I did earlier this month.
I may go to a once a week reboot on one server and see how that
goes. If all goes well, I’ll change them all to once a week.
Tom
*From:*AJLDuarte
*Sent:*Sunday, June 25, 2017 12:55 PM
*To:*opensim-users@opensimulator.org
<mailto:opensim-users@opensimulator.org>
*Subject:*Re: [Opensim-users] What is going on with the latest
release ofopensim for OSgrid
Hi
What was the state of folder bin/AssetsCache on those 2 tests?
According to options, map generation my require full assets
fetch from grid and that can take ages if not cached.
Of course, make sure FloatSam cache is proper configure. It
should no longer be optional, since it is needed and others don’t
do the job now.
Also seems your t2 is not doing map and not using ubOde ?
Even with all assets present in cache map is heavy, can take a bit
ubOde does operations before scene Ready so that ready does mean
it a bit more
Other engines do it later so you don’t see it and region Ready
really doesn’t mean much.
Ubit
*From:*opensim-users-boun...@opensimulator.org
<mailto:opensim-users-boun...@opensimulator.org>
[mailto:opensim-users-boun...@opensimulator.org
<mailto:opensim-users-boun...@opensimulator.org>] *On Behalf Of
*tring...@gmail.com <mailto:tring...@gmail.com>
*Sent:* Sunday, June 25, 2017 17:32
*To:* OS-Opensim Users
*Subject:* [Opensim-users] What is going on with the latest
release of opensim for OSgrid
I just did a update to the latest release of opensim released by
OSgrid and it took 1 hour and 12 minutes from the time the
command was give to load before it enabled logins.
This is on a 4 by 4 var region that previously took 17 minutes to
load. While that is nothing to write home about, an hour and 12
minutes is totally insane.
This region runs on a dedicated server with nothing else running
on it at all. It is a 6 core AMD with 16GB of ram.
Using “top” to view cpu usage the most I ever noticed it going to
as 16% total with only one core hitting 100% for a few MS every
now and then.
It looked like it too most og that time generating the 16 tile
textures for the grid display, but scripts did take about 11
minutes to get started. That part did not appear to be
significantly different than before.
Here are some key log entries so you can see how long each thing
is taking.
2017-06-25 10:34:15,481 INFO [OPENSIM MAIN]: configured log4net
using default OpenSim.exe.config <—first log entry
2017-06-25 10:34:25,743 DEBUG [REGION DB]: Loaded 5694 objects
using 47381 prims
2017-06-25 10:34:31,207 DEBUG [REGION DB]: Loaded inventory from
3563 objects
2017-06-25 10:34:31,209 INFO [SCENE]: Loaded 5694 objects from
the datastore
2017-06-25 10:34:40,244 DEBUG [WORLD MAP]: Generating map image
for TSim 1
<—big delay
2017-06-25 11:17:53,580 DEBUG [WORLD MAP]: Storing map image
c393c67c-eb69-43b1-b6bd-650d06c59e99 for TSim 1
2017-06-25 11:17:55,255 INFO [SCENE]: Initializing script
instances in TSim 1
2017-06-25 11:19:14,586 INFO [SCENE]: Initialized 6604 script
instances in TSim 1
2017-06-25 11:19:14,979 INFO [ubOde] start processing pending
actor operations
<—big delay
2017-06-25 11:29:23,776 INFO [XEngine]: Started 50 scripts in TSim 1
<—big delay but not much different than previous version
2017-06-25 11:45:33,268 INFO [XEngine]: Completed starting 6583
scripts on TSim 1
2017-06-25 11:45:57,478 DEBUG [RegionReady]: Region "TSim 1" is
ready: "server_startup,1,21," on channel -800
This is a very limited log entry list as there are hundreds of
entries many of them listing bad textures or other things
indicating corrupt assets coming from the asset server.
This was a cold start where it does have to generate and compile
stuff, but it has never taken any where near this long for a
region to come online before.
No one else has noticed this huge increase in start-up time?
I’m running 4 servers, so I tried this same region on my test
server, it only has 8gb or ram and a bit slower 6 core AMD cpu.
It took 2 hours on that server.
My other 4 by 4 region TSim 2 takes 12 minutes to come online.
This is not the cold start time, which does take longer.
2017-06-25 04:02:57,726 INFO [OPENSIM MAIN]: configured log4net
using default OpenSim.exe.config
2017-06-25 04:04:22,724 DEBUG [REGION DB]: Loaded 3068 objects
using 33622 prims
2017-06-25 04:03:25,024 INFO [XEngine]: Initializing scripts in
region TSim 2
2017-06-25 04:09:34,146 INFO [XEngine]: Started 50 scripts in TSim 2
2017-06-25 04:15:45,075 INFO [XEngine]: Completed starting 4534
scripts on TSim 2
2017-06-25 04:15:45,138 DEBUG [RegionReady]: Region "TSim 2" is
ready: "server_startup,1,27," on channel -800
------------------------------------------------------------------------
_______________________________________________
Opensim-users mailing list
Opensim-users@opensimulator.org
<mailto:Opensim-users@opensimulator.org>
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
<http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users>
_______________________________________________
Opensim-users mailing list
Opensim-users@opensimulator.org
<mailto:Opensim-users@opensimulator.org>
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
<http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users>
--
OpenSim: 10 Region Standalone on 0.8.1 Dev
Physics: Open Dynamics Engine
OS: Windows 7 (x64)
CPU: AMD FX 8320 8-Core 3.5 GHz
Memory: 16 GB DDR3
Database: MySQL 5.1.63 (x64)
_______________________________________________ Opensim-users
mailing list Opensim-users@opensimulator.org
<mailto:Opensim-users@opensimulator.org>
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
<http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users>
_______________________________________________
Opensim-users mailing list
Opensim-users@opensimulator.org
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users