Re: [Opensim-users] Are unused assets deleted? (Fred Beckhusen)

[Ethan Gardener]

On Mon, Aug 27, 2018, at 1:47 PM, Haravikk wrote:
> 
> > On 27 Aug 2018, at 09:57, Ethan Gardener  > > wrote:
> > 
> > On Mon, Aug 27, 2018, at 8:06 AM, Haravikk wrote:
> >> 
> >>> On 26 Aug 2018, at 15:05, Ethan Gardener  >>> > wrote:
> >>> 
> >>> However, everything I've ever learned about databases says that it's a 
> >>> bad idea to just look at the cost of the disks.  The cost of *accessing* 
> >>> the data (with acceptable performance) goes up exponentially with the 
> >>> size of the dataset.
> >> 
> >> This shouldn't be the case; any properly indexed database should have 
> >> roughly constant, or at worst logarithmic, access times for data.
> > 
> > *sigh* "Logarithmic" is another way of saying "exponential'.  It is not a 
> > trivial matter unless the data set is comparatively small.  In asking about 
> > these things, I'm particularly thinking of InWorldz, where asset data was 
> > growing at a terabyte a month.  Not small.  
> 
> Logarithmic complexity isn't the same as exponential complexity, in fact 
> it's the opposite; logarithmic complexity plateaus as the search space 
> increases, meaning it effectively becomes constant time, an algorithm 
> with exponential complexity would grow by a power of the search space 
> size (e.g- the search time might quadruple each time the search space 
> doubles in size). See here:
> https://en.wikipedia.org/wiki/Logarithmic_growth 
> 
> https://en.wikipedia.org/wiki/Exponential_growth 
> 

Oh I'm sorry, I was reasoning from a bad premise.  I later remembered I 
regularly use a program which has logarithmic search time.  (It's not even a 
database, it's the grep command in Plan 9 from Bell Labs.)  Thanks for your 
explanations.

This leaves the problem as an even bigger puzzle to me, I don't think I can 
even begin to reason on possible causes.  Perhaps I should just leave it at 
this, as I'm reducing my involvement in virtual worlds for personal reasons 
anyway.  (I'm taking up programming instead.  Maybe in a few years, I'll have 
something to offer. :)  

-- 
The lyf so short, the craft so long to lerne. -- Chaucer
___
Opensim-users mailing list
Opensim-users@opensimulator.org
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users

Re: [Opensim-users] Are unused assets deleted? (Fred Beckhusen)

[Haravikk]

> On 27 Aug 2018, at 09:57, Ethan Gardener  > wrote:
> 
> On Mon, Aug 27, 2018, at 8:06 AM, Haravikk wrote:
>> 
>>> On 26 Aug 2018, at 15:05, Ethan Gardener >> > wrote:
>>> 
>>> However, everything I've ever learned about databases says that it's a bad 
>>> idea to just look at the cost of the disks.  The cost of *accessing* the 
>>> data (with acceptable performance) goes up exponentially with the size of 
>>> the dataset.
>> 
>> This shouldn't be the case; any properly indexed database should have 
>> roughly constant, or at worst logarithmic, access times for data.
> 
> *sigh* "Logarithmic" is another way of saying "exponential'.  It is not a 
> trivial matter unless the data set is comparatively small.  In asking about 
> these things, I'm particularly thinking of InWorldz, where asset data was 
> growing at a terabyte a month.  Not small.  

Logarithmic complexity isn't the same as exponential complexity, in fact it's 
the opposite; logarithmic complexity plateaus as the search space increases, 
meaning it effectively becomes constant time, an algorithm with exponential 
complexity would grow by a power of the search space size (e.g- the search time 
might quadruple each time the search space doubles in size). See here:
https://en.wikipedia.org/wiki/Logarithmic_growth 

https://en.wikipedia.org/wiki/Exponential_growth 


A logarithmic algorithm is usually one that partitions the search space, for 
example a binary search which divides the search space in two with each 
iteration until only the target result (or nothing) remains. Some databases do 
use such algorithms but usually for intermediate operations that asset lookups 
shouldn't normally require.

For the vast majority of asset lookups (should be all of them, but it may 
depend on database used) a lookup will be constant time, e.g- using a form of 
hash-table to immediately find (or not) a request GUID. The amount of data 
stored shouldn't have a big impact on such lookups, the only case where you 
might notice a difference is if previously the entire index fit within RAM, but 
after growing no longer does (so has to come from disk), but most database 
engines are optimised on the assumption that you'll never have enough RAM to 
rely on keeping a whole index in memory anyway.

>> Most (all?) external asset requests are now served to the viewer via 
>> HTTP, so you could use Squid proxy or similar to cache those requests 
>> before they touch your asset server, it might even be possible to use a 
>> reverse proxy such as Cloudflare to do this for you, though I've never 
>> tried it (you might need to use page rules to force it to cache asset 
>> URLs).
> 
> I'm sure this would help, but wouldn't it miss the case of large vehicles 
> making sim crossings?  A vehicle may be non-physical and not phantom, so the 
> sim needs to know what shape it is and the shape may be complex.  In any 
> case, sim crossings seem to be the first thing to go as a grid grows.

I'm not overly familiar with how OpenSim handles inter-region asset exchanges; 
but with regions belonging to the same grid there should be no need to send the 
asset to the asset server, it should either be sent directly to the destination 
region, or be retained by the origin region on failure. An increase in 
failures/laggy crossings would normally be an issue of a slow or overloaded 
connection between regions, or the destination region itself being overloaded.

In cases where a region does send an asset to the asset server a cache wouldn't 
help anyway, as you'd be storing a new asset (or overwriting an existing one, 
in the case of worn attachments). But this again wouldn't be an issue with the 
size of the asset server, but the amount of traffic being sent to it.
___
Opensim-users mailing list
Opensim-users@opensimulator.org
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users

Re: [Opensim-users] Are unused assets deleted? (Fred Beckhusen)

[Ethan Gardener]

On Mon, Aug 27, 2018, at 8:06 AM, Haravikk wrote:
> 
> > On 26 Aug 2018, at 15:05, Ethan Gardener  wrote:
> > 
> > However, everything I've ever learned about databases says that it's a bad 
> > idea to just look at the cost of the disks.  The cost of *accessing* the 
> > data (with acceptable performance) goes up exponentially with the size of 
> > the dataset.
> 
> This shouldn't be the case; any properly indexed database should have 
> roughly constant, or at worst logarithmic, access times for data.

*sigh* "Logarithmic" is another way of saying "exponential'.  It is not a 
trivial matter unless the data set is comparatively small.  In asking about 
these things, I'm particularly thinking of InWorldz, where asset data was 
growing at a terabyte a month.  Not small.  

> You 
> should only really see big performance drops if the number of assets 
> requests increases beyond what the hardware can cope with (i.e- too many 
> users online simultaneously).
> 
> > There is a solution which I think practically returns the problem to just 
> > counting the cost of the disks:  Cache the asset server.  The actual asset 
> > server which never deletes anything can then be very slow, with all 
> > regularly used assets served from the much faster cache.  
> > 
> > I'm almost certain Second Life implemented such a cache years ago.  I 
> > remember when I logged in there after a long absence, things which I'd made 
> > or which were old-style took minutes to rez.  I don't ever remember seeing 
> > such an effect in InWorldz, so I think it's possible InWorldz didn't have 
> > such a cache.
> 
> Most (all?) external asset requests are now served to the viewer via 
> HTTP, so you could use Squid proxy or similar to cache those requests 
> before they touch your asset server, it might even be possible to use a 
> reverse proxy such as Cloudflare to do this for you, though I've never 
> tried it (you might need to use page rules to force it to cache asset 
> URLs).

I'm sure this would help, but wouldn't it miss the case of large vehicles 
making sim crossings?  A vehicle may be non-physical and not phantom, so the 
sim needs to know what shape it is and the shape may be complex.  In any case, 
sim crossings seem to be the first thing to go as a grid grows.
___
Opensim-users mailing list
Opensim-users@opensimulator.org
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users

Re: [Opensim-users] Are unused assets deleted? (Fred Beckhusen)

[Haravikk]

> On 26 Aug 2018, at 15:05, Ethan Gardener  wrote:
> 
> However, everything I've ever learned about databases says that it's a bad 
> idea to just look at the cost of the disks.  The cost of *accessing* the data 
> (with acceptable performance) goes up exponentially with the size of the 
> dataset.

This shouldn't be the case; any properly indexed database should have roughly 
constant, or at worst logarithmic, access times for data. You should only 
really see big performance drops if the number of assets requests increases 
beyond what the hardware can cope with (i.e- too many users online 
simultaneously).

> There is a solution which I think practically returns the problem to just 
> counting the cost of the disks:  Cache the asset server.  The actual asset 
> server which never deletes anything can then be very slow, with all regularly 
> used assets served from the much faster cache.  
> 
> I'm almost certain Second Life implemented such a cache years ago.  I 
> remember when I logged in there after a long absence, things which I'd made 
> or which were old-style took minutes to rez.  I don't ever remember seeing 
> such an effect in InWorldz, so I think it's possible InWorldz didn't have 
> such a cache.

Most (all?) external asset requests are now served to the viewer via HTTP, so 
you could use Squid proxy or similar to cache those requests before they touch 
your asset server, it might even be possible to use a reverse proxy such as 
Cloudflare to do this for you, though I've never tried it (you might need to 
use page rules to force it to cache asset URLs).

- Haravikk
___
Opensim-users mailing list
Opensim-users@opensimulator.org
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users

Re: [Opensim-users] Are unused assets deleted? (Fred Beckhusen)

[Ethan Gardener]

Thank you for your reply, melanie.  Reasoning on the problem, I've come to 
agree that deleting unused assets is an extremely difficult problem, perhaps 
not worth solving.

However, everything I've ever learned about databases says that it's a bad idea 
to just look at the cost of the disks.  The cost of *accessing* the data (with 
acceptable performance) goes up exponentially with the size of the dataset.  
There is a solution which I think practically returns the problem to just 
counting the cost of the disks:  Cache the asset server.  The actual asset 
server which never deletes anything can then be very slow, with all regularly 
used assets served from the much faster cache.  

I'm almost certain Second Life implemented such a cache years ago.  I remember 
when I logged in there after a long absence, things which I'd made or which 
were old-style took minutes to rez.  I don't ever remember seeing such an 
effect in InWorldz, so I think it's possible InWorldz didn't have such a cache. 
 

Does OpenSim cache the asset server?  I'm thinking it doesn't, because some 
grid techs will tell you the first thing they did for their new grid was 
rewrite the asset server.  Of course, maybe it does now but didn't when they 
were getting started.  I'd be interested to know.

On Sun, Aug 12, 2018, at 6:08 PM, Melanie wrote:
> There isn't a real reason to expend that sort of time and effort on 
> cleaning assets. First off, for assets named in scripts, a heuristic is 
> used. It can't detect all assets with certainty and it will also have 
> false positives. Therefore, some things may break an some unused assets 
> will be kept. However, and that is the more important part, even a large 
> commercial grid operating for years accumulates assets that still easily 
> fit on a modern hard disk. Furthermore, if one disk fills up another can 
> be added. So, for many grids I know and certainly for myself, the mantra 
> is "disk space is cheap" and that way there isn't a risk of breaking 
> users' creations. Deduplicating asset systems like fsassets make this 
> even more relevant, because most large assets are meshes and textures 
> that are uploade by many people, but are actually identical. 
> Deduplicating is a massive help there. At the end of the day, there is 
> only one way to remove a prim, even one without contents, completely. If 
> you rez a prim and then delete it, it will be taken off the region, 
> serialized and put into an asset to be put into your trash. So that is 
> not the solution. In fact, the only way to remove a prim from a region 
> without creating an asset is llDie(). It creates no copy. - Melanie  
> On Sun, 12 Aug 2018 16:46:15 + Fred Beckhusen  
> wrote  1. Are unused assets deleted? (Ethan A. Gardener) Dumping and 
> reloading breaks all Firestorm links to Outfits. Take a look at thi web 
> page at Wizardry and Steamworks (grimore.org/opensim/database/
> asset_cleaner) for a possible way to clean a db It uses PHP with OARS 
> and IARS to make lists of what to keep, then deletes the rest. Fred 
> ___ Opensim-users mailing 
> list Opensim-users@opensimulator.org 
> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
> ___
> Opensim-users mailing list
> Opensim-users@opensimulator.org
> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users


-- 
The lyf so short, the craft so long to lerne. -- Chaucer
___
Opensim-users mailing list
Opensim-users@opensimulator.org
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users

Re: [Opensim-users] Are unused assets deleted? (Fred Beckhusen)

[Melanie]

There isn't a real reason to expend that sort of time and effort on cleaning 
assets. First off, for assets named in scripts, a heuristic is used. It can't 
detect all assets with certainty and it will also have false positives. 
Therefore, some things may break an some unused assets will be kept. However, 
and that is the more important part, even a large commercial grid operating for 
years accumulates assets that still easily fit on a modern hard disk. 
Furthermore, if one disk fills up another can be added. So, for many grids I 
know and certainly for myself, the mantra is "disk space is cheap" and that way 
there isn't a risk of breaking users' creations. Deduplicating asset systems 
like fsassets make this even more relevant, because most large assets are 
meshes and textures that are uploade by many people, but are actually 
identical. Deduplicating is a massive help there. At the end of the day, there 
is only one way to remove a prim, even one without contents, completely. If you 
rez a prim and then delete it, it will be taken off the region, serialized and 
put into an asset to be put into your trash. So that is not the solution. In 
fact, the only way to remove a prim from a region without creating an asset is 
llDie(). It creates no copy. - Melanie  On Sun, 12 Aug 2018 16:46:15 + 
Fred Beckhusen  wrote  1. Are unused assets deleted? 
(Ethan A. Gardener) Dumping and reloading breaks all Firestorm links to 
Outfits. Take a look at thi web page at Wizardry and Steamworks 
(grimore.org/opensim/database/asset_cleaner) for a possible way to clean a db 
It uses PHP with OARS and IARS to make lists of what to keep, then deletes the 
rest. Fred ___ Opensim-users 
mailing list Opensim-users@opensimulator.org 
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
___
Opensim-users mailing list
Opensim-users@opensimulator.org
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users

[Opensim-users] Are unused assets deleted? (Fred Beckhusen)

[Fred Beckhusen]

1. Are unused assets deleted? (Ethan A. Gardener)

Dumping and reloading breaks all Firestorm links to Outfits.

Take a look at thi web page at Wizardry and Steamworks 
(grimore.org/opensim/database/asset_cleaner) for a possible way to clean 
a db


It uses PHP with OARS and IARS to make lists of what to keep, then 
deletes the rest.


Fred

___
Opensim-users mailing list
Opensim-users@opensimulator.org
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users