On Fri, Jun 06, 2008 at 01:05:38PM +0200, Frederik Ramm wrote:
> Hi,
>
> > I think browsing will get slightly faster too:
> > - extra overhead for one tile is only 4KB, it's propably less than
> > overhead for getting file in filesystem with millions of files
>
> I disagree. Apache is extremely efficient at delivering files.
> Internally there's even operating system support ("take this file
> handle and this network socket, then stream out contents").
You talk about zerocopy ot the api "sendfile" which should a be a huge
overhead for this type of file serving. The problem with the current
setup is the huge amount of files. Copying 4k is NOTHING compared to a
seek to the right directory location. Todays CPUs have copied half a
gigabyte in memory before the disk reached the right position.
The whole issue is i/o bound not cpu bound so you gain nothing with
zerocopy.
> proposed architecture would at the very best (if done as a C/C++
> module) achieve the same performance, but is very likely to incur
> extra overhead. If, god forbid, done in PHP or something like it,
> data will have to be copied...
Look at the history. Old news systems (netnews - NNTP) used one file per
article e.g. cnews and later inn. Then someday people discovered that
unix or better all filesystems have a problem with millions of files so
they switched to containers e.g. multigigabyte files with a much more
simplified way of accessing e.g. an index saying - article A starts at
offset 55605 and its size is 5500 bytes.
The advantage is that writing articles to a storage means only appending
or linear overwriting of the containers storage and expiring the old
locations. Thing about a log structured filesystems.
Writing software to cover this for openstreetmap should be kind of
straight forward - simplified http server which directly converts the
url via a very simple in memory index to a file, offset, length.
Flo
--
Florian Lohoff [EMAIL PROTECTED] +49-171-2280134
Those who would give up a little freedom to get a little
security shall soon have neither - Benjamin Franklin
signature.asc
Description: Digital signature
_______________________________________________ Tilesathome mailing list [email protected] http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/tilesathome
