> No doubt that this file operation will be quicker, but is that all we'd > need to test for? We need to 1)Upload (no change) 2)Check if the tileset > is complete (ie, peek at every tile in the uploaded blob) 3a) move the > tile to its end destination (if it's a complete tileset) 3b) or extract > the tiles and insert them into the destination blob if only a single > tile, or a few tiles, or an incomplete tileset has been uploaded (means > copying over the whole binary blob, and inserting the new tiles into it)
Why you need to check for tileset completness? File format is defined to contain offset to all possible tiles followed by tile data (which maybe zero length). It's not possible to send incomplete tilesets. If there is some reason to allow sending incomplete tilesets then current uploading system with zip files can be kept. It will combine tiles from aggregated file and zip. This may be slower than original approach but I don't suppose this will happen too often to be significant > And then you have to consider tile serving. Right now, it's one file per > tile, no PHP etc involved. Apache and the OS can do all their file > system tricks, e.g. caching often viewed z12 tiles in memory etc. OS doesn't care who reads the data. Tile read by php will be cached in the same way the tile read by apache is. I've already mentioned how having tileset in one file will improve caching. > The one-tileset-per-file approach would require you to open the > tilesetfile, extract the data blob, wrap it into a PNG structure, and > serve it. I meant to save whole pngs into aggregated file. So extracting will just get tile offset and return part of tileset file (or blank land/sea). > If I browse the z12 layer, I don't see any caching advantages, > as rather than opening one file per tile, it has to open one tileset > file per tile and deliver parts of it. I'd expect poorer caching results > as the > bigger tileset files fall out of memory quicker, not better ones. > Admittedly at z13, you'd only have to open one file for every 2 z13 > tiles and it get's better at higher zoom levels. Still you would have to > cache whole tilesets rather than just the often-used tiles in memory, so > there are arguments for both sides. OS doesn't cache files but pages. If you open 2MB file and read 8KB from it, only that 8KB will get cached (+plus some data readeahead). The suggested file format won't require you to read whole tileset to get one tile. You can read only 8B to get tile offset and size and then tile data. That's much less than size of data read by OS when looking for file in filesystem with millions of files. Btw. Is tilesets unzipping done on hdd or ramdisc? Using ramdisc should improve performance a bit without extra effort. > As we are getting away from NFS connected disks in favor of local disks > pretty soon now (hopefully), I wouldn't recommend optimizing the whole > thing for NFS access. > I agree that unzipping a tileset with 1600 disk seeks is more expensive > than just pushing one big file. I still haven't been convinced though > that *serving* tiles, would be quicker this way though. Actually, the > whole tile serving seems to perform pretty well, I haven't heard of any > bottlenecks. And it's pretty simple, which is a plus. > I don't think it's very obvious or easy to claim the superiority of one > system over the other. The existing system has the advantage that, errr, > it exists. :-) So, I back up deelkar here, "show us" and we'll be happy > to use the better performing system. > > spaetz > > _______________________________________________ Tilesathome mailing list [email protected] http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/tilesathome
