Dirk-Lüder Kreie wrote: > spaetz schrieb: >> On Thu, Jun 26, 2008 at 10:37:41AM +0200, Frederik Ramm wrote: >> >>> If client A uploads a tar with tiles 1, 2, 3, and later B uploads a >>> tar with tiles 10, 11, 3, then you have two choices: >> >> It seems a reasonable policy change to require the uploading of >> complete tilesets. The bottleneck is always going to be the server, so >> why not push as much work to the clients as possible and make it as >> easier for the server. > > There are some problems with tilesets that even now are too big for the > current maximum upload size (but can be adapted in php.conf) > Secondly the clients have a renderFullTileset option already which can > be set quite easily via svn.
if I have seen it correctly, the renderFullTileset option, just simply keeps all the empty tiles as files, rather than deleting them if the emptiness can be inferred recursively? That option would be quite useful for the single file per tileset implementation I think, and it should also simplify the serving of empty tiles, as one doesn't need to consult the database for those. In the tileset structure proposed, this would also not add much overhead, as each empty tile would only need 4 bytes, so a maximum of 5kb extra per tileset. In the statistics posted earlier about the ratio of complete to incomplete tilesets, is it know why these tilesets are incomplete? What fraction is due to the empty tiles rather than not being able to upload as large tiles? > >> If we mandate the uploading of full tilesets as tar files and it get's >> as easy for the server to process that as to move it to the >> appropriate location, then I'm all for it. I'm just not going to code it. > > Same here. So apart from the coding of the serverside stuff we "just" > need to adapt the php.conf (or whatever then handles the uploading > itself) to allow for larger POST and upload files (I think there were 2 > config options that had to be modified) > and the clientside config option for maximum upload size (zipHardLimit > IIRC) Over the weekend I have started writing an Apache module for serving tiles from within a tileset file. As I have never written an Apache module before, this is not the fastest endeavour though, but at least with the help of the mod_tile code, I have now managed to get to a stage, where it correctly serves blank tiles from within the tileset and is close to serving normal tiles as well. It also nicely falls back to the old code, if it doesn't find the tileset file. Unfortunately the code is still too ugly to post quite yet, but hopefully after a bit of re factoring, I can post an early version soon if people are interested. Of cause, there is still a fair bit of plumbing needed to integrate this all into the upload handlers and tile statistics pages, but that should all be doable. The thing I potentially see as biggest hurdle though is how to deal with deleting the old files. Although my current approach should work correctly without deleting files, that would then double the overall storage requirements, which can't work long term. On the other hand if you need to check each of the old tiles and delete them in the upload handler, then I could imagine a big portion of the benefits of having a single file per tileset is lost. Perhaps it could work to have a clean-up daemon periodically go through the list of recently uploaded tilesets and clean up duplicate files and recombine incomplete tilesets with individual tiles. Do people have other suggestions for this issue? > >> Marking blank tiles could still be done by passing the appropriate >> blankness info through a special HTTP call. > > however we will have all that info inside our "full Tileset" uploads. > > we might want to run a simplifier for ocean-tiles from time to time. > > or we need to calculate the overhead of storing the entire sea via such > files... In the case of having only parts of a z12 tileset empty, I think the overhead shouldn't be too bad. I would think the big problem would be if one ended up storing them for the complete pacific ocean, but if an entire z12 tileset is empty, then there shouldn't be a need to store a tileset at all and fall back to the current recursive database lookup. How does the current client deal with entire z12 being empty? Kai > > > ------------------------------------------------------------------------ > > _______________________________________________ > Tilesathome mailing list > [email protected] > http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/tilesathome _______________________________________________ Tilesathome mailing list [email protected] http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/tilesathome
