spaetz wrote:
Presumably that will also mean that re-rendering all of the tiles under
the new server will take at least as long. So there will be an overlap
of old and new system for quite a while. I think that is perfectly fine,
but it does mean that the legacy system has to work well. I.e. the
blankness lookup must be implemented correctly.
I don't do database lookups for blankness in the legacy code, I just look for a real tile. Going through the
"new tile search" -> "old tile file search" -> "recursive database search"
would be quite a stretch for an operation that should be blazingly fast.
The efficiency of serving the new tile shouldn't really be effected by
the fallback mechanisms down the line and does the third fall back
option really have to be blazingly fast? Unless perhaps loads of people
look at the pacific ocean at the same time... ;-) But then I don't know
how things would scale.
I see three possible options for the legacy blankness lookup.
1) Just like the LegacyTileset, call the old blankness db. From your
numbers above, this seems quite inefficient though, although I could
imagine this could be optimised quite a bit, e.g. not spanning a new
database connection on each new lookup
2) Implement a new blankness database in some form. Not sure how this
would be populated though.
3) Use the ocean tiles db as a fallback of the fallback. From my
previous tests, this was actually quite efficient and easy to code at
least for z12 and above.
I'd appreciate a patch for solution 3, if you can make it efficient enough. The
code would be called from the serve_legacy_tile function in Tile.py in the
directory tah_intern.
I am still not 100% sure that this will be necessary or efficient enough, but
we should try it.
I do think a solution to the blankness is necessary, as it would
presumably take a long time to fill all of the blank areas with real
tilesets. Until then, the lowzoom map would continue to look less than
ideal, as it does at the moment, as most of z12 tiles seem blank. If you
zoom in to e.g. z17 then a good portion of the tiles even in well mapped
areas are blank and currently return "unknown", as they don't have fall
back tiles.
Anyway, I have attached a patch that is based upon the oceantiles
approach. For efficiency reasons, it will only return blankness
information for z12 and above. Lower zooms will have to have real
tilesets. I think this should be pretty efficient. At least when testing
it locally I could not measure a difference between the three types,
serving from a tileset, falling back to a legacy tile or falling all the
way through to the ocean tiles. In all cases wget took about 0.01 second
to get a single tile, but we will have to see how to it would hold up
under load.
There is one thing that could probably still be optimised. I used the
django database model to determine weather a layer is transparent or
not. That probably adds extra overhead that could be hard coded away if
necessary.
Any feedback for the patch is appreciated.
Kai
spaetz
_______________________________________________
Tilesathome mailing list
[email protected]
http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/tilesathome
--- Tile_orig.py 2008-07-12 23:04:59.000000000 +0100
+++ Tile.py 2008-07-12 23:14:46.000000000 +0100
@@ -1,4 +1,7 @@
-import struct,os.path
+import struct,os.path,sys
+
+os.environ["DJANGO_SETTINGS_MODULE"]="tah.settings"
+from tah.tah_intern.models import Layer
class Tile:
x=None
@@ -6,6 +9,9 @@
z=None
layer=None
blankness=None
+ basetilepath='/usr/local/src/tah/Tiles'
+ leg_basetiledir='/mnt/agami/openstreetmap/tah/Tiles'
+ blankpng=['unknown.png','sea.png','land.png','transparent.png']
def __init__(self, layer, z, x, y):
self.layer=layer
@@ -56,8 +62,8 @@
(layer, base_z,base_x,base_y) = self.basetileset()
# basetileset returns (None,None,None,None) if invalid!
# basetilepath could be take from the Settings, hardcode for efficiency
- basetilepath='/usr/local/src/tah/Tiles'
- tilesetfile = os.path.join(basetilepath,layername+'_'+str(base_z),str(base_x)+'_'+str(base_y))
+
+ tilesetfile = os.path.join(self.basetilepath,layername+'_'+str(base_z),str(base_x)+'_'+str(base_y))
try:
f = open(tilesetfile,'rb')
#calculate file offset
@@ -84,8 +90,7 @@
data = f.read(d_offset_next-d_offset)
else:
# return blankness value
- blankpng = ['unknown.png','sea.png','land.png','transparent.png']
- f_png = open(os.path.join(basetilepath,blankpng[d_offset]),'rb')
+ f_png = open(os.path.join(self.basetilepath,self.blankpng[d_offset]),'rb')
data = f_png.read()
f_png.close()
try: f.close()
@@ -95,13 +100,39 @@
def serve_legacy_tile(self,layername):
""" Return the PNG data of a legacy tile """
- leg_basetiledir='/mnt/agami/openstreetmap/tah/Tiles'
- fname = os.path.join(leg_basetiledir,layername,"%02d"%(self.z),"%03d"%(self.x//1000),"%03d"%(self.x%1000),"%03d"%(self.y//1000),"%03d.png"%(self.y%1000))
+ fname = os.path.join(self.leg_basetiledir,layername,"%02d"%(self.z),"%03d"%(self.x//1000),"%03d"%(self.x%1000),"%03d"%(self.y//1000),"%03d.png"%(self.y%1000))
try:
f = open(fname,'rb')
data = f.read()
f.close()
return data
except:
+ #There is no legacy tile either, so fall back to a blanktile database
+ layer = Layer.objects.get(name=layername);
+ type = 0
+ if layer.transparent:
+ type = 3;
+ else:
+ if self.z < 12:
+ #We only handle z12 and above this way. Lowzooms are required to have actual tiles
+ return None
+ else:
+ #Lookup in the oceans tile
+ #For z12 and above the file will tell us directly what type it is
+ try:
+ (layern, base_z,base_x,base_y) = self.basetileset()
+ oceant = open(os.path.join(self.basetilepath,"oceantiles_12.dat"),'rb')
+ oceant.seek((4096*base_y + base_x) >> 2)
+ data = ord(oceant.read(1));
+ bit_off = 3 - ((4096*base_y + base_x) % 4);
+ type = (data & (3 << 2*(bit_off))) >> 2*(bit_off)
+
+ #remap the oceantiles type to the one used on the serverside
+ type = {0: 0, 1:2, 2:1, 3:0}[type];
+ except:
return None
+ f_png = open(os.path.join(self.basetilepath,self.blankpng[type]),'rb')
+ data = f_png.read()
+ return data
\ No newline at end of file
_______________________________________________
Tilesathome mailing list
[email protected]
http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/tilesathome
