Hi,
I'm doing yet another terrain data look up service. This time for the
UK with lookup by eastings and northings in metres on a 50m grid,
rather than lat+lng. Its a big data set about 800K records and 200Mb.
A few observations:
1) I tried the bulk uploader but soon gave up - much easier to do a
simple get of e.g. ?newkey=keyval&newdata=data
2) I have made each db entry a 10x10 grid of heights with each height
encoded as 2 base 64 digits (0-4095 metres - no big mountains in the
UK). This makes 200 characters for each 500m x 500m square of the UK.
I don't bother loading all 0 squares (sea). This is a pretty compact
form of storage. Only a few lines of Python are required to read out a
single cell's value.
3) For upload, I use MyDbClass(key_name=(k)) to create a new model
instance using a custom key 'k' derived from the east/north coords of
the S/W corner of the 10x10 grid. The response to a successful upload
is the key parameter. I use a simple C# application to loop over the
records uploading them one at a time, retrying at 1, 2, 4... seconds
after a bad response. I get about 4.5 updates a second.
4) The documentation quoted string property length limit may be 500
chars, but I believe this is for 'encoded' strings. If you put a lot
of punctuation characters in your string, the length limit seems to be
lower. For my base 64 encoding I use
"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_".
5) For point read, I use key = db.Key.from_path('MyDbClass',k) and
then simply MyDbClass.get(k). Again, 'k' is derived from the S/W
corner of the 500m x 500m square containing the wanted point. I
believe this makes for the fastest recall - no indexing, no queries.
6) Please can we have a bulk delete from the console - 20 at a time
with several mouse clicks is far too slow. A chunked up query fetch +
delete soon puts you over quota. Its more tempting to create a new
app. then to try and clear out an existing one.
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en
-~----------~----~----~----~------~----~------~--~---
