Rewrote using the ArrayProperty, works great! Very fast and about 1/4 the size of pickled data.
thanks a lot! -john On Tue, Sep 1, 2009 at 7:51 AM, John Newlin<[email protected]> wrote: > Thanks for the pointer Nick, I'll give that a try. > > -john > > > On Tue, Sep 1, 2009 at 6:00 AM, Philippe<[email protected]> wrote: >> >> thank you Nick. >> I will then also use ArrayProperty in my Models ! >> I really thought that I could use ListProperty with indexed=False, >> just as a quick and cheap list. >> >> >> On Sep 1, 1:56 pm, "Nick Johnson (Google)" <[email protected]> >> wrote: >>> Hi Philippe, >>> >>> On Tue, Sep 1, 2009 at 12:43 PM, Philippe <[email protected]> wrote: >>> >>> > hello Nick, >>> >>> > Why do you advice using ArrayProperty instead of db.ListProperty >>> > (float, indexed=False) >>> > with the indexed=False, it should behave the same way, no ? >>> >>> A ListProperty requires encoding each element of the list as a separate >>> property entry in the Entity Protocol Buffer. Protocol Buffer encoding in >>> Python is less efficient than simple array encoding (which is extremely >>> straightforward), and you also incur the additional overhead of serializing >>> the key multiple times - eg, a 1000 entry list is serialized as 1000 (key, >>> value) pairs. >>> >>> -Nick Johnson >>> >>> >>> >>> >>> >>> > Philippe >>> >>> > On Sep 1, 11:48 am, "Nick Johnson (Google)" <[email protected]> >>> > wrote: >>> > > Hi John, >>> > > You might also want to look at the ArrayProperty in the App Engine >>> > > Cookbook: >>> >http://appengine-cookbook.appspot.com/recipe/store-arrays-of-numeric-.... >>> > > Pickling is not particularly efficient, so using an Array is likely to >>> > > be >>> > > both faster and more compact. >>> >>> > > -Nick Johnson >>> >>> > > On Tue, Sep 1, 2009 at 4:55 AM, John Newlin <[email protected]> wrote: >>> >>> > > > Did some more trial and error, and found that if I change the >>> > > > ListProperty(float) to BlobProperty with pickled data, the time for >>> > > > queries goes down from 700ms to 70ms, which seems pretty reasonable. >>> >>> > > > -john >>> >>> > > > On Sun, Aug 30, 2009 at 2:08 PM, jnewlin<[email protected]> wrote: >>> > > > > Hi all, >>> >>> > > > > I'm seeing from the logs that my query is taking 400ms up to around >>> > > > > 1000ms. My model looks like: >>> > > > > class Track2(db.Model): >>> > > > > name = db.StringProperty(required=True) >>> > > > > date = db.StringProperty(required=True) >>> > > > > max_speed = db.StringProperty(required=True) >>> > > > > owner = db.UserProperty(required=True) >>> > > > > latitudes = db.ListProperty(float, indexed=False) >>> > > > > longitudes = db.ListProperty(float, indexed=False) >>> > > > > altitudes = db.ListProperty(float, indexed=False) >>> > > > > heart_rates = db.ListProperty(int, indexed=False) >>> > > > > speeds = db.ListProperty(float, indexed=False) >>> >>> > > > > Each of the lists has just under 1500 entries (I only have 1 row at >>> > > > > the moment) >>> >>> > > > > You can see an example of what I do with this here: >>> >>> > > >http://mytrackviewer.appspot.com/trackView/ag1teXRyYWNrdmlld2Vycg0LEg. >>> > .. >>> >>> > > > > (I know, there are other places to put track data, I don't care, I'm >>> > > > > mostly just learning at this point) >>> >>> > > > > My fetch code looks like: >>> > > > > track_data = db.get(track_id) >>> > > > > if track_data and ( len(track_data.latitudes) == len >>> > > > > (track_data.longitudes) ): >>> > > > > logging.debug('fetched track data with %d lat/longs' % >>> > > > > len >>> > > > > (track_data.latitudes)) >>> > > > > # blah blah blah, writes out stuff to generate the map from here. >>> >>> > > > > This log entry here though happens after ~750ms from the request log >>> > > > > entry. >>> >>> > > > > So my question is this: Is there some way to speed this up? >>> >>> > > > > I also tried splitting the Track common stuff from the Trackpoints, >>> > > > > and keeping all of the points in a separate table. That was a >>> > > > > little >>> > > > > slower, so I'm guessing pulling 1 really big row with all the data >>> > > > > is >>> > > > > faster than pulling 1500 individual rows, is that correct? >>> >>> > > > > Anything else I can try to speed up the query? would turning off >>> > > > > the >>> > > > > validator help? I'm not sure what else to try. >>> >>> > > > > ok thanks >>> >>> > > > > -John >>> >>> > > -- >>> > > Nick Johnson, Developer Programs Engineer, App Engine >>> >>> -- >>> Nick Johnson, Developer Programs Engineer, App Engine >> >> >> > --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~----------~----~----~----~------~----~------~--~---
