Hi Philip. Calling to_xml() is not a great indicator of the size of your entity as stored in BigTable. Unfortunately, there is currently no straightforward way to estimate how large your entities are, although we're working on possible solutions to this problem.
Without knowing your data model or index definitions, it's certainly not impossible to rule out the size of your indexes, particularly if your application is querying across mutliple multi-valued properties, although this isn't the only scenario that could lead to huge indexes. If you have a property that you're never querying against, I recommend you try removing this single property index and see if that makes a noticeable impact or see if you can eliminate any of your custom indexes which you don't use too often. - Jason On Tue, May 12, 2009 at 8:32 PM, WeatherPhilip < [email protected]> wrote: > > I just did a test on one of my apps. Nearly all my data is in a single > model. > > I have 163189 instances, and the total size (calculated by reading > each instance and running to_xml() on it, and then adding up the > results) is 281,145,536 bytes. Most of my properties have > indexed=False. The dashboard reports using 890MB of data. I don't know > whether the dashboard calculation is wrong, whether I should be using > a different calculation to estimate my record size, or something else. > If my indexes really are consuming 600MB, then I would work on redoing > a chunk of the app to fix that problem. > > However, the only course at the moment appears to be to delete old > data, and hope that the data consumption goes down. Not really very > satisfactory. > > Philip > > On May 12, 1:38 am, Andy Freeman <[email protected]> wrote: > > Since index space can be significant, can we get some additional > > information? > > > > For example, does an indexed db.ListProperty(db.Key) with three > > elements take significantly more or less space than an indexed > > db.StringListProperty with three elements whose value is str() of the > > same keys? (The pickle of keys seems to be significantly larger than > > the pickle of the equivalent strings.) > > > > On May 11, 5:04 pm, "Jason (Google)" <[email protected]> wrote: > > > > > Hi Anthony. I'm very sorry for the late reply, and thank you for > bearing > > > with me. I've discussed this with thedatastoreteam and it's evident > that > > > the CSV file's size is not a great indicator of how much storage your > > > entities will consume. On top of the size of the raw data, each entity > has > > > associated metadata, as you've already mentioned, but I'd bet that the > > > indexes are consuming the greatest space. If you don't ever query on > one or > > > more of these 15 string properties, you may consider changing their > property > > > types to Text or declaring indexed=false in your model. If you can do > this > > > with one of your properties and re-build your indexes, I'd be > interested in > > > seeing how much your storageusagedecreases since you'll need one less > > > index. > > > > > (Note that single-property indexes are present but not listed in the > Admin > > > Console.) > > > > > - Jason > > > > > On Sat, May 9, 2009 at 4:34 PM, Kugutsumen <[email protected]> > wrote: > > > > > > Two weeks ago, I've sent my applications ID to both you and Nick and > I > > > > haven't heard from you since then. > > > > > > Thanks- Hide quoted text - > > > > > - Show quoted text - > > > > > > > --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~----------~----~----~----~------~----~------~--~---
