Jason I removed a bunch of single property indexes (by setting indexed=False and then loaded and stored each item. This didn't save much (a few percent). Also, the fact that I can't see the single property indexes makes it more tricky to figure out if they have really gone or not!
I'm now deleting 10% of the records, but I've only reclaimed 2-3% of the space (0.92GB down to 0.90GB). I don't have any significant use of multi-value fields (there is one field, but only rarely does it have more than one (2) values). Philip On May 13, 2:51 pm, "Jason (Google)" <[email protected]> wrote: > Hi Philip. Calling to_xml() is not a great indicator of the size of your > entity as stored in BigTable. Unfortunately, there is currently no > straightforward way to estimate how large your entities are, although we're > working on possible solutions to this problem. > > Without knowing your data model or index definitions, it's certainly not > impossible to rule out the size of your indexes, particularly if your > application is querying across mutliple multi-valued properties, although > this isn't the only scenario that could lead to huge indexes. If you have a > property that you're never querying against, I recommend you try removing > this single property index and see if that makes a noticeable impact or see > if you can eliminate any of your custom indexes which you don't use too > often. > > - Jason > > On Tue, May 12, 2009 at 8:32 PM, WeatherPhilip < > > [email protected]> wrote: > > > I just did a test on one of my apps. Nearly all my data is in a single > > model. > > > I have 163189 instances, and the total size (calculated by reading > > each instance and running to_xml() on it, and then adding up the > > results) is 281,145,536 bytes. Most of my properties have > > indexed=False. The dashboard reports using 890MB of data. I don't know > > whether the dashboard calculation is wrong, whether I should be using > > a different calculation to estimate my record size, or something else. > > If my indexes really are consuming 600MB, then I would work on redoing > > a chunk of the app to fix that problem. > > > However, the only course at the moment appears to be to delete old > > data, and hope that the data consumption goes down. Not really very > > satisfactory. > > > Philip > > > On May 12, 1:38 am, Andy Freeman <[email protected]> wrote: > > > Since index space can be significant, can we get some additional > > > information? > > > > For example, does an indexed db.ListProperty(db.Key) with three > > > elements take significantly more or less space than an indexed > > > db.StringListProperty with three elements whose value is str() of the > > > same keys? (The pickle of keys seems to be significantly larger than > > > the pickle of the equivalent strings.) > > > > On May 11, 5:04 pm, "Jason (Google)" <[email protected]> wrote: > > > > > Hi Anthony. I'm very sorry for the late reply, and thank you for > > bearing > > > > with me. I've discussed this with thedatastoreteam and it's evident > > that > > > > the CSV file's size is not a great indicator of how much storage your > > > > entities will consume. On top of the size of the raw data, each entity > > has > > > > associated metadata, as you've already mentioned, but I'd bet that the > > > > indexes are consuming the greatest space. If you don't ever query on > > one or > > > > more of these 15 string properties, you may consider changing their > > property > > > > types to Text or declaring indexed=false in your model. If you can do > > this > > > > with one of your properties and re-build your indexes, I'd be > > interested in > > > > seeing how much your storageusagedecreases since you'll need one less > > > > index. > > > > > (Note that single-property indexes are present but not listed in the > > Admin > > > > Console.) > > > > > - Jason > > > > > On Sat, May 9, 2009 at 4:34 PM, Kugutsumen <[email protected]> > > wrote: > > > > > > Two weeks ago, I've sent my applications ID to both you and Nick and > > I > > > > > haven't heard from you since then. > > > > > > Thanks- Hide quoted text - > > > > > - Show quoted text - > > --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~----------~----~----~----~------~----~------~--~---
