Thanks for the information and tips Robert, somehow I missed that AppEngine automatically indexes properties. Simply adding the unindexed extension to the JDO class should eventually clear the auto-generated indexes?
Cheers & thanks again, -Erik On Nov 9, 10:52 am, Robert Kluin <[email protected]> wrote: > Hi Erik, > As far as I know the persistence manager would not create any extra > entities. > > It sounds like you might want to explicitly disable indexing on any > fields you will not be querying on. See the 'Properties that Aren't > Indexed' section on the 'Queries and Indexes' > page.http://code.google.com/appengine/docs/java/datastore/queriesandindexe... > > You might also want to use shorter kind and property names. Kind > and property names are stored with every entity, so it can add up > pretty fast. > > Robert > > On Tue, Nov 9, 2010 at 08:05, Erik <[email protected]> wrote: > > Hi Robert, > > > Thanks for the response, I did wait several days before sending that > > message, but shortly afterwards the quota cleared to zero. I am using > > a persistence manager with jdo, would these create temporary > > entities? > > > I need to explicitly index my keys in descending order for mapreduce, > > and thought I needed to index my collections but as I am not > > performing any queries don't think that is necessary. My persistent > > objects consist of a single property with type of long collection, > > entities are fetched by id. > > > My latest 30MB dataset has already grown to consume 710MB, that is an > > enormous increase. Datastore statistics say only 131MB is being used > > with 71% of that as metadata. > > > Any other ideas? > > > Thanks, > > -Erik > > > On Nov 8, 1:03 pm, Robert Kluin <[email protected]> wrote: > >> Hi Erik, > >> Several common sources of datastore stats / quota number funkiness: > >> 1) The numbers are not updated in real time. Sometimes it can > >> take a day for the numbers to get updated. > >> 2) Because of 1, if you are using a session library (or something > >> similar) that create lots of temporary entities the numbers can > >> 'appear' to be out-of-sync. > > >> I assume when you say switched to explicit indexes, you mean you > >> explicitly disabled indexing of any properties that do not need > >> indexes? > > >> Also, you might want to star issue 2740. > >> http://code.google.com/p/googleappengine/issues/detail?id=2740 > > >> Robert > > >> On Sat, Nov 6, 2010 at 19:29, Erik <[email protected]> wrote: > > >> > Hello all, > > >> > I have been uploading a dataset which is composed of 100MB of CSV > >> > values. During the process of uploading with bulkloader, which never > >> > completed, the datastore expanded to consume 2GB of usage. I decided > >> > to explicitly index my data and clear the datastore before a re-import > >> > using the blobstore/mapreduce method. However, after clearing the > >> > datastore there is still a persistent .37GB of data usage remaining, > >> > while datastore statistics say 53KB is being consumed by 174 objects, > >> > and nothing shows in datastore viewer. > > >> > Any ideas on how I can recover the .37GB of quota which is > >> > mysteriously being used in my empty datastore? > > >> > Many thanks, > >> > -Erik > > >> > -- > >> > You received this message because you are subscribed to the Google > >> > Groups "Google App Engine" group. > >> > To post to this group, send email to [email protected]. > >> > To unsubscribe from this group, send email to > >> > [email protected]. > >> > For more options, visit this group > >> > athttp://groups.google.com/group/google-appengine?hl=en. > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to [email protected]. > > To unsubscribe from this group, send email to > > [email protected]. > > For more options, visit this group > > athttp://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
