Sorry, this is not my case. I do not use default indexes ("indexed = true")
on models properties from the start. Anyway default indexes cannot be
responsible for "exploding" since they are for one property only, so their
size can account for 2x 3x ... 10x size of entities only. 4 custom indexes
with more than one property in it (i really needed in development) that i
have, i already deleted to check is this is a key of the problem. besides
that there was a long time when nothing changed inside app, user base
grouth was not big and total size was <1Gb
i started migration, when it will be completed i will try to delete all
data completely. i already tryed this (making backup before) and datastore
size did not drop at all, but maybe there is time needed for stats to be
re-counted, so i`ll try later and wait day or two before making new
assumptions
пятница, 23 марта 2012 г. 19:45:17 UTC+3 пользователь Jeff Schnitzer
написал:
>
> If you have not optimized your indexes, they will almost always
> account for the bulk of your data storage.
>
> Most of the datastore APIs (python db, ndb, java low-level, jdo/jpa,
> etc) default to creating two indexes (asc and desc) on every property.
> Each of these index entries represents another row in a BigTable
> tablet; for small entities, each of these rows are only slightly
> smaller than the master blob. So 5 indexed properties could easily
> cause a 5-10X expansion in your total stored data volume.
>
> You can't vacuum these indexes; unlike multi-property indexes defined
> in datastore-indexes.xml/yaml, single-property indexes are established
> when you save an entity and are only deleted by resaving the entity.
>
> If you are concerned about space (or datastore write operations), you
> should go through your data model and flag any properties you aren't
> querying on as unindexed. Then you will need to resave your entities
> to reclaim index space.
>
> Jeff
>
> On Fri, Mar 23, 2012 at 11:01 AM, João Campos <[email protected]> wrote:
> > Thanks for your reply.
> >
> > Even if logs do account for data storage, I've got a little less than
> 115MB
> > of those. With the 15MB of application data that would make up for a
> 130MB
> > total.
> >
> > I've also ran a vacuum_indexes and I've got not custom indexes and a very
> > simple data model (only 3 entities).
> >
> > With such a simple data model and such small amount of data, can the
> indexes
> > really be responsible for the missing 894MB (1024-130)?
> >
> >
> > On Friday, March 23, 2012 11:40:53 AM UTC, de Witte wrote:
> >>
> >> Indexes and logs also consume data storage.
> >>
> >> Op donderdag 22 maart 2012 11:46:11 UTC+1 schreef Константин Сабуров het
> >> volgende:
> >>>
> >>> Hello!
> >>>
> >>> i have an application on master/slave datastore and recently noticed
> >>> really strange things: datastore size on the dashboard is much bigger
> than
> >>> expected. in my app i proactively deleting all unused info to stay
> below 1Gb
> >>> quota (and there is not many users) but datastore size do not going
> down, it
> >>> going up instead and for now it is over 4Gb.
> >>> and this is not only mine point of view: datastore admin page and
> >>> datastore statistics both shows numbers which much easier to believe
> >>> i attached screenshots, may be i overlooked something... i just whant
> to
> >>> delete "invisible entities" which take all of my billable space
> >>> please help! :)
> >>>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Google App Engine" group.
> > To view this discussion on the web visit
> > https://groups.google.com/d/msg/google-appengine/-/s00-DD85r4cJ<https://groups.google.com/d/msg/google-appengine/-/s00-DD85r4cJ>
> .
> >
> > To post to this group, send email to
> > google-appengine@googlegroups.com<[email protected]>
> .
> > To unsubscribe from this group, send email to
> > google-appengine+unsubscribe@googlegroups.com<google-appengine%[email protected]>
> .
> > For more options, visit this group at
> > http://groups.google.com/group/google-appengine?hl=en<http://groups.google.com/group/google-appengine?hl=en>
> .
>
>
пятница, 23 марта 2012 г. 19:45:17 UTC+3 пользователь Jeff Schnitzer
написал:
>
> If you have not optimized your indexes, they will almost always
> account for the bulk of your data storage.
>
> Most of the datastore APIs (python db, ndb, java low-level, jdo/jpa,
> etc) default to creating two indexes (asc and desc) on every property.
> Each of these index entries represents another row in a BigTable
> tablet; for small entities, each of these rows are only slightly
> smaller than the master blob. So 5 indexed properties could easily
> cause a 5-10X expansion in your total stored data volume.
>
> You can't vacuum these indexes; unlike multi-property indexes defined
> in datastore-indexes.xml/yaml, single-property indexes are established
> when you save an entity and are only deleted by resaving the entity.
>
> If you are concerned about space (or datastore write operations), you
> should go through your data model and flag any properties you aren't
> querying on as unindexed. Then you will need to resave your entities
> to reclaim index space.
>
> Jeff
>
> On Fri, Mar 23, 2012 at 11:01 AM, João Campos <[email protected]> wrote:
> > Thanks for your reply.
> >
> > Even if logs do account for data storage, I've got a little less than
> 115MB
> > of those. With the 15MB of application data that would make up for a
> 130MB
> > total.
> >
> > I've also ran a vacuum_indexes and I've got not custom indexes and a very
> > simple data model (only 3 entities).
> >
> > With such a simple data model and such small amount of data, can the
> indexes
> > really be responsible for the missing 894MB (1024-130)?
> >
> >
> > On Friday, March 23, 2012 11:40:53 AM UTC, de Witte wrote:
> >>
> >> Indexes and logs also consume data storage.
> >>
> >> Op donderdag 22 maart 2012 11:46:11 UTC+1 schreef Константин Сабуров het
> >> volgende:
> >>>
> >>> Hello!
> >>>
> >>> i have an application on master/slave datastore and recently noticed
> >>> really strange things: datastore size on the dashboard is much bigger
> than
> >>> expected. in my app i proactively deleting all unused info to stay
> below 1Gb
> >>> quota (and there is not many users) but datastore size do not going
> down, it
> >>> going up instead and for now it is over 4Gb.
> >>> and this is not only mine point of view: datastore admin page and
> >>> datastore statistics both shows numbers which much easier to believe
> >>> i attached screenshots, may be i overlooked something... i just whant
> to
> >>> delete "invisible entities" which take all of my billable space
> >>> please help! :)
> >>>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Google App Engine" group.
> > To view this discussion on the web visit
> > https://groups.google.com/d/msg/google-appengine/-/s00-DD85r4cJ<https://groups.google.com/d/msg/google-appengine/-/s00-DD85r4cJ>
> .
> >
> > To post to this group, send email to
> > google-appengine@googlegroups.com<[email protected]>
> .
> > To unsubscribe from this group, send email to
> > google-appengine+unsubscribe@googlegroups.com<google-appengine%[email protected]>
> .
> > For more options, visit this group at
> > http://groups.google.com/group/google-appengine?hl=en<http://groups.google.com/group/google-appengine?hl=en>
> .
>
>
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/5Gk2zUnkmm0J.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
