No. You just create 1 million "Entity Groups". Resulting in 1 million instances of customer records. Reread the answer from Robert. You can view the Bigtable DATASTORE (notice the absence of database, usually associated with tables) as one big hashtable. You have a key that designates an object instance and the datastore retrieves the object and uses the model definition to convert the stored data (pbuffer) to an instance of the Model class. Part of the datastore you have index lists that sort keys based on some property of the object instances.
2010/10/29 sodso <[email protected]>: > my apologies for the word typo, i meant the same > > but my question still remains - if we have to store 1 million customer > records, do we create 100 different "Kinds" ~= Entity Groups ~= > db.Model and distribute those 1 million records among those entity > groups ? > > then what about scalability ? > does this mean === more records = mroe entity groups to be created ? > isnt this an awkward of doing simple things ? > > thanks. > > On Oct 28, 10:51 am, Robert Kluin <[email protected]> wrote: >> There are no "tables" on App Engine, only Kinds which are arbitrary >> collections of properties. In fact, all applications' entities are >> stored in a single bigtable "table." >> >> Entity groups refers to a logical group of entities that are >> physically stored together; this allows that group of entities to be >> updated in a transaction. >> >> If you make entity groups with lots and lots of entities in them your >> data can not be sharded across bigtable tablets. That means if you >> are try to write 5 entities in that group the writes are done >> serially. Fetches will also be less efficient for the same reasons. >> >> You can read all about these >> topics:http://code.google.com/appengine/articles/datastore/overview.html >> >> Robert >> >> On Wed, Oct 27, 2010 at 21:40, sodso <[email protected]> wrote: >> > Hi Ikan Lan, >> >> > I have seen this on so many discussions here in the forums - >> > "A best practice, however, is to keep entity groups as small as >> > possible" >> >> > As far as I understand it, one Model class should not have too many >> > records i.e. one table should not have many records >> >> > Isn't this strange ? When we talk of GAE, we talk of scalability but >> > if the best practice says to limit ther records to only a few in a >> > Model class, what use is the scalability for ? If we store millions fo >> > records (for eg- customer records) in one db.Model table, are you >> > saying it would really impact the performance. >> >> > Please correct me if I am wrong. I am unclear if my app can survive on >> > GAE if its working with millions of records in one db.Model table ?? >> >> > Thanks, >> > Sodso >> >> > On Oct 27, 11:27 pm, "Ikai Lan (Google)" <[email protected]> >> > wrote: >> >> Generally speaking: no. Entity groups will guarantee a stronger chance of >> >> data locality, but this should not affect index traversal or batch reads. >> >> >> A best practice, however, is to keep entity groups as small as possible. >> >> There aren't many compelling reasons to not use root entities if you don't >> >> need transactions, as working with root entities is generally simpler. >> >> >> -- >> >> Ikai Lan >> >> Developer Programs Engineer, Google App Engine >> >> Blogger:http://googleappengine.blogspot.com >> >> Reddit:http://www.reddit.com/r/appengine >> >> Twitter:http://twitter.com/app_engine >> >> >> On Tue, Oct 26, 2010 at 3:26 PM, djidjadji <[email protected]> wrote: >> >> > Parent and child objects are stored in the same Bigtable node. This is >> >> > done for the transactions. A transaction works on a single >> >> > entity-group. >> >> > If you perform a query you use an index to find the objects needed. At >> >> > this point there is no performance penalty for parent or child objects >> >> > that match the query. The query results in a number of keys of objects >> >> > that need to be retrieved. The best query responds time is when the >> >> > objects >> >> > to fetch are stored in as many Bigtable nodes as possible (parallel >> >> > fetch). >> >> >> > Why do you need so many child objects? >> >> > Can you implement it with adding a Parent Reference Property to the >> >> > child object and thus remove the need to store the entity group all in >> >> > one Bigtable node? >> >> >> > The parent-child objects are needed >> >> > 1) if you need transactions on them >> >> > or >> >> > 2) if you want to extract the parent key given a child key >> >> > Perform a keys_only query on child objects and from these keys get >> >> > the set of parent keys of objects that you need to fetch complete. >> >> > Brett Slatkin uses this technique in a number of Google IO talks, >> >> > the child object has a ListProperty that is used in the keys_only >> >> > query. >> >> >> > Most other applications can be implemented without the explicit parent >> >> > object. >> >> >> > 2010/10/26 Charles <[email protected]>: >> >> > > Hi all, >> >> >> > > I'm wondering, since the datastore is hierarchical, does the number of >> >> > > children an entity has affect the performance on querying on the >> >> > > parents >> >> > > themselves? For example, if I have a set of parents, say... >> >> >> > > Jane >> >> > > Margaret >> >> > > Graham >> >> > > Arthur >> >> >> > > ...and I have a set of children associated with those parents... >> >> >> > > Jane >> >> > > -Sam >> >> > > -Robert >> >> > > Margaret >> >> > > -Lisa >> >> > > Graham >> >> > > Arthur >> >> > > -Rowen >> >> > > -Jerry >> >> >> > > ...will the number of children for each parent affect the performance >> >> > > of >> >> > > querying the parents themselves? For instance, if I wanted to select >> >> > > all >> >> > of >> >> > > the parents (SELECT * FROM parents), that would be easy with the data >> >> > > above. But, since the datastore is hierarchical, does the performance >> >> > get >> >> > > hampered if say the parents have many thousands or even millions of >> >> > > children? Say, like... >> >> >> > > Jane >> >> > > -Sam >> >> > > -Robert >> >> > > ...1 million more >> >> > > Margaret >> >> > > ... >> >> >> > > If so, I'm just wondering if it would make more sense to make the >> >> > children >> >> > > root entities too, so as not to affect the performance of querying on >> >> > > the >> >> > > parents. Anyways, hope I've explained my question well enough. >> >> >> > > Thanks in advance! >> >> >> > > Charles >> >> >> > > -- >> >> > > You received this message because you are subscribed to the Google >> >> > > Groups >> >> > > "Google App Engine" group. >> >> > > To post to this group, send email to >> >> > > [email protected]. >> >> > > To unsubscribe from this group, send email to >> >> > > [email protected]<google-appengine%[email protected]> >> >> > . >> >> > > For more options, visit this group at >> >> > >http://groups.google.com/group/google-appengine?hl=en. >> >> >> > -- >> >> > You received this message because you are subscribed to the Google >> >> > Groups >> >> > "Google App Engine" group. >> >> > To post to this group, send email to [email protected]. >> >> > To unsubscribe from this group, send email to >> >> > [email protected]<google-appengine%[email protected]> >> >> > . >> >> > For more options, visit this group at >> >> >http://groups.google.com/group/google-appengine?hl=en. >> >> > -- >> > You received this message because you are subscribed to the Google Groups >> > "Google App Engine" group. >> > To post to this group, send email to [email protected]. >> > To unsubscribe from this group, send email to >> > [email protected]. >> > For more options, visit this group >> > athttp://groups.google.com/group/google-appengine?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
