Hi Anastasios, Thank you VERY much for that detailed explanation. This is great and exactly what we wanted Groups to be when we set it up :). I have a feeling I may use your post as a reference for other customers :).
Keep up the amazing work. Cheers! On Monday, September 21, 2015 at 3:21:28 PM UTC-4, Anastasios Hatzis wrote: > > You didn't mention which language or API you use. But I assume you are > referring to Google Cloud Datastore in general. Here some tricks I've > learnt the hard way, and sometimes the easy way by watching Google > Developers videos and blog posts. ;-) I hope they help. > > *Minimize read operations in Google Cloud Datastore:* > Try to have larger entities, rather than many small entities with > references. What makes sense in SQL (e.g. 5th Normal Form) does make not > much sense in a non-SQL tech. Especially because you don't have joint > queries and such. > > Start with the most popular request handlers (that you expect in case of a > new app or according to analytics). What information will they need most? > Could you put them into the same entity kind? Can you get the entities by > key or do you need queries? By key is faster. And by key would also allow > easier utilisation of memcache in some use-cases. > > Try to avoid round-trips, i.e. first get some entities, then depending on > property values in the results read more entities from datastore. Better: > Try to get and query everything in one (parallel) step. Have the client > directly provide keys/IDs or whatever you need (but of course, do never > trust the client!). Much faster and cheaper. > > For example, a SalesOrderItem query would need not only Product ID, > Description, Quantity, and Price, but you also want to show Customer ID, > Date of Purchase, and Sales Order ID, and a thumbnail of the primary > product photo. > > Now, in SQL you would have SalesOrderItem table with key columns > referencing to a Customer table, and a SalesOrder table, then doing a joint > query. If you try hard to impose SQL thinking you might see this > workaround: first query the items, then for each of them the referenced > product, sales order, and customer. > > In Google Cloud Datastore though, it makes often more sense to store every > important information of a query directly in the entity kind, especially > the properties that are unlikely to change afterwards. > > Example for SalesOrderItem kind: > > - parent (key to SalesOrder entity, key contains ID of the SalesOrder > entity, too) > - quantity (float) > - price (float) > - productID (int) > - dateOfPurchase (date copied from SalesOrder, should never change > anyway) > - customer (key, but key contains the unique ID of customer account) > - productDescr (str, copy the description property from the Product > entity; if a description is changed, you probably don't need to propagate > it anyways to old sales order items, because when the sales order is > created the sold products are "as is") > - productPhoto (URL-safe encoded key of the product; store all product > thumbnails in e.g. Google Cloud Storage and use that string in the path, > so > you can automatically build the image source path) > > This way, you can query and show sales order items in one step, with-out > additionally reading sales-orders, customers, and products. This translates > to 1 read per item rather than 4 or more reads. > > Use keys-only queries if you only care for count of results. Use > fetch-page with cursors, because with offset instead, the datastore will > really go through each entity in the query until it reaches the desired > offset. > > Eventually, you can use projected queries. I've never used them, but > basically, you define in the read op which property values you need for a > query. Those are billed as "small ops". I'm not sure if projected queries > are supported in all APIs. > > *Minimize write operations:* > Use shorter property names for less storage. Especially important for high > volume kinds. > > Only index the properties you definitely need for ordering or filtering of > queries. You can index properties later if the app really needs some new > query filter or so. > > The more indexed properties are defined for entity kind, the more write > ops with every new entity and entity update (exponentially!). > > Also important aspect regarding write operations is *scalability *- > Datastore is all about scalability, but the model needs to pay respect to > some technical implications: Try to model in a way that makes sure that > there will be no excessive write ops into the same entity group. Entity > groups allows your app strong consistency with-in the entity group, but > Datastore has a write limit of 1 write per second per entity group. For > example, SalesOrder and SalesOrderItem could be in the same entity group, > if it is unlikely that more than one user will work on the same sales-order > at the same time. If you don't update individual items but the sales order > and all its items together in single requests, it is less likely you will > hit the limit. Probably bad practice would be to put all sales-orders also > into the same entity group, e.g. all of them have OrderBook as parent, your > app could very likely hit the limit frequently. > > Some more reading: > > http://googlecloudplatform.blogspot.de/2015/08/Introduction-to-data-models-in-Cloud-Datastore.html > > http://googlecloudplatform.blogspot.de/2013/12/best-practices-for-app-engine-memcache.html > > Regarding the number of read/write ops inside a transaction, I don't know > if these numbers are available. > > 2015-09-21 9:23 GMT+02:00 Anirban Silsarma <[email protected]>: > >> How can i minimize read write operation, and how can I understand how >> much read and write operation has occurred in per transaction?? >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at http://groups.google.com/group/google-appengine. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/google-appengine/2ffcfc73-5bf3-4d03-b783-62b557dc81b2%40googlegroups.com >> >> <https://groups.google.com/d/msgid/google-appengine/2ffcfc73-5bf3-4d03-b783-62b557dc81b2%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > > Mit freundlichen Grüßen / Kind regards > > i. A. > Anastasios Hatzis > > Fon: +49 8374 930813 > Fax: +49 8374 930810 > Mobil: +49 1520 8592878 > > HATZIS Edelstahlbearbeitung GmbH > Hojen 2 > 87490 Haldenwang (Allgäu) > Germany > > Handelsregister Kempten (Allgäu): HRB 4204 > Geschäftsführer: Paulos Hatzis, Charalampos Hatzis > Umsatzsteuer-Identifikationsnummer: DE 128791802 > GLN: 42 504331 0000 6 > > http://www.hatzis.de/ -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/4354ebf1-ccfd-4982-b038-8e54a4e41ee8%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
