I've had very good results on AppEngine mixing the datastore and full text search api.
I use java, so the search api is used to index java objects, and we use Objectify to store the same object in the datastore. Basic crud operations, or anything required to be transactional, are performed on the datastore. Queries are run through the search api which returns an id. We then fetch the object from the datastore using that id as the single source of 'truth'. The only downside is that the search service does not participate in transactions, and its consistency semantics are not documented. If you assume eventual consistentency then it's no worse off than the datastore. As with every service on appengine, there are limitations however, particularly with the type of data you can store and the precision of that data. For example numeric values in the search index are doubles but constrained to the range of an int, and range queries on dates only take into account the date itself, not the time. In most cases, you can make it work for you, but it isn't as easy as it could be. On Tuesday, July 22, 2014 3:13:04 AM UTC+10, David Hardwick wrote: > > Thanks, Kaan. > > I realize my post looks too mired in opinion. In retrospect, I should > have stated that we are using App Engine and the Datastore, and we ARE > operating at that scale mentioned (40,000 customers, 10million records each > across 50 or so entities), we don't have a DBA, etc.. I just wanted to > talk to someone that has either: > > > - a) figured out how to shard a relational database at multi-tenant > scale (e.g., 30,000 customers containing 10M records per customer, across > 50 tables each) such that the relational database scales and is as > reliable > as a NoSQL database. For example, did they just shard on a > 'namespace' column and shard on that using the database's sharding > capabilities vs creating a db/schema for every customer and having to > reshuffle these around. > > ...OR... > > - b) Overcome querying deficiencies of a NoSQL database by writing the > data to both a NoSQL datastore and something like Elastic Search (which > also scales easily). This would make up for a number of the 'cons' of a > NoSQL database for this use case based on a small spike project we did > last > summer. So that's another option I would like to hear from someone about > that has implemented this architecture and are fans. I've recently heard > from a person using a 200M record, multi-tentant app in mysql that used > Elastic Search in front of it too for speed reasons. > > Thanks again! > Hardwick > > > > > On Sun, Jul 20, 2014 at 3:33 PM, Kaan Soral <[email protected] > <javascript:>> wrote: > >> Relational databases also require indexes, however NoSQL indexes are much >> harder to generate and costly >> >> You generally have to build with precision when developing with NoSQL, >> always knowing what you want and how you want it, build the models and >> indexes / systems for that >> >> There are also many other gotchas you learn with NoSQL - or AppEngine >> specifically >> >> But I will say this, I've never felt hopeless while using AppEngine, >> there is always someone out there to help with issues, or an efficient >> documentation >> >> Long story short, I've been using AppEngine at large scale for a while >> now, I would suggest it >> (The taskqueue system is also pretty strong, you can easily build a >> mapping task (map of mapreduce) and iterate over all entities in a >> relatively short time, so if costs are not a big issue, you can always >> iterate over everything instead of running a flexible relational db query >> that you would do otherwise) >> >> (Issues also happen, but there is always the soothing fact that someone >> else solves them at scale, however, since you are intending for a corporate >> usage, you might take the responsibility yourself and build a custom >> system, would be much harder, much more costly) >> >> I'm guessing this isn't the detailed reply you were looking for, but just >> my .02 >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "Google App Engine" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/google-appengine/txoW1CyVaPs/unsubscribe >> . >> To unsubscribe from this group and all its topics, send an email to >> [email protected] <javascript:>. >> To post to this group, send email to [email protected] >> <javascript:>. >> Visit this group at http://groups.google.com/group/google-appengine. >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > David Hardwick, CTO > > p. 646.237.5388 > m. 703.338.0741 > a. 3405 Piedmont Road NE, Suite 325, Atlanta, GA 30305 > e. [email protected] <javascript:> > New Guide: Best Practices for Managing Google Apps Using FlashPanel > <http://www.flashpanel.com/guide/?utm_source=email&utm_medium=click&utm_campaign=BCsig> > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout.
