We are using namespaces to provide ERP SaaS solutions.. We want to do fast aggregations for reports within the namespace. Each KIND can have 5000-10000 rows within the given time range.
Combining Datastore with BigQuery is not cheap solution for small teams and for small data. Now, small teams have to worry about two data storage technologies. What about the transactions? Now, a transaction should extend to the BigQuery as well, when the data is updated in the datastore. We have to worry about the schema changes that will happen in the datastore and should be replicated to the BigQuery. Suddenly now the appengine solution has become too complex with too many technologies for writing simple apps. It defeats the purpose of choosing appengine paas. It will be beneficial to provide an aggregation layer on top of Datastore. Google team can incorporate mapreduce as an optional layer for aggregations and make it transparent to the developers and give a easy api to use. Just my 2 cents. Regards, Rajesh *www.VeersoftSolutions.com <http://www.veersoftsolutions.com/>* *www.GainERP.com <https://www.gainerp.com/>* *Accounting/Inventory/Orders/Sales/Purchase on Google Cloud Platform and Mobile* On Wed, Feb 8, 2017 at 8:06 PM, 'Nicholas (Google Cloud Support)' via Google App Engine <google-appengine@googlegroups.com> wrote: > Datastore is not designed for large scale data aggregation. Its emphasize > is on fast and horizontally scalable reads. Though it *can* be used to > query over large datasets, it is nowhere near optimized in the same way > that SQL or SQL-like technologies are. For full dataset joins and > aggregation, BigQuery really shines. The other options mentioned above > Postgres and MapReduce would also be appropriate depending on your data. I > would strongly suggest using BigQuery as it comes with a Datastore > loading solution > <https://cloud.google.com/bigquery/loading-data-cloud-datastore> and is > designed for precisely what you describe. In addition, reading all > entities in order to aggregate some data from them will be far more costly > with Datastore than the equivalent job in BigQuery. > > In your earlier comments, you mention that you cannot use BigQuery due to > an update and delete requirement. Data can be updated in BigQuery > <https://cloud.google.com/bigquery/docs/updating-data>. Do such update > capabilities meet the requirements you mentioned above? If not, what is > ability do you think is missing from these update and delete features? > > Lastly, in your example case, you mentioned that a given query could pull > anywhere from 10 to 10,000 entities. How many entities are there in > total? Knowing the total count of entities through which your query must > search is crucial to understanding the likely costs of queries in time and > quota usage. How many total entities do you expect to have in a given > dataset within the next 6-12 months based on recent month's growth? The > answer to this should help you decide whether to invest time into > replicating data into a aggregation tool like BigQuery and building useful > queries for reporting. > > On Wednesday, February 8, 2017 at 4:46:28 AM UTC-5, Rajesh Gupta wrote: >> >> All of the above solutions involve more expertise and time, which is not >> suitable for small teams. Ofcourse, they achieve good results for large >> data and big companies. Small companies will loose the benefit of using >> the appengine paas with such wide solutions. >> It is beneficial to provide some aggregation layer on top of datastore >> with built in fast google technologies. >> Interested developers will choose to use it. >> >> Regards, >> Rajesh >> *www.VeersoftSolutions.com <http://www.veersoftsolutions.com/>* >> *www.GainERP.com <https://www.gainerp.com/>* >> *Accounting/Inventory/Orders/Sales/Purchase on Google Cloud Platform and >> Mobile* >> >> On Sat, Jan 28, 2017 at 12:48 AM, Jim <jeb62...@gmail.com> wrote: >> >>> Another option is to use map-reduce against your datastore tables for >>> aggregation of truly 'big' data sets. It's nowhere near as flexible as >>> some of the other options mentioned here, but if your requirements are >>> fairly static it works great and will allow you to keep your data in one >>> place. >>> >>> >>> >>> On Thursday, January 26, 2017 at 2:52:06 PM UTC-6, George (Cloud >>> Platform Support) wrote: >>>> >>>> How large can your sales invoice data get in the end? The solutions >>>> recommended above may work well for relatively small volumes. If you need >>>> to process terabytes of data in the end, Cloud Bigtable >>>> <https://cloud.google.com/bigtable/docs/overview> might prove speedier >>>> and cost less overall. >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Google App Engine" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to google-appengine+unsubscr...@googlegroups.com. >>> To post to this group, send email to google-appengine@googlegroups.com. >>> Visit this group at https://groups.google.com/group/google-appengine. >>> To view this discussion on the web visit https://groups.google.com/d/ms >>> gid/google-appengine/9fe3d5f5-1995-461b-85cf-1006877f3dad% >>> 40googlegroups.com >>> <https://groups.google.com/d/msgid/google-appengine/9fe3d5f5-1995-461b-85cf-1006877f3dad%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> >> >> -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to google-appengine+unsubscr...@googlegroups.com. > To post to this group, send email to google-appengine@googlegroups.com. > Visit this group at https://groups.google.com/group/google-appengine. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/google-appengine/0cbca628-9706-4f0e-ac49- > 714227d9005a%40googlegroups.com > <https://groups.google.com/d/msgid/google-appengine/0cbca628-9706-4f0e-ac49-714227d9005a%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- Regards, Rajesh *www.VeersoftSolutions.com <http://www.VeersoftSolutions.com>* *www.GainERP.com <https://www.gainerp.com>* *Accounting/Inventory/Orders/Sales/Purchase on Google Cloud Platform and Mobile* -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at https://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CA%2BS7ijajLXcXPghfjBH0K6NnpKjgoHRM9mLvB5r5ppJL%2BxU62Q%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
