Re: [google-appengine] Re: Doing Totals/aggregates

If you’re doing aggregations across <10k rows, you don’t need (or want) BigQuery or map/reduce or any other “big data” solution. You want a basic SQL database. Use Cloud SQL if you want something easy to integrate with GAE. You’re not going to get SQL aggregations out of the datastore; it’s just

Re: [google-appengine] Re: Doing Totals/aggregates

Are you aware of this map/reduce framework?I have been using it for reporting on top of datastore. It is not as convenient as a SQL aggregation query but is very flexible. https://cloud.google.com/appengine/docs/python/dataprocessing/ PK gae123.com > On Feb 10, 2017, at 7:22 AM, 'Nicholas

Re: [google-appengine] Re: Doing Totals/aggregates

Exports to BigQuery can be done at regular intervals rather than at every Datastore write removing the bulk of transactional and latency challenges with doing both writes at once. Given the existing tools available (BigQuery, Cloud SQL, Dataproc, MySQL on GCE, one's own application of

Re: [google-appengine] Re: Doing Totals/aggregates

We are using namespaces to provide ERP SaaS solutions.. We want to do fast aggregations for reports within the namespace. Each KIND can have 5000-1 rows within the given time range. Combining Datastore with BigQuery is not cheap solution for small teams and for small data. Now, small teams

Re: [google-appengine] Re: Doing Totals/aggregates

Datastore is not designed for large scale data aggregation. Its emphasize is on fast and horizontally scalable reads. Though it *can* be used to query over large datasets, it is nowhere near optimized in the same way that SQL or SQL-like technologies are. For full dataset joins and

Re: [google-appengine] Re: Doing Totals/aggregates

All of the above solutions involve more expertise and time, which is not suitable for small teams. Ofcourse, they achieve good results for large data and big companies. Small companies will loose the benefit of using the appengine paas with such wide solutions. It is beneficial to provide some

[google-appengine] Re: Doing Totals/aggregates

Another option is to use map-reduce against your datastore tables for aggregation of truly 'big' data sets. It's nowhere near as flexible as some of the other options mentioned here, but if your requirements are fairly static it works great and will allow you to keep your data in one place.

[google-appengine] Re: Doing Totals/aggregates

How large can your sales invoice data get in the end? The solutions recommended above may work well for relatively small volumes. If you need to process terabytes of data in the end, Cloud Bigtable might prove speedier and cost less overall.