Are you aware of this map/reduce framework?I have been using it for reporting on top of datastore. It is not as convenient as a SQL aggregation query but is very flexible.
https://cloud.google.com/appengine/docs/python/dataprocessing/ PK gae123.com > On Feb 10, 2017, at 7:22 AM, 'Nicholas (Google Cloud Support)' via Google App > Engine <[email protected]> wrote: > > Exports to BigQuery can be done at regular intervals rather than at every > Datastore write removing the bulk of transactional and latency challenges > with doing both writes at once. Given the existing tools available > (BigQuery, Cloud SQL, Dataproc, MySQL on GCE, one's own application of > MapReduce upon an existing Datastore dataset), the above-mentioned solutions > are those available to you at the moment. > > As for the aggregation layer on top of Datastore using MapReduce, that sounds > like an appropriate idea for a feature request on the App Engine public issue > tracker. I encourage to send your feature requests there as we welcome the > opportunities to improve the platform. Be sure to include the business case > you provided above along with some more specific details about how you see > this being implemented. Any relevant details can be helpful in evaluating > and realizing such ideas. After filing said feature request, feel free to > post a link to it here so that others in the community can follow through and > star it to show their support. > > Hope this helps! > Nicholas > >> On Friday, February 10, 2017 at 3:13:41 AM UTC-5, Rajesh Gupta wrote: >> We are using namespaces to provide ERP SaaS solutions.. We want to do fast >> aggregations for reports within the namespace. Each KIND can have >> 5000-10000 rows within the given time range. >> >> Combining Datastore with BigQuery is not cheap solution for small teams and >> for small data. Now, small teams have to worry about two data storage >> technologies. What about the transactions? Now, a transaction should extend >> to the BigQuery as well, when the data is updated in the datastore. >> We have to worry about the schema changes that will happen in the datastore >> and should be replicated to the BigQuery. >> Suddenly now the appengine solution has become too complex with too many >> technologies for writing simple apps. It defeats the purpose of choosing >> appengine paas. >> >> It will be beneficial to provide an aggregation layer on top of Datastore. >> Google team can incorporate mapreduce as an optional layer for aggregations >> and make it transparent to the developers and give a easy api to use. >> >> Just my 2 cents. >> >> Regards, >> Rajesh >> www.VeersoftSolutions.com >> www.GainERP.com >> Accounting/Inventory/Orders/Sales/Purchase on Google Cloud Platform and >> Mobile >> >> >>> On Wed, Feb 8, 2017 at 8:06 PM, 'Nicholas (Google Cloud Support)' via >>> Google App Engine <[email protected]> wrote: >>> Datastore is not designed for large scale data aggregation. Its emphasize >>> is on fast and horizontally scalable reads. Though it can be used to query >>> over large datasets, it is nowhere near optimized in the same way that SQL >>> or SQL-like technologies are. For full dataset joins and aggregation, >>> BigQuery really shines. The other options mentioned above Postgres and >>> MapReduce would also be appropriate depending on your data. I would >>> strongly suggest using BigQuery as it comes with a Datastore loading >>> solution and is designed for precisely what you describe. In addition, >>> reading all entities in order to aggregate some data from them will be far >>> more costly with Datastore than the equivalent job in BigQuery. >>> >>> In your earlier comments, you mention that you cannot use BigQuery due to >>> an update and delete requirement. Data can be updated in BigQuery. Do >>> such update capabilities meet the requirements you mentioned above? If >>> not, what is ability do you think is missing from these update and delete >>> features? >>> >>> Lastly, in your example case, you mentioned that a given query could pull >>> anywhere from 10 to 10,000 entities. How many entities are there in total? >>> Knowing the total count of entities through which your query must search >>> is crucial to understanding the likely costs of queries in time and quota >>> usage. How many total entities do you expect to have in a given dataset >>> within the next 6-12 months based on recent month's growth? The answer to >>> this should help you decide whether to invest time into replicating data >>> into a aggregation tool like BigQuery and building useful queries for >>> reporting. >>> >>>> On Wednesday, February 8, 2017 at 4:46:28 AM UTC-5, Rajesh Gupta wrote: >>>> All of the above solutions involve more expertise and time, which is not >>>> suitable for small teams. Ofcourse, they achieve good results for large >>>> data and big companies. Small companies will loose the benefit of using >>>> the appengine paas with such wide solutions. >>>> It is beneficial to provide some aggregation layer on top of datastore >>>> with built in fast google technologies. >>>> Interested developers will choose to use it. >>>> >>>> Regards, >>>> Rajesh >>>> www.VeersoftSolutions.com >>>> www.GainERP.com >>>> Accounting/Inventory/Orders/Sales/Purchase on Google Cloud Platform and >>>> Mobile >>>> >>>>> On Sat, Jan 28, 2017 at 12:48 AM, Jim <[email protected]> wrote: >>>>> Another option is to use map-reduce against your datastore tables for >>>>> aggregation of truly 'big' data sets. It's nowhere near as flexible as >>>>> some of the other options mentioned here, but if your requirements are >>>>> fairly static it works great and will allow you to keep your data in one >>>>> place. >>>>> >>>>> >>>>> >>>>>> On Thursday, January 26, 2017 at 2:52:06 PM UTC-6, George (Cloud >>>>>> Platform Support) wrote: >>>>>> How large can your sales invoice data get in the end? The solutions >>>>>> recommended above may work well for relatively small volumes. If you >>>>>> need to process terabytes of data in the end, Cloud Bigtable might prove >>>>>> speedier and cost less overall. >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google Groups >>>>> "Google App Engine" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send an >>>>> email to [email protected]. >>>>> To post to this group, send email to [email protected]. >>>>> Visit this group at https://groups.google.com/group/google-appengine. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/google-appengine/9fe3d5f5-1995-461b-85cf-1006877f3dad%40googlegroups.com. >>>>> >>>>> For more options, visit https://groups.google.com/d/optout. >>>> >>>> >>>> >>>> >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Google App Engine" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at https://groups.google.com/group/google-appengine. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/google-appengine/0cbca628-9706-4f0e-ac49-714227d9005a%40googlegroups.com. >>> >>> For more options, visit https://groups.google.com/d/optout. >> >> >> >> -- >> Regards, >> Rajesh >> www.VeersoftSolutions.com >> www.GainERP.com >> Accounting/Inventory/Orders/Sales/Purchase on Google Cloud Platform and >> Mobile >> > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/google-appengine. > To view this discussion on the web visit > https://groups.google.com/d/msgid/google-appengine/d031ab92-30ed-40ee-aa69-59134d1a5b23%40googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/3A32F9D5-FA15-4386-8D72-A0909804C64B%40gae123.com. For more options, visit https://groups.google.com/d/optout.
