Yep the blobstore approach is a better solution IMHO but costs will still be prohibitive. Im afraid nothong much you can do about that. Initially i thought that we could get an export of the datastore, download it as a binary file and process it locally (like the local .bin used for devs) but i'm dreaming out loud :-)
On Dec 30, 4:55 am, Jeff Schnitzer <[email protected]> wrote: > On Thu, Dec 29, 2011 at 2:58 AM, Brandon Wirtz <[email protected]> wrote: > > > I would say GAE handles big data really well. But you have to do testing to > > make sure your structure is correct, and that your indexes are well thought > > out. > > I think we are talking about two different things. I'm thinking of > Big Data like this: > > http://en.wikipedia.org/wiki/Big_data > > Typically characterized by: > > * Large data volumes > * Batch updates > * Frequent need to analyze/sift through large quantities of data > > The GAE datastore performs poorly in this regard. Map/reduce support > is anemic at best. Per-gigabyte storage is expensive. Raw I/O > performance is *dreadful*. Indexes consume excessive amounts of > space. > > I love the GAE datastore, I think it's hands-down the Best Storage > Around for web applications that need scalability and availability. > But there's no way in hell I would use it to store a large-scale OLAP > system or any other kind of serious analytics product. You don't want > EC2 either. You need something like Hadoop on bare metal hardware > with really fat I/O pipes. It will cost you a tiny fraction of what > you'll spend at Google will cost and perform 10X better. > > Jeff -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
