On Thu, Dec 29, 2011 at 2:58 AM, Brandon Wirtz <[email protected]> wrote: > > I would say GAE handles big data really well. But you have to do testing to > make sure your structure is correct, and that your indexes are well thought > out.
I think we are talking about two different things. I'm thinking of Big Data like this: http://en.wikipedia.org/wiki/Big_data Typically characterized by: * Large data volumes * Batch updates * Frequent need to analyze/sift through large quantities of data The GAE datastore performs poorly in this regard. Map/reduce support is anemic at best. Per-gigabyte storage is expensive. Raw I/O performance is *dreadful*. Indexes consume excessive amounts of space. I love the GAE datastore, I think it's hands-down the Best Storage Around for web applications that need scalability and availability. But there's no way in hell I would use it to store a large-scale OLAP system or any other kind of serious analytics product. You don't want EC2 either. You need something like Hadoop on bare metal hardware with really fat I/O pipes. It will cost you a tiny fraction of what you'll spend at Google will cost and perform 10X better. Jeff -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
