> > > Experimental results: > > Simple Python: 1 minute 22 seconds > GAE dev server: 2 hours 17 minutes 12 seconds >
The Dev server isnt designed for performance. Its designed to be a close approximation of the functionality of the online service. Enough so developers can work. Much functionality is emulated in software, so will not perform like real 'production'. It will be a lot slower due to lots of overhead. . > > Given the staggering difference in run time, even if computation in the > cloud were free, I'd still opt to compute locally > If you *can* do the computation locally, it will nearly always be quicker. The overhead of a solution that CAN scale, is too much, if dont need that scale. The crunch as you say comes when it gets too big. > > Everything I hear about cloud computing makes it sound like the gleaming, > glossy future, but these results makes it seem expensive and slow. > Its not 'magic' you don't access to supercomputers. What you get is access to lots and lots (and lots) of small computers. And someone else manages them, and gives you pre-configured algorithms. Its horizontal scaling, rather than vertical scaling. > $8+ to do a mapreduce across 60MB of data just doesn't seem like a good > deal to me. > If you only looking at AppEngine purely for data processing, then no its not a good place. There are much better value ways of processing data. > At that rate, there's no way I can afford to process my 10GB dataset on > App Engine. I understand that with the pipeline model you get fault > tolerance and status reports and basic job management, but none of that is > worth the expense or a 100x performance hit. > > I think there's two possible problems going on here: > > 1) I made a technical mistake in my experiment and my results are invalid > Can't really comment on that. It sounds like with a bit more work, it should at least execute cleanly. > 2) I'm not understanding the benefits / value proposition of App Engine > You do seem to have a unrealistic idea of what App Engine is. Its more of a hosting platform* (that *can* do data processing) - rather than a data processing system (that can do hosting too) Technically a PaaS https://en.wikipedia.org/wiki/Platform_as_a_service I'm just digging into map reduce on Google App Engine, and my early results are discouraging. I had in mind that I'd process about 10GB of data for an analysis I want to do, and I didn't even think that'd be that big a deal (given all the talk about petabyte-scale storage and such), but it's currently looking impossible. Going back to this. It sound that AppEngine is not a ideal platform for your task. Compute Engine, sounds a better fit https://cloud.google.com/solutions/hadoop/ -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CAJCAUuL%2B20Cgpko4xZ-e9u60qPnC3tOq-Kqy8_4s3sDPsC%3DXCw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
