The application I want to implement is extremely simple conceptually: calls to a HTTP endpoint will record all GET parameters plus a timestamp as a line into a huge log file. My key performance requirements are scalability in a range from 400QPS to 10kQPS while maintaining fairly low latency (< 250ms), and not costing more than ~$3000 per billion requests.
I've been exploring GAE as a vehicle mainly due to the attractive auto-scaling functionality, zero infrastructure management and pay-per use pricing model, but I've been experiencing a fair bit of impedance prototyping something that has led me to question its suitability to this particular problem. 1. Early on I realized the Datastore was out of the question as it's too slow and too costly for the data we're putting in. 2. Using pull queues as temporary storage for datapoints so that they can be batch-saved by a worker on a backend did not scale even to our base 400QPS, the queue itself is a bottleneck 3. Ended up trading off reliability for latency by using dedicated memcache as temporary storage for datapoints while they wait to be saved. (Note, building the FE in Go really helped keep latency and cost/billion down. Thanks Go!) Now I'm dealing with the saving to long term storage part, and with the Files API deprecated it seems there is no way for me to stream the lines into a file on Google Cloud Storage (basically with the newer and supported GCS Client Library one must build the entire file in memory before sending it over to GCS). Am I missing something or does it seem like this is a really hard task to implement in GAE? Any advice the community could offer. Should I just build this on an IaaS from the ground up and incur the cost of infrastructure management while escaping the GAE limitations? I'd appreciate people's opinions/advice. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/groups/opt_out.
