If you serve 30000 req/s, you will likely need a distributed in-memory cache. Things like Terracotta or Coherence may work well for your dataset (500 GB), although I haven't used either.
-Tao ________________________________ From: [email protected] [mailto:[email protected]] On Behalf Of ext Pranas Baliuka Sent: Monday, August 10, 2009 12:11 AM To: [email protected] Subject: [Lustre-discuss] Distributed Object storage lookup of small files Dear Lustre experts/users, I looking for optimal solution of the task: Internet-scale applications must be designed to process high volumes of transactions. Describe a design for a system that must process on average 30,000 HTTP requests per second. For each request, the system must perform a lookup into a dictionary of 50 million words, using a key word passed in via the URL query string. Each response will consist of a string containing the definition of the word (10 KB or less). My initial though was using MySQL/Berkeley DB pointing to SAN, but probably lower level solution would be more affordable. Can I use e.g. QFS storage via Java without DB severer instead. Can SAN be avoided and local HDDs joined to Lustre system? Task is hypothetical, but would be nice to get feedback from specific technology experts... Some ideas ;) I've send similar request to QFS forum and really not sure which product would fit better. Both works as distributed file systems ... and both sounds as convenient storage for particular task. Thanks, Pranas
_______________________________________________ Lustre-discuss mailing list [email protected] http://lists.lustre.org/mailman/listinfo/lustre-discuss
