Hi. I think I am confusing you guys with talking about various things at the same time. I am mostly (99.9%) after Sequential throughput but sometimes I need massively fast Random Access. And I would never ever pay for the BIG kahoona of machine(s) that would be needed to be able to give me great RA-IO and S-IO at the same time since I'm very economic or perhaps like uncle Scrooge.
You are very accurate in pointing out how our architecture looks like or should look like. I am not trying to advocate memcached at all just trying to explain why we chose memcached as part of our system. We use memcached during PR calculations to store the node's temporary score so whenever you calculate the score for another node which is dependent on the node in question you can access the previously calculated scores. Was that explicit or just confusing ? Trying once more... One MR-node updates the score of a web-node (N1) and another MR-node is processing the incoming inlinks to another web-node (N2) where surprisingly convenient N1 is one of the inlinks. Should not N2 be wanting to be aware of the freshest possible state of N1 ? We feel that but hey we are no PR experts. I believe that it can be accomplished without using shared state but my own intelligence is way too low to be able to address that so I did what I do, solved the problem with the tools I know. The choosing process looked something like this: Sharing state hmmm we need something which looks like a hashmap... hmm mysql memory table no! too slow... hmm HBase no! too slow, hmm BDB over GlusterFs/NFS ? No! too slow or we get into locking issues. Hey why not try memcached a friend of mine said. I refused initially since I do not like it for some historical reason... but we have not found anything that solves the need where many nodes need to access the same memory state at the same time which beats memcached. And I am really a performance freak and I breath caching (look at http://dev.tailsweep.com/projects/mammatus/) which is my baby but so little time, been developing it for five years now. Ahh Voldemort... was not aware of that project, is it former Facebook app ? No they have Cassandra right ? I must test this and compare to my own Mammatus just for fun. 10k per sec, looks promising. But if it solves the same use-case as memcached with basically the same performance then it is no good to us in this use-case since memcached instances are cleared each round and just used during the MR task when Voldemort is more thought of as long term storage right ? Please give me an alternate solution without shared state and I would try that and compare to the current. I agree that PR itself is not that hard but we found the above situation challenging. Again I agree on 99% of the stuff you said. Crawling = 95% hadoop + loading into a sharded DB Parsing = 95% hadoop again + loading into a sharded DB Indexing = Hadoop again + loading from sharded DB -> SOLR PR = hadoop + Tuned Lucene Link index + memcached for sharing node state Frontend: SOLR + Sharded DB Wherever you read Sharded DB you most definitely could replace that with HBase and perhaps we will. I do NOT want Random Access on the backend just in the PR case for sharing state which very well could be a stupid solution from my part but we complete 10 iterations quite fast so something is right at least. I've read something about that Google partitions the matrix locally across machines and that is the reasoning for the fast processing without shared state, guess that is what Pregel does. Long enough email, no I need to work instead of writing :) Cheers lads, it is very educating. /Marcus Yes you are 100% correct that the backend is hadoop. On Fri, Jul 3, 2009 at 8:28 PM, Ted Dunning <[email protected]> wrote: > Not my baby. > > I designed it out of my system at about the same time you did. With 0.20, > however, we are re-evaluating it. > > I still think you are thinking about random access which is a mistake for > batch computations like PageRank. > > On Fri, Jul 3, 2009 at 12:28 AM, Marcus Herou <[email protected] > >wrote: > > > Ted: > > Don't take it personally that I hammered your baby and don't take me > wrong. > > I DO like HBase but being a java app it consumed a lot of resources and > > gave > > me too little back last year when I did my tests. > > I think HBase is definitely the right way to go (can Google be wrong ?) > but > > at the time it was not snappy enough and perhaps my head was not really > > tuned in on hadoop at the time. > > > -- Marcus Herou CTO and co-founder Tailsweep AB +46702561312 [email protected] http://www.tailsweep.com/
