Scalable Custom Subscriptions

2008-06-08 Thread Jason Rutherglen
The in memory optimized Hadoop implementation sounds like it would be useful for a realtime scalable subscription system. The example I'm interested in testing is using Lucene MemoryIndex to execute millions of queries for notification of clients. Where the Hadoop map is a serialized MemoryIndex

Re: In memory Map Reduce

2008-06-08 Thread Martin Jaggi
Is there some statistics available to monitor which percentage of the pairs remains in memory, and which percentage was written to disk? Or which are these exceptional cases that you mention? Hadoop goes to some lengths to make sure that things can stay in memory as much as possible.

Initial Execution Issue

2008-06-08 Thread Ravi Shankar (Google)
Dear all, I am a newbie started using Haddop yesterday. I am having WIndows XP, and following is the output of grep program, which I had exceuted exactly after following instructions in QuickStart guide:- $ bin/hadoop jar hadoop-0.17.0-examples.jar grp input output 'dfs[a-z.]+' cygpath: cannot

compute document frequency with hadoop-streaming

2008-06-08 Thread xinfan meng
In hadoopstreaming, we accept input from stdin. If we want to compute the document frequncy of words, the somplest way is to output words as keys and file name as values. then how can we get the input file name passed to this MapReduce job? Thanks. -- Best Wishes Meng Xinfan(蒙新泛) Institute of