Re: reduce more than one way
On Nov 7, 2008, at 12:35 PM, Elia Mazzawi wrote: I have 2 hadooop map/reduce programs that have the same map, but a different reduce methods. can i run them in a way so that the map only happens once? If the input to the reduces is the same, you can put the two reduces together and use one of the multiple output libraries. That will let your reducer produce two different output directories. -- Owen
Re: reduce more than one way
Elia Mazzawi wrote: Hello, I'm writing hadoop programs in Java, I have 2 hadooop map/reduce programs that have the same map, but a different reduce methods. Look how MultipleOutputFormat is used. This provides the facility to write to multiple files. Amar can i run them in a way so that the map only happens once? maybe store the intermediate result or something?
Re: reading input for a map function from 2 different files?
some speed wrote: I was wondering if it was possible to read the input for a map function from 2 different files: 1st file --- user-input file from a particular location(path) 2nd file=--- A resultant file (has just one key,value pair) from a previous MapReduce job. (I am implementing a chain MapReduce function) Now, for every key,value pair in the user-input file, I would like to use the same key,value pair from the 2nd file for some calculations. I guess you might need to do some kind of join on the 2 files. Look at contrib/data_join for more details. Amar Is it possible for me to do so? Can someone guide me in the right direction please? Thanks!
hadoop with tomcat
Hello: I have implemented a Map/Reduce job, which will receive data from several web services and deal with it based on hadoop. But I want to build a web application to manange this web service and Monitor the Map/Reduce job process, which will be deployed with Tomcat , I have read several mails about the hadoop with tomcat , But I can't still get it going. I am very confused. Can anybody give me an example or where I can find something about it in much more details, Thanks.
Re: error with inout for the Reduce method
Change the name of the reduce method to be all lower case -- public void
reduce(...
Right now the compiler is complaining that you haven't overridden the
correct abstract method in the base class.
/ Per
On Sat, Nov 8, 2008 at 10:44 PM, pols cut [EMAIL PROTECTED] wrote:
I am trying to get a simple map reduce function going but getting the
following error. could anyone please help/guide me
filename.java is not abstract and does not override abstract method
reduceorg.apache.io.text,java.util.iteratororg.apache.hadoop.io.text,org.apache.hadoop.mapred.OutputCollectororg.apache.hadoop.io.Text.org.apache.hadoop.mapred.Reporter
in org.apache.hadoop.mapred.Reducer
public static class Reduce extends MapReduceBase implements
ReducerText,Text,Text,Text
My Map method is like this:
public static class map extends MapReduceBase implements
MapperLongWritable, Text, Text, Text
{
public void map(LongWritable key, Text value, OutputCollectorText, Text
output, Reporter reporter) throws IOException
{
output.collect(word_key,new Text(str1)); //output.collect(Text,Text)
}
}
public static class Reduce extends MapReduceBase implements ReducerText,
Text, Text, Text
{
public void Reduce(Text key, IteratorText values,
OutputCollectorText, Text output, Reporter reporter) throws IOException
{
output.collect(new Text(Test), new Text(Me));
}
}
configuration is like :
conf.setMapOutputKeyClass(Text.class);
conf.setMapOutputValueClass(Text.class);
conf.setOutputKeyClass(Text.class);
conf.setOutputValueClass(Text.class);
conf.setMapperClass(map.class);
conf.setCombinerClass(Reduce.class);
conf.setReducerClass(Reduce.class);
conf.setInputFormat(TextInputFormat.class);
conf.setOutputFormat(TextOutputFormat.class);
Add more friends to your messenger and enjoy! Go to
http://messenger.yahoo.com/invite/
Re: reading input for a map function from 2 different files?
some speed wrote: I was wondering if it was possible to read the input for a map function from 2 different files: 1st file --- user-input file from a particular location(path) 2nd file=--- A resultant file (has just one key,value pair) from a previous MapReduce job. (I am implementing a chain MapReduce function) Now, for every key,value pair in the user-input file, I would like to use the same key,value pair from the 2nd file for some calculations. I think you can use DistributedCache for distributing your second file among maps. Please see more documentation at http://hadoop.apache.org/core/docs/current/mapred_tutorial.html#DistributedCache Thanks Amareshwari Is it possible for me to do so? Can someone guide me in the right direction please? Thanks!
reading input for a map function from 2 different files?
I was wondering if it was possible to read the input for a map function from 2 different files: 1st file --- user-input file from a particular location(path) 2nd file=--- A resultant file (has just one key,value pair) from a previous MapReduce job. (I am implementing a chain MapReduce function) Now, for every key,value pair in the user-input file, I would like to use the same key,value pair from the 2nd file for some calculations. Is it possible for me to do so? Can someone guide me in the right direction please? Thanks!
Re: Question on opening file info from namenode in DFSClient
Taeho Kang wrote: Hi, thanks for your reply Dhruba, One of my co-workers is writing a BigTable-like application that could be used for online, near-real-time, services. Can your co-worker be convinced to instead spend his time helping-along the ongoing bigtable-like efforts? I think HBase developers would have run into similar issues as well. In hbase, we open the file once and keep it open. File is shared amongst all clients. St.Ack
