Re: Sorting the OutputCollector

2008-04-09 Thread Owen O'Malley
On Apr 8, 2008, at 4:54 AM, Aayush Garg wrote: I construct this type of key, value pairs in the outputcollector of reduce phase. Now I want to SORT this outputcollector in decreasing order of the value of frequency in Custom class. Could some one suggest the possible way to do this? In

Lucene on Hadoop

2008-04-09 Thread Yingfeng Zhang
Hi, From this URL http://www.mail-archive.com/[EMAIL PROTECTED]/msg00998.html I see that Hadoop is not suitable for incremental updates if the inverted files is based on it, and what's more, Nutch has adopted Hadoop, that means the incremental updates ability provided by Lucene will not work in

RE: Reduce Sort

2008-04-09 Thread Natarajan, Senthil
Ted, I am using IntWritable, the default is as you mentioned Ascending order sort, do you know how to set this to sort descending order. I checked the API for IntWritable, LongWritable and JobConf. I couldn't find any methods. Thanks, Senthil -Original Message- From: Ted Dunning

Re: Sorting the OutputCollector

2008-04-09 Thread Aayush Garg
But the problem is that I need to sort according to freq which is the part of my value field... Any inputs?? Could you provide smal piece of code of your thought On Wed, Apr 9, 2008 at 9:45 AM, Owen O'Malley [EMAIL PROTECTED] wrote: On Apr 8, 2008, at 4:54 AM, Aayush Garg wrote: I construct

need help

2008-04-09 Thread krishna prasanna
Hi I started using hadoop very recently I am struct with the basic example when i am trying to run bin/hadoop jar hadoop-*-examples.jar grep input output 'dfs[a-z.]+' i am getting output as 08/04/09 21:23:12 INFO mapred.FileInputFormat: Total input paths to process : 2 java.io.IOException:

Does any one tried to build Hadoop..

2008-04-09 Thread krishna prasanna
Does any one tried to build Hadoop ? Thanks Regards, Krishna. Meet people who discuss and share your passions. Go to http://in.promos.yahoo.com/groups/bestofyahoo/

Re: Does any one tried to build Hadoop..

2008-04-09 Thread Jean-Daniel Cryans
The ANT script works well also. Jean-Daniel 2008/4/9, Khalil Honsali [EMAIL PROTECTED]: Hi, With eclise it's easy, you just have to add it as a new project, make sure you add all libraries in folder lib and should compile fine There is also an eclipse plugin for running hadoop jobs directly

Re: Does any one tried to build Hadoop..

2008-04-09 Thread Khalil Honsali
Mr. Jean-Daniel, where is the ant script please? On 10/04/2008, Jean-Daniel Cryans [EMAIL PROTECTED] wrote: The ANT script works well also. Jean-Daniel 2008/4/9, Khalil Honsali [EMAIL PROTECTED]: Hi, With eclise it's easy, you just have to add it as a new project, make sure you

Hadoop VS MogileFS

2008-04-09 Thread Garri Santos
Hi! Im new to DFS and wondering what are the advantages and disadvantages of hadoop and mogileFS to anyone that have experienced both. Thanks, Garri

Hadoop performance on EC2?

2008-04-09 Thread Nate Carlson
Hey all, We've got a job that we're running in both a development environment, and out on EC2. I've been rather displeased with the performance on EC2, and was curious if the results that we've been seeing are similar to other people's, or if I've got something misconfigured. ;) In both

hdfs 100T?

2008-04-09 Thread Todd Troxell
Hello list, I was unable to access the archives for this list as http://hadoop.apache.org/mail/core-user/ returns 403. I am interested in using HDFS for storage, and for map/reduce only tangentially. I see clusters mentioned in the docs with many many nodes and 9TB of disk. Is HDFS expected to

Re: Hadoop performance on EC2?

2008-04-09 Thread Chris K Wensel
a few things.. make sure all nodes are running in the same 'availability zone', http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1347 and that you are using the new xen kernels. http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1353categoryID=101