Re: sort example

;> > 212 becomes 000212 >> > 123123 >> > >> > After you do the sorting, trim out the preceding zeros ... >> > >> > Rgds, >> > Ricky >> > -Original Message- >> > From: David Rio [mailto:driodei...@gmail.com] >&

Re: sort example

ecomes 001324 > > 212 becomes 000212 > > 123123 > > > > After you do the sorting, trim out the preceding zeros ... > > > > Rgds, > > Ricky > > -Original Message- > > From: David Rio [mailto:driodei...@gmail.com] > > Sent: Sunday, May

Re: sort example

@gmail.com] > Sent: Sunday, May 17, 2009 8:34 AM > To: core-user@hadoop.apache.org > Subject: Re: sort example > > On Sun, May 17, 2009 at 10:18 AM, Ricky Ho wrote: >> >> I think using a single reducer causes the sorting to be done sequentially >> and hence defeats

RE: sort example

Rio [mailto:driodei...@gmail.com] Sent: Sunday, May 17, 2009 8:34 AM To: core-user@hadoop.apache.org Subject: Re: sort example On Sun, May 17, 2009 at 10:18 AM, Ricky Ho wrote: > > I think using a single reducer causes the sorting to be done sequentially and > hence defeats the purpose

Re: sort example

On Sun, May 17, 2009 at 10:18 AM, Ricky Ho wrote: > > I think using a single reducer causes the sorting to be done sequentially and > hence defeats the purpose of using Hadoop in the first place. I agree, but this is just for testing. Actually I used two reducers in my example. > Perhaps you ca

Re: sort example

Thanks for the reply Peter but that's not it. I use the comparator class to pass the -n flag but the shuffling does not sort the keys numerically. Tell me if this is wrong: 1. input (text file): 1324 212 123123 2332 145455 . 2. The mapper job will spawn a process that will run my ruby code pass

RE: sort example

h subrange. Rgds, Ricky -Original Message- From: Peter Skomoroch [mailto:peter.skomor...@gmail.com] Sent: Saturday, May 16, 2009 9:42 PM To: core-user@hadoop.apache.org Subject: Re: sort example I just copy and pasted that comparator option from the docs, the -n part is what you want in this

Re: sort example

I just copy and pasted that comparator option from the docs, the -n part is what you want in this case. On Sun, May 17, 2009 at 12:40 AM, Peter Skomoroch wrote: > 1) It is doing alphabetical sort by default, you can force Hadoop streaming > to sort numerically with: > > -D mapred.text.key.compar

Re: sort example

1) It is doing alphabetical sort by default, you can force Hadoop streaming to sort numerically with: -D mapred.text.key.comparator.options=-k2,2nr\ see the section "A Useful Comparator Class" in the streaming docs: http://hadoop.apache.org/core/docs/current/streaming.html and https://issues.apa

Re: sort example

BTW, Basically, this is the unix equivalent to what I am trying to do: $ cat input_file.txt | sort -n -drd On Sat, May 16, 2009 at 11:10 PM, David Rio wrote: > Hi, > I am trying to sort some data with hadoop(streaming mode). The input looks > like: > $ cat small_numbers.txt > 9971681 > 9686036