Hi,
I was wondering is it possible to submit MapReduce job on remote Hadoop cluster.
(i.e) Submitting the job from the machine which doesn't have Hadoop installed
and submitting to different machine where Hadoop installed.
Is it possible to do this?
I guess at least data can be uploaded to HDFS
Subject: Re: Hadoop Permission Problem
Hi Senthil,
In the error message, it says that the permission for datastore is 755. Are
you sure that you have changed it to 777?
Nicholas
- Original Message
From: Natarajan, Senthil [EMAIL PROTECTED]
To: core-user@hadoop.apache.org core-user
2 test test 4096 May 9 09:29 hadoop-test
From the output format, the directories above seem not HDFS directories. Are
you running map/red jobs over local file system (e.g. Linux)?
Nicholas
- Original Message
From: Natarajan, Senthil [EMAIL PROTECTED]
To: core-user
@hadoop.apache.org
Subject: Re: Hadoop Permission Problem
Hi Senthil,
I cannot see why it does not work. Could you try again, do a fs -ls right
after you see the error message?
Nicholas
- Original Message
From: Natarajan, Senthil [EMAIL PROTECTED]
To: core-user@hadoop.apache.org core-user
4096 May 7 18:02 datastore
This one is your local directory. I think you might have mixed up the local
and hdfs directories.
Nicholas
- Original Message
From: Natarajan, Senthil [EMAIL PROTECTED]
To: core-user@hadoop.apache.org core-user@hadoop.apache.org
Sent: Friday, May 9
Hi,
I have some rudimentary question.
In order to use Hadoop (both HDFS and MapReduce) does each user whoever wants
to run the job needs to start their jobtracker and datanode and submit the job.
Or
is it possible to start the jobtracker and datanode using the user hadoop and
other users
May be due to firewall.
Try after stopping the iptables.
If it works add firewall rules to allow communication between master and slaves
(better allow all nodes in the subnet)
-Original Message-
From: Amit Kumar Singh [mailto:[EMAIL PROTECTED]
Sent: Thursday, May 08, 2008 4:50 PM
To:
Hi,
My datanode and jobtracker are started by user hadoop.
And user Test needs to submit the job. So if the user Test copies file to
HDFS, there is a permission error.
/usr/local/hadoop/bin/hadoop dfs -copyFromLocal /home/Test/somefile.txt myapps
copyFromLocal:
:[EMAIL PROTECTED]
Sent: Monday, April 14, 2008 2:20 PM
To: core-user@hadoop.apache.org
Subject: Re: Reduce Output
Try using Text, Text as the output type and use something like a
StringBuffer or Formatter to construct a tab-separated list.
On 4/14/08 11:13 AM, Natarajan, Senthil [EMAIL PROTECTED
Hi,
I have the reduce output like this.
1.0.2.92206475
1.0.2.9 316475847
1.0.3.933846495
1.0.4.93316975
But I want to display like this...
1.0.2.92206 475
1.0.2.9 316 475 847
1.0.3.93
the
reduce output using conventional methods. This works very well if you can
process your data in less time than hadoop takes to start your job.
On 4/14/08 9:42 AM, Natarajan, Senthil [EMAIL PROTECTED] wrote:
Hi,
I have the reduce output like this.
1.0.2.92206475
The format of the reduce output is the responsibility of the reducer. You
can store the output any way you like.
On 4/14/08 10:17 AM, Natarajan, Senthil [EMAIL PROTECTED] wrote:
Thanks Ted.
Actually I was trying to do the third option by myself before posting this
question.
Problem is I couldn't
so that you get
descending order. Make sure you set the number of reducers to 1 so that you
only get a single output file.
If you have less than 10 million values, the conventional sort is likely to
be faster simply because of hadoop's startup time.
On 4/8/08 8:37 AM, Natarajan, Senthil [EMAIL
values, the conventional sort is likely to
be faster simply because of hadoop's startup time.
On 4/8/08 8:37 AM, Natarajan, Senthil [EMAIL PROTECTED] wrote:
Hi,
I am new to MapReduce.
After slightly modifying the example wordcount, to count the IP Address.
I have two files part-0 and part
/hosts
with mapping from hostname to ip address of all nodes in cluster.
I hope my experience will be helpful for you.
On 3/27/08, Natarajan, Senthil [EMAIL PROTECTED] wrote:
Hi,
I have small Hadoop cluster, one master and three slaves.
When I try the example wordcount on one of our log file
Hi,
I have small Hadoop cluster, one master and three slaves.
When I try the example wordcount on one of our log file (size ~350 MB)
Map runs fine but reduce always hangs (sometime around 19%,60% ...) after very
long time it finishes.
I am seeing this error
Shuffle Error: Exceeded
I guess the following file might have localhost entry, change to hostname
HADOOP_INSTALL/conf/masters
HADOOP_INSTALL/conf/slaves
-Original Message-
From: Colin Freas [mailto:[EMAIL PROTECTED]
Sent: Friday, March 21, 2008 12:25 PM
To: core-user@hadoop.apache.org
Subject: NFS mounted
17 matches
Mail list logo
