When installing hadoop on slave machines, do we have to install hadoop
at same locations on each machine ?
Can we have hadoop installation at different location on different
machines at same cluster ?
If yes, what things we have to take care in that case
Thanks,
Praveenesh
do the simple think and make all of your DN the same?
Sent from my iPhone
On Dec 23, 2011, at 6:51 AM, praveenesh kumar praveen...@gmail.com wrote:
When installing hadoop on slave machines, do we have to install hadoop
at same locations on each machine ?
Can we have hadoop installation
Okay so I have one question in mind.
Suppose I have a replication factor of 3 on my cluster of some N
nodes, where N3 and there is a data block B1 that exists on some 3
Data nodes -- DD1, DD2, DD3.
I want to run some Mapper function on this block.. My JT will
communicate with NN, to know where
Hey Guys,
So I have a very naive question in my mind regarding Hadoop cluster nodes ?
more cores or more nodes – Shall I spend money on going from 2-4 core
machines, or spend money on buying more nodes less core eg. say 2
machines of 2 cores for example?
Thanks,
Praveenesh
Did anyone tried HIVE on Hadoop 0.20.205.
I am trying to build HIVE from svn. but I am seeing its downloading
hadoop-0.20.3-CDH3-SNAPSHOT.tar.gz and hadoop-0.20.1.tar.gz.
If I am trying to do ant -Dhadoop.version=”0.20.205″ package ,but build is
failing.
Any ideas or suggestion on what I may be
/build.xml:130: The following error
occurred while executing this line:
/usr/local/hadoop/hive/release-0.7.1/jdbc/build.xml:51: Compile failed; see
the compiler error output for details.
Total time: 29 minutes 46 seconds
Thanks,
Praveenesh
On Fri, Dec 9, 2011 at 2:08 PM, praveenesh kumar praveen
in 0.21 version onwards.
From: praveenesh kumar [praveen...@gmail.com]
Sent: Wednesday, December 07, 2011 12:40 PM
To: common-user@hadoop.apache.org
Subject: HDFS Backup nodes
Does hadoop 0.20.205 supports configuring HDFS backup nodes ?
Thanks
How to avoid Warning: $HADOOP_HOME is deprecated messages on hadoop
0.20.205 ?
I tried adding *export HADOOP_HOME_WARN_SUPPRESS= *in hadoop-env.sh on
Namenode.
But its still coming. Am I doing the right thing ?
Thanks,
Praveenesh
- Alex
On Wed, Dec 7, 2011 at 11:37 AM, praveenesh kumar praveen...@gmail.com
wrote:
How to avoid Warning: $HADOOP_HOME is deprecated messages on hadoop
0.20.205 ?
I tried adding *export HADOOP_HOME_WARN_SUPPRESS= *in hadoop-env.sh on
Namenode.
But its still coming. Am I doing
Does hadoop 0.20.205 supports configuring HDFS backup nodes ?
Thanks,
Praveenesh
Hi everyone,
So I have this blade server with 4x500 GB hard disks.
I want to use all these hard disks for hadoop HDFS.
How can I achieve this target ?
If I install hadoop on 1 hard disk and use other hard disk as normal
partitions eg. -
/dev/sda1, -- HDD 1 -- Primary partition -- Linux +
Hi all,
Any Idea, when is hadoop 0.20.205 is officially going to release ?
Is Hadoop-0.20.205 rc2 stable enough to start into production ?
I am using hadoop-0.20-append now with hbase 0.90.3, want to switch to 205.
But looking for some valubale suggestions/recommendations ?
Thanks,
Praveenesh
try commenting 127.0.0.1 localhost line in your /etc/hosts and then restart
the cluster and then try again.
Thanks,
Praveenesh
On Sun, Oct 16, 2011 at 2:00 PM, Humayun gmail humayun0...@gmail.comwrote:
we are using hadoop on virtual box. when it is a single node then it works
fine for big
gmail humayun0...@gmail.comwrote:
commenting the line 127.0.0.1 in /etc/hosts is not working. if i format the
namenode then automatically this line is added.
any other solution?
On 16 October 2011 19:13, praveenesh kumar praveen...@gmail.com wrote:
try commenting 127.0.0.1 localhost line
cluster- add ub16 entry
in /etc/hosts on where the task running.
On 10/5/2011 12:15 PM, praveenesh kumar wrote:
I am trying to use distcp to copy a file from one HDFS to another.
But while copying I am getting the following exception :
hadoop distcp hdfs://ub13:54310/user/hadoop/weblog
hdfs
I am trying to use distcp to copy a file from one HDFS to another.
But while copying I am getting the following exception :
hadoop distcp hdfs://ub13:54310/user/hadoop/weblog
hdfs://ub16:54310/user/hadoop/weblog
11/10/05 10:41:01 INFO mapred.JobClient: Task Id :
Hi,
I want to know can we use SAN storage for Hadoop cluster setup ?
If yes, what should be the best pratices ?
Is it a good way to do considering the fact the underlining power of Hadoop
is co-locating the processing power (CPU) with the data storage and thus it
must be local storage to be
Hi,
Suppose I am having 10 windows machines and if I have 10 VM individual
instances running on these machines independently, can I use these VM
instances to communicate with each other so that I can make hadoop cluster
using those VM instances.
Did anyone tried that thing ?
I know we can setup
the difference, it's a pure vmware
stuff.
Obviously, it's not something you can do for production nor performance
analysis.
Cheers,
N.
On Wed, Sep 28, 2011 at 8:38 AM, praveenesh kumar praveen...@gmail.com
wrote:
Hi,
Suppose I am having 10 windows machines and if I have 10 VM individual
Hey,
I have this code written using mahout. I am able to run the code from
eclipse
How can I run the code written in mahout from command line ?
My question is do I have to make a jar file and run it as hadoop jar
jarfilename.jar class
or shall I run it using simple java command ?
Can anyone
space for
NN in big clusters.
Configuring good configuration DN which has very good amount of space. And
NN has less space to store your files metadata info then its of no use to
have more space in DNs right :-)
Regards,
Uma
- Original Message -
From: praveenesh kumar praveen
Guys,
As far as I know hadoop, I think, to copy the files to HDFS, first it needs
to be copied to the NameNode's local filesystem. Is it right ??
So does it mean that even if I have a hadoop cluster of 10 nodes with
overall capacity of 6TB, but if my NameNode's hard disk capacity is 500 GB,
I can
-
From: praveenesh kumar praveen...@gmail.com
Date: Wednesday, September 21, 2011 2:14 pm
Subject: Any other way to copy to HDFS ?
To: common-user@hadoop.apache.org
Guys,
As far as I know hadoop, I think, to copy the files to HDFS,
first
it needs
to be copied
back also will the sameway.
I hope you will understand better now :-)
Regards,
Uma
- Original Message -
From: praveenesh kumar praveen...@gmail.com
Date: Wednesday, September 21, 2011 3:11 pm
Subject: Re: Any other way to copy to HDFS ?
To: common-user@hadoop.apache.org
So I want
Regards,
Uma
- Original Message -
From: praveenesh kumar praveen...@gmail.com
Date: Wednesday, September 21, 2011 3:41 pm
Subject: Fwd: Any other way to copy to HDFS ?
To: common-user@hadoop.apache.org
Thanks a lot. I am trying to run the following code on my windows
machinethat
Is there any way that we can run a particular job in a hadoop on subset of
datanodes ?
My problem is I don't want to use all the nodes to run some job,
I am trying to make Job completion Vs No. of nodes graph for a particular
job.
One way to do is I can remove datanodes, and then see how much
for?
(There are ways to achieve the exclusion dynamically, by writing a
scheduler, but hard to tell without knowing what you need
specifically, and why do you require it?)
On Wed, Sep 21, 2011 at 6:32 PM, praveenesh kumar praveen...@gmail.com
wrote:
Is there any way that we can run a particular job
clients and DNs.
Also Does Namenode/JobTracker machine's configuration needs to be better
than datanodes/tasktracker's ??
I did not get this question.
Regards,
Uma
- Original Message -
From: praveenesh kumar praveen...@gmail.com
Date: Thursday, September 22, 2011 10:13 am
Subject
Harsh, Can you please tell how can we use MultipleInputs using Job Object on
hadoop 0.20.2. As you can see, in MultipleInputs, its using JobConf object.
I want to use Job object as mentioned in new hadoop 0.21 API.
I remember you talked about pulling out things from new API and add it into
out
backported in
it).
On Wed, Aug 24, 2011 at 2:40 PM, praveenesh kumar praveen...@gmail.com
wrote:
Hello guys,
I am looking to use MultipleInputs.addInputPath() method in hadoop
0.20.2.
But when I am looking to its signature in the API, its like this :
*
public static void
Hello guys,
I am looking to use MultipleInputs.addInputPath() method in hadoop 0.20.2.
But when I am looking to its signature in the API, its like this :
*
public static void addInputPath(JobConf conf,
Path path,
Hi,
Anyone working on YCSB (Yahoo Cloud Service Benchmarking) for HBase ??
I am trying to run it, its giving me error:
$ java -cp build/ycsb.jar com.yahoo.ycsb.CommandLine -db
com.yahoo.ycsb.db.HBaseClient
YCSB Command Line client
Type help for command line help
Start with -help for usage info
Hi,
How can I give filename as key to mapper ?
I want to know the occurence of word in set of docs, so I want to keep key
as filename. Is it possible to give input key as filename in map function ?
Thanks,
Praveenesh
/r0.20.2/mapred_tutorial.html#Task+JVM+Reuse
for more such goodies.
On Fri, Jul 15, 2011 at 5:44 PM, praveenesh kumar praveen...@gmail.com
wrote:
Hi,
How can I give filename as key to mapper ?
I want to know the occurence of word in set of docs, so I want to keep
key
as filename
hocked with SequenceFileInputFormat where
my input files are key,value records written by a regular java program
(or
parser). Then my Hadoop job will look at the keys and values that I wrote.
I hope this helps a little,
Mark
On Thu, May 5, 2011 at 4:31 AM, praveenesh kumar praveen
. Need your guidance.
Thanks,
Praveenesh
On Sat, Jul 2, 2011 at 1:36 PM, Joey Echeverria j...@cloudera.com wrote:
Try replacing the hadoop jar from the pig lib directory with the one from
your cluster.
-Joey
On Jul 2, 2011, at 0:38, praveenesh kumar praveen...@gmail.com wrote:
Hi guys
Hi guys..
I am previously using hadoop and Hbase...
So for Hbase to run perfectly fine we need Hadoop-0.20-append for Hbase jar
files.. So I am using Hadoop-0.20-append jar files.. which made both my
hadoop and hbase to work fine..
Now I want to use pig for my hadoop and hbase clusters..
I
eclipse which is provided with
hadoop-0.20-append release, it will work fine.
Devaraj K
-Original Message-
From: praveenesh kumar [mailto:praveen...@gmail.com]
Sent: Wednesday, June 22, 2011 11:25 AM
To: common-user@hadoop.apache.org
Subject: Hadoop eclipse plugin stopped working
Guys,
I was using hadoop eclipse plugin on hadoop 0.20.2 cluster..
It was working fine for me.
I was using Eclipse SDK Helios 3.6.2 with the plugin
hadoop-eclipse-plugin-0.20.3-SNAPSHOT.jar downloaded from JIRA
MAPREDUCE-1280
Now for Hbase installation.. I had to use hadoop-0.20-append compiled
Helloo..
My namenode is running with the following exceptions and going to safemode
everytime its trying to start the datanodes.. why so ?
I deleted all the files in the HDFS.. and ran it again..!!
2011-06-07 15:02:19,467 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG:
wrote:
On 06/07/2011 10:50 AM, praveenesh kumar wrote:
The logs say
The ratio of reported blocks 0.9091 has not reached the threshold 0.9990.
Safe mode will be turned off automatically.
not enough datanodes reported in, or they are missing data
the
namenode.
It should definitely work then
Cheers,
Jagaran
From: praveenesh kumar praveen...@gmail.com
To: common-user@hadoop.apache.org
Sent: Tue, 7 June, 2011 3:14:01 AM
Subject: Re: NameNode is starting with exceptions whenever its trying
,
Jagaran
From: praveenesh kumar praveen...@gmail.com
To: common-user@hadoop.apache.org
Sent: Tue, 7 June, 2011 3:14:01 AM
Subject: Re: NameNode is starting with exceptions whenever its trying to
start
datanodes
But I dnt have any data on my HDFS.. I
...@yahoo.co.inwrote:
Yes Correct
Password less SSH between your name node and some of your datanode is not
working
From: praveenesh kumar praveen...@gmail.com
To: common-user@hadoop.apache.org
Sent: Tue, 7 June, 2011 10:56:08 AM
Subject: Re: NameNode is starting
and formatting the name node may
help
you
From: praveenesh kumar praveen...@gmail.com
To: common-user@hadoop.apache.org
Sent: Tue, 7 June, 2011 11:05:03 AM
Subject: Re: NameNode is starting with exceptions whenever its trying to
start
datanodes
Sorry
Hello guys..!!!
I am currently working on Hbase 0.90.3 and Hadoop 0.20.2
Since this hadoop version does not support rsync hdfs..
so I copied the *hadoop-core-append jar* file from *hbase/lib* folder
into*hadoop folder
* and replaced it with* hadoop-0.20.2-core.jar*
which was suggested in the
Hi,
Not able to see my email in the mail archive..So sending it again...!!!
Guys.. need your feedback..!!
Thanks,
Praveenesh
-- Forwarded message --
From: praveenesh kumar praveen...@gmail.com
Date: Mon, Jun 6, 2011 at 12:09 PM
Subject: Hadoop is not working after adding
hadoop
Hello guys..
I am not able to run my hbase 0.90.3 cluster on top of hadop 0.20.2
cluster I dnt know why its happening..onlye 1 time its running .. after
that its not..
HBASE WEB URL is showing the following exception
Why its happening...
Please help..!!
Thanks,
Praveenesh
HTTP ERROR
Hello guys..
Changing the name of the hadoop-apppend-core.jar file to
hadoop-0.20.2-core.jar did the trick..
Its working now..
But is this the right solution to this problem ??
Thanks,
Praveenesh
On Mon, Jun 6, 2011 at 2:18 PM, praveenesh kumar praveen...@gmail.comwrote:
Hi,
Not able
the exceptions in my other email.. I hope it will reach the
mailing group after some time.
Thanks,
Praveenesh
On Mon, Jun 6, 2011 at 8:59 PM, Stack st...@duboce.net wrote:
On Mon, Jun 6, 2011 at 6:23 AM, praveenesh kumar praveen...@gmail.com
wrote:
Changing the name of the hadoop-apppend-core.jar file
Guys,
I am in a very big big confusion. Please.. I really need your feedbacks and
suggestions..
The scenario is like this...
I set up *Hadoop 0.20.2 cluster* of *12 nodes*..
Now I set up* Hbase 0.90.3* *12 node cluster* on top of it.
But after all that experimenting and struggling.. I read
Hey guys..!!
Any suggestions..!!!
-- Forwarded message --
From: praveenesh kumar praveen...@gmail.com
Date: Wed, Jun 1, 2011 at 2:48 PM
Subject: Data node is taking time to start.. Error register
getProtocolVersion in namenode..!!
To: common-user@hadoop.apache.org
Hello Hadoop
Hello guys.
I just have installed hbase on my hadoop cluster.
HMaster,HRegionServer,HQuorum Peer all are working fine.. as I can see these
processes running through JPS.
Is there any way to know which regionservers are running right and not ?
I mean is there some kind of hbase web UI or anyway
Hello Hadoop users.!!!
Well.. I am doing simple hadoop single node installation.. but my datanode
is taking some time to run..
If I go through the namenode logs.. I am getting some strange exception.
2011-06-02 03:59:59,959 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG:
Hello guys,
In case any of you are working on HBASE, I just wrote a program by reading
some tutorials..
But no where its mentioned how to run codes on HBASE. In case anyone of you
has done some coding on HBASE , can you please tell me how to run it.
I am able to compile my code by adding
: Berlin und München, Deutschland;
Registergericht: Berlin Charlottenburg, HRB 12300, München, HRB 6684;
WEEE-Reg.-Nr. DE 23691322
-Ursprüngliche Nachricht-
Von: praveenesh kumar [mailto:praveen...@gmail.com]
Gesendet: Dienstag, 24. Mai 2011 11:08
An: common-user@hadoop.apache.org
, i.e. '.')
so Java can't really find it. This is not a HBase issue.
HTH.
On Tue, May 24, 2011 at 3:23 PM, praveenesh kumar praveen...@gmail.com
wrote:
I am simply using HBase API, not doing any Map-reduce work on it.
Following is the code I have written , simply creating the file on HBase
, May 24, 2011 at 4:59 PM, Harsh J ha...@cloudera.com wrote:
Praveenesh,
On Tue, May 24, 2011 at 4:31 PM, praveenesh kumar praveen...@gmail.com
wrote:
Hey Harsh,
Actually I mailed to HBase mailing list also.. but since I wanted to get
this thing done as soon as possible so I mailed
Please any suggestions..!!
-- Forwarded message --
From: praveenesh kumar praveen...@gmail.com
Date: Sun, May 22, 2011 at 2:23 PM
Subject: Hbase question,,!!
To: common-user@hadoop.apache.org
Okay guys.. so I have hadoop cluster of 5 nodes.. the configuration look
like
OR you can refer to following tutorial for a referal..!!
http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-single-node-cluster/
On Mon, May 23, 2011 at 11:06 PM, jgroups mohitanch...@gmail.com wrote:
I am trying to install hadoop in cluster env with multiple nodes.
Okay guys.. so I have hadoop cluster of 5 nodes.. the configuration look
like this.
162.192.100.53 -- Master as well as slave
Slave nodes :
162.192.100.52
162.192.100.51
162.192.100.50
162.192.100.49
Now I want to implement HBASE on my hadoop cluster.. What can be the best
configuration for my
in there which says:
mapred.reduce.tasks = 12
The reason I chose 12 was because it was recommended that I choose 2x
number of tasktrackers.
On 5/20/11, praveenesh kumar praveen...@gmail.com wrote:
Hello everyone,
I am using wordcount application to test on my hadoop cluster
to use how many reducers to make our program run in the
best way possible ??
Thanks,
Praveenesh
On Mon, May 23, 2011 at 10:08 AM, praveenesh kumar praveen...@gmail.comwrote:
My program is a basic program like this :
import java.io.IOException;
import java.util.*;
import org.apache.hadoop.fs.Path
hey..!!
I have a question.
If I copy some file on HDFS file system, it will get split into blocks and
Namenode will keep all these meta info with it.
How can I see that info.
I copied 5 GB file on NameNode, but I see that file only on the NameNode..
It doesnot get split into blocks..??
How can I
Hello everyone,
I am using wordcount application to test on my hadoop cluster of 5 nodes.
The file size is around 5 GB.
Its taking around 2 min - 40 sec for execution.
But when I am checking the JobTracker web portal, I am seeing only one
reducer is running. Why so ??
How can I change the code
one reducer
On 2011-05-20, at 7:19 AM, praveenesh kumar praveen...@gmail.com wrote:
Hello everyone,
I am using wordcount application to test on my hadoop cluster of 5 nodes.
The file size is around 5 GB.
Its taking around 2 min - 40 sec for execution.
But when I am checking
Hi,
As we know hadoop mapper takes input as (Key,Value) pairs and generate
intermediate (Key,Value) pairs and usually we give input to our Mapper as a
text file.
How hadoop understand this and parse our input text file into (Key,Value)
Pairs
Usually our mapper looks like --
*public* *void*
hey,
Can we access NameNode's hdfs on our slave machines ??
I am just running command hadoop dfs -ls on my slave machine ( running
tasktracker and Datanode), and its giving me the following output :
hadoop@ub12:~$ hadoop dfs -ls
11/05/05 18:31:54 INFO ipc.Client: Retrying connect to server:
Hi,
I am new to hadoop and the scenario is like this :
I have hadoop installed on a linux machine having IP as (162.192.100.46)
and I have another window machine with eclipse and hadoop plugin installed..
I am able to connect to linux hadoop machine and can see the dfs location
and mapred folder
you have a valid path /user/hadoop/In (it must be a file, or a
directory with files)?
On Mon, Apr 25, 2011 at 11:32 AM, praveenesh kumar praveen...@gmail.com
wrote:
Hi,
I am new to hadoop and the scenario is like this :
I have hadoop installed on a linux machine having IP
Hi,
I am running the following code (Gender.java) on my hadoop .
import java.io.IOException;
import java.util.*;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapred.*;
import org.apache.hadoop.util.*;
public class
, Text, LongWritable {
...
}
-Joey
On Mon, Apr 25, 2011 at 4:38 AM, praveenesh kumar praveen...@gmail.com
wrote:
Hi,
I am running the following code (Gender.java) on my hadoop .
import java.io.IOException;
import java.util.*;
import org.apache.hadoop.fs.Path;
import
Hi,
I am learning hadoop.
Whenever we use hadoop dfs -copyFromLocal input-file name output-file
name
I assume the file is copied from linux file system to hadoop file system
However the output of the command shows us that file is somewhere stored in
/user/hadoop/*
But if we search it from
The problem I am facing is
1 I have 1 Windows System. I am running eclipse with hadoop - plugin.. Its
not a part of hadoop cluster. I am able to connect to hadoop systems and can
view DFS and MAPRED folders using this plugin. If I am able to view the
contents of the hadoop, so I am assuming that
Hello everyone,
Thanks everyone for guiding me everytime. I am able to setup hadoop cluster
of 10 nodes.
Now comes HBASE..!!!
I am new to all this...
My problem is I have huge data to analyze.
so shall I go for single node Hbase installation on all nodes or go for
distributed Hbase
Hi,
Have you checked the ports on which map-reduce server and hdfs are running.
I guess the plugin gives by default its own ports. you have to replace it
with the ports on which you are running your map reduce and hdfs.
I guess that might help you..!!
Thanks,
Praveenesh
On Mon, Apr 18, 2011 at
Hello everyone,
I am new to hadoop...
I set up a hadoop cluster of 4 ubuntu systems. ( Hadoop 0.20.2)
and I am running the well known word count (gutenberg) example to test how
fast my hadoop is working..
But whenever I am running wordcount example..I am not able to see any much
processing time
Hello,
I am new to hadoop.
I am using hadoop 0.20.2 on ubuntu.
I recently installed and configured hadoop using the available tutorials on
internet.
My hadoop is running properly.
But Whenever I am trying to run a wordcount example, the wordcount program
got stuck at the reduce part. After long
14, 2011 at 3:42 PM, praveenesh kumar praveen...@gmail.com
wrote:
attempt_201104142306_0001_m_00_0, Status : FAILED
Too many fetch-failures
11/04/14 23:32:50 WARN mapred.JobClient: Error reading task outputInvalid
argument or cannot assign requested address
11/04/14 23:32:50 WARN
101 - 179 of 179 matches
Mail list logo
