Re: Jobtracker history logs missing
Hi Prashant, The userlogs for job are deleted after time specified by * mapred.userlog.retain.hours* property defined in mapred-site.xml (default is 24 Hrs). Thanks, Nitin On 5 April 2012 14:26, Prashant Kommireddi prash1...@gmail.com wrote: I am noticing something strange with JobTracker history logs on my cluster. I see configuration files (*_conf.xml) under /logs/history/ but none of the actual job logs. Anyone has ideas on what might be happening? Thanks, -- Nitin Khandelwal
Re: How do i set number of tasktrackers per node ?
Hi , You can set using properties mapred.tasktracker.map.tasks.maximum, mapred.tasktracker.reduce.tasks.maximum in mapred-site.xml Thanks, Nition On 15 December 2011 10:44, ArunKumar arunk...@gmail.com wrote: Hi guys ! I see that we can set number of map/reduce tasks per tasktracker but how do i set number of tasktrackers per node in a cluster ? Arun -- View this message in context: http://lucene.472066.n3.nabble.com/How-do-i-set-number-of-tasktrackers-per-node-tp3587672p3587672.html Sent from the Hadoop lucene-users mailing list archive at Nabble.com. -- Nitin Khandelwal
Hadoop 0.22.0 Documentation: Config Files default point to diff(current) version Config
Hi All, I was going through Hadoop 0.22.0 Documentation at http://hadoop.apache.org/common/docs/r0.22.0/cluster_setup.html#Configuration+Files. The default config files (core-default, mapred-default, hdfs-default) are pointing to* CURRENT Hadoop release and not 0.22.0. * *For Example :* *core-default.xml* points to * http://hadoop.apache.org/common/docs/current/core-default.html* instead of : *http://hadoop.apache.org/common/docs/r0.22.0/core-default.html* Also, link to mapred-queues.xml.templatehttp://hadoop.apache.org/mapreduce/docs/current/mapred_queues.xmlis broken. Please change the links (if it is wrong) as it is pretty confusing. Thanks, -- Nitin Khandelwal
Re: Not able to post a job in Hadoop 0.23.0
Hi Arun, Thanks for your reply. There is one NodeManager running ; Following is the NodeManager UI : Rack Node State Node Address Node HTTP Address Health-status Last health-update Health-report Containers Mem Used Mem Avail /default-rack RUNNING germinait93:50033 germinait93: Healthy 9-Dec-2011 13:03:33 Healthy 0 0 KB 1 GB Also, I get to see only following Logs relevant to the job posting : 2011-12-09 13:10:57,300 INFO fifo.FifoScheduler (FifoScheduler.java: addApplication(288)) - Application Submission: application_1323416004722_0002 from minal.kothari, currently active: 1 2011-12-09 13:10:57,300 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:handle(464)) - Processing event for appattempt_1323416004722_0002_01 of type APP_ACCEPTED 2011-12-09 13:10:57,317 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:handle(476)) - appattempt_1323416004722_0002_01 State change from SUBMITTED to SCHEDULED 2011-12-09 13:10:57,318 INFO rmapp.RMAppImpl (RMAppImpl.java:handle(416)) - Processing event for application_1323416004722_0002 of type APP_ACCEPTED 2011-12-09 13:10:57,318 INFO rmapp.RMAppImpl (RMAppImpl.java:handle(428)) - application_1323416004722_0002 State change from SUBMITTED to ACCEPTED 2011-12-09 13:10:57,320 INFO resourcemanager.RMAuditLogger (RMAuditLogger.java:logSuccess(140)) - USER=minal.kothari IP=192.168.0.93 OPERATION=Submit Application Request TARGET=ClientRMService RESULT=SUCCESS APPID=application_1323416004722_0002 Please let me know if you need some other logs . Thanks, Nitin On 9 December 2011 12:44, Arun C Murthy a...@hortonworks.com wrote: Moving to mapreduce-user@, bcc common-user@. Can you see any errors in the logs? Typically this happens when you have no NodeManagers. Check the 'nodes' link and then RM logs. Arun On Nov 29, 2011, at 8:36 PM, Nitin Khandelwal wrote: HI , I have successfully setup Hadoop 0.23.0 in a single m/c. When i post a job, it gets posted successfully (i can see the job in UI), but the job is never ASSIGNED and waits forever. Here are details of what i see for that Job in UI Name: random-writer State: ACCEPTED FinalStatus: UNDEFINED Started: 30-Nov-2011 10:08:55 Elapsed: 49sec Tracking URL: UNASSIGNED http://192.168.0.93:8900/cluster/app/application_1322627869620_0001# Diagnostics: AM container logs: AM not yet registered with RM Cluster ID: 1322627869620 ResourceManager state: STARTED ResourceManager started on: 30-Nov-2011 10:07:49 ResourceManager version: 0.23.0 from 722cd694fc4ab6d040c0a34f9fb5b476e330ee60 by hortonmu source checksum 4975bf112aa7faa5673f604045ced798 on Thu Nov 3 09:07:31 UTC 2011 Hadoop version: 0.23.0 from d4fee83ec1462ab9824add6449320617caa7c605 by hortonmu source checksum 4e42b2d96c899a98a8ab8c7cc23f27ae on Thu Nov 3 08:59:12 UTC 2011 Can some one tell where am i going wrong?? Thanks, -- Nitin Khandelwal -- Nitin Khandelwal
Re: Not able to post a job in Hadoop 0.23.0
CC : mapreduce-user On 9 December 2011 13:14, Nitin Khandelwal nitin.khandel...@germinait.comwrote: Hi Arun, Thanks for your reply. There is one NodeManager running ; Following is the NodeManager UI : Rack Node State Node Address Node HTTP Address Health-status Last health-update Health-report Containers Mem Used Mem Avail /default-rack RUNNING germinait93:50033 germinait93: Healthy 9-Dec-2011 13:03:33 Healthy 0 0 KB 1 GB Also, I get to see only following Logs relevant to the job posting : 2011-12-09 13:10:57,300 INFO fifo.FifoScheduler (FifoScheduler.java: addApplication(288)) - Application Submission: application_1323416004722_0002 from minal.kothari, currently active: 1 2011-12-09 13:10:57,300 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:handle(464)) - Processing event for appattempt_1323416004722_0002_01 of type APP_ACCEPTED 2011-12-09 13:10:57,317 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:handle(476)) - appattempt_1323416004722_0002_01 State change from SUBMITTED to SCHEDULED 2011-12-09 13:10:57,318 INFO rmapp.RMAppImpl (RMAppImpl.java:handle(416)) - Processing event for application_1323416004722_0002 of type APP_ACCEPTED 2011-12-09 13:10:57,318 INFO rmapp.RMAppImpl (RMAppImpl.java:handle(428)) - application_1323416004722_0002 State change from SUBMITTED to ACCEPTED 2011-12-09 13:10:57,320 INFO resourcemanager.RMAuditLogger (RMAuditLogger.java:logSuccess(140)) - USER=minal.kothari IP=192.168.0.93 OPERATION=Submit Application Request TARGET=ClientRMService RESULT=SUCCESS APPID=application_1323416004722_0002 Please let me know if you need some other logs . Thanks, Nitin On 9 December 2011 12:44, Arun C Murthy a...@hortonworks.com wrote: Moving to mapreduce-user@, bcc common-user@. Can you see any errors in the logs? Typically this happens when you have no NodeManagers. Check the 'nodes' link and then RM logs. Arun On Nov 29, 2011, at 8:36 PM, Nitin Khandelwal wrote: HI , I have successfully setup Hadoop 0.23.0 in a single m/c. When i post a job, it gets posted successfully (i can see the job in UI), but the job is never ASSIGNED and waits forever. Here are details of what i see for that Job in UI Name: random-writer State: ACCEPTED FinalStatus: UNDEFINED Started: 30-Nov-2011 10:08:55 Elapsed: 49sec Tracking URL: UNASSIGNED http://192.168.0.93:8900/cluster/app/application_1322627869620_0001# Diagnostics: AM container logs: AM not yet registered with RM Cluster ID: 1322627869620 ResourceManager state: STARTED ResourceManager started on: 30-Nov-2011 10:07:49 ResourceManager version: 0.23.0 from 722cd694fc4ab6d040c0a34f9fb5b476e330ee60 by hortonmu source checksum 4975bf112aa7faa5673f604045ced798 on Thu Nov 3 09:07:31 UTC 2011 Hadoop version: 0.23.0 from d4fee83ec1462ab9824add6449320617caa7c605 by hortonmu source checksum 4e42b2d96c899a98a8ab8c7cc23f27ae on Thu Nov 3 08:59:12 UTC 2011 Can some one tell where am i going wrong?? Thanks, -- Nitin Khandelwal -- Nitin Khandelwal -- Nitin Khandelwal
Re: Re: [help]how to stop HDFS
Hi, Even i am facing the same problem. There may be some issue with script . The doc says to start namenode type : bin/hdfs namenode start But start is not recognized. There is a hack to start namenode with command bin/hdfs namenode , but no idea how to stop. If it had been a issue with config , the later also should not have worked. Thanks, Nitin 2011/11/30 cat fa boost.subscrib...@gmail.com In fact it's me to say sorry. I used the word install which was misleading. In fact I downloaded a tar file and extracted it to ／usr／bin／hadoop Could you please tell me where to point those variables？ 2011/11/30, Prashant Sharma prashant.ii...@gmail.com: I am sorry, I had no idea you have done a rpm install, my suggestion was based on the assumption that you have done a tar extract install where all three distribution have to extracted and then export variables. Also I have no experience with rpm based installs - so no comments about what went wrong in your case. Basically from the error i can say that it is not able to find the jars needed on classpath which is referred by scripts through HADOOP_COMMON_HOME. I would say check with the access permission as in which user was it installed with and which user is it running with ? On Tue, Nov 29, 2011 at 10:48 PM, cat fa boost.subscrib...@gmail.com wrote: Thank you for your help, but I'm still a little confused. Suppose I installed hadoop in /usr/bin/hadoop/ .Should I point HADOOP_COMMON_HOME to /usr/bin/hadoop ? Where should I point HADOOP_HDFS_HOME? Also to /usr/bin/hadoop/ ? 2011/11/30 Prashant Sharma prashant.ii...@gmail.com I mean, you have to export the variables export HADOOP_CONF_DIR=/path/to/your/configdirectory. also export HADOOP_HDFS_HOME ,HADOOP_COMMON_HOME. before your run your command. I suppose this should fix the problem. -P On Tue, Nov 29, 2011 at 6:23 PM, cat fa boost.subscrib...@gmail.com wrote: it didn't work. It gave me the Usage information. 2011/11/29 hailong.yang1115 hailong.yang1...@gmail.com Try $HADOOP_PREFIX_HOME/bin/hdfs namenode stop --config $HADOOP_CONF_DIR and $HADOOP_PREFIX_HOME/bin/hdfs datanode stop --config $HADOOP_CONF_DIR. It would stop namenode and datanode separately. The HADOOP_CONF_DIR is the directory where you store your configuration files. Hailong *** * Hailong Yang, PhD. Candidate * Sino-German Joint Software Institute, * School of Computer ScienceEngineering, Beihang University * Phone: (86-010)82315908 * Email: hailong.yang1...@gmail.com * Address: G413, New Main Building in Beihang University, * No.37 XueYuan Road,HaiDian District, * Beijing,P.R.China,100191 *** From: cat fa Date: 2011-11-29 20:22 To: common-user Subject: Re: [help]how to stop HDFS use $HADOOP_CONF or $HADOOP_CONF_DIR ? I'm using hadoop 0.23. you mean which class? the class of hadoop or of java? 2011/11/29 Prashant Sharma prashant.ii...@gmail.com Try making $HADOOP_CONF point to right classpath including your configuration folder. On Tue, Nov 29, 2011 at 3:58 PM, cat fa boost.subscrib...@gmail.com wrote: I used the command : $HADOOP_PREFIX_HOME/bin/hdfs start namenode --config $HADOOP_CONF_DIR to sart HDFS. This command is in Hadoop document (here http://hadoop.apache.org/common/docs/r0.23.0/hadoop-yarn/hadoop-yarn-site/ClusterSetup.html ) However, I got errors as Exception in thread main java.lang.NoClassDefFoundError:start Could anyone tell me how to start and stop HDFS? By the way, how to set Gmail so that it doesn't top post my reply? -- Nitin Khandelwal
Re: Re: [help]how to stop HDFS
I am using Hadoop 0.23.0
There is no hadoop-daemon.sh in bin directory..
Thanks,
Nitin
On 30 November 2011 09:49, Harsh J ha...@cloudera.com wrote:
I simply use the /sbin/hadoop-daemon.sh {start|stop} {service} script
to control daemons at my end.
Does this not work for you? Or perhaps this thread is more about
documenting that?
2011/11/30 Nitin Khandelwal nitin.khandel...@germinait.com:
Hi,
Even i am facing the same problem. There may be some issue with script .
The doc says to start namenode type :
bin/hdfs namenode start
But start is not recognized. There is a hack to start namenode with
command bin/hdfs namenode , but no idea how to stop.
If it had been a issue with config , the later also should not have
worked.
Thanks,
Nitin
2011/11/30 cat fa boost.subscrib...@gmail.com
In fact it's me to say sorry. I used the word install which was
misleading.
In fact I downloaded a tar file and extracted it to ／usr／bin／hadoop
Could you please tell me where to point those variables？
2011/11/30, Prashant Sharma prashant.ii...@gmail.com:
I am sorry, I had no idea you have done a rpm install, my suggestion
was
based on the assumption that you have done a tar extract install where
all
three distribution have to extracted and then export variables.
Also I have no experience with rpm based installs - so no comments
about
what went wrong in your case.
Basically from the error i can say that it is not able to find the
jars
needed on classpath which is referred by scripts through
HADOOP_COMMON_HOME. I would say check with the access permission as in
which user was it installed with and which user is it running with ?
On Tue, Nov 29, 2011 at 10:48 PM, cat fa boost.subscrib...@gmail.com
wrote:
Thank you for your help, but I'm still a little confused.
Suppose I installed hadoop in /usr/bin/hadoop/ .Should I
point HADOOP_COMMON_HOME to /usr/bin/hadoop ? Where should I
point HADOOP_HDFS_HOME? Also to /usr/bin/hadoop/ ?
2011/11/30 Prashant Sharma prashant.ii...@gmail.com
I mean, you have to export the variables
export HADOOP_CONF_DIR=/path/to/your/configdirectory.
also export HADOOP_HDFS_HOME ,HADOOP_COMMON_HOME. before your run
your
command. I suppose this should fix the problem.
-P
On Tue, Nov 29, 2011 at 6:23 PM, cat fa
boost.subscrib...@gmail.com
wrote:
it didn't work. It gave me the Usage information.
2011/11/29 hailong.yang1115 hailong.yang1...@gmail.com
Try $HADOOP_PREFIX_HOME/bin/hdfs namenode stop --config
$HADOOP_CONF_DIR
and $HADOOP_PREFIX_HOME/bin/hdfs datanode stop --config
$HADOOP_CONF_DIR.
It would stop namenode and datanode separately.
The HADOOP_CONF_DIR is the directory where you store your
configuration
files.
Hailong
***
* Hailong Yang, PhD. Candidate
* Sino-German Joint Software Institute,
* School of Computer ScienceEngineering, Beihang University
* Phone: (86-010)82315908
* Email: hailong.yang1...@gmail.com
* Address: G413, New Main Building in Beihang University,
* No.37 XueYuan Road,HaiDian District,
* Beijing,P.R.China,100191
***
From: cat fa
Date: 2011-11-29 20:22
To: common-user
Subject: Re: [help]how to stop HDFS
use $HADOOP_CONF or $HADOOP_CONF_DIR ? I'm using hadoop 0.23.
you mean which class? the class of hadoop or of java?
2011/11/29 Prashant Sharma prashant.ii...@gmail.com
Try making $HADOOP_CONF point to right classpath including
your
configuration folder.
On Tue, Nov 29, 2011 at 3:58 PM, cat fa
boost.subscrib...@gmail.com
wrote:
I used the command :
$HADOOP_PREFIX_HOME/bin/hdfs start namenode --config
$HADOOP_CONF_DIR
to sart HDFS.
This command is in Hadoop document (here
http://hadoop.apache.org/common/docs/r0.23.0/hadoop-yarn/hadoop-yarn-site/ClusterSetup.html
)
However, I got errors as
Exception in thread main
java.lang.NoClassDefFoundError:start
Could anyone tell me how to start and stop HDFS?
By the way, how to set Gmail so that it doesn't top post my
reply?
--
Nitin Khandelwal
--
Harsh J
--
Nitin Khandelwal
Re: Re: [help]how to stop HDFS
Thanks,
I missed the sbin directory, was using the normal bin directory.
Thanks,
Nitin
On 30 November 2011 09:54, Harsh J ha...@cloudera.com wrote:
Like I wrote earlier, its in the $HADOOP_HOME/sbin directory. Not the
regular bin/ directory.
On Wed, Nov 30, 2011 at 9:52 AM, Nitin Khandelwal
nitin.khandel...@germinait.com wrote:
I am using Hadoop 0.23.0
There is no hadoop-daemon.sh in bin directory..
Thanks,
Nitin
On 30 November 2011 09:49, Harsh J ha...@cloudera.com wrote:
I simply use the /sbin/hadoop-daemon.sh {start|stop} {service} script
to control daemons at my end.
Does this not work for you? Or perhaps this thread is more about
documenting that?
2011/11/30 Nitin Khandelwal nitin.khandel...@germinait.com:
Hi,
Even i am facing the same problem. There may be some issue with
script .
The doc says to start namenode type :
bin/hdfs namenode start
But start is not recognized. There is a hack to start namenode with
command bin/hdfs namenode , but no idea how to stop.
If it had been a issue with config , the later also should not have
worked.
Thanks,
Nitin
2011/11/30 cat fa boost.subscrib...@gmail.com
In fact it's me to say sorry. I used the word install which was
misleading.
In fact I downloaded a tar file and extracted it to ／usr／bin／hadoop
Could you please tell me where to point those variables？
2011/11/30, Prashant Sharma prashant.ii...@gmail.com:
I am sorry, I had no idea you have done a rpm install, my
suggestion
was
based on the assumption that you have done a tar extract install
where
all
three distribution have to extracted and then export variables.
Also I have no experience with rpm based installs - so no comments
about
what went wrong in your case.
Basically from the error i can say that it is not able to find the
jars
needed on classpath which is referred by scripts through
HADOOP_COMMON_HOME. I would say check with the access permission
as in
which user was it installed with and which user is it running with
?
On Tue, Nov 29, 2011 at 10:48 PM, cat fa
boost.subscrib...@gmail.com
wrote:
Thank you for your help, but I'm still a little confused.
Suppose I installed hadoop in /usr/bin/hadoop/ .Should I
point HADOOP_COMMON_HOME to /usr/bin/hadoop ? Where should I
point HADOOP_HDFS_HOME? Also to /usr/bin/hadoop/ ?
2011/11/30 Prashant Sharma prashant.ii...@gmail.com
I mean, you have to export the variables
export HADOOP_CONF_DIR=/path/to/your/configdirectory.
also export HADOOP_HDFS_HOME ,HADOOP_COMMON_HOME. before your
run
your
command. I suppose this should fix the problem.
-P
On Tue, Nov 29, 2011 at 6:23 PM, cat fa
boost.subscrib...@gmail.com
wrote:
it didn't work. It gave me the Usage information.
2011/11/29 hailong.yang1115 hailong.yang1...@gmail.com
Try $HADOOP_PREFIX_HOME/bin/hdfs namenode stop --config
$HADOOP_CONF_DIR
and $HADOOP_PREFIX_HOME/bin/hdfs datanode stop --config
$HADOOP_CONF_DIR.
It would stop namenode and datanode separately.
The HADOOP_CONF_DIR is the directory where you store your
configuration
files.
Hailong
***
* Hailong Yang, PhD. Candidate
* Sino-German Joint Software Institute,
* School of Computer ScienceEngineering, Beihang University
* Phone: (86-010)82315908
* Email: hailong.yang1...@gmail.com
* Address: G413, New Main Building in Beihang University,
* No.37 XueYuan Road,HaiDian District,
* Beijing,P.R.China,100191
***
From: cat fa
Date: 2011-11-29 20:22
To: common-user
Subject: Re: [help]how to stop HDFS
use $HADOOP_CONF or $HADOOP_CONF_DIR ? I'm using hadoop
0.23.
you mean which class? the class of hadoop or of java?
2011/11/29 Prashant Sharma prashant.ii...@gmail.com
Try making $HADOOP_CONF point to right classpath including
your
configuration folder.
On Tue, Nov 29, 2011 at 3:58 PM, cat fa
boost.subscrib...@gmail.com
wrote:
I used the command :
$HADOOP_PREFIX_HOME/bin/hdfs start namenode --config
$HADOOP_CONF_DIR
to sart HDFS.
This command is in Hadoop document (here
http://hadoop.apache.org/common/docs/r0.23.0/hadoop-yarn/hadoop-yarn-site/ClusterSetup.html
)
However, I got errors as
Exception in thread main
java.lang.NoClassDefFoundError:start
Could anyone tell me how to start
Not able to post a job in Hadoop 0.23.0
HI , I have successfully setup Hadoop 0.23.0 in a single m/c. When i post a job, it gets posted successfully (i can see the job in UI), but the job is never ASSIGNED and waits forever. Here are details of what i see for that Job in UI Name: random-writer State: ACCEPTED FinalStatus: UNDEFINED Started: 30-Nov-2011 10:08:55 Elapsed: 49sec Tracking URL: UNASSIGNEDhttp://192.168.0.93:8900/cluster/app/application_1322627869620_0001# Diagnostics: AM container logs: AM not yet registered with RM Cluster ID: 1322627869620 ResourceManager state: STARTED ResourceManager started on: 30-Nov-2011 10:07:49 ResourceManager version: 0.23.0 from 722cd694fc4ab6d040c0a34f9fb5b476e330ee60 by hortonmu source checksum 4975bf112aa7faa5673f604045ced798 on Thu Nov 3 09:07:31 UTC 2011 Hadoop version: 0.23.0 from d4fee83ec1462ab9824add6449320617caa7c605 by hortonmu source checksum 4e42b2d96c899a98a8ab8c7cc23f27ae on Thu Nov 3 08:59:12 UTC 2011 Can some one tell where am i going wrong?? Thanks, -- Nitin Khandelwal
Problem running Hadoop 0.23.0
Hi, I was trying to setup Hadoop 0.23.0 with help of http://hadoop.apache.org/common/docs/r0.23.0/hadoop-yarn/hadoop-yarn-site/SingleCluster.html. After starting resourcemanager and nodemanager, I get following error when i try to hit Hadoop UI *���)org**.apache.hadoop.**ipc.RPC$Version** Mismatch���Ser**ver IPC version 5 cannot communicate with client version 47 *. There is no significant error in Hadoop logs (it shows everything started successfully). Do you have any idea about this error? Thanks, -- Nitin Khandelwal
Re: Problem running Hadoop 0.23.0
Thanks, Tom It worked.. Thanks, Nitin On 28 November 2011 22:12, Tom White t...@cloudera.com wrote: Hi Nitin, It looks like you may be using the wrong port number - try 8088 for the resource manager UI. Cheers, Tom On Mon, Nov 28, 2011 at 4:02 AM, Nitin Khandelwal nitin.khandel...@germinait.com wrote: Hi, I was trying to setup Hadoop 0.23.0 with help of http://hadoop.apache.org/common/docs/r0.23.0/hadoop-yarn/hadoop-yarn-site/SingleCluster.html . After starting resourcemanager and nodemanager, I get following error when i try to hit Hadoop UI *���)org**.apache.hadoop.**ipc.RPC$Version** Mismatch���Ser**ver IPC version 5 cannot communicate with client version 47 *. There is no significant error in Hadoop logs (it shows everything started successfully). Do you have any idea about this error? Thanks, -- Nitin Khandelwal -- Nitin Khandelwal
Problem in Getting Job Status
Hi All, We are migrating from Hadoop 0.19.2 to Hadoop 0.20.203. The normal flow ( job posting / running ) works fine. In one of the flow, we are required to check Task Reports for a running Job to fetch its state. In the older code (0.19.2) we had following piece of code which does the required tasks runningJob is an instance of Job TaskReport[] reports = jobClient.getMapTaskReports( runningJob.getID()); Wanted to know how it will change in the newer version. I tried with TaskReport[] reports = jobClient.getMapTaskReports((JobID) runningJob.getJobID()); This did not work as there is no JobID is set in Job instance. Requesting your assistance in this. Let me know if any other details are required. Thanks. -- Nitin Khandelwal
Re: Error in 9000 and 9001 port in hadoop-0.20.2
Plz change ur* fs.default.name* to hdfs://192.168.1.101:9000 Thanks, Nitin On 28 July 2011 17:46, Doan Ninh uitnetw...@gmail.com wrote: In the first time, i use *hadoop-cluster-1* for 192.168.1.101. That is the hostname of the master node. But, the same error occurs How can i fix it? On Thu, Jul 28, 2011 at 7:07 PM, madhu phatak phatak@gmail.com wrote: I had issue using IP address in XML files . You can try to use host names in the place of IP address . On Thu, Jul 28, 2011 at 5:22 PM, Doan Ninh uitnetw...@gmail.com wrote: Hi, I run Hadoop in 4 Ubuntu 11.04 on VirtualBox. On the master node (192.168.1.101), I configure fs.default.name = hdfs:// 127.0.0.1:9000. Then i configure everything on 3 other node When i start the cluster by entering $HADOOP_HOME/bin/start-all.sh on the master node Everything is ok, but the slave can't connect to the master on 9000, 9001 port. I manually telnet to 192.168.1.101 in 9000, 9001. And the result is connection refused Then, i'm on the master node, telnet to localhost, 127.0.0.1:9000. The result is connected. But, on the master node, i telnet to 192.168.1.101:9000 = Connection Refused Can somebody help me? -- Nitin Khandelwal
Hadoop Question
Hi All, How can I determine if a file is being written to (by any thread) in HDFS. I have a continuous process on the master node, which is tracking a particular folder in HDFS for files to process. On the slave nodes, I am creating files in the same folder using the following code : At the slave node: import org.apache.commons.io.IOUtils; import org.apache.hadoop.fs.FileSystem; import java.io.OutputStream; OutputStream oStream = fileSystem.create(path); IOUtils.write(Some String, oStream); IOUtils.closeQuietly(oStream); At the master node, I am getting the earliest modified file in the folder. At times when I try reading the file, I get nothing in the file, mostly because the slave might be still finishing writing to the file. Is there any way, to somehow tell the master, that the slave is still writing to the file and to check the file sometime later for actual content. Thanks, -- Nitin Khandelwal
Job Priority Hadoop 0.20.203
Hi, I am using Hadoop 0.20.203 with the new API ( mapreduce package) . I want to use Jobpriority, but unfortunately there is no option to set that in Job ( the option is there in 0.21.0). Can somebody plz tell me is there is a walkaround to set job priority? Thanks, -- Nitin Khandelwal
Isssue with Job Scheduling
Hi, I want ot use Capacity Scheduler for my Hadoop Jobs. I have currently three Queues defined and are configured and working properly. I am using Hadoop 0.20.2 And in the new library, we are not supposed to use JobConf. So, I need to set Queue name as a property in Configuration ( mapred.job.queue.name ) . I want to use a single instance of Configuration from Spring ( since i have some other properties like plugin info. ebbedded in Configuration) . I tried DeepCloning ( throes an exception as it tries to make clone of Logger too) and Shalloe Cloning ( not working properly, may be beacuse properties in Configuration are loaded in object when called for) . Could you tell me what is the best approach to use Capacity Scheduler?? Thanks And Regards, -- Nitin Khandelwal
Re: Lost Task Tracker because of no heartbeat
Hi, Just do context.progress() after small interval of time inside Your Map/reduce. That will do. If you are using Older package then, you can use reporter.progress(). Thanks Regards, Nitin Khandelwal On 16 March 2011 21:30, Baran_Cakici barancak...@gmail.com wrote: Hi Everyone, I make a Project with Hadoop-MapRedeuce for my master-Thesis. I have a strange problem on my System. First of all, I use Hadoop-0.20.2 on Windows XP Pro with Eclipse Plug-In. When I start a job with big Input(4GB - it`s may be not to big, but algorithm require some time), then i lose my Task Tracker in several minutes or seconds. I mean, Seconds since heartbeat increase and then after 600 Seconds I lose TaskTracker. I read somewhere, that can be occured because of small number of open files (ulimit -n). I try to increase this value, but i can write as max value in Cygwin 3200.(ulimit -n 3200) and default value is 256. Actually I don`t know, is it helps or not. In my job and task tracker.log have I some Errors, I posted those to. Jobtracker.log -Call to localhost/127.0.0.1:9000 failed on local exception: java.io.IOException: An existing connection was forcibly closed by the remote host another : - 2011-03-15 12:13:30,718 INFO org.apache.hadoop.mapred.JobTracker: attempt_201103151143_0002_m_91_0 is 97125 ms debug. 2011-03-15 12:16:50,718 INFO org.apache.hadoop.mapred.JobTracker: attempt_201103151143_0002_m_91_0 is 297125 ms debug. 2011-03-15 12:20:10,718 INFO org.apache.hadoop.mapred.JobTracker: attempt_201103151143_0002_m_91_0 is 497125 ms debug. 2011-03-15 12:23:30,718 INFO org.apache.hadoop.mapred.JobTracker: attempt_201103151143_0002_m_91_0 is 697125 ms debug. Error launching task Lost tracker 'tracker_apple:localhost/127.0.0.1:2654' there are my logs(jobtracker.log, tasktracker.log ...) in attachment I need really Help, I don`t have so much time for my Thessis. Thanks a lot for your Helps, Baran http://old.nabble.com/file/p31164785/logs.rar logs.rar -- View this message in context: http://old.nabble.com/Lost-Task-Tracker-because-of-no-heartbeat-tp31164785p31164785.html Sent from the Hadoop core-user mailing list archive at Nabble.com. -- Nitin Khandelwal
Re: How to make zip files as Hadoop input
Hi, You can actually make your own input format and reader which will read one file from a directory and give it to a node. If You are using hadoop 0.19 then extending MultiFilesplit format can do this task for you . If you are using Hadoop 0.20 or greater then your your inputformat can extend fileInputformat and yor reader can extend recordreader. Thanks and Regards, Nitin On 2 March 2011 11:15, Mark Kerzner markkerz...@gmail.com wrote: Hi, I have a bunch of zip files that I want to serve as input to a MapReduce job. My initial design was to list them in a text file and then give this list file as input. The list file would be read, and each line would be handed off to a node to process, which would pick up the corresponding zip file and work on it. But I feel that a better design is possible, and that my way is redundant. Can I just give the input directory as input? How do I make sure each node gets a file to process? Thank you, Mark -- Nitin Khandelwal
Re: Map Task Fails.........
Hi, Thanks for taking time to read my mail. Actually i get error only in master which is both master and slave . If i try to run only on one machine ( master and slave itself) then also i get error On 16 February 2011 20:00, madhu phatak phatak@gmail.com wrote: tasktracker log * On Wed, Feb 16, 2011 at 8:00 PM, madhu phatak phatak@gmail.com wrote: See the tasklog of the slave to see why the task attempt is failing... On Wed, Feb 16, 2011 at 7:29 PM, Nitin Khandelwal nitin.khandel...@germinait.com wrote: Hi, I am using Hadoop 0.21.0. I am getting Exception as java.lang.Throwable: Child Error at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:249) Caused by: java.io.IOException: Task process exit with nonzero status of 1. at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:236) when i am trying to run Map Red. This error comes in one of the slaves ( which is also master in my case) . Cam any body tell why i may be getting this? Thanks, -- Nitin Khandelwal -- Nitin Khandelwal
Map Task Fails.........
Hi, I am using Hadoop 0.21.0. I am getting Exception as java.lang.Throwable: Child Error at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:249) Caused by: java.io.IOException: Task process exit with nonzero status of 1. at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:236) when i am trying to run Map Red. This error comes in one of the slaves ( which is also master in my case) . Cam any body tell why i may be getting this? Thanks, -- Nitin Khandelwal
Custom Input format...
Hi, I want to give a folder as input path to Map Red. Each Task should read one file out of that folder at once . i was using it before in 0.19 using multiFileSplit Format and my own Input format extending it. can u plz tell how to do the same in 0.21.0. Thanks, -- Nitin Khandelwal
