Hi Ravishankar, I don't see two very important processes in your jps output. Just like there's JobTracker and NameNode, you should also have "TaskTracker" and "DataNode". The JobTracker only schedules jobs. To actually run the map reduce tasks, it needs TaskTrackers. This is why you see the jobtracker accepting your jobs and then getting stuck: because it doesn't have TaskTrackers to run that job on.
If I were you, I'd first check why the Datanode is not coming up. All 4 daemons are necessary for running jobs. The logs for those two should be in the same directory in which you find the JT's logs. Hope this helps. Ravi. On Fri, May 18, 2012 at 5:17 AM, Ravishankar Nair < ravishankar.n...@gmail.com> wrote: > Additionally, attached is the output of the job that I run( I mean the > example program named grep) > > > On Fri, May 18, 2012 at 6:15 AM, Ravishankar Nair < > ravishankar.n...@gmail.com> wrote: > >> Hi Ravi, >> >> Yes , it Running. Here is the output:- >> rn13067@WSUSJXLHRN13067 /home/hadoop-1.0.3 >> $ jps >> 5068 NameNode >> 5836 Jps >> 3516 JobTracker >> >> >> Here are the logs from JOBTRACKER:- >> >> 2012-05-17 21:41:31,772 INFO org.apache.hadoop.mapred.TaskTracker: >> STARTUP_MSG: >> /************************************************************ >> STARTUP_MSG: Starting TaskTracker >> >> STARTUP_MSG: host = WSUSJXLHRN13067/192.168.0.16 >> STARTUP_MSG: args = [] >> STARTUP_MSG: version = 1.0.3 >> STARTUP_MSG: build = >> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r >> 1335192; compiled by 'hortonfo' on Tue May 8 20:31:25 UTC 2012 >> ************************************************************/ >> 2012-05-17 21:41:31,944 INFO >> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from >> hadoop-metrics2.properties >> 2012-05-17 21:41:31,990 INFO >> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source >> MetricsSystem,sub=Stats registered. >> 2012-05-17 21:41:31,990 INFO >> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot >> period at 10 second(s). >> 2012-05-17 21:41:31,990 INFO >> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: TaskTracker metrics >> system started >> 2012-05-17 21:41:32,256 INFO >> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi >> registered. >> 2012-05-17 21:41:32,256 WARN >> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already >> exists! >> 2012-05-17 21:41:32,365 INFO org.mortbay.log: Logging to >> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via >> org.mortbay.log.Slf4jLog >> 2012-05-17 21:41:32,412 INFO org.apache.hadoop.http.HttpServer: Added >> global filtersafety >> (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter) >> 2012-05-17 21:41:32,428 INFO org.apache.hadoop.mapred.TaskLogsTruncater: >> Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1 >> 2012-05-17 21:41:32,444 INFO org.apache.hadoop.mapred.TaskTracker: >> Starting tasktracker with owner as SYSTEM >> 2012-05-17 21:41:32,444 INFO org.apache.hadoop.mapred.TaskTracker: Good >> mapred local directories are: /tmp/hadoop-SYSTEM/mapred/local >> 2012-05-17 21:41:32,459 WARN org.apache.hadoop.util.NativeCodeLoader: >> Unable to load native-hadoop library for your platform... using >> builtin-java classes where applicable >> 2012-05-17 21:41:32,459 ERROR org.apache.hadoop.mapred.TaskTracker: Can >> not start task tracker because java.io.IOException: Failed to set >> permissions of path: \tmp\hadoop-SYSTEM\mapred\local\ttprivate to 0700 >> at org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:689) >> at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:662) >> at >> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509) >> at >> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344) >> at >> org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189) >> at >> org.apache.hadoop.mapred.TaskTracker.initialize(TaskTracker.java:728) >> at org.apache.hadoop.mapred.TaskTracker.<init>(TaskTracker.java:1459) >> at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3742) >> >> 2012-05-17 21:41:32,459 INFO org.apache.hadoop.mapred.TaskTracker: >> SHUTDOWN_MSG: >> /************************************************************ >> SHUTDOWN_MSG: Shutting down TaskTracker at WSUSJXLHRN13067/192.168.0.16 >> ************************************************************/ >> >> Any clue? Thanks >> Regards, >> ravi >> >> >> >> On Fri, May 18, 2012 at 12:01 AM, Ravi Prakash <ravihad...@gmail.com>wrote: >> >>> Ravishankar, >>> >>> If you run $ jps, do you see a TaskTracker process running? Can you >>> please >>> post the tasktracker logs as well? >>> >>> On Thu, May 17, 2012 at 8:49 PM, Ravishankar Nair < >>> ravishankar.n...@gmail.com> wrote: >>> >>> > Dear experts, >>> > >>> > Today is my tenth day working with Hadoop on installing on my windows >>> > machine. I am trying again and again because , some where someone has >>> > written that it works on Windows with CYGWIN.(And noone has written >>> that >>> > Hadoop wont work on Windows). I am attaching my config files. >>> > >>> > Kindly help me, if anything can make this work.A feeble and humble >>> request >>> > to all experts out there. >>> > >>> > Here is the error, if you search , you can see thousands have reported >>> > this and there is no solution I got yet, though I tried all ways >>> possible. >>> > I am using Windows XP SP3, Hadoop (tried with five versions so far >>> > including 1.0.3). I am running on a single node.(machine >>> WSUSJXLHRN13067 >>> > IP:- 192.168.0.16) >>> > When I start Hadoop, no issues in any of the versions >>> > >>> > rn13067@WSUSJXLHRN13067 /home/hadoop-1.0.3 >>> > $ bin/start-all.sh >>> > starting namenode, logging to >>> > >>> /home/hadoop-1.0.3/libexec/../logs/hadoop-SUNDOOP-namenode-WSUSJXLHRN13067.out >>> > localhost: starting datanode, logging to >>> > >>> /home/hadoop-1.0.3/libexec/../logs/hadoop-SUNDOOP-datanode-WSUSJXLHRN13067.out >>> > localhost: starting secondarynamenode, logging to >>> > >>> /home/hadoop-1.0.3/libexec/../logs/hadoop-SUNDOOP-secondarynamenode-WSUSJXLHRN13067.out >>> > starting jobtracker, logging to >>> > >>> /home/hadoop-1.0.3/libexec/../logs/hadoop-SUNDOOP-jobtracker-WSUSJXLHRN13067.out >>> > localhost: starting tasktracker, logging to >>> > >>> /home/hadoop-1.0.3/libexec/../logs/hadoop-SUNDOOP-tasktracker-WSUSJXLHRN13067.out >>> > >>> > >>> > >>> > When I run the example program, this is what is printed on my console:- >>> > $ bin/hadoop jar hadoop-examples-1.0.3.jar grep input output >>> 'dfs[a-z.]+' >>> > 12/05/17 21:44:46 WARN util.NativeCodeLoader: Unable to load >>> native-hadoop >>> > library for your platform... using builtin-java classes where >>> applicable >>> > 12/05/17 21:44:46 WARN snappy.LoadSnappy: Snappy native library not >>> loaded >>> > 12/05/17 21:44:46 INFO mapred.FileInputFormat: Total input paths to >>> > process : 16 >>> > 12/05/17 21:44:47 INFO mapred.JobClient: Running job: >>> job_201205172141_0001 >>> > 12/05/17 21:44:48 INFO mapred.JobClient: map 0% reduce 0% >>> > >>> > >>> > Now it is HUNG!!. IN most of the versions this is the behaviour. >>> > >>> > Here is the log from JOBTRACKER:- >>> > >>> > >>> > 2012-05-17 21:41:28,037 INFO org.apache.hadoop.mapred.JobTracker: >>> > STARTUP_MSG: >>> > /************************************************************ >>> > STARTUP_MSG: Starting JobTracker >>> > STARTUP_MSG: host = WSUSJXLHRN13067/192.168.0.16 >>> > STARTUP_MSG: args = [] >>> > STARTUP_MSG: version = 1.0.3 >>> > STARTUP_MSG: build = >>> > https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r >>> > 1335192; compiled by 'hortonfo' on Tue May 8 20:31:25 UTC 2012 >>> > ************************************************************/ >>> > 2012-05-17 21:41:28,147 INFO >>> > org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from >>> > hadoop-metrics2.properties >>> > 2012-05-17 21:41:28,147 INFO >>> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source >>> > MetricsSystem,sub=Stats registered. >>> > 2012-05-17 21:41:28,162 INFO >>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot >>> > period at 10 second(s). >>> > 2012-05-17 21:41:28,162 INFO >>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: JobTracker metrics >>> > system started >>> > 2012-05-17 21:41:28,209 INFO >>> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source >>> > QueueMetrics,q=default registered. >>> > 2012-05-17 21:41:28,428 INFO >>> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source >>> ugi >>> > registered. >>> > 2012-05-17 21:41:28,428 WARN >>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi >>> already >>> > exists! >>> > 2012-05-17 21:41:28,428 INFO >>> > >>> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: >>> > Updating the current master key for generating delegation tokens >>> > 2012-05-17 21:41:28,428 INFO >>> > >>> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: >>> > Starting expired delegation token remover thread, >>> > tokenRemoverScanInterval=60 min(s) >>> > 2012-05-17 21:41:28,428 INFO >>> > >>> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: >>> > Updating the current master key for generating delegation tokens >>> > 2012-05-17 21:41:28,428 INFO org.apache.hadoop.mapred.JobTracker: >>> > Scheduler configured with (memSizeForMapSlotOnJT, >>> memSizeForReduceSlotOnJT, >>> > limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1) >>> > 2012-05-17 21:41:28,428 INFO org.apache.hadoop.util.HostsFileReader: >>> > Refreshing hosts (include/exclude) list >>> > 2012-05-17 21:41:28,444 INFO org.apache.hadoop.mapred.JobTracker: >>> Starting >>> > jobtracker with owner as rn13067 >>> > 2012-05-17 21:41:28,475 INFO org.apache.hadoop.ipc.Server: Starting >>> > SocketReader >>> > 2012-05-17 21:41:28,475 INFO >>> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source >>> > RpcDetailedActivityForPort47111 registered. >>> > 2012-05-17 21:41:28,475 INFO >>> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source >>> > RpcActivityForPort47111 registered. >>> > 2012-05-17 21:41:28,522 INFO org.mortbay.log: Logging to >>> > org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via >>> > org.mortbay.log.Slf4jLog >>> > 2012-05-17 21:41:28,584 INFO org.apache.hadoop.http.HttpServer: Added >>> > global filtersafety >>> > (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter) >>> > 2012-05-17 21:41:28,615 WARN org.apache.hadoop.util.NativeCodeLoader: >>> > Unable to load native-hadoop library for your platform... using >>> > builtin-java classes where applicable >>> > 2012-05-17 21:41:28,615 INFO org.apache.hadoop.http.HttpServer: Port >>> > returned by webServer.getConnectors()[0].getLocalPort() before open() >>> is >>> > -1. Opening the listener on 50030 >>> > 2012-05-17 21:41:28,615 INFO org.apache.hadoop.http.HttpServer: >>> > listener.getLocalPort() returned 50030 >>> > webServer.getConnectors()[0].getLocalPort() returned 50030 >>> > 2012-05-17 21:41:28,615 INFO org.apache.hadoop.http.HttpServer: Jetty >>> > bound to port 50030 >>> > 2012-05-17 21:41:28,615 INFO org.mortbay.log: jetty-6.1.26 >>> > 2012-05-17 21:41:28,834 INFO org.mortbay.log: Started >>> > SelectChannelConnector@0.0.0.0:50030 >>> > 2012-05-17 21:41:28,834 INFO >>> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source >>> jvm >>> > registered. >>> > 2012-05-17 21:41:28,834 INFO >>> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source >>> > JobTrackerMetrics registered. >>> > 2012-05-17 21:41:28,850 INFO org.apache.hadoop.mapred.JobTracker: >>> > JobTracker up at: 47111 >>> > 2012-05-17 21:41:28,850 INFO org.apache.hadoop.mapred.JobTracker: >>> > JobTracker webserver: 50030 >>> > 2012-05-17 21:41:29,225 INFO org.apache.hadoop.mapred.JobTracker: >>> Cleaning >>> > up the system directory >>> > 2012-05-17 21:41:29,772 INFO org.apache.hadoop.mapred.JobHistory: >>> Creating >>> > DONE folder at file:/C:/cygwin/home/hadoop-1.0.3/logs/history/done >>> > 2012-05-17 21:41:29,787 INFO org.apache.hadoop.mapred.JobTracker: >>> History >>> > server being initialized in embedded mode >>> > 2012-05-17 21:41:29,787 INFO org.apache.hadoop.mapred.JobHistoryServer: >>> > Started job history server at: localhost:50030 >>> > 2012-05-17 21:41:29,787 INFO org.apache.hadoop.mapred.JobTracker: Job >>> > History Server web address: localhost:50030 >>> > 2012-05-17 21:41:29,787 INFO >>> > org.apache.hadoop.mapred.CompletedJobStatusStore: Completed job store >>> is >>> > inactive >>> > 2012-05-17 21:41:29,990 INFO org.apache.hadoop.mapred.JobTracker: >>> > Refreshing hosts information >>> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.util.HostsFileReader: >>> > Setting the includes file to >>> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.util.HostsFileReader: >>> > Setting the excludes file to >>> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.util.HostsFileReader: >>> > Refreshing hosts (include/exclude) list >>> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.mapred.JobTracker: >>> > Decommissioning 0 nodes >>> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server >>> > Responder: starting >>> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server >>> > listener on 47111: starting >>> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server >>> > handler 0 on 47111: starting >>> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server >>> > handler 1 on 47111: starting >>> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server >>> > handler 2 on 47111: starting >>> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server >>> > handler 3 on 47111: starting >>> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server >>> > handler 4 on 47111: starting >>> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server >>> > handler 5 on 47111: starting >>> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server >>> > handler 6 on 47111: starting >>> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server >>> > handler 7 on 47111: starting >>> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.mapred.JobTracker: >>> Starting >>> > RUNNING >>> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server >>> > handler 8 on 47111: starting >>> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server >>> > handler 9 on 47111: starting >>> > 2012-05-17 21:44:47,069 INFO org.apache.hadoop.mapred.JobInProgress: >>> > job_201205172141_0001: nMaps=16 nReduces=1 max=-1 >>> > 2012-05-17 21:44:47,069 INFO org.apache.hadoop.mapred.JobTracker: Job >>> > job_201205172141_0001 added successfully for user 'rn13067' to queue >>> > 'default' >>> > 2012-05-17 21:44:47,069 INFO org.apache.hadoop.mapred.JobTracker: >>> > Initializing job_201205172141_0001 >>> > 2012-05-17 21:44:47,069 INFO org.apache.hadoop.mapred.JobInProgress: >>> > Initializing job_201205172141_0001 >>> > 2012-05-17 21:44:47,069 INFO org.apache.hadoop.mapred.AuditLogger: >>> > USER=rn13067 IP=192.168.0.16 OPERATION=SUBMIT_JOB >>> > TARGET=job_201205172141_0001 RESULT=SUCCESS >>> > *2012-05-17 21:44:47,084 ERROR org.apache.hadoop.mapred.JobHistory: >>> > Failed creating job history log file for job job_201205172141_0001 >>> > java.io.IOException: Failed to set permissions of path: >>> > >>> C:\cygwin\home\hadoop-1.0.3\logs\history\job_201205172141_0001_1337305487022_rn13067_grep-search >>> > to 0744* >>> > at >>> org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:689) >>> > at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:678) >>> > at >>> > >>> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509) >>> > at >>> > >>> org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:286) >>> > at >>> > >>> org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:385) >>> > at >>> > >>> org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:364) >>> > at >>> > >>> org.apache.hadoop.mapred.JobHistory$JobInfo.logSubmitted(JobHistory.java:1696) >>> > at >>> org.apache.hadoop.mapred.JobInProgress$3.run(JobInProgress.java:681) >>> > at java.security.AccessController.doPrivileged(Native Method) >>> > at javax.security.auth.Subject.doAs(Subject.java:396) >>> > at >>> > >>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121) >>> > at >>> > >>> org.apache.hadoop.mapred.JobInProgress.initTasks(JobInProgress.java:678) >>> > at >>> org.apache.hadoop.mapred.JobTracker.initJob(JobTracker.java:4207) >>> > at >>> > >>> org.apache.hadoop.mapred.EagerTaskInitializationListener$InitJob.run(EagerTaskInitializationListener.java:79) >>> > at >>> > >>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>> > at >>> > >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>> > at java.lang.Thread.run(Thread.java:662) >>> > >>> > Kindly help me. CYGWIN is in the path. (As some one suggested in some >>> > thread). >>> > >>> > >>> > >>> > >>> > -- >>> > Warmest Regards, >>> > >>> > Ravi >>> > >>> >> >> >> >> -- >> Warmest Regards, >> >> Ravi Shankar >> > > > > -- > Warmest Regards, > > Ravi Shankar >
