Re: Modeling WordCount in a different way
Hey Did u find any class or way out for storing results of Job1 map/reduce in memory and using that as an input to job2 map/Reduce?I am facing a situation where I need to do similar thing.If anyone can help me out.. Pankil On Wed, Apr 8, 2009 at 12:51 AM, Sharad Agarwal shara...@yahoo-inc.comwrote: I have confusion how would I start the next job after finishing the one, could you just make it clear by some rough example. See JobControl class to chain the jobs. You can specify dependencies as well. You can checkout the TestJobControl class for example code. Also do I need to use SequenceFileInputFormat to maintain the results in the memory and then accessing it. Not really. You have to use the corresponding reader to read the data. For example if you have written it using TextOutputFormat(default), you can then read it using TextInputFormat. The reader can be created in the reducer initialization code. In the new api (org.apache.hadoop.mapreduce.Reducer) it can be done in setup method. Here you can load the word,count mappings in a HashMap. In case you don't want to load all data in memory, you can create the reader in setup method and keep on doing the next (LineRecordReader#nextKeyValue()) in the reduce function if the reduce key is greater than the current key from the reader. - Sharad
Re: Modeling WordCount in a different way
Hey, I am trying complex queries on hadoop and in which i require more than one job to run to get final result..results of job one captures few joins of the query and I want to pass those results as input to 2nd job and again do processing so that I can get final results.queries are such that I cant do all types of joins and filterin in job1 and so I require two jobs. right now I write results of job 1 to hdfs and read dem for job2..but thats take unecessary IO time.So was looking for something that I can store my results of job1 in memory and use them as input for job 2. do let me know if you need any more details. Pankil On Mon, Apr 13, 2009 at 9:51 PM, sharad agarwal shara...@yahoo-inc.comwrote: Pankil Doshi wrote: Hey Did u find any class or way out for storing results of Job1 map/reduce in memory and using that as an input to job2 map/Reduce?I am facing a situation where I need to do similar thing.If anyone can help me out.. Normally you would write the job output to a file and input that to the next job. Any reason why you want to store the map reduce output in memory ? If you can describe your problem, perhaps it could be solved in more mapreduce-ish way. - Sharad
Directory /tmp/hadoop-hadoop/dfs/name is in an inconsistent state: storage directory does not exist
Hello Everyone, At time I get following error,when i restart my cluster desktops.(Before that I shutdown mapred and dfs properly though). Temp folder contains of the directory its looking for.Still I get this error. Only solution I found to get rid with this error is I have to format my dfs entirely and then load the data again. and start whole process. But in that I loose my data on HDFS and I have to reload it. Does anyone has any clue abt it? Error from log fil e:- 2009-04-14 19:40:29,963 INFO org.apache.hadoop.dfs.NameNode: STARTUP_MSG: / STARTUP_MSG: Starting NameNode STARTUP_MSG: host = Semantic002/192.168.1.133 STARTUP_MSG: args = [] STARTUP_MSG: version = 0.18.3 STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.18 -r 736250; compiled by 'ndaley' on Thu Jan 22 23:12:08 UTC 2009 / 2009-04-14 19:40:30,958 INFO org.apache.hadoop.ipc.metrics.RpcMetrics: Initializing RPC Metrics with hostName=NameNode, port=9000 2009-04-14 19:40:30,996 INFO org.apache.hadoop.dfs.NameNode: Namenode up at: Semantic002/192.168.1.133:9000 2009-04-14 19:40:31,007 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=NameNode, sessionId=null 2009-04-14 19:40:31,014 INFO org.apache.hadoop.dfs.NameNodeMetrics: Initializing NameNodeMeterics using context object:org.apache.hadoop.metrics.spi.NullCont ext 2009-04-14 19:40:31,160 INFO org.apache.hadoop.fs.FSNamesystem: fsOwner=hadoop,hadoop,adm,dialout,fax,cdrom,floppy,tape,audio,dip,plugdev,scanner,fuse,admin 2009-04-14 19:40:31,161 INFO org.apache.hadoop.fs.FSNamesystem: supergroup=supergroup 2009-04-14 19:40:31,161 INFO org.apache.hadoop.fs.FSNamesystem: isPermissionEnabled=true 2009-04-14 19:40:31,183 INFO org.apache.hadoop.dfs.FSNamesystemMetrics: Initializing FSNamesystemMeterics using context object:org.apache.hadoop.metrics.spi. NullContext 2009-04-14 19:40:31,184 INFO org.apache.hadoop.fs.FSNamesystem: Registered FSNamesystemStatusMBean 2009-04-14 19:40:31,248 INFO org.apache.hadoop.dfs.Storage: Storage directory /tmp/hadoop-hadoop/dfs/name does not exist. 2009-04-14 19:40:31,251 ERROR org.apache.hadoop.fs.FSNamesystem: FSNamesystem initialization failed. org.apache.hadoop.dfs.InconsistentFSStateException: Directory /tmp/hadoop-hadoop/dfs/name is in an inconsistent state: storage directory does not exist or is not accessible. at org.apache.hadoop.dfs.FSImage.recoverTransitionRead(FSImage.java:211) at org.apache.hadoop.dfs.FSDirectory.loadFSImage(FSDirectory.java:80) at org.apache.hadoop.dfs.FSNamesystem.initialize(FSNamesystem.java:294) at org.apache.hadoop.dfs.FSNamesystem.init(FSNamesystem.java:273) at org.apache.hadoop.dfs.NameNode.initialize(NameNode.java:148) at org.apache.hadoop.dfs.NameNode.init(NameNode.java:193) at org.apache.hadoop.dfs.NameNode.init(NameNode.java:179) at org.apache.hadoop.dfs.NameNode.createNameNode(NameNode.java:830) at org.apache.hadoop.dfs.NameNode.main(NameNode.java:839) 2009-04-14 19:40:31,261 INFO org.apache.hadoop.ipc.Server: Stopping server on 9000 2009-04-14 19:40:31,262 ERROR org.apache.hadoop.dfs.NameNode: org.apache.hadoop.dfs.InconsistentFSStateException: Directory /tmp/hadoop-hadoop/dfs/name is in an inconsistent state: storage directory does not exist or is not accessible. at org.apache.hadoop.dfs.FSImage.recoverTransitionRead(FSImage.java:211) at org.apache.hadoop.dfs.FSDirectory.loadFSImage(FSDirectory.java:80) at org.apache.hadoop.dfs.FSNamesystem.initialize(FSNamesystem.java:294) at org.apache.hadoop.dfs.FSNamesystem.init(FSNamesystem.java:273) at org.apache.hadoop.dfs.NameNode.initialize(NameNode.java:148) at org.apache.hadoop.dfs.NameNode.init(NameNode.java:193) at org.apache.hadoop.dfs.NameNode.init(NameNode.java:179) at org.apache.hadoop.dfs.NameNode.createNameNode(NameNode.java:830) at org.apache.hadoop.dfs.NameNode.main(NameNode.java:839) 2009-04-14 19:40:31,267 INFO org.apache.hadoop.dfs.NameNode: SHUTDOWN_MSG: / : Thanks Pankil
Re: Modeling WordCount in a different way
On Wed, Apr 15, 2009 at 1:26 AM, Sharad Agarwal shara...@yahoo-inc.comwrote: I am trying complex queries on hadoop and in which i require more than one job to run to get final result..results of job one captures few joins of the query and I want to pass those results as input to 2nd job and again do processing so that I can get final results.queries are such that I cant do all types of joins and filterin in job1 and so I require two jobs. right now I write results of job 1 to hdfs and read dem for job2..but thats take unecessary IO time.So was looking for something that I can store my results of job1 in memory and use them as input for job 2. do let me know if you need any more details. How big is your input and output data ? And my total data is of 7.8 gb out of which for Job 1 i use around 3 gb.output of job1 is of about 1gb and I use this output as input to job 2. How many nodes you are using? Well Right now due to lack of Resources I have only 4 nodes each dual core processors with 1GB og ram and about 80gb hard disk in each.. What is your job runtime? My first jobs takes long time after reaching 90% of reduce phase as it does in-memory merge sort and so that is also an big issue.I will have to arrange for more memory for my clusters I suppose. I will have look at jvm reuse feature. thanks Pankil
Re: hadoop-a small doubt
Hey , You can do that.That system should have same usrname like those of cluster and ofcourse it should be able to ssh name node.Also it should have hadoop and its hadoop-site.xml should be similar .Then u can access namenode,hdfs etc. if you are willing to see the web interface that can be done easily using any system. deepya wrote: Hi, I am SreeDeepya doing MTech in IIIT.I am working on a project named cost effective and scalable storage server.I configured a small hadoop cluster with only two nodes one namenode and one datanode.I am new to hadoop. I have a small doubt. Can a system not in the hadoop cluster access the namenode or the datanodeIf yes,then can you please tell me the necessary configurations that has to be done. Thanks in advance. SreeDeepya -- View this message in context: http://www.nabble.com/hadoop-a-small-doubt-tp22764615p23061794.html Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
Re: Directory /tmp/hadoop-hadoop/dfs/name is in an inconsistent state: storage directory does not exist
Thanks Pankil On Wed, Apr 15, 2009 at 5:09 PM, Alex Loddengaard a...@cloudera.com wrote: Data stored to /tmp has no consistency / reliability guarantees. Your OS can delete that data at any time. Configure hadoop-site.xml to store data elsewhere. Grep for /tmp in hadoop-default.xml to see all the configuration options you'll have to change. Here's the list I came up with: hadoop.tmp.dir fs.checkpoint.dir dfs.name.dir dfs.data.dir mapred.local.dir mapred.system.dir mapred.temp.dir Again, you need to be storing your data somewhere other than /tmp. Alex On Tue, Apr 14, 2009 at 6:06 PM, Pankil Doshi forpan...@gmail.com wrote: Hello Everyone, At time I get following error,when i restart my cluster desktops.(Before that I shutdown mapred and dfs properly though). Temp folder contains of the directory its looking for.Still I get this error. Only solution I found to get rid with this error is I have to format my dfs entirely and then load the data again. and start whole process. But in that I loose my data on HDFS and I have to reload it. Does anyone has any clue abt it? Error from log fil e:- 2009-04-14 19:40:29,963 INFO org.apache.hadoop.dfs.NameNode: STARTUP_MSG: / STARTUP_MSG: Starting NameNode STARTUP_MSG: host = Semantic002/192.168.1.133 STARTUP_MSG: args = [] STARTUP_MSG: version = 0.18.3 STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.18 -r 736250; compiled by 'ndaley' on Thu Jan 22 23:12:08 UTC 2009 / 2009-04-14 19:40:30,958 INFO org.apache.hadoop.ipc.metrics.RpcMetrics: Initializing RPC Metrics with hostName=NameNode, port=9000 2009-04-14 19:40:30,996 INFO org.apache.hadoop.dfs.NameNode: Namenode up at: Semantic002/192.168.1.133:9000 2009-04-14 19:40:31,007 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=NameNode, sessionId=null 2009-04-14 19:40:31,014 INFO org.apache.hadoop.dfs.NameNodeMetrics: Initializing NameNodeMeterics using context object:org.apache.hadoop.metrics.spi.NullCont ext 2009-04-14 19:40:31,160 INFO org.apache.hadoop.fs.FSNamesystem: fsOwner=hadoop,hadoop,adm,dialout,fax,cdrom,floppy,tape,audio,dip,plugdev,scanner,fuse,admin 2009-04-14 19:40:31,161 INFO org.apache.hadoop.fs.FSNamesystem: supergroup=supergroup 2009-04-14 19:40:31,161 INFO org.apache.hadoop.fs.FSNamesystem: isPermissionEnabled=true 2009-04-14 19:40:31,183 INFO org.apache.hadoop.dfs.FSNamesystemMetrics: Initializing FSNamesystemMeterics using context object:org.apache.hadoop.metrics.spi. NullContext 2009-04-14 19:40:31,184 INFO org.apache.hadoop.fs.FSNamesystem: Registered FSNamesystemStatusMBean 2009-04-14 19:40:31,248 INFO org.apache.hadoop.dfs.Storage: Storage directory /tmp/hadoop-hadoop/dfs/name does not exist. 2009-04-14 19:40:31,251 ERROR org.apache.hadoop.fs.FSNamesystem: FSNamesystem initialization failed. org.apache.hadoop.dfs.InconsistentFSStateException: Directory /tmp/hadoop-hadoop/dfs/name is in an inconsistent state: storage directory does not exist or is not accessible. at org.apache.hadoop.dfs.FSImage.recoverTransitionRead(FSImage.java:211) at org.apache.hadoop.dfs.FSDirectory.loadFSImage(FSDirectory.java:80) at org.apache.hadoop.dfs.FSNamesystem.initialize(FSNamesystem.java:294) at org.apache.hadoop.dfs.FSNamesystem.init(FSNamesystem.java:273) at org.apache.hadoop.dfs.NameNode.initialize(NameNode.java:148) at org.apache.hadoop.dfs.NameNode.init(NameNode.java:193) at org.apache.hadoop.dfs.NameNode.init(NameNode.java:179) at org.apache.hadoop.dfs.NameNode.createNameNode(NameNode.java:830) at org.apache.hadoop.dfs.NameNode.main(NameNode.java:839) 2009-04-14 19:40:31,261 INFO org.apache.hadoop.ipc.Server: Stopping server on 9000 2009-04-14 19:40:31,262 ERROR org.apache.hadoop.dfs.NameNode: org.apache.hadoop.dfs.InconsistentFSStateException: Directory /tmp/hadoop-hadoop/dfs/name is in an inconsistent state: storage directory does not exist or is not accessible. at org.apache.hadoop.dfs.FSImage.recoverTransitionRead(FSImage.java:211) at org.apache.hadoop.dfs.FSDirectory.loadFSImage(FSDirectory.java:80) at org.apache.hadoop.dfs.FSNamesystem.initialize(FSNamesystem.java:294) at org.apache.hadoop.dfs.FSNamesystem.init(FSNamesystem.java:273) at org.apache.hadoop.dfs.NameNode.initialize(NameNode.java:148) at org.apache.hadoop.dfs.NameNode.init(NameNode.java:193) at org.apache.hadoop.dfs.NameNode.init(NameNode.java:179) at org.apache.hadoop.dfs.NameNode.createNameNode(NameNode.java:830) at org.apache.hadoop.dfs.NameNode.main(NameNode.java:839) 2009-04-14 19:40:31,267 INFO
Re: No space left on device Exception
Hey what's your input size? from the info you gave it seems you have used 4.2GB and so probably if thats your input size your intermediate results mostly is less then your input.but that too depends on your map function. Make sure about the size of intermediate results. Pankil On Thu, Apr 16, 2009 at 3:25 AM, Rakhi Khatwani rakhi.khatw...@gmail.comwrote: Thanks, I will check tht Regards, Raakhi On Thu, Apr 16, 2009 at 1:42 PM, Miles Osborne mi...@inf.ed.ac.uk wrote: it may be that intermediate results are filling your disks and when the jobs crash, this all gets deleted. so it would look like you have spare space when in reality you don't. i would check on the file system as your jobs run and see if indeed they are filling-up. Miles 2009/4/16 Rakhi Khatwani rakhi.khatw...@gmail.com: Hi, following is the output on the df command [r...@domu-12-31-39-00-e5-d2 conf]# df -h FilesystemSize Used Avail Use% Mounted on /dev/sda1 9.9G 4.2G 5.2G 45% / /dev/sdb 414G 924M 392G 1% /mnt from the o/p it seems that i have quite an amount of memory available. but i still get the exception :( Thanks Raakhi On Thu, Apr 16, 2009 at 1:18 PM, Desai, Milind B milind.de...@hp.com wrote: From the exception it appears that there is no space left on machine. You can check using 'df' Thanks Milind -Original Message- From: Rakhi Khatwani [mailto:rakhi.khatw...@gmail.com] Sent: Thursday, April 16, 2009 1:15 PM To: hbase-u...@hadoop.apache.org; core-user@hadoop.apache.org Subject: No space left on device Exception Hi, I am running a map-reduce program on 6-Node ec2 cluster. and after a couple of hours all my tasks gets hanged. so i started digging into the logs there were no logs for regionserver no logs for tasktracker. However for jobtracker i get the following: 2009-04-16 03:00:29,691 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 50002, call heartbeat(org.apache.hadoop.mapred.tasktrackersta...@2eed7d11, false, true, 10745) from 10.254.27.79:44222: error: java.io.IOException: org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device java.io.IOException: org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:199) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65) at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109) at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:47) at java.io.DataOutputStream.write(DataOutputStream.java:90) at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.writeChunk(ChecksumFileSystem.java:346) at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:150) at org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:100) at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:86) at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:47) at java.io.DataOutputStream.write(DataOutputStream.java:90) at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202) at sun.nio.cs.StreamEncoder.implClose(StreamEncoder.java:297) at sun.nio.cs.StreamEncoder.close(StreamEncoder.java:130) at java.io.OutputStreamWriter.close(OutputStreamWriter.java:216) at java.io.BufferedWriter.close(BufferedWriter.java:248) at java.io.PrintWriter.close(PrintWriter.java:295) at org.apache.hadoop.mapred.JobHistory$JobInfo.logFinished(JobHistory.java:1024) at org.apache.hadoop.mapred.JobInProgress.jobComplete(JobInProgress.java:1906) at org.apache.hadoop.mapred.JobInProgress.comp following are the disk information on dfs UI domU-12-31-39-00-0C-A1 http://domu-12-31-39-00-0c-a1.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070dir=%2F 0In Service413.380.8321.19391.360.2 94.672353 domU-12-31-39-00-16-F1 http://domu-12-31-39-00-16-f1.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070dir=%2F 1In Service413.380.4621.24391.670.11 94.752399 domU-12-31-39-00-45-71 http://domu-12-31-39-00-45-71.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070dir=%2F 1In Service413.380.6421.34391.40.16 94.682303 domU-12-31-39-00-E5-D2 http://domu-12-31-39-00-e5-d2.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070dir=%2F 0In Service413.380.6621.53391.180.16 94.632319 domU-12-31-39-01-64-12
Re: HDFS files naming convention
hey you can surely do that using MulipleFileOutputFormat. We have already implemented that Pankil On Fri, Apr 24, 2009 at 8:58 PM, Aaron Kimball aa...@cloudera.com wrote: Alternatively, just use FileSystem.rename() on the normal output files after reducing is complete? On Sat, Apr 25, 2009 at 10:57 AM, Aaron Kimball aa...@cloudera.com wrote: I think you can do this with MultiFileOutputFormat; specifically override the getLeafFileName() method. - Aaron On Sat, Apr 25, 2009 at 4:29 AM, Parul Kudtarkar parul_kudtar...@hms.harvard.edu wrote: The HDFS files generated after mapreduce run are strored in HDFS as part-0 and so on.part-n Is it possible to name these output files stored in HDFS as per my own convention i.e. I would like to name these files my_file_1 and so onmy_files_n Please advice how this can be achieved? Thanks, Parul V. Kudtarkar -- View this message in context: http://www.nabble.com/HDFS-files-naming-convention-tp23223348p23223348.html Sent from the Hadoop core-user mailing list archive at Nabble.com.
Datanodes fail to start
Hello Everyone, Actually I had a cluster which was up. But i stopped the cluster as i wanted to format it.But cant start it back. 1)when i give start-dfs.sh I get following on screen starting namenode, logging to /Hadoop/hadoop-0.18.3/bin/../logs/hadoop-hadoop-namenode-hadoopmaster.out slave1.local: starting datanode, logging to /Hadoop/hadoop-0.18.3/bin/../logs/hadoop-hadoop-datanode-Slave1.out slave3.local: starting datanode, logging to /Hadoop/hadoop-0.18.3/bin/../logs/hadoop-hadoop-datanode-Slave3.out slave4.local: starting datanode, logging to /Hadoop/hadoop-0.18.3/bin/../logs/hadoop-hadoop-datanode-Slave4.out slave2.local: starting datanode, logging to /Hadoop/hadoop-0.18.3/bin/../logs/hadoop-hadoop-datanode-Slave2.out slave5.local: starting datanode, logging to /Hadoop/hadoop-0.18.3/bin/../logs/hadoop-hadoop-datanode-Slave5.out slave6.local: starting datanode, logging to /Hadoop/hadoop-0.18.3/bin/../logs/hadoop-hadoop-datanode-Slave6.out slave9.local: starting datanode, logging to /Hadoop/hadoop-0.18.3/bin/../logs/hadoop-hadoop-datanode-Slave9.out slave8.local: starting datanode, logging to /Hadoop/hadoop-0.18.3/bin/../logs/hadoop-hadoop-datanode-Slave8.out slave7.local: starting datanode, logging to /Hadoop/hadoop-0.18.3/bin/../logs/hadoop-hadoop-datanode-Slave7.out slave10.local: starting datanode, logging to /Hadoop/hadoop-0.18.3/bin/../logs/hadoop-hadoop-datanode-Slave10.out hadoopmaster.local: starting secondarynamenode, logging to /Hadoop/hadoop-0.18.3/bin/../logs/hadoop-hadoop-secondarynamenode-hadoopmaster.out 2) from log file named hadoop-hadoop-namenode-hadoopmaster.log I get following 2009-05-14 20:28:23,515 INFO org.apache.hadoop.dfs.NameNode: STARTUP_MSG: / STARTUP_MSG: Starting NameNode STARTUP_MSG: host = hadoopmaster/127.0.0.1 STARTUP_MSG: args = [] STARTUP_MSG: version = 0.18.3 STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.18 -r 736250; compiled by 'ndaley' on Thu Jan 22 23:12:08 UTC 2009 / 2009-05-14 20:28:23,717 INFO org.apache.hadoop.ipc.metrics.RpcMetrics: Initializing RPC Metrics with hostName=NameNode, port=9000 2009-05-14 20:28:23,728 INFO org.apache.hadoop.dfs.NameNode: Namenode up at: hadoopmaster.local/192.168.0.1:9000 2009-05-14 20:28:23,733 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=NameNode, sessionId=null 2009-05-14 20:28:23,743 INFO org.apache.hadoop.dfs.NameNodeMetrics: Initializing NameNodeMeterics using context object:org.apache.hadoop.metrics.spi.NullContext 2009-05-14 20:28:23,856 INFO org.apache.hadoop.fs.FSNamesystem: fsOwner=hadoop,hadoop,adm,dialout,fax,cdrom,floppy,tape,audio,dip,video,plugdev,fuse,lpadmin,admin,sambashare 2009-05-14 20:28:23,856 INFO org.apache.hadoop.fs.FSNamesystem: supergroup=supergroup 2009-05-14 20:28:23,856 INFO org.apache.hadoop.fs.FSNamesystem: isPermissionEnabled=true 2009-05-14 20:28:23,883 INFO org.apache.hadoop.dfs.FSNamesystemMetrics: Initializing FSNamesystemMeterics using context object:org.apache.hadoop.metrics.spi.NullContext 2009-05-14 20:28:23,885 INFO org.apache.hadoop.fs.FSNamesystem: Registered FSNamesystemStatusMBean 2009-05-14 20:28:23,964 INFO org.apache.hadoop.dfs.Storage: Number of files = 1 2009-05-14 20:28:23,971 INFO org.apache.hadoop.dfs.Storage: Number of files under construction = 0 2009-05-14 20:28:23,971 INFO org.apache.hadoop.dfs.Storage: Image file of size 80 loaded in 0 seconds. 2009-05-14 20:28:23,972 INFO org.apache.hadoop.dfs.Storage: Edits file edits of size 4 edits # 0 loaded in 0 seconds. 2009-05-14 20:28:23,974 INFO org.apache.hadoop.fs.FSNamesystem: Finished loading FSImage in 155 msecs 2009-05-14 20:28:23,976 INFO org.apache.hadoop.fs.FSNamesystem: Total number of blocks = 0 2009-05-14 20:28:23,988 INFO org.apache.hadoop.fs.FSNamesystem: Number of invalid blocks = 0 2009-05-14 20:28:23,988 INFO org.apache.hadoop.fs.FSNamesystem: Number of under-replicated blocks = 0 2009-05-14 20:28:23,988 INFO org.apache.hadoop.fs.FSNamesystem: Number of over-replicated blocks = 0 2009-05-14 20:28:23,988 INFO org.apache.hadoop.dfs.StateChange: STATE* Leaving safe mode after 0 secs. *2009-05-14 20:28:23,989 INFO org.apache.hadoop.dfs.StateChange: STATE* Network topology has 0 racks and 0 datanodes* 2009-05-14 20:28:23,989 INFO org.apache.hadoop.dfs.StateChange: STATE* UnderReplicatedBlocks has 0 blocks 2009-05-14 20:28:29,128 INFO org.mortbay.util.Credential: Checking Resource aliases 2009-05-14 20:28:29,243 INFO org.mortbay.http.HttpServer: Version Jetty/5.1.4 2009-05-14 20:28:29,244 INFO org.mortbay.util.Container: Started HttpContext[/static,/static] 2009-05-14 20:28:29,245 INFO org.mortbay.util.Container: Started HttpContext[/logs,/logs] 2009-05-14 20:28:29,750 INFO org.mortbay.util.Container: Started org.mortbay.jetty.servlet.webapplicationhand...@7fcebc9f 2009-05-14 20:28:29,838 INFO
Re: Datanodes fail to start
Can u guide me where can I find datanode log files? As I cannot find it in $hadoop/logs and so. I can only find following files in logs folder :- hadoop-hadoop-namenode-hadoopmaster.log hadoop-hadoop-namenode-hadoopmaster.out hadoop-hadoop-namenode-hadoopmaster.out.1 hadoop-hadoop-secondarynamenode-hadoopmaster.log hadoop-hadoop-secondarynamenode-hadoopmaster.out hadoop-hadoop-secondarynamenode-hadoopmaster.out.1 history Thanks Pankil On Thu, May 14, 2009 at 11:27 PM, jason hadoop jason.had...@gmail.comwrote: You have to examine the datanode log files the namenode does not start the datanodes, the start script does. The name node passively waits for the datanodes to connect to it. On Thu, May 14, 2009 at 6:43 PM, Pankil Doshi forpan...@gmail.com wrote: Hello Everyone, Actually I had a cluster which was up. But i stopped the cluster as i wanted to format it.But cant start it back. 1)when i give start-dfs.sh I get following on screen starting namenode, logging to /Hadoop/hadoop-0.18.3/bin/../logs/hadoop-hadoop-namenode-hadoopmaster.out slave1.local: starting datanode, logging to /Hadoop/hadoop-0.18.3/bin/../logs/hadoop-hadoop-datanode-Slave1.out slave3.local: starting datanode, logging to /Hadoop/hadoop-0.18.3/bin/../logs/hadoop-hadoop-datanode-Slave3.out slave4.local: starting datanode, logging to /Hadoop/hadoop-0.18.3/bin/../logs/hadoop-hadoop-datanode-Slave4.out slave2.local: starting datanode, logging to /Hadoop/hadoop-0.18.3/bin/../logs/hadoop-hadoop-datanode-Slave2.out slave5.local: starting datanode, logging to /Hadoop/hadoop-0.18.3/bin/../logs/hadoop-hadoop-datanode-Slave5.out slave6.local: starting datanode, logging to /Hadoop/hadoop-0.18.3/bin/../logs/hadoop-hadoop-datanode-Slave6.out slave9.local: starting datanode, logging to /Hadoop/hadoop-0.18.3/bin/../logs/hadoop-hadoop-datanode-Slave9.out slave8.local: starting datanode, logging to /Hadoop/hadoop-0.18.3/bin/../logs/hadoop-hadoop-datanode-Slave8.out slave7.local: starting datanode, logging to /Hadoop/hadoop-0.18.3/bin/../logs/hadoop-hadoop-datanode-Slave7.out slave10.local: starting datanode, logging to /Hadoop/hadoop-0.18.3/bin/../logs/hadoop-hadoop-datanode-Slave10.out hadoopmaster.local: starting secondarynamenode, logging to /Hadoop/hadoop-0.18.3/bin/../logs/hadoop-hadoop-secondarynamenode-hadoopmaster.out 2) from log file named hadoop-hadoop-namenode-hadoopmaster.log I get following 2009-05-14 20:28:23,515 INFO org.apache.hadoop.dfs.NameNode: STARTUP_MSG: / STARTUP_MSG: Starting NameNode STARTUP_MSG: host = hadoopmaster/127.0.0.1 STARTUP_MSG: args = [] STARTUP_MSG: version = 0.18.3 STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.18 -r 736250; compiled by 'ndaley' on Thu Jan 22 23:12:08 UTC 2009 / 2009-05-14 20:28:23,717 INFO org.apache.hadoop.ipc.metrics.RpcMetrics: Initializing RPC Metrics with hostName=NameNode, port=9000 2009-05-14 20:28:23,728 INFO org.apache.hadoop.dfs.NameNode: Namenode up at: hadoopmaster.local/192.168.0.1:9000 2009-05-14 20:28:23,733 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=NameNode, sessionId=null 2009-05-14 20:28:23,743 INFO org.apache.hadoop.dfs.NameNodeMetrics: Initializing NameNodeMeterics using context object:org.apache.hadoop.metrics.spi.NullContext 2009-05-14 20:28:23,856 INFO org.apache.hadoop.fs.FSNamesystem: fsOwner=hadoop,hadoop,adm,dialout,fax,cdrom,floppy,tape,audio,dip,video,plugdev,fuse,lpadmin,admin,sambashare 2009-05-14 20:28:23,856 INFO org.apache.hadoop.fs.FSNamesystem: supergroup=supergroup 2009-05-14 20:28:23,856 INFO org.apache.hadoop.fs.FSNamesystem: isPermissionEnabled=true 2009-05-14 20:28:23,883 INFO org.apache.hadoop.dfs.FSNamesystemMetrics: Initializing FSNamesystemMeterics using context object:org.apache.hadoop.metrics.spi.NullContext 2009-05-14 20:28:23,885 INFO org.apache.hadoop.fs.FSNamesystem: Registered FSNamesystemStatusMBean 2009-05-14 20:28:23,964 INFO org.apache.hadoop.dfs.Storage: Number of files = 1 2009-05-14 20:28:23,971 INFO org.apache.hadoop.dfs.Storage: Number of files under construction = 0 2009-05-14 20:28:23,971 INFO org.apache.hadoop.dfs.Storage: Image file of size 80 loaded in 0 seconds. 2009-05-14 20:28:23,972 INFO org.apache.hadoop.dfs.Storage: Edits file edits of size 4 edits # 0 loaded in 0 seconds. 2009-05-14 20:28:23,974 INFO org.apache.hadoop.fs.FSNamesystem: Finished loading FSImage in 155 msecs 2009-05-14 20:28:23,976 INFO org.apache.hadoop.fs.FSNamesystem: Total number of blocks = 0 2009-05-14 20:28:23,988 INFO org.apache.hadoop.fs.FSNamesystem: Number of invalid blocks = 0 2009-05-14 20:28:23,988 INFO
Re: Datanodes fail to start
This is log from datanode. 2009-05-14 00:36:14,559 INFO org.apache.hadoop.dfs.DataNode: BlockReport of 82 blocks got processed in 12 msecs 2009-05-14 01:36:15,768 INFO org.apache.hadoop.dfs.DataNode: BlockReport of 82 blocks got processed in 8 msecs 2009-05-14 02:36:13,975 INFO org.apache.hadoop.dfs.DataNode: BlockReport of 82 blocks got processed in 9 msecs 2009-05-14 03:36:15,189 INFO org.apache.hadoop.dfs.DataNode: BlockReport of 82 blocks got processed in 12 msecs 2009-05-14 04:36:13,384 INFO org.apache.hadoop.dfs.DataNode: BlockReport of 82 blocks got processed in 9 msecs 2009-05-14 05:36:14,592 INFO org.apache.hadoop.dfs.DataNode: BlockReport of 82 blocks got processed in 9 msecs 2009-05-14 06:36:15,806 INFO org.apache.hadoop.dfs.DataNode: BlockReport of 82 blocks got processed in 12 msecs 2009-05-14 07:36:14,008 INFO org.apache.hadoop.dfs.DataNode: BlockReport of 82 blocks got processed in 12 msecs 2009-05-14 08:36:15,204 INFO org.apache.hadoop.dfs.DataNode: BlockReport of 82 blocks got processed in 9 msecs 2009-05-14 09:36:13,430 INFO org.apache.hadoop.dfs.DataNode: BlockReport of 82 blocks got processed in 12 msecs 2009-05-14 10:36:14,642 INFO org.apache.hadoop.dfs.DataNode: BlockReport of 82 blocks got processed in 12 msecs 2009-05-14 11:36:15,850 INFO org.apache.hadoop.dfs.DataNode: BlockReport of 82 blocks got processed in 9 msecs 2009-05-14 12:36:14,193 INFO org.apache.hadoop.dfs.DataNode: BlockReport of 82 blocks got processed in 11 msecs 2009-05-14 13:36:15,454 INFO org.apache.hadoop.dfs.DataNode: BlockReport of 82 blocks got processed in 12 msecs 2009-05-14 14:36:13,662 INFO org.apache.hadoop.dfs.DataNode: BlockReport of 82 blocks got processed in 9 msecs 2009-05-14 15:36:14,930 INFO org.apache.hadoop.dfs.DataNode: BlockReport of 82 blocks got processed in 13 msecs 2009-05-14 16:36:16,151 INFO org.apache.hadoop.dfs.DataNode: BlockReport of 82 blocks got processed in 12 msecs 2009-05-14 17:36:14,407 INFO org.apache.hadoop.dfs.DataNode: BlockReport of 82 blocks got processed in 9 msecs 2009-05-14 18:36:15,659 INFO org.apache.hadoop.dfs.DataNode: BlockReport of 82 blocks got processed in 10 msecs 2009-05-14 19:27:02,188 WARN org.apache.hadoop.dfs.DataNode: java.io.IOException: Call to hadoopmaster.utdallas.edu/10.110.95.61:9000failed on local except$ at org.apache.hadoop.ipc.Client.wrapException(Client.java:751) at org.apache.hadoop.ipc.Client.call(Client.java:719) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216) at org.apache.hadoop.dfs.$Proxy4.sendHeartbeat(Unknown Source) at org.apache.hadoop.dfs.DataNode.offerService(DataNode.java:690) at org.apache.hadoop.dfs.DataNode.run(DataNode.java:2967) at java.lang.Thread.run(Thread.java:619) Caused by: java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:375) at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:500) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:442) 2009-05-14 19:27:06,198 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoopmaster.utdallas.edu/10.110.95.61:9000. Already tried 0 time(s). 2009-05-14 19:27:06,436 INFO org.apache.hadoop.dfs.DataNode: SHUTDOWN_MSG: / SHUTDOWN_MSG: Shutting down DataNode at Slave1/127.0.1.1 / 2009-05-14 19:27:21,737 INFO org.apache.hadoop.dfs.DataNode: STARTUP_MSG: / STARTUP_MSG: Starting DataNode STARTUP_MSG: host = Slave1/127.0.1.1 On Thu, May 14, 2009 at 11:43 PM, jason hadoop jason.had...@gmail.comwrote: The data node logs are on the datanode machines in the log directory. You may wish to buy my book and read chapter 4 on hdfs management. On Thu, May 14, 2009 at 9:39 PM, Pankil Doshi forpan...@gmail.com wrote: Can u guide me where can I find datanode log files? As I cannot find it in $hadoop/logs and so. I can only find following files in logs folder :- hadoop-hadoop-namenode-hadoopmaster.log hadoop-hadoop-namenode-hadoopmaster.out hadoop-hadoop-namenode-hadoopmaster.out.1 hadoop-hadoop-secondarynamenode-hadoopmaster.log hadoop-hadoop-secondarynamenode-hadoopmaster.out hadoop-hadoop-secondarynamenode-hadoopmaster.out.1 history Thanks Pankil On Thu, May 14, 2009 at 11:27 PM, jason hadoop jason.had...@gmail.com wrote: You have to examine the datanode log files the namenode does not start the datanodes, the start script does. The name node passively waits for the datanodes to connect to it. On Thu, May 14, 2009 at 6:43 PM, Pankil Doshi forpan...@gmail.com wrote: Hello Everyone, Actually I had a cluster which was up. But i stopped the cluster as i wanted to format it.But cant start it back. 1)when i give start-dfs.sh I get following on screen
Re: Datanodes fail to start
I got the solution..
Namespace IDs where some how incompatible.So I had to clean data dir and
temp dir ,format the cluster and make a fresh start
Pankil
On Fri, May 15, 2009 at 2:25 AM, jason hadoop jason.had...@gmail.comwrote:
There should be a few more lines at the end.
We only want the part from last the STARTUP_MSG to the end
On one of mine a successfull start looks like this:
STARTUP_MSG: Starting DataNode
STARTUP_MSG: host = at/192.168.1.119
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.19.1-dev
STARTUP_MSG: build = -r ; compiled by 'jason' on Tue Mar 17 04:03:57 PDT
2009
/
2009-03-17 03:08:11,884 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Registered
FSDatasetStatusMBean
2009-03-17 03:08:11,886 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Opened info server at
50010
2009-03-17 03:08:11,889 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Balancing bandwith is
1048576 bytes/s
2009-03-17 03:08:12,142 INFO org.mortbay.http.HttpServer: Version
Jetty/5.1.4
2009-03-17 03:08:12,155 INFO org.mortbay.util.Credential: Checking Resource
aliases
2009-03-17 03:08:12,518 INFO org.mortbay.util.Container: Started
org.mortbay.jetty.servlet.webapplicationhand...@1e184cb
2009-03-17 03:08:12,578 INFO org.mortbay.util.Container: Started
WebApplicationContext[/static,/static]
2009-03-17 03:08:12,721 INFO org.mortbay.util.Container: Started
org.mortbay.jetty.servlet.webapplicationhand...@1d9e282
2009-03-17 03:08:12,722 INFO org.mortbay.util.Container: Started
WebApplicationContext[/logs,/logs]
2009-03-17 03:08:12,878 INFO org.mortbay.util.Container: Started
org.mortbay.jetty.servlet.webapplicationhand...@14a75bb
2009-03-17 03:08:12,884 INFO org.mortbay.util.Container: Started
WebApplicationContext[/,/]
2009-03-17 03:08:12,951 INFO org.mortbay.http.SocketListener: Started
SocketListener on 0.0.0.0:50075
2009-03-17 03:08:12,951 INFO org.mortbay.util.Container: Started
org.mortbay.jetty.ser...@1358f03
2009-03-17 03:08:12,957 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=DataNode, sessionId=null
2009-03-17 03:08:13,242 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
Initializing RPC Metrics with hostName=DataNode, port=50020
2009-03-17 03:08:13,264 INFO org.apache.hadoop.ipc.Server: IPC Server
Responder: starting
2009-03-17 03:08:13,304 INFO org.apache.hadoop.ipc.Server: IPC Server
listener on 50020: starting
2009-03-17 03:08:13,343 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 0 on 50020: starting
2009-03-17 03:08:13,343 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: dnRegistration =
DatanodeRegistration(192.168.1.119:50010,
storageID=DS-540597485-192.168.1.119-50010-1237022386925, infoPort=50075,
ipcPort=50020)
2009-03-17 03:08:13,344 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 1 on 50020: starting
2009-03-17 03:08:13,344 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 2 on 50020: starting
2009-03-17 03:08:13,351 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
192.168.1.119:50010,
storageID=DS-540597485-192.168.1.119-50010-1237022386925, infoPort=50075,
ipcPort=50020)In DataNode.run, data =
FSDataset{dirpath='/tmp/hadoop-0.19.0-jason/dfs/data/current'}
2009-03-17 03:08:13,352 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: using BLOCKREPORT_INTERVAL
of 360msec Initial delay: 0msec
2009-03-17 03:08:13,391 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: BlockReport of 14 blocks
got processed in 27 msecs
2009-03-17 03:08:13,392 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Starting Periodic block
scanner.
On Thu, May 14, 2009 at 9:51 PM, Pankil Doshi forpan...@gmail.com wrote:
This is log from datanode.
2009-05-14 00:36:14,559 INFO org.apache.hadoop.dfs.DataNode: BlockReport
of
82 blocks got processed in 12 msecs
2009-05-14 01:36:15,768 INFO org.apache.hadoop.dfs.DataNode: BlockReport
of
82 blocks got processed in 8 msecs
2009-05-14 02:36:13,975 INFO org.apache.hadoop.dfs.DataNode: BlockReport
of
82 blocks got processed in 9 msecs
2009-05-14 03:36:15,189 INFO org.apache.hadoop.dfs.DataNode: BlockReport
of
82 blocks got processed in 12 msecs
2009-05-14 04:36:13,384 INFO org.apache.hadoop.dfs.DataNode: BlockReport
of
82 blocks got processed in 9 msecs
2009-05-14 05:36:14,592 INFO org.apache.hadoop.dfs.DataNode: BlockReport
of
82 blocks got processed in 9 msecs
2009-05-14 06:36:15,806 INFO org.apache.hadoop.dfs.DataNode: BlockReport
of
82 blocks got processed in 12 msecs
2009-05-14 07:36:14,008 INFO org.apache.hadoop.dfs.DataNode: BlockReport
of
82 blocks got processed in 12 msecs
2009-05-14 08:36:15,204 INFO org.apache.hadoop.dfs.DataNode: BlockReport
of
82 blocks got processed in 9 msecs
2009-05-14 09:36:13,430 INFO org.apache.hadoop.dfs.DataNode: BlockReport
of
82 blocks
ssh issues
Hello everyone, I got hint how to solve the problem where clusters have different usernames.but now other problem I face is that i can ssh a machine by using -i path/to key/ ..I cant ssh them directly but I will have to always pass the key. Now i face problem in ssh-ing my machines.Does anyone have any ideas how to deal with that?? Regards Pankil
Re: ssh issues
The problem is that it also prompts for the pass phrase. On Thu, May 21, 2009 at 2:14 PM, Brian Bockelman bbock...@cse.unl.eduwrote: Hey Pankil, Use ~/.ssh/config to set the default key location to the proper place for each host, if you're going down that route. I'd remind you that SSH is only used as a convenient method to launch daemons. If you have a preferred way to start things up on your cluster, you can use that (I think most large clusters don't use ssh... could be wrong). Brian On May 21, 2009, at 2:07 PM, Pankil Doshi wrote: Hello everyone, I got hint how to solve the problem where clusters have different usernames.but now other problem I face is that i can ssh a machine by using -i path/to key/ ..I cant ssh them directly but I will have to always pass the key. Now i face problem in ssh-ing my machines.Does anyone have any ideas how to deal with that?? Regards Pankil
Re: ssh issues
Well i made ssh with passphares. as the system in which i need to login requires ssh with pass phrases and those systems have to be part of my cluster. and so I need a way where I can specify -i path/to key/ and passphrase to hadoop in before hand. Pankil On Thu, May 21, 2009 at 9:35 PM, Aaron Kimball aa...@cloudera.com wrote: Pankil, That means that either you're using the wrong ssh key and it's falling back to password authentication, or else you created your ssh keys with passphrases attached; try making new ssh keys with ssh-keygen and distributing those to start again? - Aaron On Thu, May 21, 2009 at 3:49 PM, Pankil Doshi forpan...@gmail.com wrote: The problem is that it also prompts for the pass phrase. On Thu, May 21, 2009 at 2:14 PM, Brian Bockelman bbock...@cse.unl.edu wrote: Hey Pankil, Use ~/.ssh/config to set the default key location to the proper place for each host, if you're going down that route. I'd remind you that SSH is only used as a convenient method to launch daemons. If you have a preferred way to start things up on your cluster, you can use that (I think most large clusters don't use ssh... could be wrong). Brian On May 21, 2009, at 2:07 PM, Pankil Doshi wrote: Hello everyone, I got hint how to solve the problem where clusters have different usernames.but now other problem I face is that i can ssh a machine by using -i path/to key/ ..I cant ssh them directly but I will have to always pass the key. Now i face problem in ssh-ing my machines.Does anyone have any ideas how to deal with that?? Regards Pankil
Re: org.apache.hadoop.ipc.client : trying connect to server failed
make sure u can ping that data node and ssh it. On Thu, May 28, 2009 at 12:02 PM, ashish pareek pareek...@gmail.com wrote: HI , I am trying to step up a hadoop cluster on 512 MB machine and using hadoop 0.18 and have followed procedure given in apache hadoop site for hadoop cluster. I included in conf/slaves two datanode i.e including the namenode vitrual machine and other machine virtual machine . and have set up passwordless ssh between both virtual machines. But now problem is when is run command bin/hadoop start-all.sh It start only one datanode on the same namenode vitrual machine but it doesn't start the datanode on other machine. in logs/hadoop-datanode i get message INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop1/192.168.1.28:9000. Already tried 1 time(s). 2009-05-09 18:35:14,266 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop1/192.168.1.28:9000. Already tried 2 time(s). 2009-05-09 18:35:14,266 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop1/192.168.1.28:9000. Already tried 3 time(s). . . . . . . . . . . . . So can any one help in solving this problem. :) Thanks Regards Ashish Pareek
Re: Hadoop scheduling question
Hello Kristi, I am Research Assistant at University of Texas at Dallas. We are working of RDF data and we come across many joins in our queries. But We are not able to carry out all joins in a single job..we also tried our hadoop code using Pig scripts and found that for each join in PIG script new job is used.So basically what i think its a sequential process to handle typesof join where output of one job is required s an input to other one. do let me know what you think about my view point. Thanks Pankil On Thu, Jun 4, 2009 at 7:12 PM, Kristi Morton kmor...@cs.washington.eduwrote: Hi, I'm a Hadoop 17 user who is doing research with Prof. Magda Balazinska at the University of Washington on an improved progress indicator for Pig Latin. We have a question regarding how Hadoop schedules Pig Latin queries with JOIN operators. Does Hadoop schedule all MapReduce jobs in a script sequentially or does it ever schedule two MapReduce jobs in parallel. For example, if the output of two Map-Reduce jobs is later joined and each of these jobs only needs a subset of the cluster resources, would they be scheduled in parallel or in series? I apologize if I sent this to the wrong list, but please let me know which list is most appropriate for this type of question. Thanks, Kristi
Re: Making sure the tmp directory is cleaned?
Yes, If your job gets completed successfully .possibly it removes after completion of both map and reduce tasks. Pankil On Mon, Jun 22, 2009 at 3:15 PM, Qin Gao q...@cs.cmu.edu wrote: Hi All, Do you know if the tmp directory on every map/reduce task will be deleted automatically after the map task finishes or will do I have to delete them? I mean the tmp directory that automatically created by on current directory. Thanks a lot --Q
Re: Problem in viewing WEB UI
I am not sure but sometimes you might see that datanodes are working from cmd prompt.. But actually when you look at the logs you find sme kind of error in that..Check the logs of datanode.. Pankil On Wed, Jun 17, 2009 at 1:42 AM, ashish pareek pareek...@gmail.com wrote: Hi, When I run command *bin/hadoop dfsadmin -report *it shows that 2 datanodes are alive but when i try to http://hadoopmster:50070/ but the problem is that it opens doesnot opne http://hadoopmaster:50070/dfshealth.jsp page and throws *error HTTP: 404 . So why is't happening like this? * Regards, Ashish Pareek On Wed, Jun 17, 2009 at 10:06 AM, Sugandha Neaolekar sugandha@gmail.com wrote: Well, You just have to specify the address in the URL address bar as:: http://hadoopmaster:50070 U'll be able to see the web UI..! On Tue, Jun 16, 2009 at 7:17 PM, ashish pareek pareek...@gmail.com wrote: HI Sugandha, Hmmm your suggestion helped and Now I am able to run two datanode one on the same machine as name node and other on the different machine Thanks a lot :) But the problem is now I am not able to see web UI . for both datanode and as well as name node should I have to consider some more things in the site.xml ? if so please help... Thanking you again, regards, Ashish Pareek. On Tue, Jun 16, 2009 at 3:10 PM, Sugandha Naolekar sugandha@gmail.com wrote: hi,,! First of all, get your concepts clear of hadoop. You can refer to the following site:: http://www.google.co.in/url?sa=tsource=webct=rescd=1url=http%3A%2F%2Fwww.michael-noll.com%2Fwiki%2FRunning_Hadoop_On_Ubuntu_Linux_(Single-Node_Cluster)ei=lGU3Spv2FZbLjAe19KmiDQusg=AFQjCNFbmVGsoChOSMzCB3tRhoV0ylHOzAsig2=t2AJ_nf24SFtveN4PHS_TAhttp://www.google.co.in/url?sa=tsource=webct=rescd=1url=http%3A%2F%2Fwww.michael-noll.com%2Fwiki%2FRunning_Hadoop_On_Ubuntu_Linux_%28Single-Node_Cluster%29ei=lGU3Spv2FZbLjAe19KmiDQusg=AFQjCNFbmVGsoChOSMzCB3tRhoV0ylHOzAsig2=t2AJ_nf24SFtveN4PHS_TA http://www.google.co.in/url?sa=tsource=webct=rescd=1url=http%3A%2F%2Fwww.michael-noll.com%2Fwiki%2FRunning_Hadoop_On_Ubuntu_Linux_%28Single-Node_Cluster%29ei=lGU3Spv2FZbLjAe19KmiDQusg=AFQjCNFbmVGsoChOSMzCB3tRhoV0ylHOzAsig2=t2AJ_nf24SFtveN4PHS_TA I have small doubt whether in the mater.xml and slave.xml we can have same port numbers to both of them like for slave : property namefs.default.name/name valuehdfs://hadoopslave: 9000/value /property for master::: property namefs.default.name/name valuehdfs://hadoopmaster:9000/value /property Well, any two daemons or services can run on the same port unless, they are not run on the same machine.If you wish to run DN and NN on the same machine, their port numbers have to be different. On Tue, Jun 16, 2009 at 2:55 PM, ashish pareek pareek...@gmail.com wrote: HI sugandha, and one more thing can we have in slave::: property namedfs.datanode.address/ name valuehadoopmaster:9000/value valuehadoopslave:9001/value /property Also, fs,default.name is the tag which specifies the default filesystem. And generaLLY, it is run on namenode. So, it;s value has to be a namenode's address only and not slave's. Else if you have complete procedure for installing and running Hadoop in cluster can you please send it to me .. I need to step up hadoop with in two days and show it to my guide.Currently I am doing my masters. Thanks for your spending time Try for the above, and this should work! regards, Ashish Pareek On Tue, Jun 16, 2009 at 2:33 PM, Sugandha Naolekar sugandha@gmail.com wrote: Following changes are to be done:: Under master folder:: - put slaves address as well under the values of tag(dfs.datanode.address) - You want to make namenode as datanode as well. As per your config file, you have specified hadoopmaster in your slave file. If you don't want that, remove ti from slaves file. UNder slave folder:: - put only slave's (the m/c where you intend to run your datanode)'s address.under datanode.address tag. Else it should go as such:: property namedfs.datanode.address/name valuehadoopmaster:9000/value valuehadoopslave:9001/value /property Also, your port numbers hould be different. the daemons NN,DN,JT,TT should run independently on different ports. On Tue, Jun 16, 2009 at 2:05 PM, Sugandha Naolekar sugandha@gmail.com wrote: -- Forwarded message -- From: ashish pareek pareek...@gmail.com Date: Tue, Jun 16, 2009 at 2:00 PM Subject: Re: org.apache.hadoop.ipc.client : trying connect to server failed To: Sugandha Naolekar sugandha@gmail.com On Tue, Jun 16, 2009 at 1:58 PM, ashish pareek pareek...@gmail.com wrote:
Re: Disk Usage Overhead of Hadoop Upgrade
hi Stu, which block conversion are you talking about? If you are talking abt block size of data then it remains same in upgrade unless and until you change it. Pankil On Tue, Jun 16, 2009 at 5:16 PM, Stu Hood stuart.h...@rackspace.com wrote: Hey gang, We're preparing to upgrade our cluster from Hadoop 0.15.3 to 0.18.3. How much disk usage overhead can we expect from the block conversion before we finalize the upgrade? In the worst case, will the upgrade cause our disk usage to double? Thanks, Stu Hood Search Team Technical Lead Email Apps Division, Rackspace Hosting
Re: HDFS out of space
Hey Alex, Will Hadoop balancer utility work in this case? Pankil On Mon, Jun 22, 2009 at 4:30 PM, Alex Loddengaard a...@cloudera.com wrote: Are you seeing any exceptions because of the disk being at 99% capacity? Hadoop should do something sane here and write new data to the disk with more capacity. That said, it is ideal to be balanced. As far as I know, there is no way to balance an individual DataNode's hard drives (Hadoop does round-robin scheduling when writing data). Alex On Mon, Jun 22, 2009 at 10:12 AM, Kris Jirapinyo kjirapi...@biz360.com wrote: Hi all, How does one handle a mount running out of space for HDFS? We have two disks mounted on /mnt and /mnt2 respectively on one of the machines that are used for HDFS, and /mnt is at 99% while /mnt2 is at 30%. Is there a way to tell the machine to balance itself out? I know for the cluster, you can balance it using start-balancer.sh but I don't think that it will tell the individual machine to balance itself out. Our hack right now would be just to delete the data on /mnt, since we have replication of 3x, we should be OK. But I'd prefer not to do that. Any thoughts?
Re: Making sure the tmp directory is cleaned?
No..If your job gets killed or failed.Temp wont clean up.. and In that case you will have to carefully clean that on your own. If you dont clean it up yourself it will eat up your disk space. Pankil On Mon, Jun 22, 2009 at 4:24 PM, Qin Gao q...@cs.cmu.edu wrote: Thanks! But what if the jobs get killed or failed? Does hadoop try to clean it? we are considering bad situations - if job gets killed, will the tmp dirs sit on local disks forever and eats up all the diskspace? I guess this should be considered in distributed cache, but those files are read-only, and our program will generate new temporary files. --Q On Mon, Jun 22, 2009 at 4:19 PM, Pankil Doshi forpan...@gmail.com wrote: Yes, If your job gets completed successfully .possibly it removes after completion of both map and reduce tasks. Pankil On Mon, Jun 22, 2009 at 3:15 PM, Qin Gao q...@cs.cmu.edu wrote: Hi All, Do you know if the tmp directory on every map/reduce task will be deleted automatically after the map task finishes or will do I have to delete them? I mean the tmp directory that automatically created by on current directory. Thanks a lot --Q
Re: HDFS out of space
Matt. Kris can give that info.. I am one of the users from mailing list. PAnkil On Mon, Jun 22, 2009 at 4:37 PM, Matt Massie m...@cloudera.com wrote: Pankil- I'd be interested to know the size of the /mnt and /mnt2 partitions. Are they the same? Can you run the following and report the output... % df -h /mnt /mnt2 Thanks. -Matt On Jun 22, 2009, at 1:32 PM, Pankil Doshi wrote: Hey Alex, Will Hadoop balancer utility work in this case? Pankil On Mon, Jun 22, 2009 at 4:30 PM, Alex Loddengaard a...@cloudera.com wrote: Are you seeing any exceptions because of the disk being at 99% capacity? Hadoop should do something sane here and write new data to the disk with more capacity. That said, it is ideal to be balanced. As far as I know, there is no way to balance an individual DataNode's hard drives (Hadoop does round-robin scheduling when writing data). Alex On Mon, Jun 22, 2009 at 10:12 AM, Kris Jirapinyo kjirapi...@biz360.com wrote: Hi all, How does one handle a mount running out of space for HDFS? We have two disks mounted on /mnt and /mnt2 respectively on one of the machines that are used for HDFS, and /mnt is at 99% while /mnt2 is at 30%. Is there a way to tell the machine to balance itself out? I know for the cluster, you can balance it using start-balancer.sh but I don't think that it will tell the individual machine to balance itself out. Our hack right now would be just to delete the data on /mnt, since we have replication of 3x, we should be OK. But I'd prefer not to do that. Any thoughts?
