It doesnot seem to be helping.... my name node got down again and on trying
to restart it I got this error....
2009-02-24 06:06:59,063 ERROR
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem
initialization failed.
java.io.FileNotFoundException: /data/hadoop/dfs/name/in_use.lock
(Input/output error)
        at java.io.RandomAccessFile.open(Native Method)
        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
        at
org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.tryLock(Storage.java:524)
        at
org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.lock(Storage.java:505)
        at
org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:363)
        at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:273)
        at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:87)
        at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:311)
        at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:290)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:163)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:208)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:194)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:859)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:868)
2009-02-24 06:06:59,065 INFO org.apache.hadoop.ipc.Server: Stopping server
on 9000
2009-02-24 06:06:59,065 ERROR
org.apache.hadoop.hdfs.server.namenode.NameNode:
java.io.FileNotFoundException: /data/hadoop/dfs/name/in_use.lock (Inpu
/output error)
        at java.io.RandomAccessFile.open(Native Method)
        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
        at
org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.tryLock(Storage.java:524)
        at
org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.lock(Storage.java:505)
        at
org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:363)
        at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:273)
        at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:87)
        at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:311)
        at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:290)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:163)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:208)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:194)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:859)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:868)



I am lost please helpppppppppppp

Parag

On Tue, Feb 24, 2009 at 7:01 PM, Parag Dhanuka <[email protected]>wrote:

> I think I know what the problem is... It was related to maximum number of
> file descriptors I can had opened. The limit was 1024 and I was exceeding
> that limit. In order to fix it I increased the number to 65535 using help
> from here
> http://www.cs.uwaterloo.ca/~brecht/servers/openfiles.html
>
> The process takes 2 hrs to finish but I am hoping it does finish this time
> without causing any pains.
>
> Parag
>
>
> On Tue, Feb 24, 2009 at 5:29 PM, Parag Dhanuka <[email protected]>wrote:
>
>> I re ran after formatting and got this error.. Can it be because I have
>> too many files opened?
>> 2009-02-24 02:46:19,284 ERROR
>> org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
>> 192.168.1.201:50010, storageID=DS-1261214527-127.0
>> .1.1-50010-1235462901573, infoPort=50075, ipcPort=50020):DataXceiver
>> java.io.IOException: Input/output error
>>         at java.io.UnixFileSystem.createFileExclusively(Native Method)
>>         at java.io.File.createNewFile(File.java:883)
>>         at
>> org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.createTmpFile(FSDataset.java:398)
>>         at
>> org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.createTmpFile(FSDataset.java:371)
>>         at
>> org.apache.hadoop.hdfs.server.datanode.FSDataset.createTmpFile(FSDataset.java:1108)
>>         at
>> org.apache.hadoop.hdfs.server.datanode.FSDataset.writeToBlock(FSDataset.java:997)
>>         at
>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:98)
>>         at
>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:258)
>>         at
>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:102)
>>         at java.lang.Thread.run(Thread.java:619)
>>
>> On Tue, Feb 24, 2009 at 12:58 PM, Parag Dhanuka 
>> <[email protected]>wrote:
>>
>>> Hi Mithila,
>>> Thanks for a quick reply...
>>>
>>> I did check the name node and data node logs... They have died and I ll
>>> restart them but I just want to know what happened before doing anything.
>>>
>>> On the data node the interesting error which I saw is this:
>>>
>>>
>>> java.io.EOFException: while trying to read 65557 bytes
>>>         at
>>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(BlockReceiver.java:254)
>>>         at
>>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPacket(BlockReceiver.java:298)
>>>         at
>>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:362)
>>>         at
>>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:514)
>>>         at
>>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:356)
>>>         at
>>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:102)
>>>
>>> At the same time I saw the following error on name node
>>>  2009-02-23 10:02:57,181
>>> FATAL  org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Fatal Error :
>>> All storage directories are inaccessible.
>>>
>>> Can you please elaborate as to what this might be.
>>>
>>> Parag
>>>
>>> On Tue, Feb 24, 2009 at 12:49 PM, Mithila Nagendra <[email protected]>wrote:
>>>
>>>> Hey Parag
>>>> Check if the namenode and the datanode are up and running. Use the 'jps'
>>>> command to do so. If they are not running U ll have to do a stop-all and
>>>> reformat the namenode using hadoop namenode -format (make sure u have no
>>>> data on the HDFS). Then restart hadoop using start-all.sh.
>>>>
>>>> If U have the datanode and the namenode running, then check the log
>>>> files
>>>> for errors.
>>>>
>>>> Mithila
>>>>
>>>> On Tue, Feb 24, 2009 at 10:02 AM, Parag Dhanuka <
>>>> [email protected]>wrote:
>>>>
>>>> > I have setup hadoop in pseudo distributed mode with namenode,
>>>> datanode,
>>>> > jobtracker and tasktracker all on the same machine...
>>>> > I also have a code which I use to write my data into hadoop. The code
>>>> of my
>>>> > mine reads data from the local disk does some preprocessing and after
>>>> that
>>>> > uses (multiple) FSDataOutputStream to write data to hadoop. I have
>>>> multiple
>>>> > FSDataOutputStreams open at one time cause I want to write data into
>>>> > different files based on some logic I have.
>>>> >
>>>> > Now the problem... While the process was writing data to hadoop I got
>>>> this
>>>> > error Problem renewing lease for DFSClient_1637324984. On going to
>>>> name
>>>> > node
>>>> > logs I found this
>>>> > 2009-02-23 10:02:57,181 FATAL
>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Fatal Error : All
>>>> > storage directories are inaccessible.
>>>> >
>>>> > I have absolutely no idea as to what might have caused this. Can some
>>>> one
>>>> > please help.
>>>> >
>>>> > --
>>>> > Parag Dhanuka
>>>> >
>>>>
>>>
>>>
>>>
>>> --
>>> Parag Dhanuka
>>>
>>
>>
>>
>> --
>> Parag Dhanuka
>>
>
>
>
> --
> Parag Dhanuka
>



-- 
Parag Dhanuka

Reply via email to