in datanode logs ， I found a new error message.
Would like to help solve the problem
2009-07-19 18:40:43,464 ERROR
org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
192.168.1.97:50010, storageID=DS-148204255-192.168.1.97-50010-1247630449653,
infoPort=50075, ipcPort=50020):DataXceiver
java.io.IOException: Version Mismatch
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:81)
at java.lang.Thread.run(Thread.java:619)
2009-07-19 18:40:43,464 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
blk_9198148243909908835_1418899 src: /192.168.1.94:21486 dest: /
192.168.1.97:50010
2009-07-19 18:41:43,363 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
blk_4464496078754951058_1418901 src: /192.168.1.105:17434 dest: /
192.168.1.97:50010
2009-07-19 18:41:55,318 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
blk_-998769390764365324_1418903 src: /192.168.1.105:17436 dest: /
192.168.1.97:50010
2009-07-19 18:42:51,983 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
blk_7357805261059813870_1418903 src: /192.168.1.94:16148 dest: /
192.168.1.97:50010
2009-07-19 18:43:00,936 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
blk_6766332640664853809_1418903 src: /192.168.1.105:60392 dest: /
192.168.1.97:50010
2009-07-19 18:44:03,112 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
blk_8833189579369772797_1418903 src: /192.168.1.94:16152 dest: /
192.168.1.97:50010
2009-07-19 18:44:07,105 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
blk_-92272670489114760_1418903 src: /192.168.1.94:16153 dest: /
192.168.1.97:50010
2009-07-19 18:45:15,301 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
blk_-3189060218922169017_1418903 src: /192.168.1.94:16157 dest: /
192.168.1.97:50010
2009-07-19 18:45:19,299 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
blk_2093621896162975184_1418903 src: /192.168.1.105:60400 dest: /
192.168.1.97:50010
2009-07-19 18:46:27,602 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
blk_-2879021713701012781_1418905 src: /192.168.1.105:60404 dest: /
192.168.1.97:50010
2009-07-19 18:46:27,602 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
blk_356680792598474797_1418907 src: /192.168.1.105:60406 dest: /
192.168.1.97:50010
2009-07-19 18:47:21,366 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in receiveBlock
for block blk_-8134427742713358791_1418899 java.io.EOFException: while
trying to read 65557 bytes
2009-07-19 18:47:21,367 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for block
blk_-8134427742713358791_1418899 Interrupted.
2009-07-19 18:47:21,367 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for block
blk_-8134427742713358791_1418899 terminating
2009-07-19 18:47:21,367 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock
blk_-8134427742713358791_1418899 received exception java.io.EOFException:
while trying to read 65557 bytes
2009-07-19 18:47:21,367 ERROR
org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
192.168.1.97:50010, storageID=DS-148204255-192.168.1.97-50010-1247630449653,
infoPort=50075, ipcPort=50020):DataXceiver
java.io.EOFException: while trying to read 65557 bytes
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(BlockReceiver.java:264)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPacket(BlockReceiver.java:308)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:372)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:524)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:357)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:103)
at java.lang.Thread.run(Thread.java:619)
2009-07-19 18:47:21,368 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in receiveBlock
for block blk_-346503319966521195_1418897 java.io.EOFException: while trying
to read 65557 bytes
2009-07-19 18:47:21,371 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for block
blk_-346503319966521195_1418897 Interrupted.
2009-07-19 18:47:21,373 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for block
blk_-346503319966521195_1418897 terminating
2009/7/19 mingyang <[email protected]>
> But I added datanode still appear in this phenomenon, and I added a new
> datanode, the above data only 10g
> Strangely enough, wait about 10 minutes, hadoop will resume, I do not know
> what the problem is this, ask for help
>
>
> it‘s log
> 09/07/19 18:46:13 INFO hdfs.DFSClient: Waiting to find target node:
> 192.168.1.97:50010
> 09/07/19 18:46:13 INFO hdfs.DFSClient: Exception in createBlockOutputStream
> java.net.SocketTimeoutException: 66000 millis timeout wh
> ile waiting for channel to be ready for read. ch :
> java.nio.channels.SocketChannel[connected
> local=/192.168.1.94:16153remote=/192.1
> 68.1.97:50010]
> 09/07/19 18:46:08 INFO hdfs.DFSClient: Waiting to find target node:
> 192.168.1.97:50010
> 09/07/19 18:46:08 INFO hdfs.DFSClient: Exception in createBlockOutputStream
> java.net.SocketTimeoutException: 66000 millis timeout wh
> ile waiting for channel to be ready for read. ch :
> java.nio.channels.SocketChannel[connected
> local=/192.168.1.94:16152remote=/192.1
> 68.1.97:50010]
> 09/07/19 18:45:01 INFO hdfs.DFSClient: Waiting to find target node:
> 192.168.1.105:50010
> 09/07/19 18:45:01 INFO hdfs.DFSClient: Exception in createBlockOutputStream
> java.io.IOException: Bad connect ack with firstBadLink 1
> 92.168.1.97:50010
> 09/07/19 18:44:56 INFO hdfs.DFSClient: Waiting to find target node:
> 192.168.1.97:50010
> 09/07/19 18:44:56 INFO hdfs.DFSClient: Exception in createBlockOutputStream
> java.net.SocketTimeoutException: 66000 millis timeout wh
> ile waiting for channel to be ready for read. ch :
> java.nio.channels.SocketChannel[connected
> local=/192.168.1.94:16148remote=/192.1
> 68.1.97:50010]
>
>
>
>
>
> 2009/7/17 mingyang <[email protected]>
>
> I run this, the result is far less than eight minutes, is not where I set
>> up right? I just changed the fs.default.name, other settings are default
>> values
>> <name> fs.default.name </ name>
>> <value> hdfs: / / named: 9000 </ value>
>>
>>
>>
>> <name> mapred.job.tracker </ name>
>> <value> hdfs: / / named: 9001 </ value>
>>
>>
>>
>> <name> hadoop.tmp.dir </ name>
>> <value> / hadoop / tmp </ value>
>>
>> <name> dfs.name.dir </ name>
>> <value> / hadoop / name </ value>
>>
>>
>> <name> dfs.data.dir </ name>
>> <value> / b, / c, / d, / e, / f, / g, / h, / i, / j, / k, / l </ value>
>>
>>
>>
>>
>> <name> dfs.replication </ name>
>> <value> 1 </ value>
>>
>>
>> <name> dfs.block.size </ name>
>> <value> 2560000 </ value>
>>
>>
>>
>> time find / b / c / d / e / f / g / h / i / j / k / l-type f-print> / dev
>> / null
>>
>> real 0m4.608s
>> user 0m1.489s
>> sys 0m3.068s
>>
>>
>> 2009/7/17 jason hadoop <[email protected]>
>>
>>> From the shell,
>>> time find /b /c /d /e /f /g /h /i /j /k /l -type f -print > /dev/null
>>>
>>> Also, unless your distribution is magic, white space is not ignored in
>>> value
>>> statements. Hopefully your actual value is
>>> <value>/b,/c,/d,/e,/f,/g,/h,/i,/j,/k,/l</value>
>>>
>>> On Fri, Jul 17, 2009 at 6:29 AM, mingyang <[email protected]> wrote:
>>>
>>> > Thank you for your reply, I am novice, I do not quite understand how to
>>> > check dfs.data.dir more than eight minutes?
>>> > My settings are dfs.data.dir
>>> > <property>
>>> > <name> dfs.data.dir </ name>
>>> > <value> / b, / c, / d, / e, / f, / g, / h, / i, / j, / k, / l </
>>> value>
>>> > </ property>
>>> >
>>> > Each directory has a disk mount
>>> >
>>> > 2009/7/17 jason hadoop <[email protected]>
>>> >
>>> > > Most likely problem is that the block report is taking more than 10
>>> > > minutes.
>>> > > Due to the placement of the sync blocks in the core Datanode code,
>>> the
>>> > > block
>>> > > report locks out the heartbeat.
>>> > > This can cause the namenode to think the datanode has vanished.
>>> > >
>>> > > A simple way to check, is to run a find on the directory set
>>> specified
>>> > for
>>> > > dfs.data.dir. If this find takes more than 8 minutes or so, you are
>>> in
>>> > > trouble.
>>> > > The only solutions are to add more datanodes, reducing the block
>>> count,
>>> > or
>>> > > increase your system io speed so that the block report may complete
>>> in
>>> > > time.
>>> > >
>>> > > On Fri, Jul 17, 2009 at 6:12 AM, mingyang <[email protected]>
>>> wrote:
>>> > >
>>> > > > i using hadoop storage my media files，, but when the number of
>>> > documents
>>> > > > when more than one million,
>>> > > > Hadoop start about 10-20 minutes, my datanode automatically down,
>>> > > > namenode log shows that the loss of heart, but I see my normal
>>> > datanode,
>>> > > > port 50010 can be a normal telnet, use jps to see can see datanode
>>> > still
>>> > > > running, but at this time have been unable to put data to a hadoop,
>>> I
>>> > > guess
>>> > > > datanode services is dead, hadoop does not support more than one
>>> > million
>>> > > > documents? How do I adjust those parameters? I have already set up
>>> at
>>> > the
>>> > > > same time the number of open file 65535
>>> > > >
>>> > > >
>>> > > > namenode log
>>> > > >
>>> > > > 2009-07-17 18:14:29,330 INFO
>>> > > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit:
>>> > > >
>>> > > > ugi=root,root,bin,daemon,sys,adm,disk,wheelip=/192.168.1.96
>>> > > > cmd=setPermission
>>> > > >
>>> > > > src=/hadoop/tmp/mapred/system/jobtracker.info dst=null
>>> > > > perm=root:supergroup
>>> > > > :rw-------
>>> > > > 2009-07-17 18:14:29,336 INFO org.apache.hadoop.hdfs.StateChange:
>>> BLOCK*
>>> > > > NameSystem.allocateBlock:
>>> > > >
>>> > > > /hadoop/tmp/mapred/system/jobtrack
>>> > > > er.info. blk_-2148480138731090754_1403179
>>> > > > 2009-07-17 18:14:32,958 INFO org.apache.hadoop.hdfs.StateChange:
>>> BLOCK*
>>> > > > NameSystem.addStoredBlock: blockMap updated:
>>> > > >
>>> > > > 192.168.1.97:50
>>> > > > 010 is added to blk_-2148480138731090754_1403179 size 4
>>> > > > 2009-07-17 18:14:33,340 INFO org.apache.hadoop.hdfs.StateChange:
>>> DIR*
>>> > > > NameSystem.completeFile: file
>>> > > >
>>> > > > /hadoop/tmp/mapred/system/jobtra
>>> > > > cker.info is closed by DFSClient_1037557306
>>> > > > 2009-07-17 18:16:21,349 INFO
>>> > > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
>>> from
>>> > > > 192.168.1.96
>>> > > > 2009-07-17 18:16:21,349 INFO
>>> > > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of
>>> > > > transactions:
>>> > > > 7 Total time for
>>> > > >
>>> > > > transacti
>>> > > > ons(ms): 1Number of transactions batched in Syncs: 1 Number of
>>> syncs: 6
>>> > > > SyncTimes(ms): 9
>>> > > > 2009-07-17 18:17:12,171 INFO
>>> > > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll FSImage
>>> from
>>> > > > 192.168.1.96
>>> > > > 2009-07-17 18:17:12,171 INFO
>>> > > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of
>>> > > > transactions:
>>> > > > 0 Total time for
>>> > > >
>>> > > > transacti
>>> > > > ons(ms): 0Number of transactions batched in Syncs: 0 Number of
>>> syncs: 1
>>> > > > SyncTimes(ms): 0
>>> > > > 2009-07-17 18:51:00,566 INFO org.apache.hadoop.hdfs.StateChange:
>>> BLOCK*
>>> > > > NameSystem.heartbeatCheck: lost heartbeat from
>>> > > >
>>> > > > 192.168.1.97:
>>> > > > 50010
>>> > > > 2009-07-17 18:51:25,383 INFO org.apache.hadoop.net.NetworkTopology:
>>> > > > Removing
>>> > > > a node: /default-rack/192.168.1.97:50010
>>> > > > 2009-07-17 19:10:48,564 INFO
>>> > > > org.apache.hadoop.hdfs.server.namenode.LeaseManager: Lease [Lease.
>>> > > Holder:
>>> > > > DFSClient_-
>>> > > >
>>> > > > 1624377199, pend
>>> > > > ingcreates: 69] has expired hard limit
>>> > > > 2009-07-17 19:10:48,564 INFO
>>> > > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Recovering
>>> > > > lease=[Lease. Holder:
>>> > > >
>>> > > > DFSClient_-16243
>>> > > > 77199, pendingcreates: 69],
>>> > > >
>>> src=/unp/01/video/B3/94/{B394EDB2-0302-34B9-5357-4904FFFEFF36}_100.unp
>>> > > >
>>> > > > datanode log
>>> > > >
>>> > > > 2009-07-17 18:52:40,719 INFO
>>> > > > org.apache.hadoop.hdfs.server.datanode.DataBlockScanner:
>>> Verification
>>> > > > succeeded for blk_664105388033641
>>> > > > 1514_601647
>>> > > > 2009-07-17 18:52:12,421 INFO
>>> > > > org.apache.hadoop.hdfs.server.datanode.DataBlockScanner:
>>> Verification
>>> > > > succeeded for blk_-10747966535898
>>> > > > 19594_1392025
>>> > > > 2009-07-17 18:51:44,074 INFO
>>> > > > org.apache.hadoop.hdfs.server.datanode.DataBlockScanner:
>>> Verification
>>> > > > succeeded for blk_-63504301593802
>>> > > > 31402_155334
>>> > > > 2009-07-17 18:51:12,760 INFO
>>> > > > org.apache.hadoop.hdfs.server.datanode.DataBlockScanner:
>>> Verification
>>> > > > succeeded for blk_460729998775184
>>> > > > 5359_395290
>>> > > > 2009-07-17 18:50:39,977 INFO
>>> > > > org.apache.hadoop.hdfs.server.datanode.DataBlockScanner:
>>> Verification
>>> > > > succeeded for blk_802918354954113
>>> > > > 9011_474989
>>> > > > 2009-07-17 18:50:11,707 INFO
>>> > > > org.apache.hadoop.hdfs.server.datanode.DataBlockScanner:
>>> Verification
>>> > > > succeeded for blk_846865664811904
>>> > > > 9754_1065465
>>> > > > 2009-07-17 18:49:39,421 INFO
>>> > > > org.apache.hadoop.hdfs.server.datanode.DataBlockScanner:
>>> Verification
>>> > > > succeeded for blk_473953565994615
>>> > > > 8302_532204
>>> > > > 2009-07-17 18:49:11,213 INFO
>>> > > > org.apache.hadoop.hdfs.server.datanode.DataBlockScanner:
>>> Verification
>>> > > > succeeded for blk_-14950858387931
>>> > > > 09024_354553
>>> > > >
>>> > > > 09/07/17 18:02:09 WARN hdfs.DFSClient: DFSOutputStream
>>> > ResponseProcessor
>>> > > > exception for block
>>> > > > blk_-2536746364442878375_1403164java.net.SocketTimeoutException:
>>> 63000
>>> > > > millis timeout while waiting for channel to be ready for read. ch :
>>> > > > java.nio.channels.SocketChannel[connected local=/
>>> 192.168.1.94:54783
>>> > > remote=/
>>> > > > 192.168.1.97:50010]
>>> > > > 11096473 09/07/17 18:02:12 INFO hdfs.DFSClient: Exception in
>>> > > > createBlockOutputStream java.net.SocketTimeoutException: 63000
>>> millis
>>> > > > timeout while waiting for channel to be ready for read. ch :
>>> > > > java.nio.channels.SocketChannel[connected local=/
>>> 192.168.1.94:54790
>>> > > remote=/
>>> > > > 192.168.1.97:50010]
>>> > > > 11096475 09/07/17 18:02:12 INFO hdfs.DFSClient: Exception in
>>> > > > createBlockOutputStream java.net.SocketTimeoutException: 63000
>>> millis
>>> > > > timeout while waiting for channel to be ready for read. ch :
>>> > > > java.nio.channels.SocketChannel[connected local=/
>>> 192.168.1.94:54791
>>> > > remote=/
>>> > > > 192.168.1.97:50010]
>>> > > >
>>> > >
>>> > >
>>> > >
>>> > > --
>>> > > Pro Hadoop, a book to guide you from beginner to hadoop mastery,
>>> > > http://www.amazon.com/dp/1430219424?tag=jewlerymall
>>> > > www.prohadoopbook.com a community for Hadoop Professionals
>>> > >
>>> >
>>> >
>>> >
>>> > --
>>> > 致
>>> > 礼！
>>> >
>>> >
>>> > 王明阳
>>> >
>>>
>>>
>>>
>>> --
>>> Pro Hadoop, a book to guide you from beginner to hadoop mastery,
>>> http://www.amazon.com/dp/1430219424?tag=jewlerymall
>>> www.prohadoopbook.com a community for Hadoop Professionals
>>>
>>
>>
>>
>> --
>> 致
>> 礼！
>>
>>
>> 王明阳
>>
>
>
>
> --
> 致
> 礼！
>
>
> 王明阳
>
--
致
礼！
王明阳
