Hi, All I'm using nutch 1.0 on a 12 nodes cluster. When using crawler to index intranet, it crashed after 12 hours crawling. One of my slave crashed too. Following are logs of crashed node(tasktacker and datanode) and the job: Here is the question: 1. What's the reason it crashes? 2. How to resume the 12-hours crawler?
Thanks! Xiao log for tasktracker: /jobcache/job_200911151557_0057/attempt_200911151557_0057_r_000000_0/output/file.out in any of the configured local directories 2009-11-16 03:22:27,736 INFO TaskTracker - attempt_200911151557_0057_r_000000_0 0.90161085% reduce > reduce 2009-11-16 03:22:30,004 INFO TaskTracker - org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200911151557_0057/attempt_200911151557_0057_r_000000_0/output/file.out in any of the configured local directories 2009-11-16 03:22:30,737 INFO TaskTracker - attempt_200911151557_0057_r_000000_0 0.9031211% reduce > reduce 2009-11-16 03:22:33,739 INFO TaskTracker - attempt_200911151557_0057_r_000000_0 0.9044596% reduce > reduce 2009-11-16 03:22:35,006 INFO TaskTracker - org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200911151557_0057/attempt_200911151557_0057_r_000000_0/output/file.out in any of the configured local directories 2009-11-16 03:22:36,740 INFO TaskTracker - attempt_200911151557_0057_r_000000_0 0.90573466% reduce > reduce 2009-11-16 03:22:39,742 INFO TaskTracker - attempt_200911151557_0057_r_000000_0 0.90727556% reduce > reduce 2009-11-16 03:22:40,008 INFO TaskTracker - org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200911151557_0057/attempt_200911151557_0057_r_000000_0/output/file.out in any of the configured local directories 2009-11-16 03:22:42,743 INFO TaskTracker - attempt_200911151557_0057_r_000000_0 0.9089966% reduce > reduce 2009-11-16 03:22:45,010 INFO TaskTracker - org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200911151557_0057/attempt_200911151557_0057_r_000000_0/output/file.out in any of the configured local directories 2009-11-16 03:22:45,745 INFO TaskTracker - attempt_200911151557_0057_r_000000_0 0.9105506% reduce > reduce 2009-11-16 03:22:48,256 INFO TaskTracker - SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down TaskTracker at cluster11/10.214.10.146 ************************************************************/ log for datanode: 2009-11-16 02:04:52,350 INFO DataNode - Received block blk_-8957378818358413862_5195 src: /10.214.10.140:50154 dest: /10.214.10.146:50010 of size 25956230 2009-11-16 02:04:56,531 INFO clienttrace - src: /10.214.10.146:50010, dest: /10.214.10.144:54110, bytes: 26159014, op: HDFS_READ, cliID: DFSClient_327276617, srvID: DS-1562203209-10.214.10.146-50010-1258032593077, blockid: blk_-8957378818358413862_5195 2009-11-16 02:04:57,102 INFO DataNode - DatanodeRegistration(10.214.10.146:50010, storageID=DS-1562203209-10.214.10.146-50010-1258032593077, infoPort=50075, ipcPort=50020) Starting thread to transfer block blk_-8957378818358413862_5195 to 10.214.10.141:50010, 10.214.10.138:50010 2009-11-16 02:04:57,615 WARN DataNode - DatanodeRegistration(10.214.10.146:50010, storageID=DS-1562203209-10.214.10.146-50010-1258032593077, infoPort=50075, ipcPort=50020):Failed to transfer blk_-8957378818358413862_5195 to 10.214.10.141:50010 got java.net.SocketException: Original Exception : java.io.IOException: Connection reset by peer at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:415) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:516) at org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:199) at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendChunks(BlockSender.java:313) at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:400) at org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer.run(DataNode.java:1108) at java.lang.Thread.run(Thread.java:619) Caused by: java.io.IOException: Connection reset by peer ... 8 more 2009-11-16 02:04:58,348 INFO clienttrace - src: /10.214.10.146:50010, dest: /10.214.10.139:55389, bytes: 26159014, op: HDFS_READ, cliID: DFSClient_-1671031248, srvID: DS-1562203209-10.214.10.146-50010-1258032593077, blockid: blk_-8957378818358413862_5195 2009-11-16 02:06:41,822 INFO DataNode - Deleting block blk_-8957378818358413862_5195 file /home/hadoop/cluster/udms/filesystem/data/current/subdir40/blk_-8957378818358413862 2009-11-16 02:06:44,575 INFO DataNode - Receiving block blk_4719227011094553537_5201 src: /10.214.10.145:52848 dest: /10.214.10.146:50010 2009-11-16 02:06:48,847 INFO DataNode - Received block blk_4719227011094553537_5201 src: /10.214.10.145:52848 dest: /10.214.10.146:50010 of size 25956230 2009-11-16 02:06:50,806 INFO DataNode - DatanodeRegistration(10.214.10.146:50010, storageID=DS-1562203209-10.214.10.146-50010-1258032593077, infoPort=50075, ipcPort=50020) Starting thread to transfer block blk_4719227011094553537_5201 to 10.214.10.140:50010, 10.214.10.139:50010 2009-11-16 02:06:53,299 INFO DataBlockScanner - Verification succeeded for blk_-8639412427047222274_4014 2009-11-16 02:06:54,911 INFO DataNode - DatanodeRegistration(10.214.10.146:50010, storageID=DS-1562203209-10.214.10.146-50010-1258032593077, infoPort=50075, ipcPort=50020):Transmitted block blk_4719227011094553537_5201 to /10.214.10.140:50010 2009-11-16 02:07:08,831 INFO DataNode - Deleting block blk_4719227011094553537_5201 file /home/hadoop/cluster/udms/filesystem/data/current/subdir40/blk_4719227011094553537 2009-11-16 02:07:09,604 INFO DataNode - Receiving block blk_-4430275452309527039_5209 src: /10.214.10.135:60831 dest: /10.214.10.146:50010 2009-11-16 02:07:09,617 INFO clienttrace - src: /10.214.10.135:60831, dest: /10.214.10.146:50010, bytes: 313, op: HDFS_WRITE, cliID: DFSClient_-674969858, srvID: DS-1562203209-10.214.10.146-50010-1258032593077, blockid: blk_-4430275452309527039_5209 2009-11-16 02:07:09,617 INFO DataNode - PacketResponder 1 for block blk_-4430275452309527039_5209 terminating 2009-11-16 02:07:10,809 INFO DataNode - Receiving block blk_-5882543011393808875_5212 src: /10.214.10.135:60837 dest: /10.214.10.146:50010 2009-11-16 02:07:10,829 INFO clienttrace - src: /10.214.10.135:60837, dest: /10.214.10.146:50010, bytes: 25761, op: HDFS_WRITE, cliID: DFSClient_-1147035365, srvID: DS-1562203209-10.214.10.146-50010-1258032593077, blockid: blk_-5882543011393808875_5212 2009-11-16 02:07:10,829 INFO DataNode - PacketResponder 1 for block blk_-5882543011393808875_5212 terminating 2009-11-16 02:07:11,383 INFO DataNode - Receiving block blk_-1452382892993124154_5208 src: /10.214.10.144:48921 dest: /10.214.10.146:50010 2009-11-16 02:07:13,736 INFO DataNode - Received block blk_-1452382892993124154_5208 src: /10.214.10.144:48921 dest: /10.214.10.146:50010 of size 25956230 2009-11-16 02:07:18,052 INFO clienttrace - src: /10.214.10.146:50010, dest: /10.214.10.140:33184, bytes: 26159014, op: HDFS_READ, cliID: DFSClient_-1445553813, srvID: DS-1562203209-10.214.10.146-50010-1258032593077, blockid: blk_-1452382892993124154_5208 2009-11-16 02:16:53,392 INFO DataBlockScanner - Verification succeeded for blk_3620983791866783945_1807 2009-11-16 02:26:53,482 INFO DataBlockScanner - Verification succeeded for blk_4317683625220242951_3971 2009-11-16 02:36:53,561 INFO DataBlockScanner - Verification succeeded for blk_-7119627449348512_4655 2009-11-16 02:40:27,992 INFO DataNode - Receiving block blk_-4796295206230829835_5221 src: /10.214.10.140:38016 dest: /10.214.10.146:50010 2009-11-16 02:44:23,565 INFO clienttrace - src: /10.214.10.140:38016, dest: /10.214.10.146:50010, bytes: 67108864, op: HDFS_WRITE, cliID: DFSClient_attempt_200911151557_0053_r_000000_0, srvID: DS-1562203209-10.214.10.146-50010-1258032593077, blockid: blk_-4796295206230829835_5221 2009-11-16 02:44:23,565 INFO DataNode - PacketResponder 1 for block blk_-4796295206230829835_5221 terminating 2009-11-16 02:44:39,627 INFO DataNode - Receiving block blk_-3190633580536328464_5221 src: /10.214.10.145:37132 dest: /10.214.10.146:50010 2009-11-16 02:46:54,840 INFO DataBlockScanner - Verification succeeded for blk_7324649038313845675_4825 2009-11-16 02:46:54,857 INFO DataBlockScanner - Verification succeeded for blk_2278861981474298346_4670 2009-11-16 02:47:24,661 INFO DataNode - BlockReport of 505 blocks got processed in 36 msecs 2009-11-16 02:48:35,168 INFO clienttrace - src: /10.214.10.145:37132, dest: /10.214.10.146:50010, bytes: 67108864, op: HDFS_WRITE, cliID: DFSClient_attempt_200911151557_0053_r_000000_0, srvID: DS-1562203209-10.214.10.146-50010-1258032593077, blockid: blk_-3190633580536328464_5221 2009-11-16 02:48:35,168 INFO DataNode - PacketResponder 0 for block blk_-3190633580536328464_5221 terminating 2009-11-16 02:49:05,309 INFO DataNode - Receiving block blk_-905297406973631195_5225 src: /10.214.10.135:40423 dest: /10.214.10.146:50010 2009-11-16 02:49:05,334 INFO clienttrace - src: /10.214.10.135:40423, dest: /10.214.10.146:50010, bytes: 26044, op: HDFS_WRITE, cliID: DFSClient_-674969858, srvID: DS-1562203209-10.214.10.146-50010-1258032593077, blockid: blk_-905297406973631195_5225 2009-11-16 02:49:05,334 INFO DataNode - PacketResponder 1 for block blk_-905297406973631195_5225 terminating 2009-11-16 02:49:06,701 INFO DataNode - Deleting block blk_-4430275452309527039_5209 file /home/hadoop/cluster/udms/filesystem/data/current/subdir40/blk_-4430275452309527039 2009-11-16 02:49:06,714 INFO DataNode - Deleting block blk_-1452382892993124154_5208 file /home/hadoop/cluster/udms/filesystem/data/current/subdir40/blk_-1452382892993124154 2009-11-16 02:49:10,414 INFO DataNode - Receiving block blk_-4454277735412757930_5223 src: /10.214.10.138:33793 dest: /10.214.10.146:50010 2009-11-16 02:49:12,716 INFO DataNode - Received block blk_-4454277735412757930_5223 src: /10.214.10.138:33793 dest: /10.214.10.146:50010 of size 25956230 2009-11-16 02:49:12,843 INFO clienttrace - src: /10.214.10.146:50010, dest: /10.214.10.140:53832, bytes: 26248, op: HDFS_READ, cliID: DFSClient_-1445553813, srvID: DS-1562203209-10.214.10.146-50010-1258032593077, blockid: blk_-905297406973631195_5225 2009-11-16 02:49:15,173 INFO clienttrace - src: /10.214.10.146:50010, dest: /10.214.10.136:34162, bytes: 26248, op: HDFS_READ, cliID: DFSClient_349267602, srvID: DS-1562203209-10.214.10.146-50010-1258032593077, blockid: blk_-905297406973631195_5225 2009-11-16 02:49:15,265 INFO clienttrace - src: /10.214.10.146:50010, dest: /10.214.10.146:44947, bytes: 26248, op: HDFS_READ, cliID: DFSClient_1867506502, srvID: DS-1562203209-10.214.10.146-50010-1258032593077, blockid: blk_-905297406973631195_5225 2009-11-16 02:49:16,073 INFO DataNode - DatanodeRegistration(10.214.10.146:50010, storageID=DS-1562203209-10.214.10.146-50010-1258032593077, infoPort=50075, ipcPort=50020) Starting thread to transfer block blk_-4454277735412757930_5223 to 10.214.10.139:50010, 10.214.10.142:50010 2009-11-16 02:49:17,056 INFO clienttrace - src: /10.214.10.146:50010, dest: /10.214.10.138:33795, bytes: 26248, op: HDFS_READ, cliID: DFSClient_-1225643780, srvID: DS-1562203209-10.214.10.146-50010-1258032593077, blockid: blk_-905297406973631195_5225 2009-11-16 02:49:17,696 INFO clienttrace - src: /10.214.10.146:50010, dest: /10.214.10.146:44949, bytes: 26159014, op: HDFS_READ, cliID: DFSClient_1867506502, srvID: DS-1562203209-10.214.10.146-50010-1258032593077, blockid: blk_-4454277735412757930_5223 2009-11-16 02:49:20,590 INFO DataNode - DatanodeRegistration(10.214.10.146:50010, storageID=DS-1562203209-10.214.10.146-50010-1258032593077, infoPort=50075, ipcPort=50020):Transmitted block blk_-4454277735412757930_5223 to /10.214.10.139:50010 2009-11-16 02:49:21,187 INFO clienttrace - src: /10.214.10.146:50010, dest: /10.214.10.146:44956, bytes: 198144, op: HDFS_READ, cliID: DFSClient_attempt_200911151557_0054_m_000005_0, srvID: DS-1562203209-10.214.10.146-50010-1258032593077, blockid: blk_-3190633580536328464_5221 2009-11-16 02:49:21,883 INFO DataNode - DatanodeRegistration(10.214.10.146:50010, storageID=DS-1562203209-10.214.10.146-50010-1258032593077, infoPort=50075, ipcPort=50020) Starting thread to transfer block blk_-4454277735412757930_5223 to 10.214.10.144:50010, 10.214.10.142:50010 2009-11-16 02:49:24,670 INFO DataNode - DatanodeRegistration(10.214.10.146:50010, storageID=DS-1562203209-10.214.10.146-50010-1258032593077, infoPort=50075, ipcPort=50020):Transmitted block blk_-4454277735412757930_5223 to /10.214.10.144:50010 2009-11-16 02:50:09,240 INFO clienttrace - src: /10.214.10.146:50010, dest: /10.214.10.135:40449, bytes: 132096, op: HDFS_READ, cliID: DFSClient_attempt_200911151557_0054_m_000005_1, srvID: DS-1562203209-10.214.10.146-50010-1258032593077, blockid: blk_-3190633580536328464_5221 2009-11-16 02:51:13,615 INFO clienttrace - src: /10.214.10.146:50010, dest: /10.214.10.136:34168, bytes: 132096, op: HDFS_READ, cliID: DFSClient_attempt_200911151557_0054_m_000004_0, srvID: DS-1562203209-10.214.10.146-50010-1258032593077, blockid: blk_-3190633580536328464_5221 2009-11-16 02:51:26,149 INFO clienttrace - src: /10.214.10.146:50010, dest: /10.214.10.146:44958, bytes: 67632120, op: HDFS_READ, cliID: DFSClient_attempt_200911151557_0054_m_000005_0, srvID: DS-1562203209-10.214.10.146-50010-1258032593077, blockid: blk_-3190633580536328464_5221 2009-11-16 02:51:32,713 INFO clienttrace - src: /10.214.10.146:50010, dest: /10.214.10.135:40450, bytes: 57329664, op: HDFS_READ, cliID: DFSClient_attempt_200911151557_0054_m_000005_1, srvID: DS-1562203209-10.214.10.146-50010-1258032593077, blockid: blk_-3190633580536328464_5221 2009-11-16 02:52:33,665 INFO DataNode - Receiving block blk_-1526009892521129534_5229 src: /10.214.10.138:41961 dest: /10.214.10.146:50010 2009-11-16 02:53:53,218 INFO clienttrace - src: /10.214.10.138:41961, dest: /10.214.10.146:50010, bytes: 67108864, op: HDFS_WRITE, cliID: DFSClient_attempt_200911151557_0054_r_000000_0, srvID: DS-1562203209-10.214.10.146-50010-1258032593077, blockid: blk_-1526009892521129534_5229 2009-11-16 02:53:53,219 INFO DataNode - PacketResponder 1 for block blk_-1526009892521129534_5229 terminating 2009-11-16 02:56:29,108 INFO DataNode - Receiving block blk_5351211997239525112_5229 src: /10.214.10.138:58478 dest: /10.214.10.146:50010 2009-11-16 02:56:35,591 INFO clienttrace - src: /10.214.10.138:58478, dest: /10.214.10.146:50010, bytes: 10877088, op: HDFS_WRITE, cliID: DFSClient_attempt_200911151557_0054_r_000000_0, srvID: DS-1562203209-10.214.10.146-50010-1258032593077, blockid: blk_5351211997239525112_5229 2009-11-16 02:56:35,591 INFO DataNode - PacketResponder 1 for block blk_5351211997239525112_5229 terminating 2009-11-16 02:56:39,634 INFO DataNode - Receiving block blk_-5764721885719636747_5231 src: /10.214.10.138:58483 dest: /10.214.10.146:50010 2009-11-16 02:56:42,253 INFO clienttrace - src: /10.214.10.138:58483, dest: /10.214.10.146:50010, bytes: 25956230, op: HDFS_WRITE, cliID: DFSClient_-674969858, srvID: DS-1562203209-10.214.10.146-50010-1258032593077, blockid: blk_-5764721885719636747_5231 2009-11-16 02:56:42,253 INFO DataNode - PacketResponder 0 for block blk_-5764721885719636747_5231 terminating 2009-11-16 02:56:42,278 INFO DataNode - Receiving block blk_-7825621483773334521_5232 src: /10.214.10.135:47895 dest: /10.214.10.146:50010 2009-11-16 02:56:42,287 INFO clienttrace - src: /10.214.10.135:47895, dest: /10.214.10.146:50010, bytes: 822, op: HDFS_WRITE, cliID: DFSClient_-674969858, srvID: DS-1562203209-10.214.10.146-50010-1258032593077, blockid: blk_-7825621483773334521_5232 2009-11-16 02:56:42,288 INFO DataNode - PacketResponder 1 for block blk_-7825621483773334521_5232 terminating 2009-11-16 02:56:42,312 INFO DataNode - Receiving block blk_2168658662190289020_5233 src: /10.214.10.137:47308 dest: /10.214.10.146:50010 2009-11-16 02:56:42,323 INFO clienttrace - src: /10.214.10.137:47308, dest: /10.214.10.146:50010, bytes: 26371, op: HDFS_WRITE, cliID: DFSClient_-674969858, srvID: DS-1562203209-10.214.10.146-50010-1258032593077, blockid: blk_2168658662190289020_5233 2009-11-16 02:56:42,323 INFO DataNode - PacketResponder 0 for block blk_2168658662190289020_5233 terminating 2009-11-16 02:56:42,887 INFO DataNode - Deleting block blk_-4454277735412757930_5223 file /home/hadoop/cluster/udms/filesystem/data/current/subdir40/blk_-4454277735412757930 2009-11-16 02:56:42,888 INFO DataNode - Deleting block blk_-905297406973631195_5225 file /home/hadoop/cluster/udms/filesystem/data/current/subdir40/blk_-905297406973631195 2009-11-16 02:56:43,615 INFO DataNode - Receiving block blk_5000861293225981216_5235 src: /10.214.10.139:48907 dest: /10.214.10.146:50010 2009-11-16 02:56:43,627 INFO clienttrace - src: /10.214.10.139:48907, dest: /10.214.10.146:50010, bytes: 26369, op: HDFS_WRITE, cliID: DFSClient_-1147035365, srvID: DS-1562203209-10.214.10.146-50010-1258032593077, blockid: blk_5000861293225981216_5235 2009-11-16 02:56:43,627 INFO DataNode - PacketResponder 0 for block blk_5000861293225981216_5235 terminating 2009-11-16 02:56:49,147 INFO clienttrace - src: /10.214.10.146:50010, dest: /10.214.10.136:56767, bytes: 26579, op: HDFS_READ, cliID: DFSClient_349267602, srvID: DS-1562203209-10.214.10.146-50010-1258032593077, blockid: blk_2168658662190289020_5233 2009-11-16 02:56:51,854 INFO clienttrace - src: /10.214.10.146:50010, dest: /10.214.10.141:35836, bytes: 26579, op: HDFS_READ, cliID: DFSClient_-929047123, srvID: DS-1562203209-10.214.10.146-50010-1258032593077, blockid: blk_2168658662190289020_5233 2009-11-16 02:56:53,946 INFO DataBlockScanner - Verification succeeded for blk_2699840809375917103_3856 2009-11-16 02:57:05,236 INFO clienttrace - src: /10.214.10.146:50010, dest: /10.214.10.137:47311, bytes: 26159014, op: HDFS_READ, cliID: DFSClient_-669679079, srvID: DS-1562203209-10.214.10.146-50010-1258032593077, blockid: blk_-5764721885719636747_5231 2009-11-16 02:57:55,130 INFO clienttrace - src: /10.214.10.146:50010, dest: /10.214.10.139:58635, bytes: 132096, op: HDFS_READ, cliID: DFSClient_attempt_200911151557_0055_m_000003_0, srvID: DS-1562203209-10.214.10.146-50010-1258032593077, blockid: blk_5351211997239525112_5229 2009-11-16 02:58:11,057 INFO DataNode - Receiving block blk_-5202153540192609250_5236 src: /10.214.10.137:47322 dest: /10.214.10.146:50010 2009-11-16 02:58:19,931 INFO clienttrace - src: /10.214.10.137:47322, dest: /10.214.10.146:50010, bytes: 7259817, op: HDFS_WRITE, cliID: DFSClient_attempt_200911151557_0055_r_000000_0, srvID: DS-1562203209-10.214.10.146-50010-1258032593077, blockid: blk_-5202153540192609250_5236 2009-11-16 02:58:19,931 INFO DataNode - PacketResponder 1 for block blk_-5202153540192609250_5236 terminating 2009-11-16 02:58:24,914 INFO DataNode - Deleting block blk_-7825621483773334521_5232 file /home/hadoop/cluster/udms/filesystem/data/current/subdir13/blk_-7825621483773334521 2009-11-16 02:58:24,927 INFO DataNode - Deleting block blk_-5764721885719636747_5231 file /home/hadoop/cluster/udms/filesystem/data/current/subdir13/blk_-5764721885719636747 2009-11-16 02:58:24,928 INFO DataNode - Deleting block blk_2168658662190289020_5233 file /home/hadoop/cluster/udms/filesystem/data/current/subdir13/blk_2168658662190289020 2009-11-16 02:58:25,102 INFO DataNode - Receiving block blk_-1825317570820697486_5238 src: /10.214.10.139:58638 dest: /10.214.10.146:50010 2009-11-16 02:58:25,110 INFO clienttrace - src: /10.214.10.139:58638, dest: /10.214.10.146:50010, bytes: 371, op: HDFS_WRITE, cliID: DFSClient_-674969858, srvID: DS-1562203209-10.214.10.146-50010-1258032593077, blockid: blk_-1825317570820697486_5238 2009-11-16 02:58:25,110 INFO DataNode - PacketResponder 0 for block blk_-1825317570820697486_5238 terminating 2009-11-16 02:58:33,831 INFO DataNode - Receiving block blk_5033878573521283155_5237 src: /10.214.10.137:47327 dest: /10.214.10.146:50010 2009-11-16 02:58:41,213 INFO clienttrace - src: /10.214.10.146:50010, dest: /10.214.10.139:58645, bytes: 3764736, op: HDFS_READ, cliID: DFSClient_attempt_200911151557_0056_m_000001_0, srvID: DS-1562203209-10.214.10.146-50010-1258032593077, blockid: blk_-5202153540192609250_5236 2009-11-16 02:58:41,784 INFO DataNode - Received block blk_5033878573521283155_5237 src: /10.214.10.137:47327 dest: /10.214.10.146:50010 of size 25956230 2009-11-16 02:58:42,796 INFO DataNode - Receiving block blk_5033878573521283155_5237 src: /10.214.10.144:42027 dest: /10.214.10.146:50010 2009-11-16 02:58:42,797 INFO DataNode - writeBlock blk_5033878573521283155_5237 received exception org.apache.hadoop.hdfs.server.datanode.BlockAlreadyExistsException: Block blk_5033878573521283155_5237 is valid, and cannot be written to. 2009-11-16 02:58:42,797 ERROR DataNode - DatanodeRegistration(10.214.10.146:50010, storageID=DS-1562203209-10.214.10.146-50010-1258032593077, infoPort=50075, ipcPort=50020):DataXceiver org.apache.hadoop.hdfs.server.datanode.BlockAlreadyExistsException: Block blk_5033878573521283155_5237 is valid, and cannot be written to. at org.apache.hadoop.hdfs.server.datanode.FSDataset.writeToBlock(FSDataset.java:975) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:97) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:259) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:103) at java.lang.Thread.run(Thread.java:619) 2009-11-16 02:58:44,753 INFO DataNode - Receiving block blk_2845293546445463780_5242 src: /10.214.10.146:45619 dest: /10.214.10.146:50010 2009-11-16 02:58:45,649 INFO clienttrace - src: /10.214.10.146:45619, dest: /10.214.10.146:50010, bytes: 2607080, op: HDFS_WRITE, cliID: DFSClient_attempt_200911151557_0056_r_000001_0, srvID: DS-1562203209-10.214.10.146-50010-1258032593077, blockid: blk_2845293546445463780_5242 2009-11-16 02:58:45,650 INFO DataNode - PacketResponder 2 for block blk_2845293546445463780_5242 terminating 2009-11-16 02:58:45,914 INFO DataNode - DatanodeRegistration(10.214.10.146:50010, storageID=DS-1562203209-10.214.10.146-50010-1258032593077, infoPort=50075, ipcPort=50020) Starting thread to transfer block blk_5033878573521283155_5237 to 10.214.10.143:50010 2009-11-16 02:58:48,102 INFO DataNode - DatanodeRegistration(10.214.10.146:50010, storageID=DS-1562203209-10.214.10.146-50010-1258032593077, infoPort=50075, ipcPort=50020):Transmitted block blk_5033878573521283155_5237 to /10.214.10.143:50010 2009-11-16 02:59:00,924 INFO DataNode - Deleting block blk_-5202153540192609250_5236 file /home/hadoop/cluster/udms/filesystem/data/current/subdir13/blk_-5202153540192609250 2009-11-16 02:59:00,924 INFO DataNode - Deleting block blk_-1825317570820697486_5238 file /home/hadoop/cluster/udms/filesystem/data/current/subdir13/blk_-1825317570820697486 2009-11-16 02:59:00,938 INFO DataNode - Deleting block blk_5033878573521283155_5237 file /home/hadoop/cluster/udms/filesystem/data/current/subdir13/blk_5033878573521283155 2009-11-16 02:59:09,808 INFO DataNode - Receiving block blk_-7871239562527928275_5244 src: /10.214.10.140:60000 dest: /10.214.10.146:50010 2009-11-16 02:59:12,457 INFO DataNode - Received block blk_-7871239562527928275_5244 src: /10.214.10.140:60000 dest: /10.214.10.146:50010 of size 25956230 2009-11-16 02:59:36,927 INFO clienttrace - src: /10.214.10.146:50010, dest: /10.214.10.135:37693, bytes: 2627448, op: HDFS_READ, cliID: DFSClient_attempt_200911151557_0057_m_000001_0, srvID: DS-1562203209-10.214.10.146-50010-1258032593077, blockid: blk_2845293546445463780_5242 2009-11-16 03:01:51,003 INFO DataBlockScanner - Verification succeeded for blk_1056570906855815251_5041 2009-11-16 03:06:53,068 INFO DataBlockScanner - Verification succeeded for blk_-7719389212684200506_4360 2009-11-16 03:16:25,148 INFO DataNode - Receiving block blk_-8300321570118896035_5257 src: /10.214.10.146:55374 dest: /10.214.10.146:50010 2009-11-16 03:16:25,421 INFO DataNode - Receiving block blk_-496211706537946421_5257 src: /10.214.10.146:55377 dest: /10.214.10.146:50010 2009-11-16 03:16:26,037 INFO DataNode - Receiving block blk_3673766215710182727_5257 src: /10.214.10.146:55379 dest: /10.214.10.146:50010 2009-11-16 03:16:26,115 INFO DataNode - Receiving block blk_-2359856724295619565_5257 src: /10.214.10.146:55381 dest: /10.214.10.146:50010 2009-11-16 03:16:26,368 INFO DataNode - Receiving block blk_7157170921783486836_5257 src: /10.214.10.146:55384 dest: /10.214.10.146:50010 2009-11-16 03:16:53,212 INFO DataBlockScanner - Verification succeeded for blk_4247076805664458192_3123 2009-11-16 03:19:03,370 INFO clienttrace - src: /10.214.10.146:55374, dest: /10.214.10.146:50010, bytes: 67108864, op: HDFS_WRITE, cliID: DFSClient_attempt_200911151557_0057_r_000000_0, srvID: DS-1562203209-10.214.10.146-50010-1258032593077, blockid: blk_-8300321570118896035_5257 2009-11-16 03:19:03,370 INFO DataNode - PacketResponder 2 for block blk_-8300321570118896035_5257 terminating 2009-11-16 03:19:03,382 INFO DataNode - Receiving block blk_1839674181473840998_5257 src: /10.214.10.146:55427 dest: /10.214.10.146:50010 2009-11-16 03:22:48,269 INFO DataNode - SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down DataNode at cluster11/10.214.10.146 ************************************************************/ log for job: Lost task tracker: tracker_cluster11:localhost/127.0.0.1:54814 org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to create file /user/hadoop/crawl/segments/20091116025700/crawl_fetch/part-00000/index for DFSClient_attempt_200911151557_0057_r_000000_1 on client 10.214.10.137 because current leaseholder is trying to recreate file. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1055) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:998) at org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:301) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894) at org.apache.hadoop.ipc.Client.call(Client.java:697) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216) at $Proxy1.create(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59) at $Proxy1.create(Unknown Source) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.(DFSClient.java:2585) at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:454) at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:190) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:487) at org.apache.hadoop.io.SequenceFile$BlockCompressWriter.(SequenceFile.java:1198) at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:401) at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:306) at org.apache.hadoop.io.MapFile$Writer.(MapFile.java:160) at org.apache.hadoop.io.MapFile$Writer.(MapFile.java:134) at org.apache.hadoop.io.MapFile$Writer.(MapFile.java:92) at org.apache.nutch.fetcher.FetcherOutputFormat.getRecordWriter(FetcherOutputFormat.java:66) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:404) at org.apache.hadoop.mapred.Child.main(Child.java:158) org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to create file /user/hadoop/crawl/segments/20091116025700/crawl_fetch/part-00000/index for DFSClient_attempt_200911151557_0057_r_000000_2 on client 10.214.10.137 because current leaseholder is trying to recreate file. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1055) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:998) at org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:301) at sun.reflect.GeneratedMethodAccessor77.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894) at org.apache.hadoop.ipc.Client.call(Client.java:697) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216) at $Proxy1.create(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59) at $Proxy1.create(Unknown Source) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.(DFSClient.java:2585) at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:454) at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:190) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:487) at org.apache.hadoop.io.SequenceFile$BlockCompressWriter.(SequenceFile.java:1198) at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:401) at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:306) at org.apache.hadoop.io.MapFile$Writer.(MapFile.java:160) at org.apache.hadoop.io.MapFile$Writer.(MapFile.java:134) at org.apache.hadoop.io.MapFile$Writer.(MapFile.java:92) at org.apache.nutch.fetcher.FetcherOutputFormat.getRecordWriter(FetcherOutputFormat.java:66) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:404) at org.apache.hadoop.mapred.Child.main(Child.java:158)
