Hi,
Is there any restriction on the amount of putting files? I tried to
put/copyFromLocal about 50,573 files to HDFS, but I faced a problem:
================================================================================================================================
09/06/22 11:34:34 INFO dfs.DFSClient: Exception in createBlockOutputStream
java.io.IOException: Bad connect ack with firstBadLink 140.96.89.57:51010
09/06/22 11:34:34 INFO dfs.DFSClient: Abandoning block
blk_8245450203753506945_65955
09/06/22 11:34:40 INFO dfs.DFSClient: Exception in createBlockOutputStream
java.io.IOException: Bad connect ack with firstBadLink 140.96.89.57:51010
09/06/22 11:34:40 INFO dfs.DFSClient: Abandoning block
blk_-8257846965500649510_65956
09/06/22 11:34:46 INFO dfs.DFSClient: Exception in createBlockOutputStream
java.io.IOException: Bad connect ack with firstBadLink 140.96.89.57:51010
09/06/22 11:34:46 INFO dfs.DFSClient: Abandoning block
blk_4751737303082929912_65956
09/06/22 11:34:56 INFO dfs.DFSClient: Exception in createBlockOutputStream
java.io.IOException: Bad connect ack with firstBadLink 140.96.89.57:51010
09/06/22 11:34:56 INFO dfs.DFSClient: Abandoning block
blk_5912850890372596972_66040
09/06/22 11:35:02 INFO dfs.DFSClient: Exception in createBlockOutputStream
java.io.IOException: Bad connect ack with firstBadLink 140.96.89.193:51010
09/06/22 11:35:02 INFO dfs.DFSClient: Abandoning block
blk_6609198685444611538_66040
09/06/22 11:35:08 INFO dfs.DFSClient: Exception in createBlockOutputStream
java.io.IOException: Bad connect ack with firstBadLink 140.96.89.193:51010
09/06/22 11:35:08 INFO dfs.DFSClient: Abandoning block
blk_6696101244177965180_66040
09/06/22 11:35:17 INFO dfs.DFSClient: Exception in createBlockOutputStream
java.io.IOException: Bad connect ack with firstBadLink 140.96.89.57:51010
09/06/22 11:35:17 INFO dfs.DFSClient: Abandoning block
blk_-5430033105510098342_66105
09/06/22 11:35:26 INFO dfs.DFSClient: Exception in createBlockOutputStream
java.io.IOException: Bad connect ack with firstBadLink 140.96.89.57:51010
09/06/22 11:35:26 INFO dfs.DFSClient: Abandoning block
blk_5325140471333041601_66165
09/06/22 11:35:32 INFO dfs.DFSClient: Exception in createBlockOutputStream
java.io.IOException: Bad connect ack with firstBadLink 140.96.89.205:51010
09/06/22 11:35:32 INFO dfs.DFSClient: Abandoning block
blk_1121864992752821949_66165
09/06/22 11:35:39 INFO dfs.DFSClient: Exception in createBlockOutputStream
java.io.IOException: Bad connect ack with firstBadLink 140.96.89.205:51010
09/06/22 11:35:39 INFO dfs.DFSClient: Abandoning block
blk_-2096783021040778965_66184
09/06/22 11:35:45 INFO dfs.DFSClient: Exception in createBlockOutputStream
java.io.IOException: Bad connect ack with firstBadLink 140.96.89.205:51010
09/06/22 11:35:45 INFO dfs.DFSClient: Abandoning block
blk_6949821898790162970_66184
09/06/22 11:35:51 INFO dfs.DFSClient: Exception in createBlockOutputStream
java.io.IOException: Bad connect ack with firstBadLink 140.96.89.205:51010
09/06/22 11:35:51 INFO dfs.DFSClient: Abandoning block
blk_4708848202696905125_66184
09/06/22 11:35:57 INFO dfs.DFSClient: Exception in createBlockOutputStream
java.io.IOException: Bad connect ack with firstBadLink 140.96.89.205:51010
09/06/22 11:35:57 INFO dfs.DFSClient: Abandoning block
blk_8031882012801762201_66184
09/06/22 11:36:03 WARN dfs.DFSClient: DataStreamer Exception:
java.io.IOException: Unable to create new block.
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2359)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1800(DFSClient.java:1745)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1922)
09/06/22 11:36:03 WARN dfs.DFSClient: Error Recovery for block
blk_8031882012801762201_66184 bad datanode[2]
put: Could not get block locations. Aborting...
Exception closing file /osmFiles/a/109103.gpx.txt
java.io.IOException: Could not get block locations. Aborting...
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2153)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1400(DFSClient.java:1745)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1899)
===================================================================================================================================
And I checked the log file on one of the datanode:
===================================================================================================================================
2009-06-22 11:34:47,888 INFO org.apache.hadoop.dfs.DataNode: PacketResponder
2 for block blk_1759242372147720864_66183 terminating
2009-06-22 11:34:47,926 INFO org.apache.hadoop.dfs.DataNode: Receiving block
blk_-2096783021040778965_66184 src: /140.96.89.224:53984 dest: /
140.96.89.224:51010
2009-06-22 11:34:47,926 INFO org.apache.hadoop.dfs.DataNode: writeBlock
blk_-2096783021040778965_66184 received exception java.io.IOException: Too
many open files
2009-06-22 11:34:47,926 ERROR org.apache.hadoop.dfs.DataNode:
DatanodeRegistration(140.96.89.193:51010,
storageID=DS-1452520190-140.96.89.193-51010-1241603100681, infoPort=51075,
ipcPort=51020):DataXceiver: java.io.IOException: Too many open files
at sun.nio.ch.IOUtil.initPipe(Native Method)
at sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:49)
at
sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
at sun.nio.ch.Util.getTemporarySelector(Util.java:123)
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:92)
at
org.apache.hadoop.dfs.DataNode$DataXceiver.writeBlock(DataNode.java:1254)
at
org.apache.hadoop.dfs.DataNode$DataXceiver.run(DataNode.java:1091)
at java.lang.Thread.run(Thread.java:619)
===================================================================================================================================
Does anyone know why it occured and how to solve this problem? Thank you
very much!!
