Xiao Chen created HDFS-13642: -------------------------------- Summary: Creating a file with block size smaller than EC policy's cell size should throw Key: HDFS-13642 URL: https://issues.apache.org/jira/browse/HDFS-13642 Project: Hadoop HDFS Issue Type: Bug Components: erasure-coding Affects Versions: 3.0.0 Reporter: Xiao Chen Assignee: Xiao Chen Attachments: HDFS-13642.01.patch
The following command causes an exception: {noformat} hadoop fs -Ddfs.block.size=349696 -put -f lineitem_sixblocks.parquet /test-warehouse/tmp123ec {noformat} {noformat} 18/05/25 16:00:59 WARN hdfs.DataStreamer: DataStreamer Exception java.io.IOException: BlockSize 349696 < lastByteOffsetInBlock, #0: blk_-9223372036854574256_14634, packet seqno: 7 offsetInBlock: 349696 lastPacketInBlock: false lastByteOffsetInBlock: 350208 at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:729) at org.apache.hadoop.hdfs.StripedDataStreamer.run(StripedDataStreamer.java:46) 18/05/25 16:00:59 WARN hdfs.DFSOutputStream: Failed: offset=4096, length=512, DFSStripedOutputStream:#0: failed, blk_-9223372036854574256_14634 java.io.IOException: BlockSize 349696 < lastByteOffsetInBlock, #0: blk_-9223372036854574256_14634, packet seqno: 7 offsetInBlock: 349696 lastPacketInBlock: false lastByteOffsetInBlock: 350208 at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:729) at org.apache.hadoop.hdfs.StripedDataStreamer.run(StripedDataStreamer.java:46) {noformat} Then the streamer is confused and hangs. The local file is under 6MB, the hdfs file has a RS-3-2-1024k EC policy. Credit to [~tarasbob] for reporting this issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org