[jira] [Commented] (HDFS-13642) Creating a file with block size smaller than EC policy's cell size should fail

2019-08-22 Thread Akira Ajisaka (Jira) 


[ 
https://issues.apache.org/jira/browse/HDFS-13642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913960#comment-16913960
 ] 

Akira Ajisaka commented on HDFS-13642:
--

Cherry-picking this to branch-3.1 to apply HDFS-13596 cleanly.

> Creating a file with block size smaller than EC policy's cell size should fail
> --
>
> Key: HDFS-13642
> URL: https://issues.apache.org/jira/browse/HDFS-13642
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: erasure-coding
>Affects Versions: 3.0.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HDFS-13642.01.patch, HDFS-13642.02.patch, 
> HDFS-13642.03.patch, editsStored
>
>
> The following command causes an exception:
> {noformat}
> hadoop fs -Ddfs.block.size=349696 -put -f lineitem_sixblocks.parquet 
> /test-warehouse/tmp123ec
> {noformat}
> {noformat}
> 18/05/25 16:00:59 WARN hdfs.DataStreamer: DataStreamer Exception
> java.io.IOException: BlockSize 349696 < lastByteOffsetInBlock, #0: 
> blk_-9223372036854574256_14634, packet seqno: 7 offsetInBlock: 349696 
> lastPacketInBlock: false lastByteOffsetInBlock: 350208
>   at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:729)
>   at 
> org.apache.hadoop.hdfs.StripedDataStreamer.run(StripedDataStreamer.java:46)
> 18/05/25 16:00:59 WARN hdfs.DFSOutputStream: Failed: offset=4096, length=512, 
> DFSStripedOutputStream:#0: failed, blk_-9223372036854574256_14634
> java.io.IOException: BlockSize 349696 < lastByteOffsetInBlock, #0: 
> blk_-9223372036854574256_14634, packet seqno: 7 offsetInBlock: 349696 
> lastPacketInBlock: false lastByteOffsetInBlock: 350208
>   at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:729)
>   at 
> org.apache.hadoop.hdfs.StripedDataStreamer.run(StripedDataStreamer.java:46)
> {noformat}
> Then the streamer is confused and hangs.
> The local file is under 6MB, the hdfs file has a RS-3-2-1024k EC policy.
>  
> Credit to [~tarasbob] for reporting this issue.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-13642) Creating a file with block size smaller than EC policy's cell size should fail

2018-06-08 Thread Hudson (JIRA) 


[ 
https://issues.apache.org/jira/browse/HDFS-13642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16506656#comment-16506656
 ] 

Hudson commented on HDFS-13642:
---

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14392 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/14392/])
HDFS-13642. Creating a file with block size smaller than EC policy's (xiao: rev 
cf4108313da83e28d07676078a33016ec8856ff6)
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStream.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
* (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored
* (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestErasureCodingPolicies.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirErasureCodingOp.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestErasureCodingExerciseAPIs.java


> Creating a file with block size smaller than EC policy's cell size should fail
> --
>
> Key: HDFS-13642
> URL: https://issues.apache.org/jira/browse/HDFS-13642
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: erasure-coding
>Affects Versions: 3.0.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HDFS-13642.01.patch, HDFS-13642.02.patch, 
> HDFS-13642.03.patch, editsStored
>
>
> The following command causes an exception:
> {noformat}
> hadoop fs -Ddfs.block.size=349696 -put -f lineitem_sixblocks.parquet 
> /test-warehouse/tmp123ec
> {noformat}
> {noformat}
> 18/05/25 16:00:59 WARN hdfs.DataStreamer: DataStreamer Exception
> java.io.IOException: BlockSize 349696 < lastByteOffsetInBlock, #0: 
> blk_-9223372036854574256_14634, packet seqno: 7 offsetInBlock: 349696 
> lastPacketInBlock: false lastByteOffsetInBlock: 350208
>   at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:729)
>   at 
> org.apache.hadoop.hdfs.StripedDataStreamer.run(StripedDataStreamer.java:46)
> 18/05/25 16:00:59 WARN hdfs.DFSOutputStream: Failed: offset=4096, length=512, 
> DFSStripedOutputStream:#0: failed, blk_-9223372036854574256_14634
> java.io.IOException: BlockSize 349696 < lastByteOffsetInBlock, #0: 
> blk_-9223372036854574256_14634, packet seqno: 7 offsetInBlock: 349696 
> lastPacketInBlock: false lastByteOffsetInBlock: 350208
>   at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:729)
>   at 
> org.apache.hadoop.hdfs.StripedDataStreamer.run(StripedDataStreamer.java:46)
> {noformat}
> Then the streamer is confused and hangs.
> The local file is under 6MB, the hdfs file has a RS-3-2-1024k EC policy.
>  
> Credit to [~tarasbob] for reporting this issue.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org