[
https://issues.apache.org/jira/browse/HDFS-8220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14698009#comment-14698009
]
Walter Su commented on HDFS-8220:
---------------------------------
bq. Should we always do endBlock() and close when setting a streamer as failed?
No. We set a streamer failed to prevent OutputStream write data to it. I think
the failed flag should unset when move on to next block group. So no close()
should called.
The patch doesn't handle the situation when first LocatedStripedBlock satisfied
GroupSize, but the second doesn't. It's possible when some DNs become busy.
Since this jira already closed, and also it's more complicated, we can fix it
later. Thanks [~rakeshr] for fixing the first step.
[~szetszwo], would you mind solve the conflicts in HDFS-8838?
> Erasure Coding: StripedDataStreamer fails to handle the blocklocations which
> doesn't satisfy BlockGroupSize
> -----------------------------------------------------------------------------------------------------------
>
> Key: HDFS-8220
> URL: https://issues.apache.org/jira/browse/HDFS-8220
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Rakesh R
> Assignee: Rakesh R
> Fix For: HDFS-7285
>
> Attachments: HDFS-8220-001.patch, HDFS-8220-002.patch,
> HDFS-8220-003.patch, HDFS-8220-004.patch, HDFS-8220-HDFS-7285-09.patch,
> HDFS-8220-HDFS-7285-10.patch, HDFS-8220-HDFS-7285-merge-10.patch,
> HDFS-8220-HDFS-7285.005.patch, HDFS-8220-HDFS-7285.006.patch,
> HDFS-8220-HDFS-7285.007.patch, HDFS-8220-HDFS-7285.007.patch,
> HDFS-8220-HDFS-7285.008.patch
>
>
> During write operations {{StripedDataStreamer#locateFollowingBlock}} fails to
> validate the available datanodes against the {{BlockGroupSize}}. Please see
> the exception to understand more:
> {code}
> 2015-04-22 14:56:11,313 WARN hdfs.DFSClient (DataStreamer.java:run(538)) -
> DataStreamer Exception
> java.lang.NullPointerException
> at
> java.util.concurrent.LinkedBlockingQueue.offer(LinkedBlockingQueue.java:374)
> at
> org.apache.hadoop.hdfs.StripedDataStreamer.locateFollowingBlock(StripedDataStreamer.java:157)
> at
> org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1332)
> at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:424)
> at
> org.apache.hadoop.hdfs.StripedDataStreamer.run(StripedDataStreamer.java:1)
> 2015-04-22 14:56:11,313 INFO hdfs.MiniDFSCluster
> (MiniDFSCluster.java:shutdown(1718)) - Shutting down the Mini HDFS Cluster
> 2015-04-22 14:56:11,313 ERROR hdfs.DFSClient
> (DFSClient.java:closeAllFilesBeingWritten(608)) - Failed to close inode 16387
> java.io.IOException: DataStreamer Exception:
> at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:544)
> at
> org.apache.hadoop.hdfs.StripedDataStreamer.run(StripedDataStreamer.java:1)
> Caused by: java.lang.NullPointerException
> at
> java.util.concurrent.LinkedBlockingQueue.offer(LinkedBlockingQueue.java:374)
> at
> org.apache.hadoop.hdfs.StripedDataStreamer.locateFollowingBlock(StripedDataStreamer.java:157)
> at
> org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1332)
> at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:424)
> ... 1 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
