[ 
https://issues.apache.org/jira/browse/HDFS-10429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16118235#comment-16118235
 ] 

Andras Bokor edited comment on HDFS-10429 at 8/8/17 11:06 PM:
--------------------------------------------------------------

I am not confident in this area but I tried out the patch on a cluster with 4 
datanodes.
I have tested with 12000 random files between 1-10 MB. Before the patch I got 
exceptions. After the patch I run the copy 3 times (which means 36000 files) 
without exception so it seems the patch fixes the issue.
Related the code of patch:
* We should consider introducing some checks in finally to not to call close 
methods twice. Something like {{if(!getStreamer().isSocketClosed() || 
!getStreamer().isAlive())}} and call forced close if something is not closed 
properly. What do you think?
* I would not hide the InterruptedExceptions where it's not expected. It can 
show when our thread handling is not clean like in this case.

Other than the two bullet points I'd give a non-binding +1 since my test shows 
that the exceptions disappeared and all the files were copied to HDFS.


was (Author: boky01):
I am not confident in this area but I tried out the patch on a cluster with 4 
datanodes.
I have tested with 12000 random files between 1-10 MB. Before the patch I got 
exceptions. After the patch I run the copy 3 times (which means 36000 files) 
without exception so it seems the patch fixes the issue.
Related the code of patch:
* We should consider introducing some checks in finally to not to call close 
methods twice. Something like {{if(!getStreamer().isSocketClosed() || 
!getStreamer().isAlive())}} and call forced close if something is not closed 
properly. What do you think?
* I would not hide the InterruptedExceptions where it's not expected. It can 
show when our thread handling is not clean like in this case.

Other than the two bullet points I'd give a non-binding +1 since my test shows 
that the exceptions disappeared but and all the files were copied to HDFS.

> DataStreamer interrupted warning  always appears when using CLI upload file
> ---------------------------------------------------------------------------
>
>                 Key: HDFS-10429
>                 URL: https://issues.apache.org/jira/browse/HDFS-10429
>             Project: Hadoop HDFS
>          Issue Type: Bug
>    Affects Versions: 2.7.3
>            Reporter: Zhiyuan Yang
>            Assignee: Wei-Chiu Chuang
>            Priority: Minor
>         Attachments: HDFS-10429.1.patch, HDFS-10429.2.patch, 
> HDFS-10429.3.patch
>
>
> Every time I use 'hdfs dfs -put' upload file, this warning is printed:
> {code:java}
> 16/05/18 20:57:56 WARN hdfs.DataStreamer: Caught exception
> java.lang.InterruptedException
>       at java.lang.Object.wait(Native Method)
>       at java.lang.Thread.join(Thread.java:1245)
>       at java.lang.Thread.join(Thread.java:1319)
>       at 
> org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:871)
>       at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:519)
>       at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:696)
> {code}
> The reason is this: originally, DataStreamer::closeResponder always prints a 
> warning about InterruptedException; since HDFS-9812, 
> DFSOutputStream::closeImpl  always forces threads to close, which causes 
> InterruptedException.
> A simple fix is to use debug level log instead of warning level.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

Reply via email to