[jira] Commented: (HDFS-1479) Massive file deletion causes some timeouts in writers
[
https://issues.apache.org/jira/browse/HDFS-1479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12927633#action_12927633
]
Thanh Do commented on HDFS-1479:
Thanks, Zheng for the explanation.
The reason I couldn't find the AsyncDiskService because I was looking at 0.20.2
where deletion at datanode is done synchronously. Now I find it in 0.21.0.
In general, how do you plan to fix this?
> Massive file deletion causes some timeouts in writers
> -
>
> Key: HDFS-1479
> URL: https://issues.apache.org/jira/browse/HDFS-1479
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 0.20.2
>Reporter: Zheng Shao
>Assignee: Zheng Shao
>Priority: Minor
>
> When we do a massive deletion of files, we saw some timeouts in writers who's
> writing to HDFS. This does not happen to all DataNodes, but it's happening
> regularly enough that we would like to fix it.
> {code}
> yyy.xxx.com: 10/10/25 00:55:32 WARN hdfs.DFSClient: DFSOutputStream
> ResponseProcessor exception for block
> blk_-5459995953259765112_37619608java.net.SocketTimeoutException: 69000
> millis timeout while waiting for channel to be ready for read. ch :
> java.nio.channels.SocketChannel[connected local=/10.10.10.10:56319
> remote=/10.10.10.10:50010]
> {code}
> This is caused by the default setting of AsyncDiskService, which starts 4
> threads per volume to delete files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1479) Massive file deletion causes some timeouts in writers
[
https://issues.apache.org/jira/browse/HDFS-1479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12927581#action_12927581
]
Zheng Shao commented on HDFS-1479:
--
Thanh, we have a service that keeps the last 5 days of log data on HDFS. We do
a deletion every day to remove 1/5 of the files, which is a massive deletion.
The java file is
org/apache/hadoop/hdfs/server/datanode/FSDatasetAsyncDiskService.java
> Massive file deletion causes some timeouts in writers
> -
>
> Key: HDFS-1479
> URL: https://issues.apache.org/jira/browse/HDFS-1479
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 0.20.2
>Reporter: Zheng Shao
>Assignee: Zheng Shao
>Priority: Minor
>
> When we do a massive deletion of files, we saw some timeouts in writers who's
> writing to HDFS. This does not happen to all DataNodes, but it's happening
> regularly enough that we would like to fix it.
> {code}
> yyy.xxx.com: 10/10/25 00:55:32 WARN hdfs.DFSClient: DFSOutputStream
> ResponseProcessor exception for block
> blk_-5459995953259765112_37619608java.net.SocketTimeoutException: 69000
> millis timeout while waiting for channel to be ready for read. ch :
> java.nio.channels.SocketChannel[connected local=/10.10.10.10:56319
> remote=/10.10.10.10:50010]
> {code}
> This is caused by the default setting of AsyncDiskService, which starts 4
> threads per volume to delete files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1479) Massive file deletion causes some timeouts in writers
[
https://issues.apache.org/jira/browse/HDFS-1479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12927578#action_12927578
]
Thanh Do commented on HDFS-1479:
Zheng, can you explain why you need massive deletion? That is what kinds of
application require such operation? Thanks
> Massive file deletion causes some timeouts in writers
> -
>
> Key: HDFS-1479
> URL: https://issues.apache.org/jira/browse/HDFS-1479
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 0.20.2
>Reporter: Zheng Shao
>Assignee: Zheng Shao
>Priority: Minor
>
> When we do a massive deletion of files, we saw some timeouts in writers who's
> writing to HDFS. This does not happen to all DataNodes, but it's happening
> regularly enough that we would like to fix it.
> {code}
> yyy.xxx.com: 10/10/25 00:55:32 WARN hdfs.DFSClient: DFSOutputStream
> ResponseProcessor exception for block
> blk_-5459995953259765112_37619608java.net.SocketTimeoutException: 69000
> millis timeout while waiting for channel to be ready for read. ch :
> java.nio.channels.SocketChannel[connected local=/10.10.10.10:56319
> remote=/10.10.10.10:50010]
> {code}
> This is caused by the default setting of AsyncDiskService, which starts 4
> threads per volume to delete files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1479) Massive file deletion causes some timeouts in writers
[
https://issues.apache.org/jira/browse/HDFS-1479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926159#action_12926159
]
Thanh Do commented on HDFS-1479:
Thanks Dhruba.
I try to grep for AsyncDiskService but cannot find it any where in the source
tree.
Do i misspell it?
> Massive file deletion causes some timeouts in writers
> -
>
> Key: HDFS-1479
> URL: https://issues.apache.org/jira/browse/HDFS-1479
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 0.20.2
>Reporter: Zheng Shao
>Assignee: Zheng Shao
>Priority: Minor
>
> When we do a massive deletion of files, we saw some timeouts in writers who's
> writing to HDFS. This does not happen to all DataNodes, but it's happening
> regularly enough that we would like to fix it.
> {code}
> yyy.xxx.com: 10/10/25 00:55:32 WARN hdfs.DFSClient: DFSOutputStream
> ResponseProcessor exception for block
> blk_-5459995953259765112_37619608java.net.SocketTimeoutException: 69000
> millis timeout while waiting for channel to be ready for read. ch :
> java.nio.channels.SocketChannel[connected local=/10.10.10.10:56319
> remote=/10.10.10.10:50010]
> {code}
> This is caused by the default setting of AsyncDiskService, which starts 4
> threads per volume to delete files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1479) Massive file deletion causes some timeouts in writers
[
https://issues.apache.org/jira/browse/HDFS-1479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925680#action_12925680
]
dhruba borthakur commented on HDFS-1479:
An app decided to delete a large bunch of files from HDFS within a short period
of time. The namenode, in turn, told the relevant datanodes to delete the
blocks. Block deletion occurs in parallel on datanodes from multiple threads,
this put enough load on the datanode's storage device that writes to other
blocks on those peices of storage became very slow
> Massive file deletion causes some timeouts in writers
> -
>
> Key: HDFS-1479
> URL: https://issues.apache.org/jira/browse/HDFS-1479
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 0.20.2
>Reporter: Zheng Shao
>Assignee: Zheng Shao
>Priority: Minor
>
> When we do a massive deletion of files, we saw some timeouts in writers who's
> writing to HDFS. This does not happen to all DataNodes, but it's happening
> regularly enough that we would like to fix it.
> {code}
> yyy.xxx.com: 10/10/25 00:55:32 WARN hdfs.DFSClient: DFSOutputStream
> ResponseProcessor exception for block
> blk_-5459995953259765112_37619608java.net.SocketTimeoutException: 69000
> millis timeout while waiting for channel to be ready for read. ch :
> java.nio.channels.SocketChannel[connected local=/10.10.10.10:56319
> remote=/10.10.10.10:50010]
> {code}
> This is caused by the default setting of AsyncDiskService, which starts 4
> threads per volume to delete files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1479) Massive file deletion causes some timeouts in writers
[
https://issues.apache.org/jira/browse/HDFS-1479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925657#action_12925657
]
Thanh Do commented on HDFS-1479:
Is this due to the block deleting policy in HDFS?
> Massive file deletion causes some timeouts in writers
> -
>
> Key: HDFS-1479
> URL: https://issues.apache.org/jira/browse/HDFS-1479
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 0.20.2
>Reporter: Zheng Shao
>Assignee: Zheng Shao
>Priority: Minor
>
> When we do a massive deletion of files, we saw some timeouts in writers who's
> writing to HDFS. This does not happen to all DataNodes, but it's happening
> regularly enough that we would like to fix it.
> {code}
> yyy.xxx.com: 10/10/25 00:55:32 WARN hdfs.DFSClient: DFSOutputStream
> ResponseProcessor exception for block
> blk_-5459995953259765112_37619608java.net.SocketTimeoutException: 69000
> millis timeout while waiting for channel to be ready for read. ch :
> java.nio.channels.SocketChannel[connected local=/10.10.10.10:56319
> remote=/10.10.10.10:50010]
> {code}
> This is caused by the default setting of AsyncDiskService, which starts 4
> threads per volume to delete files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1479) Massive file deletion causes some timeouts in writers
[
https://issues.apache.org/jira/browse/HDFS-1479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925530#action_12925530
]
Thanh Do commented on HDFS-1479:
Can you give the detail scenario?
> Massive file deletion causes some timeouts in writers
> -
>
> Key: HDFS-1479
> URL: https://issues.apache.org/jira/browse/HDFS-1479
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 0.20.2
>Reporter: Zheng Shao
>Assignee: Zheng Shao
>Priority: Minor
>
> When we do a massive deletion of files, we saw some timeouts in writers who's
> writing to HDFS. This does not happen to all DataNodes, but it's happening
> regularly enough that we would like to fix it.
> {code}
> yyy.xxx.com: 10/10/25 00:55:32 WARN hdfs.DFSClient: DFSOutputStream
> ResponseProcessor exception for block
> blk_-5459995953259765112_37619608java.net.SocketTimeoutException: 69000
> millis timeout while waiting for channel to be ready for read. ch :
> java.nio.channels.SocketChannel[connected local=/10.10.10.10:56319
> remote=/10.10.10.10:50010]
> {code}
> This is caused by the default setting of AsyncDiskService, which starts 4
> threads per volume to delete files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
