[jira] [Commented] (HDFS-7060) Avoid taking locks when sending heartbeats from the DataNode

[ https://issues.apache.org/jira/browse/HDFS-7060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17500478#comment-17500478 ] chen commented on HDFS-7060: [~cheersyang]  Over time, the datanode becomes slower and slower; after

[jira] [Commented] (HDFS-7060) Avoid taking locks when sending heartbeats from the DataNode

[ https://issues.apache.org/jira/browse/HDFS-7060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877257#comment-16877257 ] Anes Mukhametov commented on HDFS-7060: --- got the same issue with cdh 5.16 (hadoop 2.6/2.7) > Avoid

[jira] [Commented] (HDFS-7060) Avoid taking locks when sending heartbeats from the DataNode

[ https://issues.apache.org/jira/browse/HDFS-7060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16307926#comment-16307926 ] Weiwei Yang commented on HDFS-7060: --- Apologies for the late response [~hexiaoqiao] and thanks for your

[jira] [Commented] (HDFS-7060) Avoid taking locks when sending heartbeats from the DataNode

[ https://issues.apache.org/jira/browse/HDFS-7060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16305967#comment-16305967 ] He Xiaoqiao commented on HDFS-7060: --- The last patch is good for me. there are some suggestions for

[jira] [Commented] (HDFS-7060) Avoid taking locks when sending heartbeats from the DataNode

[ https://issues.apache.org/jira/browse/HDFS-7060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16303660#comment-16303660 ] He Xiaoqiao commented on HDFS-7060: --- hi [~yangjiandan] [~cheersyang], we encountered this problems as

[jira] [Commented] (HDFS-7060) Avoid taking locks when sending heartbeats from the DataNode

[ https://issues.apache.org/jira/browse/HDFS-7060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16243299#comment-16243299 ] Hudson commented on HDFS-7060: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13198 (See

[jira] [Commented] (HDFS-7060) Avoid taking locks when sending heartbeats from the DataNode

[ https://issues.apache.org/jira/browse/HDFS-7060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16243263#comment-16243263 ] Weiwei Yang commented on HDFS-7060: --- Thanks [~elgoiri], please see the attachment [^HDFS Status Post

[jira] [Commented] (HDFS-7060) Avoid taking locks when sending heartbeats from the DataNode

[ https://issues.apache.org/jira/browse/HDFS-7060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242948#comment-16242948 ] Íñigo Goiri commented on HDFS-7060: --- I went through the failed unit tests and everything is the same OOM

[jira] [Commented] (HDFS-7060) Avoid taking locks when sending heartbeats from the DataNode

[ https://issues.apache.org/jira/browse/HDFS-7060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241442#comment-16241442 ] Weiwei Yang commented on HDFS-7060: --- The UT failures were not related to this patch, they were failing

[jira] [Commented] (HDFS-7060) Avoid taking locks when sending heartbeats from the DataNode

[ https://issues.apache.org/jira/browse/HDFS-7060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241433#comment-16241433 ] Hadoop QA commented on HDFS-7060: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem ||

[jira] [Commented] (HDFS-7060) Avoid taking locks when sending heartbeats from the DataNode

[ https://issues.apache.org/jira/browse/HDFS-7060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241333#comment-16241333 ] Weiwei Yang commented on HDFS-7060: --- Thank you [~elgoiri], I have uploaded v5 patch to format the comment

[jira] [Commented] (HDFS-7060) Avoid taking locks when sending heartbeats from the DataNode

[ https://issues.apache.org/jira/browse/HDFS-7060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16240582#comment-16240582 ] Íñigo Goiri commented on HDFS-7060: --- Thanks [~cheersyang], the comments now make clear the

[jira] [Commented] (HDFS-7060) Avoid taking locks when sending heartbeats from the DataNode

[ https://issues.apache.org/jira/browse/HDFS-7060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16240324#comment-16240324 ] Hadoop QA commented on HDFS-7060: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem ||

[jira] [Commented] (HDFS-7060) Avoid taking locks when sending heartbeats from the DataNode

[ https://issues.apache.org/jira/browse/HDFS-7060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16240293#comment-16240293 ] Weiwei Yang commented on HDFS-7060: --- bq. is it worth making these assumptions explicit in a comment in

[jira] [Commented] (HDFS-7060) Avoid taking locks when sending heartbeats from the DataNode

[ https://issues.apache.org/jira/browse/HDFS-7060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16240291#comment-16240291 ] Weiwei Yang commented on HDFS-7060: --- Hi [~goirix] Thanks for commenting on this issue. The patch was

[jira] [Commented] (HDFS-7060) Avoid taking locks when sending heartbeats from the DataNode

[ https://issues.apache.org/jira/browse/HDFS-7060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16238362#comment-16238362 ] Íñigo Goiri commented on HDFS-7060: --- [^HDFS-7060.003.patch] removed both {{statsLock}} and the

[jira] [Commented] (HDFS-7060) Avoid taking locks when sending heartbeats from the DataNode

[ https://issues.apache.org/jira/browse/HDFS-7060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16224451#comment-16224451 ] Weiwei Yang commented on HDFS-7060: --- Hi [~yangjiandan] Your latest patch looks good to me. Thanks for

[jira] [Commented] (HDFS-7060) Avoid taking locks when sending heartbeats from the DataNode

[ https://issues.apache.org/jira/browse/HDFS-7060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16224402#comment-16224402 ] Hadoop QA commented on HDFS-7060: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem ||

[jira] [Commented] (HDFS-7060) Avoid taking locks when sending heartbeats from the DataNode

[ https://issues.apache.org/jira/browse/HDFS-7060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16221789#comment-16221789 ] Jiandan Yang commented on HDFS-7060: - We have deploy Hadoop including patch of HDFS-7060-003.patch,

[jira] [Commented] (HDFS-7060) Avoid taking locks when sending heartbeats from the DataNode

[ https://issues.apache.org/jira/browse/HDFS-7060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16221765#comment-16221765 ] Jiandan Yang commented on HDFS-7060: - upload HDFS-7060-003.patch > Avoid taking locks when sending

[jira] [Commented] (HDFS-7060) Avoid taking locks when sending heartbeats from the DataNode

[ https://issues.apache.org/jira/browse/HDFS-7060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16212722#comment-16212722 ] Hadoop QA commented on HDFS-7060: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem ||

[jira] [Commented] (HDFS-7060) Avoid taking locks when sending heartbeats from the DataNode

[ https://issues.apache.org/jira/browse/HDFS-7060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16212333#comment-16212333 ] Jiandan Yang commented on HDFS-7060: - [~xinwei] [~brahmareddy] [~jojochuang] We encountered the same

[jira] [Commented] (HDFS-7060) Avoid taking locks when sending heartbeats from the DataNode

[ https://issues.apache.org/jira/browse/HDFS-7060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15783413#comment-15783413 ] Wei-Chiu Chuang commented on HDFS-7060: --- BlockPoolSplice objects in FsVolumeImpl are stored in

[jira] [Commented] (HDFS-7060) Avoid taking locks when sending heartbeats from the DataNode

[ https://issues.apache.org/jira/browse/HDFS-7060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15783407#comment-15783407 ] Hadoop QA commented on HDFS-7060: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem ||

[jira] [Commented] (HDFS-7060) Avoid taking locks when sending heartbeats from the DataNode

[ https://issues.apache.org/jira/browse/HDFS-7060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15783405#comment-15783405 ] Wei-Chiu Chuang commented on HDFS-7060: --- I am trying to revive this thread. The heartbeat issue with

[jira] [Commented] (HDFS-7060) Avoid taking locks when sending heartbeats from the DataNode

[ https://issues.apache.org/jira/browse/HDFS-7060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15215462#comment-15215462 ] Hadoop QA commented on HDFS-7060: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem ||

[jira] [Commented] (HDFS-7060) Avoid taking locks when sending heartbeats from the DataNode

[ https://issues.apache.org/jira/browse/HDFS-7060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15215413#comment-15215413 ] John Zhuge commented on HDFS-7060: -- If we assume dataset lock protects {{bpSlices}}. The following

[jira] [Commented] (HDFS-7060) Avoid taking locks when sending heartbeats from the DataNode

[ https://issues.apache.org/jira/browse/HDFS-7060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15118300#comment-15118300 ] Kihwal Lee commented on HDFS-7060: -- We have recently tried it on a big cluster. I can't tell how much it

[jira] [Commented] (HDFS-7060) Avoid taking locks when sending heartbeats from the DataNode

[ https://issues.apache.org/jira/browse/HDFS-7060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15118535#comment-15118535 ] Tsz Wo Nicholas Sze commented on HDFS-7060: --- Without holding the dataset lock, we may hit

[jira] [Commented] (HDFS-7060) Avoid taking locks when sending heartbeats from the DataNode

[ https://issues.apache.org/jira/browse/HDFS-7060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15118556#comment-15118556 ] Tsz Wo Nicholas Sze commented on HDFS-7060: --- > ... Maybe it will be better with the lifeline

[jira] [Commented] (HDFS-7060) Avoid taking locks when sending heartbeats from the DataNode

[ https://issues.apache.org/jira/browse/HDFS-7060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15023692#comment-15023692 ] Colin Patrick McCabe commented on HDFS-7060: I took another look at this, and it seems

[jira] [Commented] (HDFS-7060) Avoid taking locks when sending heartbeats from the DataNode

[ https://issues.apache.org/jira/browse/HDFS-7060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15018828#comment-15018828 ] Nathan Roberts commented on HDFS-7060: -- Has anyone given any further thought on this patch? It seems

[jira] [Commented] (HDFS-7060) Avoid taking locks when sending heartbeats from the DataNode

[ https://issues.apache.org/jira/browse/HDFS-7060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15019286#comment-15019286 ] Haohui Mai commented on HDFS-7060: -- Hi [~nroberts], we're being conservative as you never know whether it

[jira] [Commented] (HDFS-7060) Avoid taking locks when sending heartbeats from the DataNode

[ https://issues.apache.org/jira/browse/HDFS-7060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14516265#comment-14516265 ] Xinwei Qin commented on HDFS-7060: --- [~jnp] Thanks for your comment. This test was done

[jira] [Commented] (HDFS-7060) Avoid taking locks when sending heartbeats from the DataNode

[ https://issues.apache.org/jira/browse/HDFS-7060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14514511#comment-14514511 ] Jitendra Nath Pandey commented on HDFS-7060: bq. We create 400 threads to write

[jira] [Commented] (HDFS-7060) Avoid taking locks when sending heartbeats from the DataNode

[ https://issues.apache.org/jira/browse/HDFS-7060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14481372#comment-14481372 ] Brahma Reddy Battula commented on HDFS-7060: [~cmccabe] and [~wheat9] can you

[jira] [Commented] (HDFS-7060) Avoid taking locks when sending heartbeats from the DataNode

[ https://issues.apache.org/jira/browse/HDFS-7060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14392672#comment-14392672 ] Xinwei Qin commented on HDFS-7060: --- [~cmccabe] , [~wheat9] Thanks a lot for your

[jira] [Commented] (HDFS-7060) Avoid taking locks when sending heartbeats from the DataNode

[ https://issues.apache.org/jira/browse/HDFS-7060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14391982#comment-14391982 ] Brahma Reddy Battula commented on HDFS-7060: Thanks a lot for taking look into

[jira] [Commented] (HDFS-7060) Avoid taking locks when sending heartbeats from the DataNode

[ https://issues.apache.org/jira/browse/HDFS-7060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14391538#comment-14391538 ] Colin Patrick McCabe commented on HDFS-7060: [~Xinwei Qin], [~brahmareddy], do

[jira] [Commented] (HDFS-7060) Avoid taking locks when sending heartbeats from the DataNode

[ https://issues.apache.org/jira/browse/HDFS-7060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14391604#comment-14391604 ] Haohui Mai commented on HDFS-7060: -- I agree with Colin that it needs to be well thought