[jira] [Commented] (HADOOP-12107) long running apps may have a huge number of StatisticsData instances under FileSystem

[ https://issues.apache.org/jira/browse/HADOOP-12107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209500#comment-15209500 ] Jason Lowe commented on HADOOP-12107: - This appears to be triggering OOM errors in Tez during task

[jira] [Commented] (HADOOP-12107) long running apps may have a huge number of StatisticsData instances under FileSystem

[ https://issues.apache.org/jira/browse/HADOOP-12107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15099221#comment-15099221 ] Hudson commented on HADOOP-12107: - FAILURE: Integrated in Hadoop-trunk-Commit #9116 (See

[jira] [Commented] (HADOOP-12107) long running apps may have a huge number of StatisticsData instances under FileSystem

[ https://issues.apache.org/jira/browse/HADOOP-12107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15096632#comment-15096632 ] Jason Lowe commented on HADOOP-12107: - I tried pulling this into branch-2.7. It comes in cleanly

[jira] [Commented] (HADOOP-12107) long running apps may have a huge number of StatisticsData instances under FileSystem

[ https://issues.apache.org/jira/browse/HADOOP-12107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15089972#comment-15089972 ] Jason Lowe commented on HADOOP-12107: - I recently ran across this on a NodeManager running 2.6 that

[jira] [Commented] (HADOOP-12107) long running apps may have a huge number of StatisticsData instances under FileSystem

[ https://issues.apache.org/jira/browse/HADOOP-12107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090150#comment-15090150 ] Sangjin Lee commented on HADOOP-12107: -- +1 > long running apps may have a huge number of

[jira] [Commented] (HADOOP-12107) long running apps may have a huge number of StatisticsData instances under FileSystem

[ https://issues.apache.org/jira/browse/HADOOP-12107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14610306#comment-14610306 ] Hudson commented on HADOOP-12107: - FAILURE: Integrated in Hadoop-Hdfs-trunk #2172 (See

[jira] [Commented] (HADOOP-12107) long running apps may have a huge number of StatisticsData instances under FileSystem

[ https://issues.apache.org/jira/browse/HADOOP-12107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14610348#comment-14610348 ] Hudson commented on HADOOP-12107: - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #233

[jira] [Commented] (HADOOP-12107) long running apps may have a huge number of StatisticsData instances under FileSystem

[ https://issues.apache.org/jira/browse/HADOOP-12107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14608516#comment-14608516 ] Hudson commented on HADOOP-12107: - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2190

[jira] [Commented] (HADOOP-12107) long running apps may have a huge number of StatisticsData instances under FileSystem

[ https://issues.apache.org/jira/browse/HADOOP-12107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14608460#comment-14608460 ] Hudson commented on HADOOP-12107: - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8

[jira] [Commented] (HADOOP-12107) long running apps may have a huge number of StatisticsData instances under FileSystem

[ https://issues.apache.org/jira/browse/HADOOP-12107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14609256#comment-14609256 ] Colin Patrick McCabe commented on HADOOP-12107: --- Thanks, guys. long

[jira] [Commented] (HADOOP-12107) long running apps may have a huge number of StatisticsData instances under FileSystem

[ https://issues.apache.org/jira/browse/HADOOP-12107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14606431#comment-14606431 ] Gera Shegalov commented on HADOOP-12107: bq. You could make the same argument to

[jira] [Commented] (HADOOP-12107) long running apps may have a huge number of StatisticsData instances under FileSystem

[ https://issues.apache.org/jira/browse/HADOOP-12107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14606271#comment-14606271 ] Colin Patrick McCabe commented on HADOOP-12107: --- Guys, we clearly define

[jira] [Commented] (HADOOP-12107) long running apps may have a huge number of StatisticsData instances under FileSystem

[ https://issues.apache.org/jira/browse/HADOOP-12107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14606495#comment-14606495 ] Hudson commented on HADOOP-12107: - FAILURE: Integrated in Hadoop-trunk-Commit #8089 (See

[jira] [Commented] (HADOOP-12107) long running apps may have a huge number of StatisticsData instances under FileSystem

[ https://issues.apache.org/jira/browse/HADOOP-12107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14606514#comment-14606514 ] Ming Ma commented on HADOOP-12107: -- Also thanks for [~walter.k.su] and [~sandyr] for

[jira] [Commented] (HADOOP-12107) long running apps may have a huge number of StatisticsData instances under FileSystem

[ https://issues.apache.org/jira/browse/HADOOP-12107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14606509#comment-14606509 ] Ming Ma commented on HADOOP-12107: -- I have committed this to trunk and branch-2. Thanks

[jira] [Commented] (HADOOP-12107) long running apps may have a huge number of StatisticsData instances under FileSystem

[ https://issues.apache.org/jira/browse/HADOOP-12107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14606552#comment-14606552 ] Sangjin Lee commented on HADOOP-12107: -- Thanks [~mingma] for the commit! Many

[jira] [Commented] (HADOOP-12107) long running apps may have a huge number of StatisticsData instances under FileSystem

[ https://issues.apache.org/jira/browse/HADOOP-12107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14603218#comment-14603218 ] Sandy Ryza commented on HADOOP-12107: - I don't know of anyone using the

[jira] [Commented] (HADOOP-12107) long running apps may have a huge number of StatisticsData instances under FileSystem

[ https://issues.apache.org/jira/browse/HADOOP-12107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14603211#comment-14603211 ] Sangjin Lee commented on HADOOP-12107: -- [~cmccabe], your thoughts on this?

[jira] [Commented] (HADOOP-12107) long running apps may have a huge number of StatisticsData instances under FileSystem

[ https://issues.apache.org/jira/browse/HADOOP-12107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14603216#comment-14603216 ] Sangjin Lee commented on HADOOP-12107: -- FWIW, when I look at the spark code (which

[jira] [Commented] (HADOOP-12107) long running apps may have a huge number of StatisticsData instances under FileSystem

[ https://issues.apache.org/jira/browse/HADOOP-12107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14603433#comment-14603433 ] Colin Patrick McCabe commented on HADOOP-12107: --- OK, at the risk of being

[jira] [Commented] (HADOOP-12107) long running apps may have a huge number of StatisticsData instances under FileSystem

[ https://issues.apache.org/jira/browse/HADOOP-12107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14603982#comment-14603982 ] Gera Shegalov commented on HADOOP-12107: I agree with the sentiment what API

[jira] [Commented] (HADOOP-12107) long running apps may have a huge number of StatisticsData instances under FileSystem

[ https://issues.apache.org/jira/browse/HADOOP-12107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14602230#comment-14602230 ] Gera Shegalov commented on HADOOP-12107: bq StatisticsData is a public class,

[jira] [Commented] (HADOOP-12107) long running apps may have a huge number of StatisticsData instances under FileSystem

[ https://issues.apache.org/jira/browse/HADOOP-12107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14600376#comment-14600376 ] Sangjin Lee commented on HADOOP-12107: -- I believe the test failure is unrelated.

[jira] [Commented] (HADOOP-12107) long running apps may have a huge number of StatisticsData instances under FileSystem

[ https://issues.apache.org/jira/browse/HADOOP-12107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14600361#comment-14600361 ] Hadoop QA commented on HADOOP-12107: \\ \\ | (x) *{color:red}-1 overall{color}* | \\

[jira] [Commented] (HADOOP-12107) long running apps may have a huge number of StatisticsData instances under FileSystem

[ https://issues.apache.org/jira/browse/HADOOP-12107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14599861#comment-14599861 ] Colin Patrick McCabe commented on HADOOP-12107: --- bq. 003 patch is good. I

[jira] [Commented] (HADOOP-12107) long running apps may have a huge number of StatisticsData instances under FileSystem

[ https://issues.apache.org/jira/browse/HADOOP-12107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14597851#comment-14597851 ] Sangjin Lee commented on HADOOP-12107: -- Thanks for pointing that out

[jira] [Commented] (HADOOP-12107) long running apps may have a huge number of StatisticsData instances under FileSystem

[ https://issues.apache.org/jira/browse/HADOOP-12107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14598388#comment-14598388 ] Ming Ma commented on HADOOP-12107: -- Thanks [~sjlee0]. Latest patch LGTM. [~cmccabe],

[jira] [Commented] (HADOOP-12107) long running apps may have a huge number of StatisticsData instances under FileSystem

[ https://issues.apache.org/jira/browse/HADOOP-12107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14598397#comment-14598397 ] Gera Shegalov commented on HADOOP-12107: Thanks for v3 [~sjlee0]!

[jira] [Commented] (HADOOP-12107) long running apps may have a huge number of StatisticsData instances under FileSystem

[ https://issues.apache.org/jira/browse/HADOOP-12107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14597297#comment-14597297 ] Walter Su commented on HADOOP-12107: bq. This will create one additional thread per

[jira] [Commented] (HADOOP-12107) long running apps may have a huge number of StatisticsData instances under FileSystem

[ https://issues.apache.org/jira/browse/HADOOP-12107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14598617#comment-14598617 ] Gera Shegalov commented on HADOOP-12107: +1, 004 LGTM long running apps may

[jira] [Commented] (HADOOP-12107) long running apps may have a huge number of StatisticsData instances under FileSystem

[ https://issues.apache.org/jira/browse/HADOOP-12107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14598570#comment-14598570 ] Hadoop QA commented on HADOOP-12107: \\ \\ | (x) *{color:red}-1 overall{color}* | \\

[jira] [Commented] (HADOOP-12107) long running apps may have a huge number of StatisticsData instances under FileSystem

[ https://issues.apache.org/jira/browse/HADOOP-12107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596117#comment-14596117 ] Ming Ma commented on HADOOP-12107: -- Thanks, [~sjlee0], great find. It overall looks

[jira] [Commented] (HADOOP-12107) long running apps may have a huge number of StatisticsData instances under FileSystem

[ https://issues.apache.org/jira/browse/HADOOP-12107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596543#comment-14596543 ] Colin Patrick McCabe commented on HADOOP-12107: --- Thanks for looking at

[jira] [Commented] (HADOOP-12107) long running apps may have a huge number of StatisticsData instances under FileSystem

[ https://issues.apache.org/jira/browse/HADOOP-12107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596951#comment-14596951 ] Gera Shegalov commented on HADOOP-12107: Between a single thread and a thread

[jira] [Commented] (HADOOP-12107) long running apps may have a huge number of StatisticsData instances under FileSystem

[ https://issues.apache.org/jira/browse/HADOOP-12107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596806#comment-14596806 ] Sangjin Lee commented on HADOOP-12107: -- Thanks [~mingma] and [~cmccabe] for your

[jira] [Commented] (HADOOP-12107) long running apps may have a huge number of StatisticsData instances under FileSystem

[ https://issues.apache.org/jira/browse/HADOOP-12107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14597146#comment-14597146 ] Hadoop QA commented on HADOOP-12107: \\ \\ | (x) *{color:red}-1 overall{color}* | \\

[jira] [Commented] (HADOOP-12107) long running apps may have a huge number of StatisticsData instances under FileSystem

[ https://issues.apache.org/jira/browse/HADOOP-12107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14597153#comment-14597153 ] Sangjin Lee commented on HADOOP-12107: -- The checkstyle warning: {panel}

[jira] [Commented] (HADOOP-12107) long running apps may have a huge number of StatisticsData instances under FileSystem

[ https://issues.apache.org/jira/browse/HADOOP-12107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14597058#comment-14597058 ] Hadoop QA commented on HADOOP-12107: \\ \\ | (x) *{color:red}-1 overall{color}* | \\

[jira] [Commented] (HADOOP-12107) long running apps may have a huge number of StatisticsData instances under FileSystem

[ https://issues.apache.org/jira/browse/HADOOP-12107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594570#comment-14594570 ] Hadoop QA commented on HADOOP-12107: \\ \\ | (x) *{color:red}-1 overall{color}* | \\

[jira] [Commented] (HADOOP-12107) long running apps may have a huge number of StatisticsData instances under FileSystem

[ https://issues.apache.org/jira/browse/HADOOP-12107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594054#comment-14594054 ] Sangjin Lee commented on HADOOP-12107: -- The key method in clearing this memory is