[jira] [Commented] (HDFS-7784) load fsimage in parallel

2019-07-19 Thread Hadoop QA (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16889199#comment-16889199 ] Hadoop QA commented on HDFS-7784: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem ||

[jira] [Commented] (HDFS-7784) load fsimage in parallel

2017-01-03 Thread Gang Xie (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15796824#comment-15796824 ] Gang Xie commented on HDFS-7784: About the GC activities, the following is the gstat output. And it do

[jira] [Commented] (HDFS-7784) load fsimage in parallel

2017-01-03 Thread Gang Xie (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15796809#comment-15796809 ] Gang Xie commented on HDFS-7784: The hardware info: CPU: Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz with 24

[jira] [Commented] (HDFS-7784) load fsimage in parallel

2017-01-03 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15796804#comment-15796804 ] Kai Zheng commented on HDFS-7784: - OOO today for customer visit, please expect delayed response. Thanks.

[jira] [Commented] (HDFS-7784) load fsimage in parallel

2017-01-03 Thread Gang Xie (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15796797#comment-15796797 ] Gang Xie commented on HDFS-7784: The JVM setting: -Xmx102400m -Xms102400m -Xmn5508m

[jira] [Commented] (HDFS-7784) load fsimage in parallel

2017-01-03 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15795396#comment-15795396 ] Kihwal Lee commented on HDFS-7784: -- [~xiegang112], when you have a chance to test the performance, please

[jira] [Commented] (HDFS-7784) load fsimage in parallel

2016-12-28 Thread Gang Xie (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15784660#comment-15784660 ] Gang Xie commented on HDFS-7784: After make the AclStorage synchronized, it could works. the loading time

[jira] [Commented] (HDFS-7784) load fsimage in parallel

2016-12-28 Thread Gang Xie (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15784285#comment-15784285 ] Gang Xie commented on HDFS-7784: Found a potential issue while trying to back port this patch to 2.4. Pls

[jira] [Commented] (HDFS-7784) load fsimage in parallel

2016-12-26 Thread Gang Xie (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15777928#comment-15777928 ] Gang Xie commented on HDFS-7784: Hello, Any update about this improvement? Loading huge image really takes

[jira] [Commented] (HDFS-7784) load fsimage in parallel

2015-12-21 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15067073#comment-15067073 ] Kihwal Lee commented on HDFS-7784: -- bq. protobuf seems to generate a lot of garbage during startup,

[jira] [Commented] (HDFS-7784) load fsimage in parallel

2015-12-21 Thread Colin Patrick McCabe (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15066949#comment-15066949 ] Colin Patrick McCabe commented on HDFS-7784: Thanks, [~kihwal]. Unfortunately, that's what

[jira] [Commented] (HDFS-7784) load fsimage in parallel

2015-12-18 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15064274#comment-15064274 ] Kihwal Lee commented on HDFS-7784: -- bq. ... find out that the bottleneck is deserialization taking too

[jira] [Commented] (HDFS-7784) load fsimage in parallel

2015-02-24 Thread Walter Su (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14335929#comment-14335929 ] Walter Su commented on HDFS-7784: - I use visualvm to profile the loading process and find

[jira] [Commented] (HDFS-7784) load fsimage in parallel

2015-02-17 Thread Colin Patrick McCabe (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14325162#comment-14325162 ] Colin Patrick McCabe commented on HDFS-7784: At the end of the day, there are

[jira] [Commented] (HDFS-7784) load fsimage in parallel

2015-02-13 Thread Hadoop QA (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14319717#comment-14319717 ] Hadoop QA commented on HDFS-7784: - {color:red}-1 overall{color}. Here are the results of

[jira] [Commented] (HDFS-7784) load fsimage in parallel

2015-02-13 Thread Hadoop QA (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14319840#comment-14319840 ] Hadoop QA commented on HDFS-7784: - {color:red}-1 overall{color}. Here are the results of

[jira] [Commented] (HDFS-7784) load fsimage in parallel

2015-02-13 Thread Walter Su (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14320201#comment-14320201 ] Walter Su commented on HDFS-7784: - I agree with you. A single Namenode with 64GB memory can

[jira] [Commented] (HDFS-7784) load fsimage in parallel

2015-02-13 Thread Walter Su (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14320210#comment-14320210 ] Walter Su commented on HDFS-7784: - I mean fsimage loading time drops from 371s to 159s. And

[jira] [Commented] (HDFS-7784) load fsimage in parallel

2015-02-13 Thread Walter Su (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14320261#comment-14320261 ] Walter Su commented on HDFS-7784: - In my testing, the memory usage doesn't grow. GC doesn't

[jira] [Commented] (HDFS-7784) load fsimage in parallel

2015-02-13 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14321075#comment-14321075 ] Kai Zheng commented on HDFS-7784: - Hi [~walter.k.su], It's interesting, thanks ! bq.So I

[jira] [Commented] (HDFS-7784) load fsimage in parallel

2015-02-12 Thread Haohui Mai (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14319115#comment-14319115 ] Haohui Mai commented on HDFS-7784: -- I've done some experiments in HDFS-5698. Parallelism

[jira] [Commented] (HDFS-7784) load fsimage in parallel

2015-02-12 Thread Colin Patrick McCabe (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14319097#comment-14319097 ] Colin Patrick McCabe commented on HDFS-7784: Hi Walter, this is an interesting

[jira] [Commented] (HDFS-7784) load fsimage in parallel

2015-02-12 Thread Walter Su (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14319684#comment-14319684 ] Walter Su commented on HDFS-7784: - I'll upload performance test results in 4 hours. load