[jira] [Commented] (HBASE-9983) Lower the memory footprint of HRegionLocation
[ https://issues.apache.org/jira/browse/HBASE-9983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13825598#comment-13825598 ] Nicolas Liochon commented on HBASE-9983: I haven't done a measure before/after for this one (would have been interested, I agree). Right now, so with this jira included, I'm seeing (with Yourkit) that an entry the location skip list takes ~550 bytes. This will be lowered with HBASE-9976 and HBASE-9988. Then we have one ServerName instance by entry as well... > Lower the memory footprint of HRegionLocation > - > > Key: HBASE-9983 > URL: https://issues.apache.org/jira/browse/HBASE-9983 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 0.98.0, 0.96.0 >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon >Priority: Minor > Fix For: 0.98.0, 0.96.1 > > Attachments: 9983.v2.patch, unbloated.patch > > > It"s more or less a desperate cause... > If we have a lot of regions, the size of this class becomes a problem. It's > under optimized for this case... > This patch is just about the trivial part -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9983) Lower the memory footprint of HRegionLocation
[ https://issues.apache.org/jira/browse/HBASE-9983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13825576#comment-13825576 ] Vladimir Rodionov commented on HBASE-9983: -- Its always good to measure the improvement. Any numbers, Nicolas, you can share? > Lower the memory footprint of HRegionLocation > - > > Key: HBASE-9983 > URL: https://issues.apache.org/jira/browse/HBASE-9983 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 0.98.0, 0.96.0 >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon >Priority: Minor > Fix For: 0.98.0, 0.96.1 > > Attachments: 9983.v2.patch, unbloated.patch > > > It"s more or less a desperate cause... > If we have a lot of regions, the size of this class becomes a problem. It's > under optimized for this case... > This patch is just about the trivial part -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9983) Lower the memory footprint of HRegionLocation
[ https://issues.apache.org/jira/browse/HBASE-9983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13825179#comment-13825179 ] Nicolas Liochon commented on HBASE-9983: Note that I've closed the issue as I committed the patch, but more work is needed... > Lower the memory footprint of HRegionLocation > - > > Key: HBASE-9983 > URL: https://issues.apache.org/jira/browse/HBASE-9983 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 0.98.0, 0.96.0 >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon >Priority: Minor > Fix For: 0.98.0, 0.96.1 > > Attachments: 9983.v2.patch, unbloated.patch > > > It"s more or less a desperate cause... > If we have a lot of regions, the size of this class becomes a problem. It's > under optimized for this case... > This patch is just about the trivial part -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9983) Lower the memory footprint of HRegionLocation
[ https://issues.apache.org/jira/browse/HBASE-9983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13824883#comment-13824883 ] Hudson commented on HBASE-9983: --- SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #840 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/840/]) HBASE-9983 Lower the memory footprint of HRegionLocation (nkeywal: rev 1542694) * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/HRegionLocation.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java > Lower the memory footprint of HRegionLocation > - > > Key: HBASE-9983 > URL: https://issues.apache.org/jira/browse/HBASE-9983 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 0.98.0, 0.96.0 >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon >Priority: Minor > Fix For: 0.98.0, 0.96.1 > > Attachments: 9983.v2.patch, unbloated.patch > > > It"s more or less a desperate cause... > If we have a lot of regions, the size of this class becomes a problem. It's > under optimized for this case... > This patch is just about the trivial part -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9983) Lower the memory footprint of HRegionLocation
[ https://issues.apache.org/jira/browse/HBASE-9983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13824867#comment-13824867 ] Hudson commented on HBASE-9983: --- SUCCESS: Integrated in HBase-TRUNK #4683 (See [https://builds.apache.org/job/HBase-TRUNK/4683/]) HBASE-9983 Lower the memory footprint of HRegionLocation (nkeywal: rev 1542694) * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/HRegionLocation.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java > Lower the memory footprint of HRegionLocation > - > > Key: HBASE-9983 > URL: https://issues.apache.org/jira/browse/HBASE-9983 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 0.98.0, 0.96.0 >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon >Priority: Minor > Fix For: 0.98.0, 0.96.1 > > Attachments: 9983.v2.patch, unbloated.patch > > > It"s more or less a desperate cause... > If we have a lot of regions, the size of this class becomes a problem. It's > under optimized for this case... > This patch is just about the trivial part -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9983) Lower the memory footprint of HRegionLocation
[ https://issues.apache.org/jira/browse/HBASE-9983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13824869#comment-13824869 ] Hudson commented on HBASE-9983: --- SUCCESS: Integrated in hbase-0.96 #192 (See [https://builds.apache.org/job/hbase-0.96/192/]) HBASE-9983 Lower the memory footprint of HRegionLocation (nkeywal: rev 1542696) * /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java * /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/HRegionLocation.java * /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java * /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java > Lower the memory footprint of HRegionLocation > - > > Key: HBASE-9983 > URL: https://issues.apache.org/jira/browse/HBASE-9983 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 0.98.0, 0.96.0 >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon >Priority: Minor > Fix For: 0.98.0, 0.96.1 > > Attachments: 9983.v2.patch, unbloated.patch > > > It"s more or less a desperate cause... > If we have a lot of regions, the size of this class becomes a problem. It's > under optimized for this case... > This patch is just about the trivial part -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9983) Lower the memory footprint of HRegionLocation
[ https://issues.apache.org/jira/browse/HBASE-9983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13824865#comment-13824865 ] Hudson commented on HBASE-9983: --- FAILURE: Integrated in hbase-0.96-hadoop2 #121 (See [https://builds.apache.org/job/hbase-0.96-hadoop2/121/]) HBASE-9983 Lower the memory footprint of HRegionLocation (nkeywal: rev 1542696) * /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java * /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/HRegionLocation.java * /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java * /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java > Lower the memory footprint of HRegionLocation > - > > Key: HBASE-9983 > URL: https://issues.apache.org/jira/browse/HBASE-9983 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 0.98.0, 0.96.0 >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon >Priority: Minor > Fix For: 0.98.0, 0.96.1 > > Attachments: 9983.v2.patch, unbloated.patch > > > It"s more or less a desperate cause... > If we have a lot of regions, the size of this class becomes a problem. It's > under optimized for this case... > This patch is just about the trivial part -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9983) Lower the memory footprint of HRegionLocation
[ https://issues.apache.org/jira/browse/HBASE-9983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13824799#comment-13824799 ] Nicolas Liochon commented on HBASE-9983: v2 is what I committed (rebased, ClentNoCluster removed, commented added). Thanks for the review, Stack & Ted. > Lower the memory footprint of HRegionLocation > - > > Key: HBASE-9983 > URL: https://issues.apache.org/jira/browse/HBASE-9983 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 0.98.0, 0.96.0 >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon >Priority: Minor > Fix For: 0.98.0, 0.96.1 > > Attachments: 9983.v2.patch, unbloated.patch > > > It"s more or less a desperate cause... > If we have a lot of regions, the size of this class becomes a problem. It's > under optimized for this case... > This patch is just about the trivial part -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9983) Lower the memory footprint of HRegionLocation
[ https://issues.apache.org/jira/browse/HBASE-9983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13824744#comment-13824744 ] stack commented on HBASE-9983: -- bq. What if two region servers run on the same server ? ServerName includes port. +1 Looks like another issue leaked into this one: -byte [] b = format(i); +byte [] b = format(rd.nextLong()); > Lower the memory footprint of HRegionLocation > - > > Key: HBASE-9983 > URL: https://issues.apache.org/jira/browse/HBASE-9983 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 0.98.0, 0.96.0 >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon >Priority: Minor > Fix For: 0.98.0, 0.96.1 > > Attachments: unbloated.patch > > > It"s more or less a desperate cause... > If we have a lot of regions, the size of this class becomes a problem. It's > under optimized for this case... > This patch is just about the trivial part -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9983) Lower the memory footprint of HRegionLocation
[ https://issues.apache.org/jira/browse/HBASE-9983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13824670#comment-13824670 ] Ted Yu commented on HBASE-9983: --- {code} - addresses.add(location != null ? location.getHostnamePort() : "null location"); + addresses.add(location != null ? location.getServerName().toString() : "null location"); {code} What if two region servers run on the same server ? > Lower the memory footprint of HRegionLocation > - > > Key: HBASE-9983 > URL: https://issues.apache.org/jira/browse/HBASE-9983 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 0.98.0, 0.96.0 >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon >Priority: Minor > Fix For: 0.98.0, 0.96.1 > > Attachments: unbloated.patch > > > It"s more or less a desperate cause... > If we have a lot of regions, the size of this class becomes a problem. It's > under optimized for this case... > This patch is just about the trivial part -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9983) Lower the memory footprint of HRegionLocation
[ https://issues.apache.org/jira/browse/HBASE-9983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13824658#comment-13824658 ] Hadoop QA commented on HBASE-9983: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12614244/unbloated.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 3 new or modified tests. {color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop 1.0 profile. {color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop 2.0 profile. {color:red}-1 javadoc{color}. The javadoc tool appears to have generated 10 warning messages. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 lineLengths{color}. The patch does not introduce lines longer than 100 {color:red}-1 site{color}. The patch appears to cause mvn site goal to fail. {color:green}+1 core tests{color}. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/7899//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7899//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7899//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7899//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7899//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7899//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7899//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7899//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7899//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7899//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7899//console This message is automatically generated. > Lower the memory footprint of HRegionLocation > - > > Key: HBASE-9983 > URL: https://issues.apache.org/jira/browse/HBASE-9983 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 0.98.0, 0.96.0 >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon >Priority: Minor > Fix For: 0.98.0, 0.96.1 > > Attachments: unbloated.patch > > > It"s more or less a desperate cause... > If we have a lot of regions, the size of this class becomes a problem. It's > under optimized for this case... > This patch is just about the trivial part -- This message was sent by Atlassian JIRA (v6.1#6144)