[jira] [Commented] (HBASE-15202) Reduce garbage while setting response
[ https://issues.apache.org/jira/browse/HBASE-15202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15132198#comment-15132198 ] Anoop Sam John commented on HBASE-15202: +1 > Reduce garbage while setting response > - > > Key: HBASE-15202 > URL: https://issues.apache.org/jira/browse/HBASE-15202 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15202.patch, HBASE-15202_1.patch, > HBASE-15202_2.patch, HBASE-15202_3.patch > > > Probably this can move under HBASE-15177. During RpcServer#setResponse we do > IPCUtil.getDelimitedMessageAsByteBuffer for the header and result. This > internally creates a byte[], CodedOutputStream and a BB. Then the totalSize > is also wrapped in a BB. > Finally all these BB are passed to BufferChain along with the Cellblock. > This JIRA is to reduce the number of objects that we create here and allow > one BB from this header, result and total Size along with one > CodedOutputStream and pass this BB along with the Cellblocks to the > Bufferchain. From the JMC tool can observe around 2% lesser object allocation > arising out of these objects. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15203) Reduce garbage created by path.toString() during Checksum verification
[ https://issues.apache.org/jira/browse/HBASE-15203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15132217#comment-15132217 ] Hudson commented on HBASE-15203: FAILURE: Integrated in HBase-Trunk_matrix #678 (See [https://builds.apache.org/job/HBase-Trunk_matrix/678/]) HBASE-HBASE-15203 Reduce garbage created by path.toString() during (ramkrishna: rev 2cf8af5bf1d501156cbb3b421cf75c1051ead7d9) * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/ChecksumUtil.java > Reduce garbage created by path.toString() during Checksum verification > -- > > Key: HBASE-15203 > URL: https://issues.apache.org/jira/browse/HBASE-15203 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15203.patch, HBASE-15203_1.patch, > HBASE-15203_2.patch, HBASE-15203_branch-1.1.patch > > > When we try to read a block we do checksum verification for which we need the > file name in which the block belongs to. So we do Path.toString() every time. > This seems to create around 163MB of char[] that is garbage collected in a > simple scan run. It is also visible in writes but the impact is lesser. In > overall write/read profile the top 2 factors are byte[] and char[]. This > toString() can easily be avoided and reduce its share from the total. To make > it more precise in 1 min of profiling, among the 1.8G of garbage created by > StringBuilder.toString - this path.toString() was contributing around 3.5%. > After the patch this is totally not there. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15214) Valid mutate Ops fail with RPC Codec in use and region moves across
[ https://issues.apache.org/jira/browse/HBASE-15214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-15214: --- Attachment: HBASE-15214.patch > Valid mutate Ops fail with RPC Codec in use and region moves across > --- > > Key: HBASE-15214 > URL: https://issues.apache.org/jira/browse/HBASE-15214 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.0 >Reporter: Anoop Sam John >Assignee: Anoop Sam John >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4, 0.98.18 > > Attachments: HBASE-15214.patch > > > Test failures in HBASE-15198 lead to this bug. Till now we are not doing cell > block (codec usage) for write requests. (Client -> server) Once we enabled > Codec usage by default, aw this issue. > A multi request came to RS with mutation for different regions. One of the > region which was in this RS got unavailable now. In RsRpcServices#multi, we > will fail that entire RegionAction (with N mutations in it) in that > MultiRequest. Then we will continue with remaining RegionActions. Those > Regions might be available. (The failed RegionAction will get retried from > client after fetching latest region location). This all works fine in pure > PB requests world. When a Codec is used, we wont convert the Mutation Cell to > PB Cells and pack them in PB Message. Instead we will pass all Cells > serialized into one byte[] cellblock. Using Decoder we will iterate over > these cells at server side. Each Mutation PB will know only the number of > cells associated with it. As in above case when an entire RegionAction was > skipped, there might be N Mutations under that which might have corresponding > Cells in the cellblock. We are not doing the skip in that Iterator. This > makes the later Mutations (for other Regions) to refer to invalid Cells and > try to put those into the a different region. This will make > HRegion#checkRow() to throw WrongRegionException which will be treated as > Sanity check failure and so throwing back a DNRIOE to client. So the op will > get failed for the user code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[ https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15204: --- Status: Open (was: Patch Available) > Try to estimate the cell count for adding into WALEdit > -- > > Key: HBASE-15204 > URL: https://issues.apache.org/jira/browse/HBASE-15204 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-15204.patch, WAlEdit_add_allocation.jpg, > WAlEdit_add_allocation_after_patch.jpg > > > The write path profiling shows that when we try to add Cells to WALEdits we > try to do a lot of Array copy inorder to grow the Arraylist backing the > WALEdits. In a simple one min profiling of the write path with 50 YCSB > threads shows around 261MB of allocation done for the Array copy to happen. > We can try to avoid that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[ https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15204: --- Attachment: HBASE-15204_1.patch Updated patch. Does not pass the cellcount from RsRpcServices. Instead tries to find the cellCount from the familyMaps associated with each of the mutation and then sets the WALEdit. So anyway we are now having a value > 1 for the ArrayList backing the WALEdit. > Try to estimate the cell count for adding into WALEdit > -- > > Key: HBASE-15204 > URL: https://issues.apache.org/jira/browse/HBASE-15204 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-15204.patch, HBASE-15204_1.patch, > WAlEdit_add_allocation.jpg, WAlEdit_add_allocation_after_patch.jpg > > > The write path profiling shows that when we try to add Cells to WALEdits we > try to do a lot of Array copy inorder to grow the Arraylist backing the > WALEdits. In a simple one min profiling of the write path with 50 YCSB > threads shows around 261MB of allocation done for the Array copy to happen. > We can try to avoid that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15202) Reduce garbage while setting response
[ https://issues.apache.org/jira/browse/HBASE-15202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15202: --- Status: Patch Available (was: Open) > Reduce garbage while setting response > - > > Key: HBASE-15202 > URL: https://issues.apache.org/jira/browse/HBASE-15202 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15202.patch, HBASE-15202_1.patch, > HBASE-15202_2.patch, HBASE-15202_3.patch > > > Probably this can move under HBASE-15177. During RpcServer#setResponse we do > IPCUtil.getDelimitedMessageAsByteBuffer for the header and result. This > internally creates a byte[], CodedOutputStream and a BB. Then the totalSize > is also wrapped in a BB. > Finally all these BB are passed to BufferChain along with the Cellblock. > This JIRA is to reduce the number of objects that we create here and allow > one BB from this header, result and total Size along with one > CodedOutputStream and pass this BB along with the Cellblocks to the > Bufferchain. From the JMC tool can observe around 2% lesser object allocation > arising out of these objects. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15202) Reduce garbage while setting response
[ https://issues.apache.org/jira/browse/HBASE-15202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15202: --- Attachment: HBASE-15202_3.patch Updated patch as per the comment from Anoop. Now directly writes the total Size to byte[] and rest is written by the COS. So we create a suitable COS on the byte[] from the required offset and length. > Reduce garbage while setting response > - > > Key: HBASE-15202 > URL: https://issues.apache.org/jira/browse/HBASE-15202 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15202.patch, HBASE-15202_1.patch, > HBASE-15202_2.patch, HBASE-15202_3.patch > > > Probably this can move under HBASE-15177. During RpcServer#setResponse we do > IPCUtil.getDelimitedMessageAsByteBuffer for the header and result. This > internally creates a byte[], CodedOutputStream and a BB. Then the totalSize > is also wrapped in a BB. > Finally all these BB are passed to BufferChain along with the Cellblock. > This JIRA is to reduce the number of objects that we create here and allow > one BB from this header, result and total Size along with one > CodedOutputStream and pass this BB along with the Cellblocks to the > Bufferchain. From the JMC tool can observe around 2% lesser object allocation > arising out of these objects. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15214) Valid mutate Ops fail with RPC Codec in use and region moves across
[ https://issues.apache.org/jira/browse/HBASE-15214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-15214: --- Fix Version/s: 0.98.18 1.0.4 1.1.4 1.2.1 1.3.0 2.0.0 > Valid mutate Ops fail with RPC Codec in use and region moves across > --- > > Key: HBASE-15214 > URL: https://issues.apache.org/jira/browse/HBASE-15214 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.0 >Reporter: Anoop Sam John >Assignee: Anoop Sam John >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4, 0.98.18 > > > Test failures in HBASE-15198 lead to this bug. Till now we are not doing cell > block (codec usage) for write requests. (Client -> server) Once we enabled > Codec usage by default, aw this issue. > A multi request came to RS with mutation for different regions. One of the > region which was in this RS got unavailable now. In RsRpcServices#multi, we > will fail that entire RegionAction (with N mutations in it) in that > MultiRequest. Then we will continue with remaining RegionActions. Those > Regions might be available. (The failed RegionAction will get retried from > client after fetching latest region location). This all works fine in pure > PB requests world. When a Codec is used, we wont convert the Mutation Cell to > PB Cells and pack them in PB Message. Instead we will pass all Cells > serialized into one byte[] cellblock. Using Decoder we will iterate over > these cells at server side. Each Mutation PB will know only the number of > cells associated with it. As in above case when an entire RegionAction was > skipped, there might be N Mutations under that which might have corresponding > Cells in the cellblock. We are not doing the skip in that Iterator. This > makes the later Mutations (for other Regions) to refer to invalid Cells and > try to put those into the a different region. This will make > HRegion#checkRow() to throw WrongRegionException which will be treated as > Sanity check failure and so throwing back a DNRIOE to client. So the op will > get failed for the user code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15214) Valid mutate Ops fail with RPC Codec in use and region moves across
[ https://issues.apache.org/jira/browse/HBASE-15214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-15214: --- Status: Patch Available (was: Open) > Valid mutate Ops fail with RPC Codec in use and region moves across > --- > > Key: HBASE-15214 > URL: https://issues.apache.org/jira/browse/HBASE-15214 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.0 >Reporter: Anoop Sam John >Assignee: Anoop Sam John >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4, 0.98.18 > > Attachments: HBASE-15214.patch > > > Test failures in HBASE-15198 lead to this bug. Till now we are not doing cell > block (codec usage) for write requests. (Client -> server) Once we enabled > Codec usage by default, aw this issue. > A multi request came to RS with mutation for different regions. One of the > region which was in this RS got unavailable now. In RsRpcServices#multi, we > will fail that entire RegionAction (with N mutations in it) in that > MultiRequest. Then we will continue with remaining RegionActions. Those > Regions might be available. (The failed RegionAction will get retried from > client after fetching latest region location). This all works fine in pure > PB requests world. When a Codec is used, we wont convert the Mutation Cell to > PB Cells and pack them in PB Message. Instead we will pass all Cells > serialized into one byte[] cellblock. Using Decoder we will iterate over > these cells at server side. Each Mutation PB will know only the number of > cells associated with it. As in above case when an entire RegionAction was > skipped, there might be N Mutations under that which might have corresponding > Cells in the cellblock. We are not doing the skip in that Iterator. This > makes the later Mutations (for other Regions) to refer to invalid Cells and > try to put those into the a different region. This will make > HRegion#checkRow() to throw WrongRegionException which will be treated as > Sanity check failure and so throwing back a DNRIOE to client. So the op will > get failed for the user code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HBASE-11262) Avoid empty columns while doing bulk-load
[ https://issues.apache.org/jira/browse/HBASE-11262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-11262 started by Ashish Kumar. > Avoid empty columns while doing bulk-load > - > > Key: HBASE-11262 > URL: https://issues.apache.org/jira/browse/HBASE-11262 > Project: HBase > Issue Type: New Feature >Reporter: Ashish Kumar >Assignee: Ashish Kumar > > In bulk load feature of hbase (ImportTsv and CompleteBulkload), it just > stores everything in the table but some times we want to avoid empty column > values to save some disk space. So, we can have a feature to skip those empty > columns values while bulk load (If user wants). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-11262) Avoid empty columns while doing bulk-load
[ https://issues.apache.org/jira/browse/HBASE-11262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Kumar reassigned HBASE-11262: Assignee: Ashish Kumar > Avoid empty columns while doing bulk-load > - > > Key: HBASE-11262 > URL: https://issues.apache.org/jira/browse/HBASE-11262 > Project: HBase > Issue Type: New Feature >Reporter: Ashish Kumar >Assignee: Ashish Kumar > > In bulk load feature of hbase (ImportTsv and CompleteBulkload), it just > stores everything in the table but some times we want to avoid empty column > values to save some disk space. So, we can have a feature to skip those empty > columns values while bulk load (If user wants). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15209) disable table in HBaseTestingUtility.truncateTable
[
https://issues.apache.org/jira/browse/HBASE-15209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15132220#comment-15132220
]
Hadoop QA commented on HBASE-15209:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
38s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s
{color} | {color:green} branch-1 passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} branch-1 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 23s
{color} | {color:green} branch-1 passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} branch-1 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 4m
9s {color} | {color:green} Patch does not cause any errors with Hadoop 2.4.1
2.5.2 2.6.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 83m 31s {color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0_72. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 86m 57s
{color} | {color:green} hbase-server in the patch passed with JDK v1.7.0_95.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 197m 6s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_72 Failed junit tests | hadoop.hbase.mapreduce.TestImportExport |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.9.1 Server=1.9.1 Image:yetus/hbase:date2016-02-04 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12786216/HBASE-15209-branch-1-v2.patch
|
| JIRA Issue | HBASE-15209 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 7f84eacea811 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64
[jira] [Commented] (HBASE-15203) Reduce garbage created by path.toString() during Checksum verification
[ https://issues.apache.org/jira/browse/HBASE-15203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15132181#comment-15132181 ] Hudson commented on HBASE-15203: FAILURE: Integrated in HBase-1.3 #533 (See [https://builds.apache.org/job/HBase-1.3/533/]) HBASE-HBASE-15203 Reduce garbage created by path.toString() during (ramkrishna: rev 1404d5a97331ecc63db53971f5cb7329cb40ce67) * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/ChecksumUtil.java > Reduce garbage created by path.toString() during Checksum verification > -- > > Key: HBASE-15203 > URL: https://issues.apache.org/jira/browse/HBASE-15203 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15203.patch, HBASE-15203_1.patch, > HBASE-15203_2.patch, HBASE-15203_branch-1.1.patch > > > When we try to read a block we do checksum verification for which we need the > file name in which the block belongs to. So we do Path.toString() every time. > This seems to create around 163MB of char[] that is garbage collected in a > simple scan run. It is also visible in writes but the impact is lesser. In > overall write/read profile the top 2 factors are byte[] and char[]. This > toString() can easily be avoided and reduce its share from the total. To make > it more precise in 1 min of profiling, among the 1.8G of garbage created by > StringBuilder.toString - this path.toString() was contributing around 3.5%. > After the patch this is totally not there. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15213) Fix increment performance regression caused by HBASE-8763 on branch-1.0
[
https://issues.apache.org/jira/browse/HBASE-15213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15132245#comment-15132245
]
Hadoop QA commented on HBASE-15213:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
20s {color} | {color:green} branch-1.0 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s
{color} | {color:green} branch-1.0 passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} branch-1.0 passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s {color} | {color:green} branch-1.0 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} branch-1.0 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 49s
{color} | {color:red} hbase-server in branch-1.0 has 61 extant Findbugs
warnings. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 33s
{color} | {color:red} hbase-server in branch-1.0 failed with JDK v1.8.0_66.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} branch-1.0 passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 4m
13s {color} | {color:green} Patch does not cause any errors with Hadoop 2.4.1
2.5.2 2.6.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 3s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 31s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0_66.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 73m 25s {color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 74m 1s
{color} | {color:green} hbase-server in the patch passed with JDK v1.7.0_91.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 173m 33s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests |
hadoop.hbase.master.handler.TestEnableTableHandler |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.9.1 Server=1.9.1 Image:yetus/hbase:date2016-02-04 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12786221/HBASE-15213.branch-1.0.patch
|
| JIRA Issue | HBASE-15213 |
| Optional Tests |
[jira] [Created] (HBASE-15214) Valid mutate Ops fail with RPC Codec in use and region moves across
Anoop Sam John created HBASE-15214: -- Summary: Valid mutate Ops fail with RPC Codec in use and region moves across Key: HBASE-15214 URL: https://issues.apache.org/jira/browse/HBASE-15214 Project: HBase Issue Type: Bug Affects Versions: 0.98.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Priority: Critical Test failures in HBASE-15198 lead to this bug. Till now we are not doing cell block (codec usage) for write requests. (Client -> server) Once we enabled Codec usage by default, aw this issue. A multi request came to RS with mutation for different regions. One of the region which was in this RS got unavailable now. In RsRpcServices#multi, we will fail that entire RegionAction (with N mutations in it) in that MultiRequest. Then we will continue with remaining RegionActions. Those Regions might be available. (The failed RegionAction will get retried from client after fetching latest region location). This all works fine in pure PB requests world. When a Codec is used, we wont convert the Mutation Cell to PB Cells and pack them in PB Message. Instead we will pass all Cells serialized into one byte[] cellblock. Using Decoder we will iterate over these cells at server side. Each Mutation PB will know only the number of cells associated with it. As in above case when an entire RegionAction was skipped, there might be N Mutations under that which might have corresponding Cells in the cellblock. We are not doing the skip in that Iterator. This makes the later Mutations (for other Regions) to refer to invalid Cells and try to put those into the a different region. This will make HRegion#checkRow() to throw WrongRegionException which will be treated as Sanity check failure and so throwing back a DNRIOE to client. So the op will get failed for the user code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15202) Reduce garbage while setting response
[ https://issues.apache.org/jira/browse/HBASE-15202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15202: --- Status: Open (was: Patch Available) > Reduce garbage while setting response > - > > Key: HBASE-15202 > URL: https://issues.apache.org/jira/browse/HBASE-15202 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15202.patch, HBASE-15202_1.patch, > HBASE-15202_2.patch > > > Probably this can move under HBASE-15177. During RpcServer#setResponse we do > IPCUtil.getDelimitedMessageAsByteBuffer for the header and result. This > internally creates a byte[], CodedOutputStream and a BB. Then the totalSize > is also wrapped in a BB. > Finally all these BB are passed to BufferChain along with the Cellblock. > This JIRA is to reduce the number of objects that we create here and allow > one BB from this header, result and total Size along with one > CodedOutputStream and pass this BB along with the Cellblocks to the > Bufferchain. From the JMC tool can observe around 2% lesser object allocation > arising out of these objects. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15202) Reduce garbage while setting response
[
https://issues.apache.org/jira/browse/HBASE-15202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15132202#comment-15132202
]
Hadoop QA commented on HBASE-15202:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
42s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 54s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 5m
50s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 58s
{color} | {color:red} hbase-client in master has 4 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 48s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
2s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 54s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 54s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 47s
{color} | {color:red} Patch generated 1 new checkstyle issues in hbase-client
(total was 50, now 51). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
22m 22s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 55s
{color} | {color:green} hbase-client in the patch passed with JDK v1.8.0_72.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 102m 0s {color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0_72. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 4s
{color} | {color:green} hbase-client in the patch passed with JDK v1.7.0_95.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 99m 54s {color}
| {color:red} hbase-server in the patch failed with JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 256m 40s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
[jira] [Updated] (HBASE-15211) Don't run the CatalogJanitor if there are regions in transition
[ https://issues.apache.org/jira/browse/HBASE-15211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-15211: -- Fix Version/s: 1.1.4 1.2.1 1.3.0 2.0.0 > Don't run the CatalogJanitor if there are regions in transition > --- > > Key: HBASE-15211 > URL: https://issues.apache.org/jira/browse/HBASE-15211 > Project: HBase > Issue Type: Improvement > Components: master >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4 > > Attachments: HBASE-15211.patch > > > This could make things a little safer to make sure that CatalogJanitor > doesn't do something weird while a cluster is starting up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15213) Fix increment performance regression caused by HBASE-8763 on branch-1.0
[ https://issues.apache.org/jira/browse/HBASE-15213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133063#comment-15133063 ] stack commented on HBASE-15213: --- I tried the patch against standalone instance of hbase (no hdfs) using the IncrementPerformanceTest and did not see much difference; a slight improvement for the patch. How were you testing it [~junegunn]? Thanks. > Fix increment performance regression caused by HBASE-8763 on branch-1.0 > --- > > Key: HBASE-15213 > URL: https://issues.apache.org/jira/browse/HBASE-15213 > Project: HBase > Issue Type: Bug > Components: Performance >Reporter: Junegunn Choi >Assignee: Junegunn Choi > Attachments: HBASE-15213.branch-1.0.patch > > > This is an attempt to fix the increment performance regression caused by > HBASE-8763 on branch-1.0. > I'm aware that hbase.increment.fast.but.narrow.consistency was added to > branch-1.0 (HBASE-15031) to address the issue and a separate work is ongoing > on master branch, but anyway, this is my take on the problem. > I read through HBASE-14460 and HBASE-8763 but it wasn't clear to me what > caused the slowdown but I could indeed reproduce the performance regression. > Test setup: > - Server: 4-core Xeon 2.4GHz Linux server running mini cluster (100 handlers, > JDK 1.7) > - Client: Another box of the same spec > - Increments on random 10k records on a single-region table, recreated every > time > Increment throughput (TPS): > || Num threads || Before HBASE-8763 (d6cc2fb) || branch-1.0 || branch-1.0 > (narrow-consistency) || > || 1| 2661 | 2486| 2359 | > || 2| 5048 | 5064| 4867 | > || 4| 7503 | 8071| 8690 | > || 8| 10471| 10886 | 13980 | > || 16 | 15515| 9418| 18601 | > || 32 | 17699| 5421| 20540 | > || 64 | 20601| 4038| 25591 | > || 96 | 19177| 3891| 26017 | > We can clearly observe that the throughtput degrades as we increase the > number of concurrent requests, which led me to believe that there's severe > context switching overhead and I could indirectly confirm that suspicion with > cs entry in vmstat output. branch-1.0 shows a much higher number of context > switches even with much lower throughput. > Here are the observations: > - WriteEntry in the writeQueue can only be removed by the very handler that > put it, only when it is at the front of the queue and marked complete. > - Since a WriteEntry is marked complete after the wait-loop, only one entry > can be removed at a time. > - This stringent condition causes O(N^2) context switches where n is the > number of concurrent handlers processing requests. > So what I tried here is to mark WriteEntry complete before we go into > wait-loop. With the change, multiple WriteEntries can be shifted at a time > without context switches. I changed writeQueue to LinkedHashSet since fast > containment check is needed as WriteEntry can be removed by any handler. > The numbers look good, it's virtually identical to pre-HBASE-8763 era. > || Num threads || branch-1.0 with fix || > || 1| 2459 | > || 2| 4976 | > || 4| 8033 | > || 8| 12292| > || 16 | 15234| > || 32 | 16601| > || 64 | 19994| > || 96 | 20052| > So what do you think about it? Please let me know if I'm missing anything. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15213) Fix increment performance regression caused by HBASE-8763 on branch-1.0
[ https://issues.apache.org/jira/browse/HBASE-15213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133074#comment-15133074 ] stack commented on HBASE-15213: --- Or give me the checkouts your compared... HBASE-15031 is probably hiding the benefit your patch brings. > Fix increment performance regression caused by HBASE-8763 on branch-1.0 > --- > > Key: HBASE-15213 > URL: https://issues.apache.org/jira/browse/HBASE-15213 > Project: HBase > Issue Type: Bug > Components: Performance >Reporter: Junegunn Choi >Assignee: Junegunn Choi > Attachments: HBASE-15213.branch-1.0.patch > > > This is an attempt to fix the increment performance regression caused by > HBASE-8763 on branch-1.0. > I'm aware that hbase.increment.fast.but.narrow.consistency was added to > branch-1.0 (HBASE-15031) to address the issue and a separate work is ongoing > on master branch, but anyway, this is my take on the problem. > I read through HBASE-14460 and HBASE-8763 but it wasn't clear to me what > caused the slowdown but I could indeed reproduce the performance regression. > Test setup: > - Server: 4-core Xeon 2.4GHz Linux server running mini cluster (100 handlers, > JDK 1.7) > - Client: Another box of the same spec > - Increments on random 10k records on a single-region table, recreated every > time > Increment throughput (TPS): > || Num threads || Before HBASE-8763 (d6cc2fb) || branch-1.0 || branch-1.0 > (narrow-consistency) || > || 1| 2661 | 2486| 2359 | > || 2| 5048 | 5064| 4867 | > || 4| 7503 | 8071| 8690 | > || 8| 10471| 10886 | 13980 | > || 16 | 15515| 9418| 18601 | > || 32 | 17699| 5421| 20540 | > || 64 | 20601| 4038| 25591 | > || 96 | 19177| 3891| 26017 | > We can clearly observe that the throughtput degrades as we increase the > number of concurrent requests, which led me to believe that there's severe > context switching overhead and I could indirectly confirm that suspicion with > cs entry in vmstat output. branch-1.0 shows a much higher number of context > switches even with much lower throughput. > Here are the observations: > - WriteEntry in the writeQueue can only be removed by the very handler that > put it, only when it is at the front of the queue and marked complete. > - Since a WriteEntry is marked complete after the wait-loop, only one entry > can be removed at a time. > - This stringent condition causes O(N^2) context switches where n is the > number of concurrent handlers processing requests. > So what I tried here is to mark WriteEntry complete before we go into > wait-loop. With the change, multiple WriteEntries can be shifted at a time > without context switches. I changed writeQueue to LinkedHashSet since fast > containment check is needed as WriteEntry can be removed by any handler. > The numbers look good, it's virtually identical to pre-HBASE-8763 era. > || Num threads || branch-1.0 with fix || > || 1| 2459 | > || 2| 4976 | > || 4| 8033 | > || 8| 12292| > || 16 | 15234| > || 32 | 16601| > || 64 | 19994| > || 96 | 20052| > So what do you think about it? Please let me know if I'm missing anything. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15211) Don't run the CatalogJanitor if there are regions in transition
[ https://issues.apache.org/jira/browse/HBASE-15211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133094#comment-15133094 ] Hudson commented on HBASE-15211: SUCCESS: Integrated in HBase-1.2 #528 (See [https://builds.apache.org/job/HBase-1.2/528/]) HBASE-15211 Don't run the CatalogJanitor if there are regions in (eclark: rev faff18fbaa219127dc27eb982622bcb419019249) * hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java > Don't run the CatalogJanitor if there are regions in transition > --- > > Key: HBASE-15211 > URL: https://issues.apache.org/jira/browse/HBASE-15211 > Project: HBase > Issue Type: Improvement > Components: master >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4 > > Attachments: HBASE-15211.patch > > > This could make things a little safer to make sure that CatalogJanitor > doesn't do something weird while a cluster is starting up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15181) A simple implementation of date based tiered compaction
[ https://issues.apache.org/jira/browse/HBASE-15181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133258#comment-15133258 ] Ted Yu commented on HBASE-15181: Please incorporate the above and submit new patch for QA run. > A simple implementation of date based tiered compaction > --- > > Key: HBASE-15181 > URL: https://issues.apache.org/jira/browse/HBASE-15181 > Project: HBase > Issue Type: New Feature > Components: Compaction >Reporter: Clara Xiong >Assignee: Clara Xiong > Fix For: 2.0.0 > > Attachments: HBASE-15181-v1.patch, HBASE-15181-v2.patch > > > This is a simple implementation of date-based tiered compaction similar to > Cassandra's for the following benefits: > 1. Improve date-range-based scan by structuring store files in date-based > tiered layout. > 2. Reduce compaction overhead. > 3. Improve TTL efficiency. > Perfect fit for the use cases that: > 1. has mostly date-based date write and scan and a focus on the most recent > data. > 2. never or rarely deletes data. > Out-of-order writes are handled gracefully so the data will still get to the > right store file for time-range-scan and re-compacton with existing store > file in the same time window is handled by ExploringCompactionPolicy. > Time range overlapping among store files is tolerated and the performance > impact is minimized. > Configuration can be set at hbase-site or overriden at per-table or > per-column-famly level by hbase shell. > Design spec is at > https://docs.google.com/document/d/1_AmlNb2N8Us1xICsTeGDLKIqL6T-oHoRLZ323MG_uy8/edit?usp=sharing -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15177) Reduce garbage created under high load
[ https://issues.apache.org/jira/browse/HBASE-15177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133266#comment-15133266 ] Hudson commented on HBASE-15177: SUCCESS: Integrated in HBase-1.3-IT #477 (See [https://builds.apache.org/job/HBase-1.3-IT/477/]) HBASE-15177 Reduce garbage created under high load (enis: rev 73d67788206c3f60773d861375f5e6934a284418) * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java * hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/PayloadCarryingRpcController.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * hbase-common/src/main/java/org/apache/hadoop/hbase/io/ByteBufferInputStream.java * hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/IPCUtil.java * hbase-common/src/main/java/org/apache/hadoop/hbase/util/Threads.java * hbase-common/src/main/java/org/apache/hadoop/hbase/util/UnsafeAccess.java * hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestClientScanner.java * hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/RpcClientImpl.java * hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/AnnotationReadingPriorityFunction.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java * hbase-client/src/test/java/org/apache/hadoop/hbase/ipc/TestIPCUtil.java * hbase-common/src/main/java/org/apache/hadoop/hbase/util/ByteBufferUtils.java * hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/AsyncRpcChannel.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java > Reduce garbage created under high load > -- > > Key: HBASE-15177 > URL: https://issues.apache.org/jira/browse/HBASE-15177 > Project: HBase > Issue Type: Improvement >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 2.0.0, 1.3.0 > > Attachments: Screen Shot 2016-01-26 at 10.03.48 PM.png, Screen Shot > 2016-01-26 at 10.03.56 PM.png, Screen Shot 2016-01-26 at 10.06.16 PM.png, > Screen Shot 2016-01-26 at 10.15.15 PM.png, hbase-15177_v0.patch, > hbase-15177_v1.patch, hbase-15177_v2.patch, hbase-15177_v3.patch, > hbase-15177_v4.patch, hbase-15177_v4.patch > > > I have been doing some profiling of the garbage being created. The idea was > to follow up on HBASE-14490 and experiment with offheap IPC byte buffers and > byte buffer re-use. However, without changing the IPC byte buffers for now, > there are a couple of (easy) improvements that I've identified from > profiling: > 1. RPCServer.Connection.processRequest() should work with ByteBuffer instead > of byte[] and not-recreate CodedInputStream a few times. > 2. RSRpcServices.getRegion() allocates two byte arrays for region, while only > 1 is needed. > 3. AnnotationReadingPriorityFunction is very expensive in allocations. Mainly > it allocates the regionName byte[] to get the table name. We already set the > priority for most of the operations (multi, get, increment, etc) but we are > only reading the priority in case of multi. We should use the priority from > the client side. > Lets do the simple improvements in this patch, we can get to IPC buffer > re-use in HBASE-14490. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15120) Backport HBASE-14883 to branch-1.1
[ https://issues.apache.org/jira/browse/HBASE-15120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133267#comment-15133267 ] Hudson commented on HBASE-15120: SUCCESS: Integrated in HBase-1.3-IT #477 (See [https://builds.apache.org/job/HBase-1.3-IT/477/]) HBASE-15120 Undo aggressive load balancer logging at tens of lines per (stack: rev 66da601cdcc915c3ccebd994105c8f7fbebf556d) * hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java > Backport HBASE-14883 to branch-1.1 > -- > > Key: HBASE-15120 > URL: https://issues.apache.org/jira/browse/HBASE-15120 > Project: HBase > Issue Type: Bug >Affects Versions: 1.1.2 >Reporter: Yu Li >Assignee: Yu Li >Priority: Minor > Attachments: HBASE-15120.branch-1.1.patch, > HBASE-15120.branch-1.1.patch > > > When checking branch-1.1 UT in HBASE-13590, found > TestSplitTransactionOnCluster#testFailedSplit will fail at a 12/50 chance. > The issue is fixed by HBASE-14883 but the change didn't go into branch-1.1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15214) Valid mutate Ops fail with RPC Codec in use and region moves across
[
https://issues.apache.org/jira/browse/HBASE-15214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133085#comment-15133085
]
Hadoop QA commented on HBASE-15214:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
1s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 3s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 4s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 6m
25s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 1s
{color} | {color:red} hbase-client in master has 4 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 53s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 53s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
6s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 58s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 58s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 12s
{color} | {color:red} Patch generated 2 new checkstyle issues in hbase-server
(total was 138, now 140). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
23m 52s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
57s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 9s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 8s
{color} | {color:green} hbase-client in the patch passed with JDK v1.8.0_72.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 86m 49s {color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0_72. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 29s
{color} | {color:green} hbase-client in the patch passed with JDK v1.7.0_95.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 103m 30s
{color} | {color:red} hbase-server in the patch failed with JDK v1.7.0_95.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 250m 32s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_72 Timed out junit tests |
org.apache.hadoop.hbase.util.TestHBaseFsckOneRS |
| |
[jira] [Commented] (HBASE-15211) Don't run the CatalogJanitor if there are regions in transition
[ https://issues.apache.org/jira/browse/HBASE-15211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133319#comment-15133319 ] Hudson commented on HBASE-15211: SUCCESS: Integrated in HBase-1.1-JDK8 #1738 (See [https://builds.apache.org/job/HBase-1.1-JDK8/1738/]) HBASE-15211 Don't run the CatalogJanitor if there are regions in (eclark: rev bc564d3a6a0e0bc41dc35eb799b2d9e3f8a10bf2) * hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java > Don't run the CatalogJanitor if there are regions in transition > --- > > Key: HBASE-15211 > URL: https://issues.apache.org/jira/browse/HBASE-15211 > Project: HBase > Issue Type: Improvement > Components: master >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4 > > Attachments: HBASE-15211.patch > > > This could make things a little safer to make sure that CatalogJanitor > doesn't do something weird while a cluster is starting up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15211) Don't run the CatalogJanitor if there are regions in transition
[ https://issues.apache.org/jira/browse/HBASE-15211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-15211: Component/s: master > Don't run the CatalogJanitor if there are regions in transition > --- > > Key: HBASE-15211 > URL: https://issues.apache.org/jira/browse/HBASE-15211 > Project: HBase > Issue Type: Improvement > Components: master >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-15211.patch > > > This could make things a little safer to make sure that CatalogJanitor > doesn't do something weird while a cluster is starting up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15200) ZooKeeper znode ACL checks should only compare the shortname
[ https://issues.apache.org/jira/browse/HBASE-15200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-15200: Component/s: security > ZooKeeper znode ACL checks should only compare the shortname > > > Key: HBASE-15200 > URL: https://issues.apache.org/jira/browse/HBASE-15200 > Project: HBase > Issue Type: Bug > Components: security >Affects Versions: 2.0.0, 1.2.0, 1.0.3, 1.1.3, 0.98.17 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4, 0.98.18 > > Attachments: HBASE-15200-branch-1.0.patch, > HBASE-15200-branch-1.1.patch, HBASE-15200.patch, HBASE-15200.patch > > > After HBASE-13768 we check at startup in secure configurations if our znodes > have the correct ACLs. However when checking the ACL we compare the Kerberos > fullname, which includes the host component. We should only compare the > shortname, the principal. Otherwise in a multimaster configuration we will > unnecessarily reset ACLs whenever any master running on a host other than the > one that initialized the ACLs makes the check. You can imagine this happening > multiple times in a rolling restart scenario. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11262) Avoid empty columns while doing bulk-load
[ https://issues.apache.org/jira/browse/HBASE-11262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133292#comment-15133292 ] Ted Yu commented on HBASE-11262: Please attach patch here for QA run. > Avoid empty columns while doing bulk-load > - > > Key: HBASE-11262 > URL: https://issues.apache.org/jira/browse/HBASE-11262 > Project: HBase > Issue Type: New Feature >Reporter: Ashish Kumar >Assignee: Ashish Kumar > > In bulk load feature of hbase (ImportTsv and CompleteBulkload), it just > stores everything in the table but some times we want to avoid empty column > values to save some disk space. So, we can have a feature to skip those empty > columns values while bulk load (If user wants). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15177) Reduce garbage created under high load
[ https://issues.apache.org/jira/browse/HBASE-15177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-15177: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I've committed this to branch-1+. Thanks for the reviews. > Reduce garbage created under high load > -- > > Key: HBASE-15177 > URL: https://issues.apache.org/jira/browse/HBASE-15177 > Project: HBase > Issue Type: Improvement >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 2.0.0, 1.3.0 > > Attachments: Screen Shot 2016-01-26 at 10.03.48 PM.png, Screen Shot > 2016-01-26 at 10.03.56 PM.png, Screen Shot 2016-01-26 at 10.06.16 PM.png, > Screen Shot 2016-01-26 at 10.15.15 PM.png, hbase-15177_v0.patch, > hbase-15177_v1.patch, hbase-15177_v2.patch, hbase-15177_v3.patch, > hbase-15177_v4.patch, hbase-15177_v4.patch > > > I have been doing some profiling of the garbage being created. The idea was > to follow up on HBASE-14490 and experiment with offheap IPC byte buffers and > byte buffer re-use. However, without changing the IPC byte buffers for now, > there are a couple of (easy) improvements that I've identified from > profiling: > 1. RPCServer.Connection.processRequest() should work with ByteBuffer instead > of byte[] and not-recreate CodedInputStream a few times. > 2. RSRpcServices.getRegion() allocates two byte arrays for region, while only > 1 is needed. > 3. AnnotationReadingPriorityFunction is very expensive in allocations. Mainly > it allocates the regionName byte[] to get the table name. We already set the > priority for most of the operations (multi, get, increment, etc) but we are > only reading the priority in case of multi. We should use the priority from > the client side. > Lets do the simple improvements in this patch, we can get to IPC buffer > re-use in HBASE-14490. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15210) Undo aggressive load balancer logging at tens of lines per millisecond
[
https://issues.apache.org/jira/browse/HBASE-15210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-15210:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 1.3.0
1.2.0
2.0.0
Status: Resolved (was: Patch Available)
Thanks [~busbey] Pushed to branch-1.2+
> Undo aggressive load balancer logging at tens of lines per millisecond
> --
>
> Key: HBASE-15210
> URL: https://issues.apache.org/jira/browse/HBASE-15210
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer
>Affects Versions: 1.2.0
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 15210.patch
>
>
> Sometimes the load balancer falls into a state of aggressive logging in 1.2.0
> writing out ten lines a millisecond like this:
> {code}
> 2016-02-02 17:15:01,382 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region server with non zero
> regions is ve0540.halxg.cloudera.com with locality 0.22908874
> 2016-02-02 17:15:01,382 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region index is 3 and its region
> server contains 4 regions
> 2016-02-02 17:15:01,384 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region server with non zero
> regions is ve0540.halxg.cloudera.com with locality 0.22908874
> 2016-02-02 17:15:01,384 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region index is 3 and its region
> server contains 4 regions
> 2016-02-02 17:15:01,386 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region server with non zero
> regions is ve0540.halxg.cloudera.com with locality 0.22908874
> 2016-02-02 17:15:01,386 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region index is 2 and its region
> server contains 4 regions
> 2016-02-02 17:15:01,387 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region server with non zero
> regions is ve0540.halxg.cloudera.com with locality 0.22908874
> 2016-02-02 17:15:01,387 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region index is 3 and its region
> server contains 4 regions
> 2016-02-02 17:15:01,388 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region server with non zero
> regions is ve0540.halxg.cloudera.com with locality 0.22908874
> 2016-02-02 17:15:01,388 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region index is 3 and its region
> server contains 4 regions
> 2016-02-02 17:15:01,388 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region server with non zero
> regions is ve0540.halxg.cloudera.com with locality 0.22908874
> {code}
> This issue is about less logging.
> This log was added recently by this:
> commit 54028140f4f19a6af81c8c8f29dda0c52491a0c9
> Author: tedyu
> Date: Thu Aug 13 09:11:59 2015 -0700
> HBASE-13376 Improvements to Stochastic load balancer (Vandana
> Ayyalasomayajula)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15181) A simple implementation of date based tiered compaction
[ https://issues.apache.org/jira/browse/HBASE-15181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133182#comment-15133182 ] Clara Xiong commented on HBASE-15181: - Yes， that serves the same purpose with the properly added check. > A simple implementation of date based tiered compaction > --- > > Key: HBASE-15181 > URL: https://issues.apache.org/jira/browse/HBASE-15181 > Project: HBase > Issue Type: New Feature > Components: Compaction >Reporter: Clara Xiong >Assignee: Clara Xiong > Fix For: 2.0.0 > > Attachments: HBASE-15181-v1.patch, HBASE-15181-v2.patch > > > This is a simple implementation of date-based tiered compaction similar to > Cassandra's for the following benefits: > 1. Improve date-range-based scan by structuring store files in date-based > tiered layout. > 2. Reduce compaction overhead. > 3. Improve TTL efficiency. > Perfect fit for the use cases that: > 1. has mostly date-based date write and scan and a focus on the most recent > data. > 2. never or rarely deletes data. > Out-of-order writes are handled gracefully so the data will still get to the > right store file for time-range-scan and re-compacton with existing store > file in the same time window is handled by ExploringCompactionPolicy. > Time range overlapping among store files is tolerated and the performance > impact is minimized. > Configuration can be set at hbase-site or overriden at per-table or > per-column-famly level by hbase shell. > Design spec is at > https://docs.google.com/document/d/1_AmlNb2N8Us1xICsTeGDLKIqL6T-oHoRLZ323MG_uy8/edit?usp=sharing -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15197) Expose filtered read requests metric to metrics framework and Web UI
[ https://issues.apache.org/jira/browse/HBASE-15197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133322#comment-15133322 ] Ted Yu commented on HBASE-15197: [~chenheng]: Please go ahead with commit. > Expose filtered read requests metric to metrics framework and Web UI > > > Key: HBASE-15197 > URL: https://issues.apache.org/jira/browse/HBASE-15197 > Project: HBase > Issue Type: Improvement >Reporter: Eungsop Yoo >Assignee: Eungsop Yoo >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15197-v1.patch, HBASE-15197-v2.patch, > HBASE-15197.patch, HBASE-15197.patch, WebUI-screenshot.png > > > Filtered read requests metric for scan is added in > [HBASE-5980|https://issues.apache.org/jira/browse/HBASE-5980]. But it can be > retrieved from Scan object only. I think that it would be more informative > when it is exposed to metrics framework. So I suggest to add filtered read > requests metric and to expose it metrics framework and Web UI. > ps. I think I found a bug; read requests count increased when Get operation > returns no record by filtering out. Should it be fixed? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15130) Backport 0.98 Scan different TimeRange for each column family
[ https://issues.apache.org/jira/browse/HBASE-15130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133186#comment-15133186 ] churro morales commented on HBASE-15130: In a review I was told to add fields to query Query.java in a specific manner, the checkstyle from hadoop QA sees this as an issue. I do not believe this is an issue, but will be happy to accommodate Hadoop QA if this feature gets upstream. I can fix the whitespaces and submit another patch. The other issues seem to be compounding and not related to this patch. > Backport 0.98 Scan different TimeRange for each column family > -- > > Key: HBASE-15130 > URL: https://issues.apache.org/jira/browse/HBASE-15130 > Project: HBase > Issue Type: Bug > Components: Client, regionserver, Scanners >Affects Versions: 0.98.17 >Reporter: churro morales >Assignee: churro morales > Fix For: 0.98.18 > > Attachments: HBASE-15130-0.98.patch, HBASE-15130-0.98.v1.patch, > HBASE-15130-0.98.v1.patch, HBASE-15130-0.98.v2.patch > > > branch 98 version backport for HBASE-14355 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HBASE-14979) Update to the newest Zookeeper release
[ https://issues.apache.org/jira/browse/HBASE-14979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi reopened HBASE-14979: --- Reopening the issue as the code change was reverted. > Update to the newest Zookeeper release > -- > > Key: HBASE-14979 > URL: https://issues.apache.org/jira/browse/HBASE-14979 > Project: HBase > Issue Type: Improvement > Components: Zookeeper >Affects Versions: 1.2.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-14979.patch > > > ZOOKEEPER-706 is nice to have for anyone running replication that sometimes > gets stalled. We should update to the latest patch version. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15218) On RS crash and replay of WAL, loosing all Tags in Cells
[ https://issues.apache.org/jira/browse/HBASE-15218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-15218: --- Attachment: HBASE-15218.patch > On RS crash and replay of WAL, loosing all Tags in Cells > > > Key: HBASE-15218 > URL: https://issues.apache.org/jira/browse/HBASE-15218 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 0.98.13, 1.0.2, 1.2.0, 1.1.1 >Reporter: Anoop Sam John >Assignee: Anoop Sam John >Priority: Critical > Attachments: HBASE-15218.patch > > > The KeyValueCodec's Decoder makes NoTagsKeyValue. The WalCellCodec also makes > this Decoder and so while reading Cells after RS crash, we will loose all > tags in Cells. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15218) On RS crash and replay of WAL, loosing all Tags in Cells
[ https://issues.apache.org/jira/browse/HBASE-15218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-15218: --- Hadoop Flags: Reviewed Status: Patch Available (was: Open) > On RS crash and replay of WAL, loosing all Tags in Cells > > > Key: HBASE-15218 > URL: https://issues.apache.org/jira/browse/HBASE-15218 > Project: HBase > Issue Type: Bug >Affects Versions: 1.1.1, 1.0.2, 0.98.13, 2.0.0, 1.2.0 >Reporter: Anoop Sam John >Assignee: Anoop Sam John >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4, 0.98.18 > > Attachments: HBASE-15218.patch > > > The KeyValueCodec's Decoder makes NoTagsKeyValue. The WalCellCodec also makes > this Decoder and so while reading Cells after RS crash, we will loose all > tags in Cells. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15218) On RS crash and replay of WAL, loosing all Tags in Cells
[ https://issues.apache.org/jira/browse/HBASE-15218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133779#comment-15133779 ] ramkrishna.s.vasudevan commented on HBASE-15218: +1 > On RS crash and replay of WAL, loosing all Tags in Cells > > > Key: HBASE-15218 > URL: https://issues.apache.org/jira/browse/HBASE-15218 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 0.98.13, 1.0.2, 1.2.0, 1.1.1 >Reporter: Anoop Sam John >Assignee: Anoop Sam John >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4, 0.98.18 > > Attachments: HBASE-15218.patch > > > The KeyValueCodec's Decoder makes NoTagsKeyValue. The WalCellCodec also makes > this Decoder and so while reading Cells after RS crash, we will loose all > tags in Cells. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15157) Add *PerformanceTest for Append, CheckAnd*
[ https://issues.apache.org/jira/browse/HBASE-15157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-15157: -- Assignee: stack Status: Patch Available (was: Open) > Add *PerformanceTest for Append, CheckAnd* > -- > > Key: HBASE-15157 > URL: https://issues.apache.org/jira/browse/HBASE-15157 > Project: HBase > Issue Type: Sub-task > Components: Performance, test >Reporter: stack >Assignee: stack > Attachments: 15157.patch > > > In a sibling issue we add IncrementPerformanceTest which is handy checking > Increment performance. Make a general tool to do all check-and-set type ops > like increment, append, checkAndPut, checkAndDelete... just to make sure no > regression in future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15157) Add *PerformanceTest for Append, CheckAnd*
[ https://issues.apache.org/jira/browse/HBASE-15157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-15157: -- Attachment: 15157.patch Add tests to PE for: increment append checkAndPut checkAndMutate checkAndDelete Tested the tool. Seems to do the right thing. Made it so if you have N clients, then they overlap in the keyspace so some contention. Usually, a Client in PE gets their own keyspace to operate in. > Add *PerformanceTest for Append, CheckAnd* > -- > > Key: HBASE-15157 > URL: https://issues.apache.org/jira/browse/HBASE-15157 > Project: HBase > Issue Type: Sub-task > Components: Performance, test >Reporter: stack > Attachments: 15157.patch > > > In a sibling issue we add IncrementPerformanceTest which is handy checking > Increment performance. Make a general tool to do all check-and-set type ops > like increment, append, checkAndPut, checkAndDelete... just to make sure no > regression in future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15213) Fix increment performance regression caused by HBASE-8763 on branch-1.0
[ https://issues.apache.org/jira/browse/HBASE-15213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133791#comment-15133791 ] stack commented on HBASE-15213: --- bq. ...but it was already so before HBASE-8763. So I couldn't see how HBASE-8763 made things worse than before. And I had to run some tests. This was lazyness on my part. The mvcc had its own counter apart from sequenceid and I just presumed that them spinning independent of each other made for less friction and so mvcc was less of a retardant. Thats a big difference in cs counts. Let me try a run. Will be back. Thanks for the great work. > Fix increment performance regression caused by HBASE-8763 on branch-1.0 > --- > > Key: HBASE-15213 > URL: https://issues.apache.org/jira/browse/HBASE-15213 > Project: HBase > Issue Type: Bug > Components: Performance >Reporter: Junegunn Choi >Assignee: Junegunn Choi > Attachments: HBASE-15213.branch-1.0.patch > > > This is an attempt to fix the increment performance regression caused by > HBASE-8763 on branch-1.0. > I'm aware that hbase.increment.fast.but.narrow.consistency was added to > branch-1.0 (HBASE-15031) to address the issue and a separate work is ongoing > on master branch, but anyway, this is my take on the problem. > I read through HBASE-14460 and HBASE-8763 but it wasn't clear to me what > caused the slowdown but I could indeed reproduce the performance regression. > Test setup: > - Server: 4-core Xeon 2.4GHz Linux server running mini cluster (100 handlers, > JDK 1.7) > - Client: Another box of the same spec > - Increments on random 10k records on a single-region table, recreated every > time > Increment throughput (TPS): > || Num threads || Before HBASE-8763 (d6cc2fb) || branch-1.0 || branch-1.0 > (narrow-consistency) || > || 1| 2661 | 2486| 2359 | > || 2| 5048 | 5064| 4867 | > || 4| 7503 | 8071| 8690 | > || 8| 10471| 10886 | 13980 | > || 16 | 15515| 9418| 18601 | > || 32 | 17699| 5421| 20540 | > || 64 | 20601| 4038| 25591 | > || 96 | 19177| 3891| 26017 | > We can clearly observe that the throughtput degrades as we increase the > number of concurrent requests, which led me to believe that there's severe > context switching overhead and I could indirectly confirm that suspicion with > cs entry in vmstat output. branch-1.0 shows a much higher number of context > switches even with much lower throughput. > Here are the observations: > - WriteEntry in the writeQueue can only be removed by the very handler that > put it, only when it is at the front of the queue and marked complete. > - Since a WriteEntry is marked complete after the wait-loop, only one entry > can be removed at a time. > - This stringent condition causes O(N^2) context switches where n is the > number of concurrent handlers processing requests. > So what I tried here is to mark WriteEntry complete before we go into > wait-loop. With the change, multiple WriteEntries can be shifted at a time > without context switches. I changed writeQueue to LinkedHashSet since fast > containment check is needed as WriteEntry can be removed by any handler. > The numbers look good, it's virtually identical to pre-HBASE-8763 era. > || Num threads || branch-1.0 with fix || > || 1| 2459 | > || 2| 4976 | > || 4| 8033 | > || 8| 12292| > || 16 | 15234| > || 32 | 16601| > || 64 | 19994| > || 96 | 20052| > So what do you think about it? Please let me know if I'm missing anything. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15210) Undo aggressive load balancer logging at tens of lines per millisecond
[
https://issues.apache.org/jira/browse/HBASE-15210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133542#comment-15133542
]
Hudson commented on HBASE-15210:
SUCCESS: Integrated in HBase-1.3-IT #478 (See
[https://builds.apache.org/job/HBase-1.3-IT/478/])
HBASE-15210 Undo aggressive load balancer logging at tens of lines per (stack:
rev 2203d55b7d58a32cd879886bd4ac02df9c0fd387)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java
HBASE-15210 Undo aggressive load balancer logging at tens of lines per (stack:
rev 846a2f31c1f9ff2298e3dbc81e526ae92e5dc692)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java
> Undo aggressive load balancer logging at tens of lines per millisecond
> --
>
> Key: HBASE-15210
> URL: https://issues.apache.org/jira/browse/HBASE-15210
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer
>Affects Versions: 1.2.0
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 15210.patch
>
>
> Sometimes the load balancer falls into a state of aggressive logging in 1.2.0
> writing out ten lines a millisecond like this:
> {code}
> 2016-02-02 17:15:01,382 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region server with non zero
> regions is ve0540.halxg.cloudera.com with locality 0.22908874
> 2016-02-02 17:15:01,382 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region index is 3 and its region
> server contains 4 regions
> 2016-02-02 17:15:01,384 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region server with non zero
> regions is ve0540.halxg.cloudera.com with locality 0.22908874
> 2016-02-02 17:15:01,384 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region index is 3 and its region
> server contains 4 regions
> 2016-02-02 17:15:01,386 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region server with non zero
> regions is ve0540.halxg.cloudera.com with locality 0.22908874
> 2016-02-02 17:15:01,386 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region index is 2 and its region
> server contains 4 regions
> 2016-02-02 17:15:01,387 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region server with non zero
> regions is ve0540.halxg.cloudera.com with locality 0.22908874
> 2016-02-02 17:15:01,387 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region index is 3 and its region
> server contains 4 regions
> 2016-02-02 17:15:01,388 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region server with non zero
> regions is ve0540.halxg.cloudera.com with locality 0.22908874
> 2016-02-02 17:15:01,388 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region index is 3 and its region
> server contains 4 regions
> 2016-02-02 17:15:01,388 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region server with non zero
> regions is ve0540.halxg.cloudera.com with locality 0.22908874
> {code}
> This issue is about less logging.
> This log was added recently by this:
> commit 54028140f4f19a6af81c8c8f29dda0c52491a0c9
> Author: tedyu
> Date: Thu Aug 13 09:11:59 2015 -0700
> HBASE-13376 Improvements to Stochastic load balancer (Vandana
> Ayyalasomayajula)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15200) ZooKeeper znode ACL checks should only compare the shortname
[ https://issues.apache.org/jira/browse/HBASE-15200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133541#comment-15133541 ] Hudson commented on HBASE-15200: SUCCESS: Integrated in HBase-1.3-IT #478 (See [https://builds.apache.org/job/HBase-1.3-IT/478/]) Amend HBASE-15200 ZooKeeper znode ACL checks should only compare the (apurtell: rev 1b420be56a530e5b07ed8f478d5aa4916abeb438) * hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java > ZooKeeper znode ACL checks should only compare the shortname > > > Key: HBASE-15200 > URL: https://issues.apache.org/jira/browse/HBASE-15200 > Project: HBase > Issue Type: Bug > Components: security >Affects Versions: 2.0.0, 1.2.0, 1.0.3, 1.1.3, 0.98.17 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4, 0.98.18 > > Attachments: HBASE-15200-branch-1.0.patch, > HBASE-15200-branch-1.1.patch, HBASE-15200.patch, HBASE-15200.patch > > > After HBASE-13768 we check at startup in secure configurations if our znodes > have the correct ACLs. However when checking the ACL we compare the Kerberos > fullname, which includes the host component. We should only compare the > shortname, the principal. Otherwise in a multimaster configuration we will > unnecessarily reset ACLs whenever any master running on a host other than the > one that initialized the ACLs makes the check. You can imagine this happening > multiple times in a rolling restart scenario. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15120) Backport HBASE-14883 to branch-1.1
[ https://issues.apache.org/jira/browse/HBASE-15120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133543#comment-15133543 ] Hudson commented on HBASE-15120: SUCCESS: Integrated in HBase-1.3-IT #478 (See [https://builds.apache.org/job/HBase-1.3-IT/478/]) Revert "HBASE-15120 Undo aggressive load balancer logging at tens of (stack: rev 8061fa4af8adb512e5c61374fdef8870e94d281a) * hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java > Backport HBASE-14883 to branch-1.1 > -- > > Key: HBASE-15120 > URL: https://issues.apache.org/jira/browse/HBASE-15120 > Project: HBase > Issue Type: Bug >Affects Versions: 1.1.2 >Reporter: Yu Li >Assignee: Yu Li >Priority: Minor > Attachments: HBASE-15120.branch-1.1.patch, > HBASE-15120.branch-1.1.patch > > > When checking branch-1.1 UT in HBASE-13590, found > TestSplitTransactionOnCluster#testFailedSplit will fail at a 12/50 chance. > The issue is fixed by HBASE-14883 but the change didn't go into branch-1.1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-8815) A replicated cross cluster client
[ https://issues.apache.org/jira/browse/HBASE-8815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133752#comment-15133752 ] cuixin commented on HBASE-8815: --- Are you talk about the https://github.com/tmalaska/HBase.MCC ? > A replicated cross cluster client > - > > Key: HBASE-8815 > URL: https://issues.apache.org/jira/browse/HBASE-8815 > Project: HBase > Issue Type: New Feature >Reporter: Varun Sharma > > I would like to float this idea for brain storming. > HBase is a strongly consistent system modelled after bigtable which means a > machine going down results in loss of availability of around 2 minutes as it > stands today. So there is a trade off. > However, for high availability and redundancy, it is common practice for > online/mission critical applications to run replicated clusters. For example, > we run replicated clusters at pinterest in different EC2 az(s) and at google, > critical data is always replicated across bigtable cells. > At high volumes, 2 minutes of downtime can also be critical, however, today > our client does not make use of the fact, that there is an available slave > replica cluster from which slightly inconsistent data can be read. It only > reads from one cluster. When you have replication, it is a very common > practice for reading from slave if the error rate from master is high. That > is how, web sites serve data out of MySQL and survive machine failures by > directing their reads to slave machines when the master goes down. > I am sure folks love the strong consistency gaurantee from HBase, but I think > that this way, we can make better use of the replica cluster, much in the > same way people use MySQL slaves for reads. In case of regions going offline, > it would be nice if, for the offline regions only (a small fraction), reads > could be directed to the slave cluster. > I know one company which follows this model. At Google, a replicated client > api is used for reads which is able to farm reads to multiple clusters and > also writes to multiple clusters depending on availability in case of Multi > master replication. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15200) ZooKeeper znode ACL checks should only compare the shortname
[ https://issues.apache.org/jira/browse/HBASE-15200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133762#comment-15133762 ] Hudson commented on HBASE-15200: FAILURE: Integrated in HBase-1.3 #536 (See [https://builds.apache.org/job/HBase-1.3/536/]) Amend HBASE-15200 ZooKeeper znode ACL checks should only compare the (apurtell: rev 1b420be56a530e5b07ed8f478d5aa4916abeb438) * hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java > ZooKeeper znode ACL checks should only compare the shortname > > > Key: HBASE-15200 > URL: https://issues.apache.org/jira/browse/HBASE-15200 > Project: HBase > Issue Type: Bug > Components: security >Affects Versions: 2.0.0, 1.2.0, 1.0.3, 1.1.3, 0.98.17 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4, 0.98.18 > > Attachments: HBASE-15200-branch-1.0.patch, > HBASE-15200-branch-1.1.patch, HBASE-15200.patch, HBASE-15200.patch > > > After HBASE-13768 we check at startup in secure configurations if our znodes > have the correct ACLs. However when checking the ACL we compare the Kerberos > fullname, which includes the host component. We should only compare the > shortname, the principal. Otherwise in a multimaster configuration we will > unnecessarily reset ACLs whenever any master running on a host other than the > one that initialized the ACLs makes the check. You can imagine this happening > multiple times in a rolling restart scenario. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15210) Undo aggressive load balancer logging at tens of lines per millisecond
[
https://issues.apache.org/jira/browse/HBASE-15210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133763#comment-15133763
]
Hudson commented on HBASE-15210:
FAILURE: Integrated in HBase-1.3 #536 (See
[https://builds.apache.org/job/HBase-1.3/536/])
HBASE-15210 Undo aggressive load balancer logging at tens of lines per (stack:
rev 846a2f31c1f9ff2298e3dbc81e526ae92e5dc692)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java
> Undo aggressive load balancer logging at tens of lines per millisecond
> --
>
> Key: HBASE-15210
> URL: https://issues.apache.org/jira/browse/HBASE-15210
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer
>Affects Versions: 1.2.0
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 15210.patch
>
>
> Sometimes the load balancer falls into a state of aggressive logging in 1.2.0
> writing out ten lines a millisecond like this:
> {code}
> 2016-02-02 17:15:01,382 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region server with non zero
> regions is ve0540.halxg.cloudera.com with locality 0.22908874
> 2016-02-02 17:15:01,382 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region index is 3 and its region
> server contains 4 regions
> 2016-02-02 17:15:01,384 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region server with non zero
> regions is ve0540.halxg.cloudera.com with locality 0.22908874
> 2016-02-02 17:15:01,384 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region index is 3 and its region
> server contains 4 regions
> 2016-02-02 17:15:01,386 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region server with non zero
> regions is ve0540.halxg.cloudera.com with locality 0.22908874
> 2016-02-02 17:15:01,386 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region index is 2 and its region
> server contains 4 regions
> 2016-02-02 17:15:01,387 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region server with non zero
> regions is ve0540.halxg.cloudera.com with locality 0.22908874
> 2016-02-02 17:15:01,387 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region index is 3 and its region
> server contains 4 regions
> 2016-02-02 17:15:01,388 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region server with non zero
> regions is ve0540.halxg.cloudera.com with locality 0.22908874
> 2016-02-02 17:15:01,388 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region index is 3 and its region
> server contains 4 regions
> 2016-02-02 17:15:01,388 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region server with non zero
> regions is ve0540.halxg.cloudera.com with locality 0.22908874
> {code}
> This issue is about less logging.
> This log was added recently by this:
> commit 54028140f4f19a6af81c8c8f29dda0c52491a0c9
> Author: tedyu
> Date: Thu Aug 13 09:11:59 2015 -0700
> HBASE-13376 Improvements to Stochastic load balancer (Vandana
> Ayyalasomayajula)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15157) Add *PerformanceTest for Append, CheckAnd*
[
https://issues.apache.org/jira/browse/HBASE-15157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-15157:
--
Release Note:
Add append, increment, checkAndMutate, checkAndPut, and checkAndDelete tests to
PerformanceEvaluation tool. Below are excerpts from new usage from PE:
{code}
Command:
append Append on each row; clients overlap on keyspace so some
concurrent operations
checkAndDelete CheckAndDelete on each row; clients overlap on keyspace so
some concurrent operations
checkAndMutate CheckAndMutate on each row; clients overlap on keyspace so
some concurrent operations
checkAndPut CheckAndPut on each row; clients overlap on keyspace so some
concurrent operations
filterScan Run scan test using a filter to find a specific row based on
it's value (make sure to use --rows=20)
increment Increment on each row; clients overlap on keyspace so some
concurrent operations
randomRead Run random read test
Examples:
...
To run 10 clients doing increments over ten rows:
$ bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 --nomapred
increment 10
{code}
> Add *PerformanceTest for Append, CheckAnd*
> --
>
> Key: HBASE-15157
> URL: https://issues.apache.org/jira/browse/HBASE-15157
> Project: HBase
> Issue Type: Sub-task
> Components: Performance, test
>Reporter: stack
>Assignee: stack
> Attachments: 15157.patch
>
>
> In a sibling issue we add IncrementPerformanceTest which is handy checking
> Increment performance. Make a general tool to do all check-and-set type ops
> like increment, append, checkAndPut, checkAndDelete... just to make sure no
> regression in future.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[
https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133792#comment-15133792
]
Hadoop QA commented on HBASE-15204:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
46s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 1s
{color} | {color:red} Patch generated 1 new checkstyle issues in hbase-server
(total was 284, now 284). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
22m 52s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 4s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 104m 9s {color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0_72. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 45s {color}
| {color:red} hbase-server in the patch failed with JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 161m 10s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_72 Failed junit tests | hadoop.hbase.mob.mapreduce.TestMobSweepJob
|
| | hadoop.hbase.client.TestBlockEvictionFromClient |
| JDK v1.8.0_72 Timed out junit tests | org.apache.hadoop.hbase.TestZooKeeper |
| | org.apache.hadoop.hbase.snapshot.TestSecureExportSnapshot |
| | org.apache.hadoop.hbase.snapshot.TestExportSnapshot |
| |
[jira] [Created] (HBASE-15218) On RS crash and replay of WAL, loosing all Tags in Cells
Anoop Sam John created HBASE-15218: -- Summary: On RS crash and replay of WAL, loosing all Tags in Cells Key: HBASE-15218 URL: https://issues.apache.org/jira/browse/HBASE-15218 Project: HBase Issue Type: Bug Affects Versions: 1.1.1, 1.0.2, 0.98.13, 2.0.0, 1.2.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Priority: Critical The KeyValueCodec's Decoder makes NoTagsKeyValue. The WalCellCodec also makes this Decoder and so while reading Cells after RS crash, we will loose all tags in Cells. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15218) On RS crash and replay of WAL, loosing all Tags in Cells
[ https://issues.apache.org/jira/browse/HBASE-15218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-15218: --- Fix Version/s: 0.98.18 1.0.4 1.1.4 1.2.1 1.3.0 2.0.0 > On RS crash and replay of WAL, loosing all Tags in Cells > > > Key: HBASE-15218 > URL: https://issues.apache.org/jira/browse/HBASE-15218 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 0.98.13, 1.0.2, 1.2.0, 1.1.1 >Reporter: Anoop Sam John >Assignee: Anoop Sam John >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4, 0.98.18 > > Attachments: HBASE-15218.patch > > > The KeyValueCodec's Decoder makes NoTagsKeyValue. The WalCellCodec also makes > this Decoder and so while reading Cells after RS crash, we will loose all > tags in Cells. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15157) Add *PerformanceTest for Append, CheckAnd*
[
https://issues.apache.org/jira/browse/HBASE-15157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-15157:
--
Release Note:
Add append, increment, checkAndMutate, checkAndPut, and checkAndDelete tests to
PerformanceEvaluation tool. Below are excerpts from new usage from PE:
Command:
append Append on each row; clients overlap on keyspace so some
concurrent operations
checkAndDelete CheckAndDelete on each row; clients overlap on keyspace so
some concurrent operations
checkAndMutate CheckAndMutate on each row; clients overlap on keyspace so
some concurrent operations
checkAndPut CheckAndPut on each row; clients overlap on keyspace so some
concurrent operations
filterScan Run scan test using a filter to find a specific row based on
it's value (make sure to use --rows=20)
increment Increment on each row; clients overlap on keyspace so some
concurrent operations
randomRead Run random read test
Examples:
...
To run 10 clients doing increments over ten rows:
$ bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 --nomapred
increment 10
Removed IncrementPerformanceTest. It is not as configurable as the additions
made here.
was:
Add append, increment, checkAndMutate, checkAndPut, and checkAndDelete tests to
PerformanceEvaluation tool. Below are excerpts from new usage from PE:
{code}
Command:
append Append on each row; clients overlap on keyspace so some
concurrent operations
checkAndDelete CheckAndDelete on each row; clients overlap on keyspace so
some concurrent operations
checkAndMutate CheckAndMutate on each row; clients overlap on keyspace so
some concurrent operations
checkAndPut CheckAndPut on each row; clients overlap on keyspace so some
concurrent operations
filterScan Run scan test using a filter to find a specific row based on
it's value (make sure to use --rows=20)
increment Increment on each row; clients overlap on keyspace so some
concurrent operations
randomRead Run random read test
Examples:
...
To run 10 clients doing increments over ten rows:
$ bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 --nomapred
increment 10
{code}
> Add *PerformanceTest for Append, CheckAnd*
> --
>
> Key: HBASE-15157
> URL: https://issues.apache.org/jira/browse/HBASE-15157
> Project: HBase
> Issue Type: Sub-task
> Components: Performance, test
>Reporter: stack
>Assignee: stack
> Attachments: 15157.patch
>
>
> In a sibling issue we add IncrementPerformanceTest which is handy checking
> Increment performance. Make a general tool to do all check-and-set type ops
> like increment, append, checkAndPut, checkAndDelete... just to make sure no
> regression in future.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15157) Add *PerformanceTest for Append, CheckAnd*
[
https://issues.apache.org/jira/browse/HBASE-15157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133788#comment-15133788
]
Hadoop QA commented on HBASE-15157:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s {color}
| {color:red} HBASE-15157 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/latest/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12786431/15157.patch |
| JIRA Issue | HBASE-15157 |
| Powered by | Apache Yetus 0.1.0 http://yetus.apache.org |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/435/console |
This message was automatically generated.
> Add *PerformanceTest for Append, CheckAnd*
> --
>
> Key: HBASE-15157
> URL: https://issues.apache.org/jira/browse/HBASE-15157
> Project: HBase
> Issue Type: Sub-task
> Components: Performance, test
>Reporter: stack
>Assignee: stack
> Attachments: 15157.patch
>
>
> In a sibling issue we add IncrementPerformanceTest which is handy checking
> Increment performance. Make a general tool to do all check-and-set type ops
> like increment, append, checkAndPut, checkAndDelete... just to make sure no
> regression in future.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15202) Reduce garbage while setting response
[ https://issues.apache.org/jira/browse/HBASE-15202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133364#comment-15133364 ] Enis Soztutar commented on HBASE-15202: --- This can also go to branch-1 I think. > Reduce garbage while setting response > - > > Key: HBASE-15202 > URL: https://issues.apache.org/jira/browse/HBASE-15202 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15202.patch, HBASE-15202_1.patch, > HBASE-15202_2.patch, HBASE-15202_3.patch > > > Probably this can move under HBASE-15177. During RpcServer#setResponse we do > IPCUtil.getDelimitedMessageAsByteBuffer for the header and result. This > internally creates a byte[], CodedOutputStream and a BB. Then the totalSize > is also wrapped in a BB. > Finally all these BB are passed to BufferChain along with the Cellblock. > This JIRA is to reduce the number of objects that we create here and allow > one BB from this header, result and total Size along with one > CodedOutputStream and pass this BB along with the Cellblocks to the > Bufferchain. From the JMC tool can observe around 2% lesser object allocation > arising out of these objects. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15211) Don't run the CatalogJanitor if there are regions in transition
[ https://issues.apache.org/jira/browse/HBASE-15211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133355#comment-15133355 ] Hudson commented on HBASE-15211: FAILURE: Integrated in HBase-1.3 #534 (See [https://builds.apache.org/job/HBase-1.3/534/]) HBASE-15211 Don't run the CatalogJanitor if there are regions in (eclark: rev 908e5a66208c1f0cda76fb4f8674e104275fb17e) * hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java > Don't run the CatalogJanitor if there are regions in transition > --- > > Key: HBASE-15211 > URL: https://issues.apache.org/jira/browse/HBASE-15211 > Project: HBase > Issue Type: Improvement > Components: master >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4 > > Attachments: HBASE-15211.patch > > > This could make things a little safer to make sure that CatalogJanitor > doesn't do something weird while a cluster is starting up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15211) Don't run the CatalogJanitor if there are regions in transition
[ https://issues.apache.org/jira/browse/HBASE-15211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133389#comment-15133389 ] Hudson commented on HBASE-15211: SUCCESS: Integrated in HBase-1.1-JDK7 #1651 (See [https://builds.apache.org/job/HBase-1.1-JDK7/1651/]) HBASE-15211 Don't run the CatalogJanitor if there are regions in (eclark: rev bc564d3a6a0e0bc41dc35eb799b2d9e3f8a10bf2) * hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java > Don't run the CatalogJanitor if there are regions in transition > --- > > Key: HBASE-15211 > URL: https://issues.apache.org/jira/browse/HBASE-15211 > Project: HBase > Issue Type: Improvement > Components: master >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4 > > Attachments: HBASE-15211.patch > > > This could make things a little safer to make sure that CatalogJanitor > doesn't do something weird while a cluster is starting up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14919) Infrastructure refactoring for In-Memory Flush
[ https://issues.apache.org/jira/browse/HBASE-14919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133328#comment-15133328 ] stack commented on HBASE-14919: --- bq. Task #4 in the umbrella issue implements another ImmutableSegment (CellBlocksSrgment), which does not take MutableSegment on construction. So, no adapter is necessary in this case? How are they made? bq. It is possible to do it this way. We believe exposing an unnecessary API which incurs costly implementation is not advisable. What you say makes sense. What you think of my continual stumbling over this ImmutableSegment vs MutableSegment and that there are sometimes Adapters? bq. This method returns a KeyValueScanner that is used by MemStoreSnapshot, but it scans the cells in the segment who generated it. My misunderstanding. Thanks. bq. Would you like better the name getKeyValueScanner()? And the other method would be getSegmentScanner(). Yes. This would be clearer I believe. Thats all my concerns addressed. Let me try and get a basic test run to make sure basically performs the same but put up a v11 w/ the above small changes and I'll commit unless objection from others ([~anoop.hbase] or [~enis]). Thanks [~eshcar]. > Infrastructure refactoring for In-Memory Flush > -- > > Key: HBASE-14919 > URL: https://issues.apache.org/jira/browse/HBASE-14919 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel > Attachments: HBASE-14919-V01.patch, HBASE-14919-V01.patch, > HBASE-14919-V02.patch, HBASE-14919-V03.patch, HBASE-14919-V04.patch, > HBASE-14919-V04.patch, HBASE-14919-V05.patch, HBASE-14919-V06.patch, > HBASE-14919-V06.patch, HBASE-14919-V07.patch, HBASE-14919-V08.patch, > HBASE-14919-V09.patch, HBASE-14919-V10.patch > > > Refactoring the MemStore hierarchy, introducing segment (StoreSegment) as > first-class citizen and decoupling memstore scanner from the memstore > implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15120) Backport HBASE-14883 to branch-1.1
[ https://issues.apache.org/jira/browse/HBASE-15120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133358#comment-15133358 ] Hudson commented on HBASE-15120: SUCCESS: Integrated in HBase-1.2-IT #419 (See [https://builds.apache.org/job/HBase-1.2-IT/419/]) HBASE-15120 Undo aggressive load balancer logging at tens of lines per (stack: rev 69b70b87aa2ddd808623912d93d627fe02e211f4) * hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java > Backport HBASE-14883 to branch-1.1 > -- > > Key: HBASE-15120 > URL: https://issues.apache.org/jira/browse/HBASE-15120 > Project: HBase > Issue Type: Bug >Affects Versions: 1.1.2 >Reporter: Yu Li >Assignee: Yu Li >Priority: Minor > Attachments: HBASE-15120.branch-1.1.patch, > HBASE-15120.branch-1.1.patch > > > When checking branch-1.1 UT in HBASE-13590, found > TestSplitTransactionOnCluster#testFailedSplit will fail at a 12/50 chance. > The issue is fixed by HBASE-14883 but the change didn't go into branch-1.1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14030) HBase Backup/Restore Phase 1
[ https://issues.apache.org/jira/browse/HBASE-14030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133438#comment-15133438 ] Vladimir Rodionov commented on HBASE-14030: --- You do not have the latest patch, Ted. It works now. Can you wait a little bit? I am going to submit new patch tomorrow. > HBase Backup/Restore Phase 1 > > > Key: HBASE-14030 > URL: https://issues.apache.org/jira/browse/HBASE-14030 > Project: HBase > Issue Type: Umbrella >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > Attachments: HBASE-14030-v0.patch, HBASE-14030-v1.patch, > HBASE-14030-v10.patch, HBASE-14030-v11.patch, HBASE-14030-v12.patch, > HBASE-14030-v13.patch, HBASE-14030-v14.patch, HBASE-14030-v15.patch, > HBASE-14030-v17.patch, HBASE-14030-v18.patch, HBASE-14030-v2.patch, > HBASE-14030-v20.patch, HBASE-14030-v21.patch, HBASE-14030-v22.patch, > HBASE-14030-v23.patch, HBASE-14030-v24.patch, HBASE-14030-v25.patch, > HBASE-14030-v26.patch, HBASE-14030-v27.patch, HBASE-14030-v28.patch, > HBASE-14030-v3.patch, HBASE-14030-v30.patch, HBASE-14030-v4.patch, > HBASE-14030-v5.patch, HBASE-14030-v6.patch, HBASE-14030-v7.patch, > HBASE-14030-v8.patch > > > This is the umbrella ticket for Backup/Restore Phase 1. See HBASE-7912 design > doc for the phase description. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15197) Expose filtered read requests metric to metrics framework and Web UI
[ https://issues.apache.org/jira/browse/HBASE-15197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heng Chen updated HBASE-15197: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) > Expose filtered read requests metric to metrics framework and Web UI > > > Key: HBASE-15197 > URL: https://issues.apache.org/jira/browse/HBASE-15197 > Project: HBase > Issue Type: Improvement >Reporter: Eungsop Yoo >Assignee: Eungsop Yoo >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15197-v1.patch, HBASE-15197-v2.patch, > HBASE-15197.patch, HBASE-15197.patch, WebUI-screenshot.png > > > Filtered read requests metric for scan is added in > [HBASE-5980|https://issues.apache.org/jira/browse/HBASE-5980]. But it can be > retrieved from Scan object only. I think that it would be more informative > when it is exposed to metrics framework. So I suggest to add filtered read > requests metric and to expose it metrics framework and Web UI. > ps. I think I found a bug; read requests count increased when Get operation > returns no record by filtering out. Should it be fixed? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15212) RRCServer should enforce max request size
[ https://issues.apache.org/jira/browse/HBASE-15212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133413#comment-15133413 ] Elliott Clark commented on HBASE-15212: --- Yeah that's wy too aggressive. We shouldn't be guarding against lots of requests being too large. That's the job of total call queue size. We should only be guarding against things that are outrageously large. I would vote 256mb flat or something like that otherwise we are going to break existing people pretty hard. > RRCServer should enforce max request size > -- > > Key: HBASE-15212 > URL: https://issues.apache.org/jira/browse/HBASE-15212 > Project: HBase > Issue Type: Improvement >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 2.0.0, 1.3.0 > > Attachments: hbase-15212_v1.patch > > > A TODO from HBASE-15177 was that we are not protecting the RPCServer in case > an RPC request with a very large size is received. This might cause the > server to go OOM because we are allocating the RPC serialization into a BB. > Instead we should reject the RPC and close the connection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15202) Reduce garbage while setting response
[ https://issues.apache.org/jira/browse/HBASE-15202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133376#comment-15133376 ] Hudson commented on HBASE-15202: FAILURE: Integrated in HBase-Trunk_matrix #680 (See [https://builds.apache.org/job/HBase-Trunk_matrix/680/]) HBASE-15202 Reduce garbage while setting response (Ram) (ramkrishna: rev 7b33a740b10b05b50f8e9d3b2a1ef37593cb6eb3) * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java > Reduce garbage while setting response > - > > Key: HBASE-15202 > URL: https://issues.apache.org/jira/browse/HBASE-15202 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15202.patch, HBASE-15202_1.patch, > HBASE-15202_2.patch, HBASE-15202_3.patch > > > Probably this can move under HBASE-15177. During RpcServer#setResponse we do > IPCUtil.getDelimitedMessageAsByteBuffer for the header and result. This > internally creates a byte[], CodedOutputStream and a BB. Then the totalSize > is also wrapped in a BB. > Finally all these BB are passed to BufferChain along with the Cellblock. > This JIRA is to reduce the number of objects that we create here and allow > one BB from this header, result and total Size along with one > CodedOutputStream and pass this BB along with the Cellblocks to the > Bufferchain. From the JMC tool can observe around 2% lesser object allocation > arising out of these objects. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15197) Expose filtered read requests metric to metrics framework and Web UI
[ https://issues.apache.org/jira/browse/HBASE-15197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133551#comment-15133551 ] Heng Chen commented on HBASE-15197: --- Push to master. > Expose filtered read requests metric to metrics framework and Web UI > > > Key: HBASE-15197 > URL: https://issues.apache.org/jira/browse/HBASE-15197 > Project: HBase > Issue Type: Improvement >Reporter: Eungsop Yoo >Assignee: Eungsop Yoo >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15197-v1.patch, HBASE-15197-v2.patch, > HBASE-15197.patch, HBASE-15197.patch, WebUI-screenshot.png > > > Filtered read requests metric for scan is added in > [HBASE-5980|https://issues.apache.org/jira/browse/HBASE-5980]. But it can be > retrieved from Scan object only. I think that it would be more informative > when it is exposed to metrics framework. So I suggest to add filtered read > requests metric and to expose it metrics framework and Web UI. > ps. I think I found a bug; read requests count increased when Get operation > returns no record by filtering out. Should it be fixed? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14030) HBase Backup/Restore Phase 1
[ https://issues.apache.org/jira/browse/HBASE-14030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133430#comment-15133430 ] Ted Yu commented on HBASE-14030: I ran backup tests based on latest patch from reviewboard. TestBackupLogCleaner seems to hang. Can you check ? > HBase Backup/Restore Phase 1 > > > Key: HBASE-14030 > URL: https://issues.apache.org/jira/browse/HBASE-14030 > Project: HBase > Issue Type: Umbrella >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > Attachments: HBASE-14030-v0.patch, HBASE-14030-v1.patch, > HBASE-14030-v10.patch, HBASE-14030-v11.patch, HBASE-14030-v12.patch, > HBASE-14030-v13.patch, HBASE-14030-v14.patch, HBASE-14030-v15.patch, > HBASE-14030-v17.patch, HBASE-14030-v18.patch, HBASE-14030-v2.patch, > HBASE-14030-v20.patch, HBASE-14030-v21.patch, HBASE-14030-v22.patch, > HBASE-14030-v23.patch, HBASE-14030-v24.patch, HBASE-14030-v25.patch, > HBASE-14030-v26.patch, HBASE-14030-v27.patch, HBASE-14030-v28.patch, > HBASE-14030-v3.patch, HBASE-14030-v30.patch, HBASE-14030-v4.patch, > HBASE-14030-v5.patch, HBASE-14030-v6.patch, HBASE-14030-v7.patch, > HBASE-14030-v8.patch > > > This is the umbrella ticket for Backup/Restore Phase 1. See HBASE-7912 design > doc for the phase description. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15120) Backport HBASE-14883 to branch-1.1
[ https://issues.apache.org/jira/browse/HBASE-15120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133512#comment-15133512 ] Hudson commented on HBASE-15120: SUCCESS: Integrated in HBase-1.2 #529 (See [https://builds.apache.org/job/HBase-1.2/529/]) HBASE-15120 Undo aggressive load balancer logging at tens of lines per (stack: rev 69b70b87aa2ddd808623912d93d627fe02e211f4) * hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java > Backport HBASE-14883 to branch-1.1 > -- > > Key: HBASE-15120 > URL: https://issues.apache.org/jira/browse/HBASE-15120 > Project: HBase > Issue Type: Bug >Affects Versions: 1.1.2 >Reporter: Yu Li >Assignee: Yu Li >Priority: Minor > Attachments: HBASE-15120.branch-1.1.patch, > HBASE-15120.branch-1.1.patch > > > When checking branch-1.1 UT in HBASE-13590, found > TestSplitTransactionOnCluster#testFailedSplit will fail at a 12/50 chance. > The issue is fixed by HBASE-14883 but the change didn't go into branch-1.1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15212) RRCServer should enforce max request size
[
https://issues.apache.org/jira/browse/HBASE-15212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133525#comment-15133525
]
Hadoop QA commented on HBASE-15212:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
38s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
59s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
52s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
23m 21s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 29m 56s {color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0_72. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 31m 53s {color}
| {color:red} hbase-server in the patch failed with JDK v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 107m 14s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_72 Timed out junit tests |
org.apache.hadoop.hbase.ipc.TestAsyncIPC |
| JDK v1.7.0_91 Timed out junit tests |
org.apache.hadoop.hbase.ipc.TestAsyncIPC |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.9.1 Server=1.9.1 Image:yetus/hbase:date2016-02-05 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12786192/hbase-15212_v1.patch |
| JIRA Issue | HBASE-15212 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 462fab42be87
[jira] [Commented] (HBASE-15120) Backport HBASE-14883 to branch-1.1
[ https://issues.apache.org/jira/browse/HBASE-15120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133572#comment-15133572 ] Hudson commented on HBASE-15120: FAILURE: Integrated in HBase-1.3 #535 (See [https://builds.apache.org/job/HBase-1.3/535/]) HBASE-15120 Undo aggressive load balancer logging at tens of lines per (stack: rev 66da601cdcc915c3ccebd994105c8f7fbebf556d) * hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java Revert "HBASE-15120 Undo aggressive load balancer logging at tens of (stack: rev 8061fa4af8adb512e5c61374fdef8870e94d281a) * hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java > Backport HBASE-14883 to branch-1.1 > -- > > Key: HBASE-15120 > URL: https://issues.apache.org/jira/browse/HBASE-15120 > Project: HBase > Issue Type: Bug >Affects Versions: 1.1.2 >Reporter: Yu Li >Assignee: Yu Li >Priority: Minor > Attachments: HBASE-15120.branch-1.1.patch, > HBASE-15120.branch-1.1.patch > > > When checking branch-1.1 UT in HBASE-13590, found > TestSplitTransactionOnCluster#testFailedSplit will fail at a 12/50 chance. > The issue is fixed by HBASE-14883 but the change didn't go into branch-1.1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15210) Undo aggressive load balancer logging at tens of lines per millisecond
[
https://issues.apache.org/jira/browse/HBASE-15210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133570#comment-15133570
]
Hudson commented on HBASE-15210:
FAILURE: Integrated in HBase-1.3 #535 (See
[https://builds.apache.org/job/HBase-1.3/535/])
HBASE-15210 Undo aggressive load balancer logging at tens of lines per (stack:
rev 2203d55b7d58a32cd879886bd4ac02df9c0fd387)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java
> Undo aggressive load balancer logging at tens of lines per millisecond
> --
>
> Key: HBASE-15210
> URL: https://issues.apache.org/jira/browse/HBASE-15210
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer
>Affects Versions: 1.2.0
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 15210.patch
>
>
> Sometimes the load balancer falls into a state of aggressive logging in 1.2.0
> writing out ten lines a millisecond like this:
> {code}
> 2016-02-02 17:15:01,382 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region server with non zero
> regions is ve0540.halxg.cloudera.com with locality 0.22908874
> 2016-02-02 17:15:01,382 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region index is 3 and its region
> server contains 4 regions
> 2016-02-02 17:15:01,384 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region server with non zero
> regions is ve0540.halxg.cloudera.com with locality 0.22908874
> 2016-02-02 17:15:01,384 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region index is 3 and its region
> server contains 4 regions
> 2016-02-02 17:15:01,386 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region server with non zero
> regions is ve0540.halxg.cloudera.com with locality 0.22908874
> 2016-02-02 17:15:01,386 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region index is 2 and its region
> server contains 4 regions
> 2016-02-02 17:15:01,387 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region server with non zero
> regions is ve0540.halxg.cloudera.com with locality 0.22908874
> 2016-02-02 17:15:01,387 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region index is 3 and its region
> server contains 4 regions
> 2016-02-02 17:15:01,388 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region server with non zero
> regions is ve0540.halxg.cloudera.com with locality 0.22908874
> 2016-02-02 17:15:01,388 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region index is 3 and its region
> server contains 4 regions
> 2016-02-02 17:15:01,388 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region server with non zero
> regions is ve0540.halxg.cloudera.com with locality 0.22908874
> {code}
> This issue is about less logging.
> This log was added recently by this:
> commit 54028140f4f19a6af81c8c8f29dda0c52491a0c9
> Author: tedyu
> Date: Thu Aug 13 09:11:59 2015 -0700
> HBASE-13376 Improvements to Stochastic load balancer (Vandana
> Ayyalasomayajula)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15177) Reduce garbage created under high load
[ https://issues.apache.org/jira/browse/HBASE-15177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133571#comment-15133571 ] Hudson commented on HBASE-15177: FAILURE: Integrated in HBase-1.3 #535 (See [https://builds.apache.org/job/HBase-1.3/535/]) HBASE-15177 Reduce garbage created under high load (enis: rev 73d67788206c3f60773d861375f5e6934a284418) * hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/RpcClientImpl.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/AnnotationReadingPriorityFunction.java * hbase-common/src/main/java/org/apache/hadoop/hbase/util/ByteBufferUtils.java * hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/AsyncRpcChannel.java * hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java * hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/PayloadCarryingRpcController.java * hbase-client/src/test/java/org/apache/hadoop/hbase/ipc/TestIPCUtil.java * hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/IPCUtil.java * hbase-common/src/main/java/org/apache/hadoop/hbase/util/UnsafeAccess.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java * hbase-common/src/main/java/org/apache/hadoop/hbase/io/ByteBufferInputStream.java * hbase-common/src/main/java/org/apache/hadoop/hbase/util/Threads.java * hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestClientScanner.java > Reduce garbage created under high load > -- > > Key: HBASE-15177 > URL: https://issues.apache.org/jira/browse/HBASE-15177 > Project: HBase > Issue Type: Improvement >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 2.0.0, 1.3.0 > > Attachments: Screen Shot 2016-01-26 at 10.03.48 PM.png, Screen Shot > 2016-01-26 at 10.03.56 PM.png, Screen Shot 2016-01-26 at 10.06.16 PM.png, > Screen Shot 2016-01-26 at 10.15.15 PM.png, hbase-15177_v0.patch, > hbase-15177_v1.patch, hbase-15177_v2.patch, hbase-15177_v3.patch, > hbase-15177_v4.patch, hbase-15177_v4.patch > > > I have been doing some profiling of the garbage being created. The idea was > to follow up on HBASE-14490 and experiment with offheap IPC byte buffers and > byte buffer re-use. However, without changing the IPC byte buffers for now, > there are a couple of (easy) improvements that I've identified from > profiling: > 1. RPCServer.Connection.processRequest() should work with ByteBuffer instead > of byte[] and not-recreate CodedInputStream a few times. > 2. RSRpcServices.getRegion() allocates two byte arrays for region, while only > 1 is needed. > 3. AnnotationReadingPriorityFunction is very expensive in allocations. Mainly > it allocates the regionName byte[] to get the table name. We already set the > priority for most of the operations (multi, get, increment, etc) but we are > only reading the priority in case of multi. We should use the priority from > the client side. > Lets do the simple improvements in this patch, we can get to IPC buffer > re-use in HBASE-14490. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15177) Reduce garbage created under high load
[ https://issues.apache.org/jira/browse/HBASE-15177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133352#comment-15133352 ] Andrew Purtell commented on HBASE-15177: bq. Findbugs came in with HBASE-15200 Oops will fix there > Reduce garbage created under high load > -- > > Key: HBASE-15177 > URL: https://issues.apache.org/jira/browse/HBASE-15177 > Project: HBase > Issue Type: Improvement >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 2.0.0, 1.3.0 > > Attachments: Screen Shot 2016-01-26 at 10.03.48 PM.png, Screen Shot > 2016-01-26 at 10.03.56 PM.png, Screen Shot 2016-01-26 at 10.06.16 PM.png, > Screen Shot 2016-01-26 at 10.15.15 PM.png, hbase-15177_v0.patch, > hbase-15177_v1.patch, hbase-15177_v2.patch, hbase-15177_v3.patch, > hbase-15177_v4.patch, hbase-15177_v4.patch > > > I have been doing some profiling of the garbage being created. The idea was > to follow up on HBASE-14490 and experiment with offheap IPC byte buffers and > byte buffer re-use. However, without changing the IPC byte buffers for now, > there are a couple of (easy) improvements that I've identified from > profiling: > 1. RPCServer.Connection.processRequest() should work with ByteBuffer instead > of byte[] and not-recreate CodedInputStream a few times. > 2. RSRpcServices.getRegion() allocates two byte arrays for region, while only > 1 is needed. > 3. AnnotationReadingPriorityFunction is very expensive in allocations. Mainly > it allocates the regionName byte[] to get the table name. We already set the > priority for most of the operations (multi, get, increment, etc) but we are > only reading the priority in case of multi. We should use the priority from > the client side. > Lets do the simple improvements in this patch, we can get to IPC buffer > re-use in HBASE-14490. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15210) Undo aggressive load balancer logging at tens of lines per millisecond
[
https://issues.apache.org/jira/browse/HBASE-15210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133386#comment-15133386
]
stack commented on HBASE-15210:
---
Reverted and reapplied because had HBASE-15120 rather than HBASE-15210 for
issue number.
> Undo aggressive load balancer logging at tens of lines per millisecond
> --
>
> Key: HBASE-15210
> URL: https://issues.apache.org/jira/browse/HBASE-15210
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer
>Affects Versions: 1.2.0
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 15210.patch
>
>
> Sometimes the load balancer falls into a state of aggressive logging in 1.2.0
> writing out ten lines a millisecond like this:
> {code}
> 2016-02-02 17:15:01,382 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region server with non zero
> regions is ve0540.halxg.cloudera.com with locality 0.22908874
> 2016-02-02 17:15:01,382 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region index is 3 and its region
> server contains 4 regions
> 2016-02-02 17:15:01,384 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region server with non zero
> regions is ve0540.halxg.cloudera.com with locality 0.22908874
> 2016-02-02 17:15:01,384 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region index is 3 and its region
> server contains 4 regions
> 2016-02-02 17:15:01,386 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region server with non zero
> regions is ve0540.halxg.cloudera.com with locality 0.22908874
> 2016-02-02 17:15:01,386 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region index is 2 and its region
> server contains 4 regions
> 2016-02-02 17:15:01,387 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region server with non zero
> regions is ve0540.halxg.cloudera.com with locality 0.22908874
> 2016-02-02 17:15:01,387 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region index is 3 and its region
> server contains 4 regions
> 2016-02-02 17:15:01,388 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region server with non zero
> regions is ve0540.halxg.cloudera.com with locality 0.22908874
> 2016-02-02 17:15:01,388 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region index is 3 and its region
> server contains 4 regions
> 2016-02-02 17:15:01,388 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region server with non zero
> regions is ve0540.halxg.cloudera.com with locality 0.22908874
> {code}
> This issue is about less logging.
> This log was added recently by this:
> commit 54028140f4f19a6af81c8c8f29dda0c52491a0c9
> Author: tedyu
> Date: Thu Aug 13 09:11:59 2015 -0700
> HBASE-13376 Improvements to Stochastic load balancer (Vandana
> Ayyalasomayajula)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-15200) ZooKeeper znode ACL checks should only compare the shortname
[ https://issues.apache.org/jira/browse/HBASE-15200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133394#comment-15133394 ] Andrew Purtell edited comment on HBASE-15200 at 2/5/16 12:19 AM: - Ok I made the change and am committing an addendum now. This time I ran a Findbugs check in Eclipse after editing ZooKeeperWatcher.java, and see 0 warnings from this file after the change. was (Author: apurtell): Ok I made the change and am committing an addendum now. This time I ran a Findbugs check in Eclipse after editing the file, and see 0 warnings from this file after the change. > ZooKeeper znode ACL checks should only compare the shortname > > > Key: HBASE-15200 > URL: https://issues.apache.org/jira/browse/HBASE-15200 > Project: HBase > Issue Type: Bug > Components: security >Affects Versions: 2.0.0, 1.2.0, 1.0.3, 1.1.3, 0.98.17 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4, 0.98.18 > > Attachments: HBASE-15200-branch-1.0.patch, > HBASE-15200-branch-1.1.patch, HBASE-15200.patch, HBASE-15200.patch > > > After HBASE-13768 we check at startup in secure configurations if our znodes > have the correct ACLs. However when checking the ACL we compare the Kerberos > fullname, which includes the host component. We should only compare the > shortname, the principal. Otherwise in a multimaster configuration we will > unnecessarily reset ACLs whenever any master running on a host other than the > one that initialized the ACLs makes the check. You can imagine this happening > multiple times in a rolling restart scenario. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15200) ZooKeeper znode ACL checks should only compare the shortname
[ https://issues.apache.org/jira/browse/HBASE-15200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133394#comment-15133394 ] Andrew Purtell commented on HBASE-15200: Ok I made the change and am committing an addendum now. This time I ran a Findbugs check in Eclipse after editing the file, and see 0 warnings from this file after the change. > ZooKeeper znode ACL checks should only compare the shortname > > > Key: HBASE-15200 > URL: https://issues.apache.org/jira/browse/HBASE-15200 > Project: HBase > Issue Type: Bug > Components: security >Affects Versions: 2.0.0, 1.2.0, 1.0.3, 1.1.3, 0.98.17 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4, 0.98.18 > > Attachments: HBASE-15200-branch-1.0.patch, > HBASE-15200-branch-1.1.patch, HBASE-15200.patch, HBASE-15200.patch > > > After HBASE-13768 we check at startup in secure configurations if our znodes > have the correct ACLs. However when checking the ACL we compare the Kerberos > fullname, which includes the host component. We should only compare the > shortname, the principal. Otherwise in a multimaster configuration we will > unnecessarily reset ACLs whenever any master running on a host other than the > one that initialized the ACLs makes the check. You can imagine this happening > multiple times in a rolling restart scenario. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15212) RRCServer should enforce max request size
[ https://issues.apache.org/jira/browse/HBASE-15212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-15212: -- Status: Patch Available (was: Open) > RRCServer should enforce max request size > -- > > Key: HBASE-15212 > URL: https://issues.apache.org/jira/browse/HBASE-15212 > Project: HBase > Issue Type: Improvement >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 2.0.0, 1.3.0 > > Attachments: hbase-15212_v1.patch > > > A TODO from HBASE-15177 was that we are not protecting the RPCServer in case > an RPC request with a very large size is received. This might cause the > server to go OOM because we are allocating the RPC serialization into a BB. > Instead we should reject the RPC and close the connection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15213) Fix increment performance regression caused by HBASE-8763 on branch-1.0
[ https://issues.apache.org/jira/browse/HBASE-15213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133472#comment-15133472 ] Junegunn Choi commented on HBASE-15213: --- bq. Its not clear how the mvcc can act as an effective region-scoped lock? You seem to be more looking at why the friction in mvcc... nice. It was clear, but it was already so before HBASE-8763. So I couldn't see how HBASE-8763 made things worse than before. And I had to run some tests. bq. You were looking at cs for the whole hbase app? I am interested in how you attributed cs to mvcc or was it just general observation? The whole app. I probably should have broken down the numbers, but I had this hypothesis on MVCC in mind and the patch worked as expected, so I kind of jumped into the conclusion. FYI, here are the cs numbers roughly measured: - less than 1K cs with no workload - 280K cs before HBASE-8763 at (approx) TPS 2 - 450K cs just after HBASE-8763 at TPS 2000 - 430K cs on branch-1.0 at TPS 3800 - 240K cs on branch-1.0 with this patch at TPS 2 bq. Can you show me to what you are referring too? Is this before HBASE-15031? It's still so after HBASE-15031. We call advanceMemstore() after we break out of the wait-loop: https://github.com/apache/hbase/blob/b43442c/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MultiVersionConsistencyControl.java#L235 then the entry is marked complete: https://github.com/apache/hbase/blob/b43442c/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MultiVersionConsistencyControl.java#L153 bq. Do you mean when you do the remove because we are just getting first entry or perhaps you are referring to elsewhere because LinkedList is just returning its pointer to first when we loop through. It's related to the above observation that WriteEntry is marked complete only after the wait-loop. The entries in the writeQueue are not marked complete until advanceMemstore and thus bulk removal is not possible. I mean the removeFirst loop in advanceMemstore is not working as expected; only the first entry is marked complete. But this also allows for head-only checking in the wait-loop possible, since a waiting handler can always be sure that its entry is not removed by another handler. https://github.com/apache/hbase/blob/b43442c/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MultiVersionConsistencyControl.java#L218-L221 With the patch this is no longer true, "w" could have been removed by another handler while this handler is waiting for it to come to the head of the queue. bq. Asking because wondering if we should pull this change to master? I don't think we should, instead we have HBASE-12751 there. bq. How were you testing it - Before HBASE-8763: d6cc2fb - branch-1.0: b43442 (tip of the branch as of yesterday) I didn't use IncrementPerformanceTest instead I wrote a simple client program. It's written in Clojure with a custom API wrapper, but it's simple and should be straightforward. https://gist.github.com/junegunn/1d53b795b22fe9939dc7 So anyway, here are the fresh numbers of IncrementPerformanceTest I just got. I used the same configuration as above, 4-core server and a separate 4-core client machine. - b43442c without fix: 75th=24.01575, 95th=35.668799, 99th=43.03161 - b43442c with fix: 75th=4.87025, 95th=7.16975, 99th=8.7261113 > Fix increment performance regression caused by HBASE-8763 on branch-1.0 > --- > > Key: HBASE-15213 > URL: https://issues.apache.org/jira/browse/HBASE-15213 > Project: HBase > Issue Type: Bug > Components: Performance >Reporter: Junegunn Choi >Assignee: Junegunn Choi > Attachments: HBASE-15213.branch-1.0.patch > > > This is an attempt to fix the increment performance regression caused by > HBASE-8763 on branch-1.0. > I'm aware that hbase.increment.fast.but.narrow.consistency was added to > branch-1.0 (HBASE-15031) to address the issue and a separate work is ongoing > on master branch, but anyway, this is my take on the problem. > I read through HBASE-14460 and HBASE-8763 but it wasn't clear to me what > caused the slowdown but I could indeed reproduce the performance regression. > Test setup: > - Server: 4-core Xeon 2.4GHz Linux server running mini cluster (100 handlers, > JDK 1.7) > - Client: Another box of the same spec > - Increments on random 10k records on a single-region table, recreated every > time > Increment throughput (TPS): > || Num threads || Before HBASE-8763 (d6cc2fb) || branch-1.0 || branch-1.0 > (narrow-consistency) || > || 1| 2661 | 2486| 2359 | > || 2| 5048 | 5064| 4867 | > || 4| 7503 | 8071|
[jira] [Commented] (HBASE-15210) Undo aggressive load balancer logging at tens of lines per millisecond
[
https://issues.apache.org/jira/browse/HBASE-15210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15132350#comment-15132350
]
Sean Busbey commented on HBASE-15210:
-
+1
> Undo aggressive load balancer logging at tens of lines per millisecond
> --
>
> Key: HBASE-15210
> URL: https://issues.apache.org/jira/browse/HBASE-15210
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer
>Affects Versions: 1.2.0
>Reporter: stack
>Assignee: stack
> Attachments: 15210.patch
>
>
> Sometimes the load balancer falls into a state of aggressive logging in 1.2.0
> writing out ten lines a millisecond like this:
> {code}
> 2016-02-02 17:15:01,382 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region server with non zero
> regions is ve0540.halxg.cloudera.com with locality 0.22908874
> 2016-02-02 17:15:01,382 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region index is 3 and its region
> server contains 4 regions
> 2016-02-02 17:15:01,384 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region server with non zero
> regions is ve0540.halxg.cloudera.com with locality 0.22908874
> 2016-02-02 17:15:01,384 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region index is 3 and its region
> server contains 4 regions
> 2016-02-02 17:15:01,386 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region server with non zero
> regions is ve0540.halxg.cloudera.com with locality 0.22908874
> 2016-02-02 17:15:01,386 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region index is 2 and its region
> server contains 4 regions
> 2016-02-02 17:15:01,387 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region server with non zero
> regions is ve0540.halxg.cloudera.com with locality 0.22908874
> 2016-02-02 17:15:01,387 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region index is 3 and its region
> server contains 4 regions
> 2016-02-02 17:15:01,388 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region server with non zero
> regions is ve0540.halxg.cloudera.com with locality 0.22908874
> 2016-02-02 17:15:01,388 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region index is 3 and its region
> server contains 4 regions
> 2016-02-02 17:15:01,388 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region server with non zero
> regions is ve0540.halxg.cloudera.com with locality 0.22908874
> {code}
> This issue is about less logging.
> This log was added recently by this:
> commit 54028140f4f19a6af81c8c8f29dda0c52491a0c9
> Author: tedyu
> Date: Thu Aug 13 09:11:59 2015 -0700
> HBASE-13376 Improvements to Stochastic load balancer (Vandana
> Ayyalasomayajula)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15203) Reduce garbage created by path.toString() during Checksum verification
[ https://issues.apache.org/jira/browse/HBASE-15203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15132416#comment-15132416 ] Hudson commented on HBASE-15203: SUCCESS: Integrated in HBase-1.3-IT #475 (See [https://builds.apache.org/job/HBase-1.3-IT/475/]) HBASE-HBASE-15203 Reduce garbage created by path.toString() during (ramkrishna: rev 1404d5a97331ecc63db53971f5cb7329cb40ce67) * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/ChecksumUtil.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java > Reduce garbage created by path.toString() during Checksum verification > -- > > Key: HBASE-15203 > URL: https://issues.apache.org/jira/browse/HBASE-15203 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15203.patch, HBASE-15203_1.patch, > HBASE-15203_2.patch, HBASE-15203_branch-1.1.patch > > > When we try to read a block we do checksum verification for which we need the > file name in which the block belongs to. So we do Path.toString() every time. > This seems to create around 163MB of char[] that is garbage collected in a > simple scan run. It is also visible in writes but the impact is lesser. In > overall write/read profile the top 2 factors are byte[] and char[]. This > toString() can easily be avoided and reduce its share from the total. To make > it more precise in 1 min of profiling, among the 1.8G of garbage created by > StringBuilder.toString - this path.toString() was contributing around 3.5%. > After the patch this is totally not there. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15213) Fix increment performance regression caused by HBASE-8763 on branch-1.0
[ https://issues.apache.org/jira/browse/HBASE-15213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15132347#comment-15132347 ] Jonathan Hsieh commented on HBASE-15213: Hey [~junegunn], take a look at HBASE-14460. [~saint@gmail.com]'s been working on this for a whlie, has made progress and probably would love to have another set of ideas on how to solve and review. > Fix increment performance regression caused by HBASE-8763 on branch-1.0 > --- > > Key: HBASE-15213 > URL: https://issues.apache.org/jira/browse/HBASE-15213 > Project: HBase > Issue Type: Bug > Components: Performance >Reporter: Junegunn Choi >Assignee: Junegunn Choi > Attachments: HBASE-15213.branch-1.0.patch > > > This is an attempt to fix the increment performance regression caused by > HBASE-8763 on branch-1.0. > I'm aware that hbase.increment.fast.but.narrow.consistency was added to > branch-1.0 (HBASE-15031) to address the issue and a separate work is ongoing > on master branch, but anyway, this is my take on the problem. > I read through HBASE-14460 and HBASE-8763 but it wasn't clear to me what > caused the slowdown but I could indeed reproduce the performance regression. > Test setup: > - Server: 4-core Xeon 2.4GHz Linux server running mini cluster (100 handlers, > JDK 1.7) > - Client: Another box of the same spec > - Increments on random 10k records on a single-region table, recreated every > time > Increment throughput (TPS): > || Num threads || Before HBASE-8763 (d6cc2fb) || branch-1.0 || branch-1.0 > (narrow-consistency) || > || 1| 2661 | 2486| 2359 | > || 2| 5048 | 5064| 4867 | > || 4| 7503 | 8071| 8690 | > || 8| 10471| 10886 | 13980 | > || 16 | 15515| 9418| 18601 | > || 32 | 17699| 5421| 20540 | > || 64 | 20601| 4038| 25591 | > || 96 | 19177| 3891| 26017 | > We can clearly observe that the throughtput degrades as we increase the > number of concurrent requests, which led me to believe that there's severe > context switching overhead and I could indirectly confirm that suspicion with > cs entry in vmstat output. branch-1.0 shows a much higher number of context > switches even with much lower throughput. > Here are the observations: > - WriteEntry in the writeQueue can only be removed by the very handler that > put it, only when it is at the front of the queue and marked complete. > - Since a WriteEntry is marked complete after the wait-loop, only one entry > can be removed at a time. > - This stringent condition causes O(N^2) context switches where n is the > number of concurrent handlers processing requests. > So what I tried here is to mark WriteEntry complete before we go into > wait-loop. With the change, multiple WriteEntries can be shifted at a time > without context switches. I changed writeQueue to LinkedHashSet since fast > containment check is needed as WriteEntry can be removed by any handler. > The numbers look good, it's virtually identical to pre-HBASE-8763 era. > || Num threads || branch-1.0 with fix || > || 1| 2459 | > || 2| 4976 | > || 4| 8033 | > || 8| 12292| > || 16 | 15234| > || 32 | 16601| > || 64 | 19994| > || 96 | 20052| > So what do you think about it? Please let me know if I'm missing anything. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15214) Valid mutate Ops fail with RPC Codec in use and region moves across
[
https://issues.apache.org/jira/browse/HBASE-15214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15132391#comment-15132391
]
Yu Li commented on HBASE-15214:
---
Good catch sir! The patch makes the problem easier to understand. :-)
Just one comment, that since no cellScanner will be set if user choose to
disable codec (set {{hbase.client.rpc.codec}} to empty), I think we could save
the skipCells loop if cellScanner is null, like
{code}
if (cellScanner != null) skipCellsForMutations(regionAction.getActionList(),
cellScanner);
{code}
> Valid mutate Ops fail with RPC Codec in use and region moves across
> ---
>
> Key: HBASE-15214
> URL: https://issues.apache.org/jira/browse/HBASE-15214
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4, 0.98.18
>
> Attachments: HBASE-15214.patch
>
>
> Test failures in HBASE-15198 lead to this bug. Till now we are not doing cell
> block (codec usage) for write requests. (Client -> server) Once we enabled
> Codec usage by default, aw this issue.
> A multi request came to RS with mutation for different regions. One of the
> region which was in this RS got unavailable now. In RsRpcServices#multi, we
> will fail that entire RegionAction (with N mutations in it) in that
> MultiRequest. Then we will continue with remaining RegionActions. Those
> Regions might be available. (The failed RegionAction will get retried from
> client after fetching latest region location). This all works fine in pure
> PB requests world. When a Codec is used, we wont convert the Mutation Cell to
> PB Cells and pack them in PB Message. Instead we will pass all Cells
> serialized into one byte[] cellblock. Using Decoder we will iterate over
> these cells at server side. Each Mutation PB will know only the number of
> cells associated with it. As in above case when an entire RegionAction was
> skipped, there might be N Mutations under that which might have corresponding
> Cells in the cellblock. We are not doing the skip in that Iterator. This
> makes the later Mutations (for other Regions) to refer to invalid Cells and
> try to put those into the a different region. This will make
> HRegion#checkRow() to throw WrongRegionException which will be treated as
> Sanity check failure and so throwing back a DNRIOE to client. So the op will
> get failed for the user code.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14919) Infrastructure refactoring for In-Memory Flush
[
https://issues.apache.org/jira/browse/HBASE-14919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15132267#comment-15132267
]
Eshcar Hillel commented on HBASE-14919:
---
bq. Should we set -1 to when create instance of this class? Define -1 as
NO_SNAPSHOT_ID?
Definitely, yes.
bq. This javadoc seems wrong.
I will fix this.
bq. So, an ImmutableSegment takes a MutableSegment on construction? You can
skip the pupa ImmutableSegmentAdapter? (It is the only implementation of
ImmutableSegment).
Task #4 in the umbrella issue implements another ImmutableSegment
(CellBlocksSrgment), which does not take MutableSegment on construction.
Would you like better the name ImmutableSegmentWrapper?
bq. Reading the MutableSegment looking at the methods it has, why can't they
all be in ImmutableSegment?
We could simplify the design by removing the distinction between mutable and
immutable segments but this would be at the cost of implementing unnecessary
API, possibly not efficiently.
Consider tailSet(firstCell), it is similar for first() and getComparator().
tailSet is used in the methods AbstractMemStore::upsert() and
AbstractMemStore::updateColumnValue() where it is applied only to the active
(mutable) segment, and in methods of MutableCellSetSegment and
MutableCellSetSegmentScanner. It is not needed in any use case for an immutable
segment. However, its implementation in CellBlocksSegment would be inefficient
since it will incur costly traversal over all cell blocks plus allocating many
new objects to be stored in the result sorted set.
It is possible to do it this way. We believe exposing an unnecessary API which
incurs costly implementation is not advisable.
bq. And why this subset of Set methods in MutableSegment?
We believe it is good practice to only expose the minimal necessary
functionality.
bq. In ImmutableSegment, there is this method getScannerForMemStoreSnapshot().
As I read it, I am getting a Scanner on a MemStoreSnapshot...
No, this is incorrect. This method returns a KeyValueScanner that is used *by*
MemStoreSnapshot, but _it scans the cells in the segment who generated it_.
Would you like better the name getKeyValueScanner()? And the other method would
be getSegmentScanner().
bq. What is special about MutableCellSetSegmentScanner? When would the Scanner
on a MutableSegment differ from Scanner on an ImmutableSegment? In other words,
could we just have a SegmentScanner implementation and it work for both Segment
types?
Each scanner depends on the segment it is scanning.
For example, consider the method MutableCellSetSegmentScanner::backwardSeek() -
it seeks forward then go backward
{code}
public boolean backwardSeek(Cell key) throws IOException {
seek(key);
if (peek() == null || segment.compareRows(peek(), key) > 0) {
return seekToPreviousRow(key);
}
return true;
}
{code}
There is no reason for CellBlocksSegment to implement it this way since the
cells are contiguous and therefore allow for much simpler traversal.
> Infrastructure refactoring for In-Memory Flush
> --
>
> Key: HBASE-14919
> URL: https://issues.apache.org/jira/browse/HBASE-14919
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Eshcar Hillel
> Attachments: HBASE-14919-V01.patch, HBASE-14919-V01.patch,
> HBASE-14919-V02.patch, HBASE-14919-V03.patch, HBASE-14919-V04.patch,
> HBASE-14919-V04.patch, HBASE-14919-V05.patch, HBASE-14919-V06.patch,
> HBASE-14919-V06.patch, HBASE-14919-V07.patch, HBASE-14919-V08.patch,
> HBASE-14919-V09.patch, HBASE-14919-V10.patch
>
>
> Refactoring the MemStore hierarchy, introducing segment (StoreSegment) as
> first-class citizen and decoupling memstore scanner from the memstore
> implementation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15202) Reduce garbage while setting response
[
https://issues.apache.org/jira/browse/HBASE-15202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15132435#comment-15132435
]
Hadoop QA commented on HBASE-15202:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 1s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 55s
{color} | {color:green} master passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
51s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
38s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s
{color} | {color:green} master passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 44s
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
24m 32s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 2s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 98m 56s {color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 93m 52s {color}
| {color:red} hbase-server in the patch failed with JDK v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 244m 7s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Timed out junit tests |
org.apache.hadoop.hbase.snapshot.TestFlushSnapshotFromClient |
| JDK v1.7.0_91 Failed junit tests |
hadoop.hbase.client.TestBlockEvictionFromClient |
| | hadoop.hbase.mapreduce.TestCopyTable |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.9.1 Server=1.9.1 Image:yetus/hbase:date2016-02-04 |
| JIRA Patch URL |
[jira] [Updated] (HBASE-15180) Reduce garbage created while reading Cells from Codec Decoder
[ https://issues.apache.org/jira/browse/HBASE-15180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-15180: --- Affects Version/s: 0.98.0 > Reduce garbage created while reading Cells from Codec Decoder > - > > Key: HBASE-15180 > URL: https://issues.apache.org/jira/browse/HBASE-15180 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Affects Versions: 0.98.0 >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0, 1.3.0, 1.2.1 > > Attachments: HBASE-15180.patch, HBASE-15180_V2.patch > > > In KeyValueDecoder#parseCell (Default Codec decoder) we use > KeyValueUtil#iscreate to read cells from the InputStream. Here we 1st create > a byte[] of length 4 and read the cell length and then an array of Cell's > length and read in cell bytes into it and create a KV. > Actually in server we read the reqs into a byte[] and CellScanner is created > on top of a ByteArrayInputStream on top of this. By default in write path, we > have MSLAB usage ON. So while adding Cells to memstore, we will copy the Cell > bytes to MSLAB memory chunks (default 2 MB size) and recreate Cells over that > bytes. So there is no issue if we create Cells over the RPC read byte[] > directly here in Decoder. No need for 2 byte[] creation and copy for every > Cell in request. > My plan is to make a Cell aware ByteArrayInputStream which can read Cells > directly from it. > Same Codec path is used in client side also. There better we can avoid this > direct Cell create and continue to do the copy to smaller byte[]s path. Plan > to introduce some thing like a CodecContext associated with every Codec > instance which can say the server/client context. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15180) Reduce garbage created while reading Cells from Codec Decoder
[ https://issues.apache.org/jira/browse/HBASE-15180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-15180: --- Fix Version/s: 1.2.1 1.3.0 > Reduce garbage created while reading Cells from Codec Decoder > - > > Key: HBASE-15180 > URL: https://issues.apache.org/jira/browse/HBASE-15180 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Affects Versions: 0.98.0 >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0, 1.3.0, 1.2.1 > > Attachments: HBASE-15180.patch, HBASE-15180_V2.patch > > > In KeyValueDecoder#parseCell (Default Codec decoder) we use > KeyValueUtil#iscreate to read cells from the InputStream. Here we 1st create > a byte[] of length 4 and read the cell length and then an array of Cell's > length and read in cell bytes into it and create a KV. > Actually in server we read the reqs into a byte[] and CellScanner is created > on top of a ByteArrayInputStream on top of this. By default in write path, we > have MSLAB usage ON. So while adding Cells to memstore, we will copy the Cell > bytes to MSLAB memory chunks (default 2 MB size) and recreate Cells over that > bytes. So there is no issue if we create Cells over the RPC read byte[] > directly here in Decoder. No need for 2 byte[] creation and copy for every > Cell in request. > My plan is to make a Cell aware ByteArrayInputStream which can read Cells > directly from it. > Same Codec path is used in client side also. There better we can avoid this > direct Cell create and continue to do the copy to smaller byte[]s path. Plan > to introduce some thing like a CodecContext associated with every Codec > instance which can say the server/client context. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15213) Fix increment performance regression caused by HBASE-8763 on branch-1.0
[ https://issues.apache.org/jira/browse/HBASE-15213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133697#comment-15133697 ] Heng Chen commented on HBASE-15213: --- Thanks! Got it now! > Fix increment performance regression caused by HBASE-8763 on branch-1.0 > --- > > Key: HBASE-15213 > URL: https://issues.apache.org/jira/browse/HBASE-15213 > Project: HBase > Issue Type: Bug > Components: Performance >Reporter: Junegunn Choi >Assignee: Junegunn Choi > Attachments: HBASE-15213.branch-1.0.patch > > > This is an attempt to fix the increment performance regression caused by > HBASE-8763 on branch-1.0. > I'm aware that hbase.increment.fast.but.narrow.consistency was added to > branch-1.0 (HBASE-15031) to address the issue and a separate work is ongoing > on master branch, but anyway, this is my take on the problem. > I read through HBASE-14460 and HBASE-8763 but it wasn't clear to me what > caused the slowdown but I could indeed reproduce the performance regression. > Test setup: > - Server: 4-core Xeon 2.4GHz Linux server running mini cluster (100 handlers, > JDK 1.7) > - Client: Another box of the same spec > - Increments on random 10k records on a single-region table, recreated every > time > Increment throughput (TPS): > || Num threads || Before HBASE-8763 (d6cc2fb) || branch-1.0 || branch-1.0 > (narrow-consistency) || > || 1| 2661 | 2486| 2359 | > || 2| 5048 | 5064| 4867 | > || 4| 7503 | 8071| 8690 | > || 8| 10471| 10886 | 13980 | > || 16 | 15515| 9418| 18601 | > || 32 | 17699| 5421| 20540 | > || 64 | 20601| 4038| 25591 | > || 96 | 19177| 3891| 26017 | > We can clearly observe that the throughtput degrades as we increase the > number of concurrent requests, which led me to believe that there's severe > context switching overhead and I could indirectly confirm that suspicion with > cs entry in vmstat output. branch-1.0 shows a much higher number of context > switches even with much lower throughput. > Here are the observations: > - WriteEntry in the writeQueue can only be removed by the very handler that > put it, only when it is at the front of the queue and marked complete. > - Since a WriteEntry is marked complete after the wait-loop, only one entry > can be removed at a time. > - This stringent condition causes O(N^2) context switches where n is the > number of concurrent handlers processing requests. > So what I tried here is to mark WriteEntry complete before we go into > wait-loop. With the change, multiple WriteEntries can be shifted at a time > without context switches. I changed writeQueue to LinkedHashSet since fast > containment check is needed as WriteEntry can be removed by any handler. > The numbers look good, it's virtually identical to pre-HBASE-8763 era. > || Num threads || branch-1.0 with fix || > || 1| 2459 | > || 2| 4976 | > || 4| 8033 | > || 8| 12292| > || 16 | 15234| > || 32 | 16601| > || 64 | 19994| > || 96 | 20052| > So what do you think about it? Please let me know if I'm missing anything. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15216) Canary does not accept few client config param from commandline( it should only be present in hbase-site.xml)
[
https://issues.apache.org/jira/browse/HBASE-15216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vishal Khandelwal updated HBASE-15216:
--
Summary: Canary does not accept few client config param from commandline(
it should only be present in hbase-site.xml) (was: Canary does not accepts few
client config param from commandline( it should only be present in
hbase-site.xml))
> Canary does not accept few client config param from commandline( it should
> only be present in hbase-site.xml)
> -
>
> Key: HBASE-15216
> URL: https://issues.apache.org/jira/browse/HBASE-15216
> Project: HBase
> Issue Type: Bug
> Components: canary
>Affects Versions: 0.98.16
>Reporter: Vishal Khandelwal
>
> At present there are few configs which needs to be present in hbase-site or
> default xml for it to work. following are the list.
> hbase.canary.threads.num
> hbase.canary.sink.class
> hbase.client.keytab.file
> hbase.client.kerberos.principal
> Execution in secure expects keytab and princ to be present
> 2016-02-05 05:58:44,024 ERROR [main] hbase.AuthUtil - Error while trying to
> perform the initial login: Running in secure mode, but config doesn't have a
> keytab
> java.io.IOException: Running in secure mode, but config doesn't have a keytab
> at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:236)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.login(User.java:392)
> at org.apache.hadoop.hbase.security.User.login(User.java:259)
> at
> org.apache.hadoop.hbase.security.UserProvider.login(UserProvider.java:116)
> at org.apache.hadoop.hbase.AuthUtil.launchAuthChore(AuthUtil.java:64)
> at org.apache.hadoop.hbase.tool.Canary.main(Canary.java:1146)
> Exception in thread "main" java.io.IOException: Running in secure mode, but
> config doesn't have a keytab
> at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:236)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.login(User.java:392)
> at org.apache.hadoop.hbase.security.User.login(User.java:259)
> at
> org.apache.hadoop.hbase.security.UserProvider.login(UserProvider.java:116)
> at org.apache.hadoop.hbase.AuthUtil.launchAuthChore(AuthUtil.java:64)
> at org.apache.hadoop.hbase.tool.Canary.main(Canary.java:1146)
> public static void main(String[] args) throws Exception {
> final Configuration conf = HBaseConfiguration.create();
> AuthUtil.launchAuthChore(conf);
> int numThreads = conf.getInt("hbase.canary.threads.num", MAX_THREADS_NUM);
> ExecutorService executor = new ScheduledThreadPoolExecutor(numThreads);
> Class sinkClass =
> conf.getClass("hbase.canary.sink.class", StdOutSink.class,
> Sink.class);
> Sink sink = ReflectionUtils.newInstance(sinkClass);
> int exitCode = ToolRunner.run(conf, new Canary(executor, sink), args);
> executor.shutdown();
> System.exit(exitCode);
> }
> In main class these params should be parsed and updated. else for any change
> to these value hbase-stie.xml needs to be updated and hbase service needs to
> be restarted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15213) Fix increment performance regression caused by HBASE-8763 on branch-1.0
[
https://issues.apache.org/jira/browse/HBASE-15213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133634#comment-15133634
]
Heng Chen commented on HBASE-15213:
---
Thanks for your explain! [~junegunn]
You are right, we could mark entry as complete after sync successfully
immediately, there is no need to wait other entries complete and current one
come to be first in writeQueue, so as your mentioned, other handlers could
remove it while current thread in waiting state.
{quote}
This stringent condition causes O(N^2) context switches where n is the number
of concurrent handlers processing requests.
{quote}
This is why you make writeQueue from list to set, right? Sorry, i have a
little confused about this sentence.
> Fix increment performance regression caused by HBASE-8763 on branch-1.0
> ---
>
> Key: HBASE-15213
> URL: https://issues.apache.org/jira/browse/HBASE-15213
> Project: HBase
> Issue Type: Bug
> Components: Performance
>Reporter: Junegunn Choi
>Assignee: Junegunn Choi
> Attachments: HBASE-15213.branch-1.0.patch
>
>
> This is an attempt to fix the increment performance regression caused by
> HBASE-8763 on branch-1.0.
> I'm aware that hbase.increment.fast.but.narrow.consistency was added to
> branch-1.0 (HBASE-15031) to address the issue and a separate work is ongoing
> on master branch, but anyway, this is my take on the problem.
> I read through HBASE-14460 and HBASE-8763 but it wasn't clear to me what
> caused the slowdown but I could indeed reproduce the performance regression.
> Test setup:
> - Server: 4-core Xeon 2.4GHz Linux server running mini cluster (100 handlers,
> JDK 1.7)
> - Client: Another box of the same spec
> - Increments on random 10k records on a single-region table, recreated every
> time
> Increment throughput (TPS):
> || Num threads || Before HBASE-8763 (d6cc2fb) || branch-1.0 || branch-1.0
> (narrow-consistency) ||
> || 1| 2661 | 2486| 2359 |
> || 2| 5048 | 5064| 4867 |
> || 4| 7503 | 8071| 8690 |
> || 8| 10471| 10886 | 13980 |
> || 16 | 15515| 9418| 18601 |
> || 32 | 17699| 5421| 20540 |
> || 64 | 20601| 4038| 25591 |
> || 96 | 19177| 3891| 26017 |
> We can clearly observe that the throughtput degrades as we increase the
> number of concurrent requests, which led me to believe that there's severe
> context switching overhead and I could indirectly confirm that suspicion with
> cs entry in vmstat output. branch-1.0 shows a much higher number of context
> switches even with much lower throughput.
> Here are the observations:
> - WriteEntry in the writeQueue can only be removed by the very handler that
> put it, only when it is at the front of the queue and marked complete.
> - Since a WriteEntry is marked complete after the wait-loop, only one entry
> can be removed at a time.
> - This stringent condition causes O(N^2) context switches where n is the
> number of concurrent handlers processing requests.
> So what I tried here is to mark WriteEntry complete before we go into
> wait-loop. With the change, multiple WriteEntries can be shifted at a time
> without context switches. I changed writeQueue to LinkedHashSet since fast
> containment check is needed as WriteEntry can be removed by any handler.
> The numbers look good, it's virtually identical to pre-HBASE-8763 era.
> || Num threads || branch-1.0 with fix ||
> || 1| 2459 |
> || 2| 4976 |
> || 4| 8033 |
> || 8| 12292|
> || 16 | 15234|
> || 32 | 16601|
> || 64 | 19994|
> || 96 | 20052|
> So what do you think about it? Please let me know if I'm missing anything.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15120) Backport HBASE-14883 to branch-1.1
[ https://issues.apache.org/jira/browse/HBASE-15120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133639#comment-15133639 ] Hudson commented on HBASE-15120: FAILURE: Integrated in HBase-Trunk_matrix #681 (See [https://builds.apache.org/job/HBase-Trunk_matrix/681/]) HBASE-15120 Undo aggressive load balancer logging at tens of lines per (stack: rev c177cfed416e0cf72ae6131c5e98c99672bef3cc) * hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java Revert "HBASE-15120 Undo aggressive load balancer logging at tens of (stack: rev 337f4830770d87e2fe629d742a3dcc64dba0c2cb) * hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java > Backport HBASE-14883 to branch-1.1 > -- > > Key: HBASE-15120 > URL: https://issues.apache.org/jira/browse/HBASE-15120 > Project: HBase > Issue Type: Bug >Affects Versions: 1.1.2 >Reporter: Yu Li >Assignee: Yu Li >Priority: Minor > Attachments: HBASE-15120.branch-1.1.patch, > HBASE-15120.branch-1.1.patch > > > When checking branch-1.1 UT in HBASE-13590, found > TestSplitTransactionOnCluster#testFailedSplit will fail at a 12/50 chance. > The issue is fixed by HBASE-14883 but the change didn't go into branch-1.1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15177) Reduce garbage created under high load
[ https://issues.apache.org/jira/browse/HBASE-15177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133658#comment-15133658 ] ramkrishna.s.vasudevan commented on HBASE-15177: Sorry for the late reviews here. bq.Had to bring back the buffer.position(offset) that was removed from v0. This is needed since in case cells are compressed, we are creating an BBIS over the BB. Okie. I can see now why it is needed bq.Ram filed issue on TestFlushSnapshotFromClient being flakey. I filed an issue for TestBlockEvictionFromclient - https://issues.apache.org/jira/browse/HBASE-15215. Not for TestFlushSnapshotFromClient. Thanks for the patch [~enis]. > Reduce garbage created under high load > -- > > Key: HBASE-15177 > URL: https://issues.apache.org/jira/browse/HBASE-15177 > Project: HBase > Issue Type: Improvement >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 2.0.0, 1.3.0 > > Attachments: Screen Shot 2016-01-26 at 10.03.48 PM.png, Screen Shot > 2016-01-26 at 10.03.56 PM.png, Screen Shot 2016-01-26 at 10.06.16 PM.png, > Screen Shot 2016-01-26 at 10.15.15 PM.png, hbase-15177_v0.patch, > hbase-15177_v1.patch, hbase-15177_v2.patch, hbase-15177_v3.patch, > hbase-15177_v4.patch, hbase-15177_v4.patch > > > I have been doing some profiling of the garbage being created. The idea was > to follow up on HBASE-14490 and experiment with offheap IPC byte buffers and > byte buffer re-use. However, without changing the IPC byte buffers for now, > there are a couple of (easy) improvements that I've identified from > profiling: > 1. RPCServer.Connection.processRequest() should work with ByteBuffer instead > of byte[] and not-recreate CodedInputStream a few times. > 2. RSRpcServices.getRegion() allocates two byte arrays for region, while only > 1 is needed. > 3. AnnotationReadingPriorityFunction is very expensive in allocations. Mainly > it allocates the regionName byte[] to get the table name. We already set the > priority for most of the operations (multi, get, increment, etc) but we are > only reading the priority in case of multi. We should use the priority from > the client side. > Lets do the simple improvements in this patch, we can get to IPC buffer > re-use in HBASE-14490. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[ https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133673#comment-15133673 ] Anoop Sam John commented on HBASE-15204: +1 > Try to estimate the cell count for adding into WALEdit > -- > > Key: HBASE-15204 > URL: https://issues.apache.org/jira/browse/HBASE-15204 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-15204.patch, HBASE-15204_1.patch, > HBASE-15204_1.patch, WAlEdit_add_allocation.jpg, > WAlEdit_add_allocation_after_patch.jpg > > > The write path profiling shows that when we try to add Cells to WALEdits we > try to do a lot of Array copy inorder to grow the Arraylist backing the > WALEdits. In a simple one min profiling of the write path with 50 YCSB > threads shows around 261MB of allocation done for the Array copy to happen. > We can try to avoid that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15214) Valid mutate Ops fail with RPC Codec in use and region moves across
[ https://issues.apache.org/jira/browse/HBASE-15214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133684#comment-15133684 ] ramkrishna.s.vasudevan commented on HBASE-15214: LGTM. good one. > Valid mutate Ops fail with RPC Codec in use and region moves across > --- > > Key: HBASE-15214 > URL: https://issues.apache.org/jira/browse/HBASE-15214 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.0 >Reporter: Anoop Sam John >Assignee: Anoop Sam John >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4, 0.98.18 > > Attachments: HBASE-15214.patch, HBASE-15214_V2.patch, > HBASE-15214_V3.patch > > > Test failures in HBASE-15198 lead to this bug. Till now we are not doing cell > block (codec usage) for write requests. (Client -> server) Once we enabled > Codec usage by default, aw this issue. > A multi request came to RS with mutation for different regions. One of the > region which was in this RS got unavailable now. In RsRpcServices#multi, we > will fail that entire RegionAction (with N mutations in it) in that > MultiRequest. Then we will continue with remaining RegionActions. Those > Regions might be available. (The failed RegionAction will get retried from > client after fetching latest region location). This all works fine in pure > PB requests world. When a Codec is used, we wont convert the Mutation Cell to > PB Cells and pack them in PB Message. Instead we will pass all Cells > serialized into one byte[] cellblock. Using Decoder we will iterate over > these cells at server side. Each Mutation PB will know only the number of > cells associated with it. As in above case when an entire RegionAction was > skipped, there might be N Mutations under that which might have corresponding > Cells in the cellblock. We are not doing the skip in that Iterator. This > makes the later Mutations (for other Regions) to refer to invalid Cells and > try to put those into the a different region. This will make > HRegion#checkRow() to throw WrongRegionException which will be treated as > Sanity check failure and so throwing back a DNRIOE to client. So the op will > get failed for the user code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15120) Backport HBASE-14883 to branch-1.1
[ https://issues.apache.org/jira/browse/HBASE-15120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133608#comment-15133608 ] Hudson commented on HBASE-15120: SUCCESS: Integrated in HBase-1.2-IT #420 (See [https://builds.apache.org/job/HBase-1.2-IT/420/]) Revert "HBASE-15120 Undo aggressive load balancer logging at tens of (stack: rev d4eb78ea3aec45dffa2ee78d8128e46d9dae4fb5) * hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java > Backport HBASE-14883 to branch-1.1 > -- > > Key: HBASE-15120 > URL: https://issues.apache.org/jira/browse/HBASE-15120 > Project: HBase > Issue Type: Bug >Affects Versions: 1.1.2 >Reporter: Yu Li >Assignee: Yu Li >Priority: Minor > Attachments: HBASE-15120.branch-1.1.patch, > HBASE-15120.branch-1.1.patch > > > When checking branch-1.1 UT in HBASE-13590, found > TestSplitTransactionOnCluster#testFailedSplit will fail at a 12/50 chance. > The issue is fixed by HBASE-14883 but the change didn't go into branch-1.1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15200) ZooKeeper znode ACL checks should only compare the shortname
[ https://issues.apache.org/jira/browse/HBASE-15200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133606#comment-15133606 ] Hudson commented on HBASE-15200: SUCCESS: Integrated in HBase-1.2-IT #420 (See [https://builds.apache.org/job/HBase-1.2-IT/420/]) Amend HBASE-15200 ZooKeeper znode ACL checks should only compare the (apurtell: rev d52625de8fc7dcd7595d3cd019ca6ad77493c96f) * hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java > ZooKeeper znode ACL checks should only compare the shortname > > > Key: HBASE-15200 > URL: https://issues.apache.org/jira/browse/HBASE-15200 > Project: HBase > Issue Type: Bug > Components: security >Affects Versions: 2.0.0, 1.2.0, 1.0.3, 1.1.3, 0.98.17 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4, 0.98.18 > > Attachments: HBASE-15200-branch-1.0.patch, > HBASE-15200-branch-1.1.patch, HBASE-15200.patch, HBASE-15200.patch > > > After HBASE-13768 we check at startup in secure configurations if our znodes > have the correct ACLs. However when checking the ACL we compare the Kerberos > fullname, which includes the host component. We should only compare the > shortname, the principal. Otherwise in a multimaster configuration we will > unnecessarily reset ACLs whenever any master running on a host other than the > one that initialized the ACLs makes the check. You can imagine this happening > multiple times in a rolling restart scenario. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15210) Undo aggressive load balancer logging at tens of lines per millisecond
[
https://issues.apache.org/jira/browse/HBASE-15210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133607#comment-15133607
]
Hudson commented on HBASE-15210:
SUCCESS: Integrated in HBase-1.2-IT #420 (See
[https://builds.apache.org/job/HBase-1.2-IT/420/])
HBASE-15210 Undo aggressive load balancer logging at tens of lines per (stack:
rev 28921e2a83e76bffcbbaa4885219c223546d876e)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java
HBASE-15210 Undo aggressive load balancer logging at tens of lines per (stack:
rev 0e36afd2d09f04ab217d69e89256b6ea79d6d00c)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java
> Undo aggressive load balancer logging at tens of lines per millisecond
> --
>
> Key: HBASE-15210
> URL: https://issues.apache.org/jira/browse/HBASE-15210
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer
>Affects Versions: 1.2.0
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 15210.patch
>
>
> Sometimes the load balancer falls into a state of aggressive logging in 1.2.0
> writing out ten lines a millisecond like this:
> {code}
> 2016-02-02 17:15:01,382 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region server with non zero
> regions is ve0540.halxg.cloudera.com with locality 0.22908874
> 2016-02-02 17:15:01,382 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region index is 3 and its region
> server contains 4 regions
> 2016-02-02 17:15:01,384 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region server with non zero
> regions is ve0540.halxg.cloudera.com with locality 0.22908874
> 2016-02-02 17:15:01,384 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region index is 3 and its region
> server contains 4 regions
> 2016-02-02 17:15:01,386 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region server with non zero
> regions is ve0540.halxg.cloudera.com with locality 0.22908874
> 2016-02-02 17:15:01,386 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region index is 2 and its region
> server contains 4 regions
> 2016-02-02 17:15:01,387 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region server with non zero
> regions is ve0540.halxg.cloudera.com with locality 0.22908874
> 2016-02-02 17:15:01,387 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region index is 3 and its region
> server contains 4 regions
> 2016-02-02 17:15:01,388 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region server with non zero
> regions is ve0540.halxg.cloudera.com with locality 0.22908874
> 2016-02-02 17:15:01,388 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region index is 3 and its region
> server contains 4 regions
> 2016-02-02 17:15:01,388 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region server with non zero
> regions is ve0540.halxg.cloudera.com with locality 0.22908874
> {code}
> This issue is about less logging.
> This log was added recently by this:
> commit 54028140f4f19a6af81c8c8f29dda0c52491a0c9
> Author: tedyu
> Date: Thu Aug 13 09:11:59 2015 -0700
> HBASE-13376 Improvements to Stochastic load balancer (Vandana
> Ayyalasomayajula)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-15216) Canary does not accepts few client config param from commandline( it should only be present in hbase-site.xml)
Vishal Khandelwal created HBASE-15216:
-
Summary: Canary does not accepts few client config param from
commandline( it should only be present in hbase-site.xml)
Key: HBASE-15216
URL: https://issues.apache.org/jira/browse/HBASE-15216
Project: HBase
Issue Type: Bug
Components: canary
Affects Versions: 0.98.16
Reporter: Vishal Khandelwal
At present there are few configs which needs to be present in hbase-site or
default xml for it to work. following are the list.
hbase.canary.threads.num
hbase.canary.sink.class
hbase.client.keytab.file
hbase.client.kerberos.principal
Execution in secure expects keytab and princ to be present
2016-02-05 05:58:44,024 ERROR [main] hbase.AuthUtil - Error while trying to
perform the initial login: Running in secure mode, but config doesn't have a
keytab
java.io.IOException: Running in secure mode, but config doesn't have a keytab
at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:236)
at
org.apache.hadoop.hbase.security.User$SecureHadoopUser.login(User.java:392)
at org.apache.hadoop.hbase.security.User.login(User.java:259)
at
org.apache.hadoop.hbase.security.UserProvider.login(UserProvider.java:116)
at org.apache.hadoop.hbase.AuthUtil.launchAuthChore(AuthUtil.java:64)
at org.apache.hadoop.hbase.tool.Canary.main(Canary.java:1146)
Exception in thread "main" java.io.IOException: Running in secure mode, but
config doesn't have a keytab
at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:236)
at
org.apache.hadoop.hbase.security.User$SecureHadoopUser.login(User.java:392)
at org.apache.hadoop.hbase.security.User.login(User.java:259)
at
org.apache.hadoop.hbase.security.UserProvider.login(UserProvider.java:116)
at org.apache.hadoop.hbase.AuthUtil.launchAuthChore(AuthUtil.java:64)
at org.apache.hadoop.hbase.tool.Canary.main(Canary.java:1146)
public static void main(String[] args) throws Exception {
final Configuration conf = HBaseConfiguration.create();
AuthUtil.launchAuthChore(conf);
int numThreads = conf.getInt("hbase.canary.threads.num", MAX_THREADS_NUM);
ExecutorService executor = new ScheduledThreadPoolExecutor(numThreads);
Class sinkClass =
conf.getClass("hbase.canary.sink.class", StdOutSink.class, Sink.class);
Sink sink = ReflectionUtils.newInstance(sinkClass);
int exitCode = ToolRunner.run(conf, new Canary(executor, sink), args);
executor.shutdown();
System.exit(exitCode);
}
In main class these params should be parsed and updated. else for any change to
these value hbase-stie.xml needs to be updated and hbase service needs to be
restarted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15216) Canary does not accept few client config param from commandline( it should only be present in hbase-site.xml)
[
https://issues.apache.org/jira/browse/HBASE-15216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matteo Bertozzi updated HBASE-15216:
Description:
At present there are few configs which needs to be present in hbase-site or
default xml for it to work. following are the list.
hbase.canary.threads.num
hbase.canary.sink.class
hbase.client.keytab.file
hbase.client.kerberos.principal
Execution in secure expects keytab and princ to be present
{noformat}
2016-02-05 05:58:44,024 ERROR [main] hbase.AuthUtil - Error while trying to
perform the initial login: Running in secure mode, but config doesn't have a
keytab
java.io.IOException: Running in secure mode, but config doesn't have a keytab
at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:236)
at
org.apache.hadoop.hbase.security.User$SecureHadoopUser.login(User.java:392)
at org.apache.hadoop.hbase.security.User.login(User.java:259)
at
org.apache.hadoop.hbase.security.UserProvider.login(UserProvider.java:116)
at org.apache.hadoop.hbase.AuthUtil.launchAuthChore(AuthUtil.java:64)
at org.apache.hadoop.hbase.tool.Canary.main(Canary.java:1146)
Exception in thread "main" java.io.IOException: Running in secure mode, but
config doesn't have a keytab
at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:236)
at
org.apache.hadoop.hbase.security.User$SecureHadoopUser.login(User.java:392)
at org.apache.hadoop.hbase.security.User.login(User.java:259)
at
org.apache.hadoop.hbase.security.UserProvider.login(UserProvider.java:116)
at org.apache.hadoop.hbase.AuthUtil.launchAuthChore(AuthUtil.java:64)
at org.apache.hadoop.hbase.tool.Canary.main(Canary.java:1146)
{noformat}
{code}
public static void main(String[] args) throws Exception {
final Configuration conf = HBaseConfiguration.create();
AuthUtil.launchAuthChore(conf);
int numThreads = conf.getInt("hbase.canary.threads.num", MAX_THREADS_NUM);
ExecutorService executor = new ScheduledThreadPoolExecutor(numThreads);
Class sinkClass =
conf.getClass("hbase.canary.sink.class", StdOutSink.class, Sink.class);
Sink sink = ReflectionUtils.newInstance(sinkClass);
int exitCode = ToolRunner.run(conf, new Canary(executor, sink), args);
executor.shutdown();
System.exit(exitCode);
}
{code}
In main class these params should be parsed and updated. else for any change to
these value hbase-stie.xml needs to be updated and hbase service needs to be
restarted.
was:
At present there are few configs which needs to be present in hbase-site or
default xml for it to work. following are the list.
hbase.canary.threads.num
hbase.canary.sink.class
hbase.client.keytab.file
hbase.client.kerberos.principal
Execution in secure expects keytab and princ to be present
2016-02-05 05:58:44,024 ERROR [main] hbase.AuthUtil - Error while trying to
perform the initial login: Running in secure mode, but config doesn't have a
keytab
java.io.IOException: Running in secure mode, but config doesn't have a keytab
at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:236)
at
org.apache.hadoop.hbase.security.User$SecureHadoopUser.login(User.java:392)
at org.apache.hadoop.hbase.security.User.login(User.java:259)
at
org.apache.hadoop.hbase.security.UserProvider.login(UserProvider.java:116)
at org.apache.hadoop.hbase.AuthUtil.launchAuthChore(AuthUtil.java:64)
at org.apache.hadoop.hbase.tool.Canary.main(Canary.java:1146)
Exception in thread "main" java.io.IOException: Running in secure mode, but
config doesn't have a keytab
at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:236)
at
org.apache.hadoop.hbase.security.User$SecureHadoopUser.login(User.java:392)
at org.apache.hadoop.hbase.security.User.login(User.java:259)
at
org.apache.hadoop.hbase.security.UserProvider.login(UserProvider.java:116)
at org.apache.hadoop.hbase.AuthUtil.launchAuthChore(AuthUtil.java:64)
at org.apache.hadoop.hbase.tool.Canary.main(Canary.java:1146)
public static void main(String[] args) throws Exception {
final Configuration conf = HBaseConfiguration.create();
AuthUtil.launchAuthChore(conf);
int numThreads = conf.getInt("hbase.canary.threads.num", MAX_THREADS_NUM);
ExecutorService executor = new ScheduledThreadPoolExecutor(numThreads);
Class sinkClass =
conf.getClass("hbase.canary.sink.class", StdOutSink.class, Sink.class);
Sink sink = ReflectionUtils.newInstance(sinkClass);
int exitCode = ToolRunner.run(conf, new Canary(executor, sink), args);
executor.shutdown();
System.exit(exitCode);
}
In main class these params should be parsed and updated. else for any change to
these value hbase-stie.xml needs to be updated and hbase service needs to be
restarted.
> Canary
[jira] [Commented] (HBASE-15210) Undo aggressive load balancer logging at tens of lines per millisecond
[
https://issues.apache.org/jira/browse/HBASE-15210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133637#comment-15133637
]
Hudson commented on HBASE-15210:
FAILURE: Integrated in HBase-Trunk_matrix #681 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/681/])
HBASE-15210 Undo aggressive load balancer logging at tens of lines per (stack:
rev 18eff3c1c337003b2a419490e621f931d16936fb)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java
HBASE-15210 Undo aggressive load balancer logging at tens of lines per (stack:
rev f4d9597e1eef3baa9266581cf762ae3b249a1dff)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java
> Undo aggressive load balancer logging at tens of lines per millisecond
> --
>
> Key: HBASE-15210
> URL: https://issues.apache.org/jira/browse/HBASE-15210
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer
>Affects Versions: 1.2.0
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 15210.patch
>
>
> Sometimes the load balancer falls into a state of aggressive logging in 1.2.0
> writing out ten lines a millisecond like this:
> {code}
> 2016-02-02 17:15:01,382 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region server with non zero
> regions is ve0540.halxg.cloudera.com with locality 0.22908874
> 2016-02-02 17:15:01,382 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region index is 3 and its region
> server contains 4 regions
> 2016-02-02 17:15:01,384 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region server with non zero
> regions is ve0540.halxg.cloudera.com with locality 0.22908874
> 2016-02-02 17:15:01,384 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region index is 3 and its region
> server contains 4 regions
> 2016-02-02 17:15:01,386 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region server with non zero
> regions is ve0540.halxg.cloudera.com with locality 0.22908874
> 2016-02-02 17:15:01,386 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region index is 2 and its region
> server contains 4 regions
> 2016-02-02 17:15:01,387 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region server with non zero
> regions is ve0540.halxg.cloudera.com with locality 0.22908874
> 2016-02-02 17:15:01,387 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region index is 3 and its region
> server contains 4 regions
> 2016-02-02 17:15:01,388 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region server with non zero
> regions is ve0540.halxg.cloudera.com with locality 0.22908874
> 2016-02-02 17:15:01,388 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region index is 3 and its region
> server contains 4 regions
> 2016-02-02 17:15:01,388 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region server with non zero
> regions is ve0540.halxg.cloudera.com with locality 0.22908874
> {code}
> This issue is about less logging.
> This log was added recently by this:
> commit 54028140f4f19a6af81c8c8f29dda0c52491a0c9
> Author: tedyu
> Date: Thu Aug 13 09:11:59 2015 -0700
> HBASE-13376 Improvements to Stochastic load balancer (Vandana
> Ayyalasomayajula)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15177) Reduce garbage created under high load
[ https://issues.apache.org/jira/browse/HBASE-15177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133638#comment-15133638 ] Hudson commented on HBASE-15177: FAILURE: Integrated in HBase-Trunk_matrix #681 (See [https://builds.apache.org/job/HBase-Trunk_matrix/681/]) HBASE-15177 Reduce garbage created under high load (enis: rev a69272efe12f7b780fbf2fa14c42d0c0b155205f) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/AnnotationReadingPriorityFunction.java * hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java * hbase-client/src/test/java/org/apache/hadoop/hbase/ipc/TestIPCUtil.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java * hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/PayloadCarryingRpcController.java * hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestClientScanner.java * hbase-common/src/main/java/org/apache/hadoop/hbase/util/Threads.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java * hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/RpcClientImpl.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/IPCUtil.java * hbase-common/src/main/java/org/apache/hadoop/hbase/io/ByteBufferInputStream.java * hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/AsyncRpcChannel.java > Reduce garbage created under high load > -- > > Key: HBASE-15177 > URL: https://issues.apache.org/jira/browse/HBASE-15177 > Project: HBase > Issue Type: Improvement >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 2.0.0, 1.3.0 > > Attachments: Screen Shot 2016-01-26 at 10.03.48 PM.png, Screen Shot > 2016-01-26 at 10.03.56 PM.png, Screen Shot 2016-01-26 at 10.06.16 PM.png, > Screen Shot 2016-01-26 at 10.15.15 PM.png, hbase-15177_v0.patch, > hbase-15177_v1.patch, hbase-15177_v2.patch, hbase-15177_v3.patch, > hbase-15177_v4.patch, hbase-15177_v4.patch > > > I have been doing some profiling of the garbage being created. The idea was > to follow up on HBASE-14490 and experiment with offheap IPC byte buffers and > byte buffer re-use. However, without changing the IPC byte buffers for now, > there are a couple of (easy) improvements that I've identified from > profiling: > 1. RPCServer.Connection.processRequest() should work with ByteBuffer instead > of byte[] and not-recreate CodedInputStream a few times. > 2. RSRpcServices.getRegion() allocates two byte arrays for region, while only > 1 is needed. > 3. AnnotationReadingPriorityFunction is very expensive in allocations. Mainly > it allocates the regionName byte[] to get the table name. We already set the > priority for most of the operations (multi, get, increment, etc) but we are > only reading the priority in case of multi. We should use the priority from > the client side. > Lets do the simple improvements in this patch, we can get to IPC buffer > re-use in HBASE-14490. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15202) Reduce garbage while setting response
[ https://issues.apache.org/jira/browse/HBASE-15202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15202: --- Fix Version/s: 1.3.0 Pushed to branch-1 also. > Reduce garbage while setting response > - > > Key: HBASE-15202 > URL: https://issues.apache.org/jira/browse/HBASE-15202 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Minor > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15202.patch, HBASE-15202_1.patch, > HBASE-15202_2.patch, HBASE-15202_3.patch > > > Probably this can move under HBASE-15177. During RpcServer#setResponse we do > IPCUtil.getDelimitedMessageAsByteBuffer for the header and result. This > internally creates a byte[], CodedOutputStream and a BB. Then the totalSize > is also wrapped in a BB. > Finally all these BB are passed to BufferChain along with the Cellblock. > This JIRA is to reduce the number of objects that we create here and allow > one BB from this header, result and total Size along with one > CodedOutputStream and pass this BB along with the Cellblocks to the > Bufferchain. From the JMC tool can observe around 2% lesser object allocation > arising out of these objects. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15212) RRCServer should enforce max request size
[ https://issues.apache.org/jira/browse/HBASE-15212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133663#comment-15133663 ] ramkrishna.s.vasudevan commented on HBASE-15212: I think sooner we will be moving over to Offheap Memstore and offheap in write path. I think once that happens we have to fine tune other parameters also like flush related params as many of that are dependent on the heap Xmx which may become irrelevant to an extent once we go to Offheap. > RRCServer should enforce max request size > -- > > Key: HBASE-15212 > URL: https://issues.apache.org/jira/browse/HBASE-15212 > Project: HBase > Issue Type: Improvement >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 2.0.0, 1.3.0 > > Attachments: hbase-15212_v1.patch > > > A TODO from HBASE-15177 was that we are not protecting the RPCServer in case > an RPC request with a very large size is received. This might cause the > server to go OOM because we are allocating the RPC serialization into a BB. > Instead we should reject the RPC and close the connection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15205) Do not find the replication scope for every WAL#append()
[ https://issues.apache.org/jira/browse/HBASE-15205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133706#comment-15133706 ] ramkrishna.s.vasudevan commented on HBASE-15205: I have a query here. In order to make things simple - Does it makes sense to add the scope of all the families in the table descriptor which are NOT having default replication scope into every WALKey? Currently we are iterating the cells in a WALEdit and get every family in that cell - compare it with the HTD families scope and then add those scopes which are NOT having default scope. (as a map). Again while doing the filtering in ScopeWalFilter - we get the scope from the WALKey, iterate over the Cells in the WALEdit, check the family of the cells - if it is not in the scope map of the WALKey or if it is having default scope (we wont have this case as we don't write families with scope default) - we remove those cells. So instead while actually adding the scope for every WALKey during WAL append, can we just add all those families with out default replication scope per WALKey? The disadvantage is that suppose a Cell is having only one family and it has default scope still we will be writing other families which has non default scope but the advantage in terms of write path is that just have your scopes calculated once (may be in HRegion) and just use that every where and that will make your write path faster because we need not check every family in the cells and see if it is scope is default or not. If we don't need to do that way due to extra space - then we could have our scopes evaluated once per HRegion and in every append we may still need to check the family of every cell and see the scope of that family in our evaluated scope map and then create a new set of map which has scopes of the families that are not having default scope. > Do not find the replication scope for every WAL#append() > > > Key: HBASE-15205 > URL: https://issues.apache.org/jira/browse/HBASE-15205 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Minor > Fix For: 2.0.0 > > Attachments: ScopeWALEdits.jpg > > > After the byte[] and char[] the other top contributor for lot of GC (though > it is only 2.86%) is the UTF_8.newDecoder. > This happens because for every WAL append we try to calculate the replication > scope associate with the families associated with the TableDescriptor. I > think per WAL append doing this is very costly and creates lot of garbage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
