[jira] [Commented] (HBASE-15082) Fix merge of MVCC and SequenceID performance regression
[ https://issues.apache.org/jira/browse/HBASE-15082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15092821#comment-15092821 ] Elliott Clark commented on HBASE-15082: --- I think we should just get this in when it's ready to land. It it lands before 1.2 is ready then I feel we can get it in without making 1.2 too delayed. Still reading the code to try and make sure that everything else looks good. > Fix merge of MVCC and SequenceID performance regression > --- > > Key: HBASE-15082 > URL: https://issues.apache.org/jira/browse/HBASE-15082 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: 15082.patch, 15082v2.patch, 15082v2.txt, 15082v3.txt, > 15082v4.patch, 15082v5.patch, 15082v6.patch, 15082v7.patch > > > This is general fix for increments (appends, checkAnd*) perf-regression > identified in the parent issue. HBASE-15031 has a narrow fix for branch-1.1 > and branch-1.0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15065) SimpleRegionNormalizer should return multiple normalization plans in one run
[ https://issues.apache.org/jira/browse/HBASE-15065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15092820#comment-15092820 ] Ted Yu commented on HBASE-15065: Currently normalizer chore would run all the normalization plans. I think we should provide some throttling in certain scenarios. e.g. when normalization is enabled for an existing table which has many (close to) empty regions to be merged or, at cluster startup, the number of normalization plans for all the tables in the cluster could be quite high Throttling in the above scenarios would make the cluster more stable. [~enis] [~mantonov]: What's your opinion ? > SimpleRegionNormalizer should return multiple normalization plans in one run > > > Key: HBASE-15065 > URL: https://issues.apache.org/jira/browse/HBASE-15065 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: 15065-v1.txt, 15065-v2.txt, 15065.addendum > > > This is follow up to HBASE-14867 w.r.t. SimpleRegionNormalizer > Outline for enhancements: > * adjustment to the period when SimpleRegionNormalizer runs > * explore merge opportunities among all neighboring region pairs > * return multiple normalization plans -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15082) Fix merge of MVCC and SequenceID performance regression
[ https://issues.apache.org/jira/browse/HBASE-15082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15093062#comment-15093062 ] stack commented on HBASE-15082: --- bq. How far away is a fix for the current 1.2 blocker? I updated current blocker. Having trouble getting it to fail for me. Will keep at it. > Fix merge of MVCC and SequenceID performance regression > --- > > Key: HBASE-15082 > URL: https://issues.apache.org/jira/browse/HBASE-15082 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: 15082.patch, 15082v2.patch, 15082v2.txt, 15082v3.txt, > 15082v4.patch, 15082v5.patch, 15082v6.patch, 15082v7.patch > > > This is general fix for increments (appends, checkAnd*) perf-regression > identified in the parent issue. HBASE-15031 has a narrow fix for branch-1.1 > and branch-1.0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15052) Use EnvironmentEdgeManager in ReplicationSource
[
https://issues.apache.org/jira/browse/HBASE-15052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15092801#comment-15092801
]
Hudson commented on HBASE-15052:
SUCCESS: Integrated in HBase-1.2 #499 (See
[https://builds.apache.org/job/HBase-1.2/499/])
HBASE-15052 Use EnvironmentEdgeManager in ReplicationSource (matteo.bertozzi:
rev 4b59f62c231a39fd4a824c0cbf4c2166c4ec2a76)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
> Use EnvironmentEdgeManager in ReplicationSource
>
>
> Key: HBASE-15052
> URL: https://issues.apache.org/jira/browse/HBASE-15052
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 2.0.0, 1.2.0, 1.1.2, 1.0.3, 0.98.16.1
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
>Priority: Trivial
> Fix For: 2.0.0, 1.2.1, 1.1.3, 0.98.17, 1.0.4
>
> Attachments: HBASE-15052-v0.patch, HBASE-15052-v00.patch
>
>
> ReplicationSource is passing System.currentTimeMillis() to
> MetricsSource.setAgeOfLastShippedOp() which is subtracting that from
> EnvironmentEdgeManager.currentTime().
> {code}
> // if there was nothing to ship and it's not an error
> // set "ageOfLastShippedOp" to to indicate that we're current
> metrics.setAgeOfLastShippedOp(System.currentTimeMillis(), walGroupId);
> public void setAgeOfLastShippedOp(long timestamp, String walGroup) {
> long age = EnvironmentEdgeManager.currentTime() - timestamp;
> {code}
> we should just use EnvironmentEdgeManager.currentTime() in ReplicationSource
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15087) Fix hbase-common findbugs complaints
[ https://issues.apache.org/jira/browse/HBASE-15087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15092817#comment-15092817 ] stack commented on HBASE-15087: --- bq. ERROR: Docker failed to build image. Let me retry > Fix hbase-common findbugs complaints > > > Key: HBASE-15087 > URL: https://issues.apache.org/jira/browse/HBASE-15087 > Project: HBase > Issue Type: Sub-task > Components: build >Reporter: stack >Assignee: Stack > Attachments: 15087.patch, 15087v2.patch, 15087v3.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15082) Fix merge of MVCC and SequenceID performance regression
[
https://issues.apache.org/jira/browse/HBASE-15082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15092831#comment-15092831
]
Sean Busbey commented on HBASE-15082:
-
{quote}
The write pipeline changed a load with HBASE-12751. Porting the 'hack' from
branch-1.1 is mostly done as part of this attached patch. I could spend time
distilling and testing just forward porting the 'hack' for a 1.2. Preferred
trying to do the proper 'fix' given HBASE-12751 opens up new vistas allowing
reordering of when edits are sync'd and when they can go to memstore.
{quote}
Now I see my confusion. HBASE-12751 is marked as only 2.0.0, so I wasn't sure
why it'd be relevant. I see that HBASE-14465 backported it to branch-1.
How far away is a fix for the current 1.2 blocker?
> Fix merge of MVCC and SequenceID performance regression
> ---
>
> Key: HBASE-15082
> URL: https://issues.apache.org/jira/browse/HBASE-15082
> Project: HBase
> Issue Type: Sub-task
> Components: Performance
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 15082.patch, 15082v2.patch, 15082v2.txt, 15082v3.txt,
> 15082v4.patch, 15082v5.patch, 15082v6.patch, 15082v7.patch
>
>
> This is general fix for increments (appends, checkAnd*) perf-regression
> identified in the parent issue. HBASE-15031 has a narrow fix for branch-1.1
> and branch-1.0.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15052) Use EnvironmentEdgeManager in ReplicationSource
[
https://issues.apache.org/jira/browse/HBASE-15052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15092872#comment-15092872
]
Hudson commented on HBASE-15052:
SUCCESS: Integrated in HBase-1.3 #490 (See
[https://builds.apache.org/job/HBase-1.3/490/])
HBASE-15052 Use EnvironmentEdgeManager in ReplicationSource (matteo.bertozzi:
rev bb5026c6a42912334bb08cc65e983e80ee49ba06)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
> Use EnvironmentEdgeManager in ReplicationSource
>
>
> Key: HBASE-15052
> URL: https://issues.apache.org/jira/browse/HBASE-15052
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 2.0.0, 1.2.0, 1.1.2, 1.0.3, 0.98.16.1
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
>Priority: Trivial
> Fix For: 2.0.0, 1.2.1, 1.1.3, 0.98.17, 1.0.4
>
> Attachments: HBASE-15052-v0.patch, HBASE-15052-v00.patch
>
>
> ReplicationSource is passing System.currentTimeMillis() to
> MetricsSource.setAgeOfLastShippedOp() which is subtracting that from
> EnvironmentEdgeManager.currentTime().
> {code}
> // if there was nothing to ship and it's not an error
> // set "ageOfLastShippedOp" to to indicate that we're current
> metrics.setAgeOfLastShippedOp(System.currentTimeMillis(), walGroupId);
> public void setAgeOfLastShippedOp(long timestamp, String walGroup) {
> long age = EnvironmentEdgeManager.currentTime() - timestamp;
> {code}
> we should just use EnvironmentEdgeManager.currentTime() in ReplicationSource
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15075) Allow region split request to carry metadata
[
https://issues.apache.org/jira/browse/HBASE-15075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15092908#comment-15092908
]
Hadoop QA commented on HBASE-15075:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
44s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 17s
{color} | {color:green} master passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 20s
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 10m
12s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
37s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
51s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 5s
{color} | {color:red} hbase-client in master has 13 extant Findbugs warnings.
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 2s
{color} | {color:red} hbase-server in master has 84 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 56s
{color} | {color:green} master passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 20s
{color} | {color:red} hbase-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 34s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 35s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0_66.
{color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 0m 35s {color} |
{color:red} hbase-server in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 35s {color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 36s
{color} | {color:red} hbase-server in the patch failed with JDK v1.7.0_91.
{color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 0m 36s {color} |
{color:red} hbase-server in the patch failed with JDK v1.7.0_91. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 36s {color}
| {color:red} hbase-server in the patch failed with JDK v1.7.0_91. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m 23s
{color} | {color:red} Patch generated 12 new checkstyle issues in hbase-client
(total was 599, now 611). {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 39s
{color} | {color:red} Patch generated 1 new checkstyle issues in hbase-server
(total was 510, now 511). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
43s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 4 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
26m 19s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
52s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 57s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
|
[jira] [Commented] (HBASE-15052) Use EnvironmentEdgeManager in ReplicationSource
[
https://issues.apache.org/jira/browse/HBASE-15052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15092954#comment-15092954
]
Hudson commented on HBASE-15052:
FAILURE: Integrated in HBase-1.1-JDK8 #1724 (See
[https://builds.apache.org/job/HBase-1.1-JDK8/1724/])
HBASE-15052 Use EnvironmentEdgeManager in ReplicationSource (matteo.bertozzi:
rev 893dff8bd39a6dbc7242fe10ed9328e46d3d5a71)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
> Use EnvironmentEdgeManager in ReplicationSource
>
>
> Key: HBASE-15052
> URL: https://issues.apache.org/jira/browse/HBASE-15052
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 2.0.0, 1.2.0, 1.1.2, 1.0.3, 0.98.16.1
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
>Priority: Trivial
> Fix For: 2.0.0, 1.2.1, 1.1.3, 0.98.17, 1.0.4
>
> Attachments: HBASE-15052-v0.patch, HBASE-15052-v00.patch
>
>
> ReplicationSource is passing System.currentTimeMillis() to
> MetricsSource.setAgeOfLastShippedOp() which is subtracting that from
> EnvironmentEdgeManager.currentTime().
> {code}
> // if there was nothing to ship and it's not an error
> // set "ageOfLastShippedOp" to to indicate that we're current
> metrics.setAgeOfLastShippedOp(System.currentTimeMillis(), walGroupId);
> public void setAgeOfLastShippedOp(long timestamp, String walGroup) {
> long age = EnvironmentEdgeManager.currentTime() - timestamp;
> {code}
> we should just use EnvironmentEdgeManager.currentTime() in ReplicationSource
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14975) Don't color the total RIT line yellow if it's zero
[ https://issues.apache.org/jira/browse/HBASE-14975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-14975: --- Fix Version/s: 0.98.17 > Don't color the total RIT line yellow if it's zero > -- > > Key: HBASE-14975 > URL: https://issues.apache.org/jira/browse/HBASE-14975 > Project: HBase > Issue Type: Bug > Components: UI >Reporter: Elliott Clark >Assignee: Pallavi Adusumilli > Fix For: 2.0.0, 1.3.0, 0.98.17 > > Attachments: HBASE-14975.patch, Screen Shot 2015-12-14 at 11.37.13 > AM.png, Screenshot 2016-01-04.png > > > Right now if there are regions in transition, sometimes the RIT over 60 > seconds line is colored yellow. It shouldn't be colored yellow if there are > no regions that have been in transition too long. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15066) Small improvements to Canary tool
[
https://issues.apache.org/jira/browse/HBASE-15066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-15066:
---
Fix Version/s: 0.98.17
> Small improvements to Canary tool
> -
>
> Key: HBASE-15066
> URL: https://issues.apache.org/jira/browse/HBASE-15066
> Project: HBase
> Issue Type: Improvement
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.3.0, 0.98.17
>
> Attachments: hbase-15066_v1.patch
>
>
> From an internal request from a user:
> {quote}
> HBase comes with a canary tool which is very useful, but a few minor changes
> would make it substantially better. Specifically, you don't know what it's
> testing until it completes the test, so if it hangs, you can't tell where
> it's hanging. For example, if I run it with the -regionserver option, it
> sends get requests to each of the regionservers in some random order...
> And the same goes for the non-default mode, which hits each region for each
> table.
> If it would just print a message prior to each 'get', it would be extremely
> helpful.
> {quote}
> This patch:
> - Adds debug statements for pre-operation steps.
> - Adds bin/hbase canary support
> - Fixes table names with patterns where the pattern was not matched
> previously.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15075) Allow region split request to carry metadata
[
https://issues.apache.org/jira/browse/HBASE-15075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15093019#comment-15093019
]
Hadoop QA commented on HBASE-15075:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 1s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
38s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 14s
{color} | {color:green} master passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 9s
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 9m
1s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
33s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
49s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 58s
{color} | {color:red} hbase-client in master has 13 extant Findbugs warnings.
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 55s
{color} | {color:red} hbase-server in master has 84 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 2s
{color} | {color:green} master passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 22s
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 19s
{color} | {color:red} hbase-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 28s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 33s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0_66.
{color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 0m 33s {color} |
{color:red} hbase-server in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 33s {color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 36s
{color} | {color:red} hbase-server in the patch failed with JDK v1.7.0_91.
{color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 0m 36s {color} |
{color:red} hbase-server in the patch failed with JDK v1.7.0_91. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 36s {color}
| {color:red} hbase-server in the patch failed with JDK v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 11m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
57s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 4 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
24m 51s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 2s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 1s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 59s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 52s
{color} | {color:green} hbase-client
[jira] [Commented] (HBASE-15075) Allow region split request to carry metadata
[
https://issues.apache.org/jira/browse/HBASE-15075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15093070#comment-15093070
]
Andrew Purtell commented on HBASE-15075:
The description on this issue implies something generic ("metadata" is a
generic term). This patch only amends method calls to pass through an extra
argument, a UUID. Why use UUID? Why not a random long? Should we add API so
split requests can pass arbitrary metadata as issue description implies? Should
every request type be extended like this? I.e. merge transactions?
> Allow region split request to carry metadata
>
>
> Key: HBASE-15075
> URL: https://issues.apache.org/jira/browse/HBASE-15075
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 15075-v0.txt, 15075-v1.txt, 15075-v2.txt,
> HBASE-15075.v2.patch, HBASE-15075.v3.patch
>
>
> During the process of improving region normalization feature, I found that if
> region split request triggered by the execution of SplitNormalizationPlan
> fails, there is no way of knowing whether the failed split originated from
> region normalization.
> The association of particular split request with outcome of split would give
> RegionNormalizer information so that it can make better normalization
> decisions in the subsequent invocations.
> One approach is to embed metadata in SplitRequest which gets passed through
> RegionStateTransitionContext when
> RegionServerServices#reportRegionStateTransition() is called.
> This way, RegionStateListener can be notified with the metadata (id of the
> requester).
> See discussion on dev mailing list
> http://search-hadoop.com/m/YGbbCXdkivihp2
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15082) Fix merge of MVCC and SequenceID performance regression
[ https://issues.apache.org/jira/browse/HBASE-15082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15092782#comment-15092782 ] stack commented on HBASE-15082: --- bq. Other current releases already have the slow increment/append so it's not making the situation worse relative to other releases in 1.2 to hold off. A sibling issue has made it so branch-1.0 and branch-1.1 have a route to fast increment (set a configuration and allow that increments are their own thing and away you go). There is no means available in 1.2. HBASE-12751 changed the write pipeline. Would be cool to have this in 1.2. You are quiet [~eclark] > Fix merge of MVCC and SequenceID performance regression > --- > > Key: HBASE-15082 > URL: https://issues.apache.org/jira/browse/HBASE-15082 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: 15082.patch, 15082v2.patch, 15082v2.txt, 15082v3.txt, > 15082v4.patch, 15082v5.patch, 15082v6.patch, 15082v7.patch > > > This is general fix for increments (appends, checkAnd*) perf-regression > identified in the parent issue. HBASE-15031 has a narrow fix for branch-1.1 > and branch-1.0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15052) Use EnvironmentEdgeManager in ReplicationSource
[
https://issues.apache.org/jira/browse/HBASE-15052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15092953#comment-15092953
]
Hudson commented on HBASE-15052:
FAILURE: Integrated in HBase-Trunk_matrix #625 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/625/])
HBASE-15052 Use EnvironmentEdgeManager in ReplicationSource (matteo.bertozzi:
rev ec47a811a2e7c0bb3f844d780049bb8a67cc4c35)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
> Use EnvironmentEdgeManager in ReplicationSource
>
>
> Key: HBASE-15052
> URL: https://issues.apache.org/jira/browse/HBASE-15052
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 2.0.0, 1.2.0, 1.1.2, 1.0.3, 0.98.16.1
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
>Priority: Trivial
> Fix For: 2.0.0, 1.2.1, 1.1.3, 0.98.17, 1.0.4
>
> Attachments: HBASE-15052-v0.patch, HBASE-15052-v00.patch
>
>
> ReplicationSource is passing System.currentTimeMillis() to
> MetricsSource.setAgeOfLastShippedOp() which is subtracting that from
> EnvironmentEdgeManager.currentTime().
> {code}
> // if there was nothing to ship and it's not an error
> // set "ageOfLastShippedOp" to to indicate that we're current
> metrics.setAgeOfLastShippedOp(System.currentTimeMillis(), walGroupId);
> public void setAgeOfLastShippedOp(long timestamp, String walGroup) {
> long age = EnvironmentEdgeManager.currentTime() - timestamp;
> {code}
> we should just use EnvironmentEdgeManager.currentTime() in ReplicationSource
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13372) Unit tests for SplitTransaction and RegionMergeTransaction listeners
[ https://issues.apache.org/jira/browse/HBASE-13372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15093079#comment-15093079 ] Andrew Purtell commented on HBASE-13372: Looks good. Like [~ted_yu] mentions, with another unit counting transitions for the merge transaction listener, this is good to go. > Unit tests for SplitTransaction and RegionMergeTransaction listeners > > > Key: HBASE-13372 > URL: https://issues.apache.org/jira/browse/HBASE-13372 > Project: HBase > Issue Type: Test >Affects Versions: 2.0.0, 1.1.0 >Reporter: Andrew Purtell >Assignee: Gabor Liptak > Labels: beginner > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-13372.2.patch > > > We have new Listener interfaces in SplitTransaction and > RegionMergeTransaction. There are no use cases for these yet, nor unit tests. > We should have unit tests for these that do something just a bit nontrivial > so as to provide a useful example. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14738) Backport HBASE-11927 (Use Native Hadoop Library for HFile checksum) to 0.98
[ https://issues.apache.org/jira/browse/HBASE-14738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-14738: --- Fix Version/s: (was: 0.98.17) Unscheduling. I'm not seeing gains worth the risk of changing this code in 0.98. Let me leave this issue open with patch parked here in case we want to revisit this later. > Backport HBASE-11927 (Use Native Hadoop Library for HFile checksum) to 0.98 > --- > > Key: HBASE-14738 > URL: https://issues.apache.org/jira/browse/HBASE-14738 > Project: HBase > Issue Type: Task >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Attachments: HBASE-14738-0.98.patch > > > Profiling 0.98.15 I see 20-30% of CPU time spent in Hadoop's PureJavaCrc32. > Not surprising given previous results described on HBASE-11927. Backport. > There are two issues with the backport: > # The patch on 11927 changes the default CRC type from CRC32 to CRC32C. > Although the changes are backwards compatible -files with either CRC type > will be handled correctly in a transparent manner - we should probably leave > the default alone in 0.98 and advise users on a site configuration change to > use CRC32C if desired, for potential hardware acceleration. > # Need a shim for differences between Hadoop's DataChecksum type. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15082) Fix merge of MVCC and SequenceID performance regression
[ https://issues.apache.org/jira/browse/HBASE-15082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15092811#comment-15092811 ] Andrew Purtell commented on HBASE-15082: bq. could we port the sibling issue to branch-1.2? Same question here > Fix merge of MVCC and SequenceID performance regression > --- > > Key: HBASE-15082 > URL: https://issues.apache.org/jira/browse/HBASE-15082 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: 15082.patch, 15082v2.patch, 15082v2.txt, 15082v3.txt, > 15082v4.patch, 15082v5.patch, 15082v6.patch, 15082v7.patch > > > This is general fix for increments (appends, checkAnd*) perf-regression > identified in the parent issue. HBASE-15031 has a narrow fix for branch-1.1 > and branch-1.0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15075) Allow region split request to carry metadata
[ https://issues.apache.org/jira/browse/HBASE-15075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-15075: --- Attachment: HBASE-15075.v3.patch Gone through whole patch - the trailing whitespace was from protobuf generated file > Allow region split request to carry metadata > > > Key: HBASE-15075 > URL: https://issues.apache.org/jira/browse/HBASE-15075 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 15075-v0.txt, 15075-v1.txt, 15075-v2.txt, > HBASE-15075.v2.patch, HBASE-15075.v3.patch > > > During the process of improving region normalization feature, I found that if > region split request triggered by the execution of SplitNormalizationPlan > fails, there is no way of knowing whether the failed split originated from > region normalization. > The association of particular split request with outcome of split would give > RegionNormalizer information so that it can make better normalization > decisions in the subsequent invocations. > One approach is to embed metadata in SplitRequest which gets passed through > RegionStateTransitionContext when > RegionServerServices#reportRegionStateTransition() is called. > This way, RegionStateListener can be notified with the metadata (id of the > requester). > See discussion on dev mailing list > http://search-hadoop.com/m/YGbbCXdkivihp2 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14962) TestSplitWalDataLoss fails on all branches
[
https://issues.apache.org/jira/browse/HBASE-14962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15093061#comment-15093061
]
stack commented on HBASE-14962:
---
Running with the test that the JIRA is about (not TestSplitLogManager) I got
this eventually:
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 5.812 sec <<<
FAILURE! - in org.apache.hadoop.hbase.regionserver.TestSplitWalDataLoss
test(org.apache.hadoop.hbase.regionserver.TestSplitWalDataLoss) Time elapsed:
5 sec <<< ERROR!
java.lang.RuntimeException: Could not create interface
org.apache.hadoop.hbase.test.MetricsAssertHelper Is the hadoop compatibility
jar on the classpath?
A different complaint. Is that what you see?
On TestSplitLogManager, the one you pasted, I can't get it to fail [~eclark]
> TestSplitWalDataLoss fails on all branches
> --
>
> Key: HBASE-14962
> URL: https://issues.apache.org/jira/browse/HBASE-14962
> Project: HBase
> Issue Type: Sub-task
>Reporter: Elliott Clark
>Assignee: stack
>Priority: Blocker
> Fix For: 1.2.0
>
>
> With some regularity I am seeing:
> {code}
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1
> action: TestSplitWalDataLoss:dataloss: 1 time,
> at
> org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:228)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1800(AsyncProcess.java:208)
> at
> org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1712)
> at
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:240)
> at
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:190)
> at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1430)
> at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1021)
> at
> org.apache.hadoop.hbase.regionserver.TestSplitWalDataLoss.test(TestSplitWalDataLoss.java:121)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14465) Backport 'Allow rowlock to be reader/write' to branch-1
[ https://issues.apache.org/jira/browse/HBASE-14465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-14465: Release Note: Locks on row are now reader/writer rather than exclusive. Moves sequenceid out of HRegion and into MVCC class; MVCC is now in charge. A WAL append is still stamped in same way (we pass MVCC context in a few places where we previously we did not). MVCC methods cleaned up. Make a bit more sense now. Less of them. Simplifies our update of MemStore/WAL. Now we update memstore AFTER we add to WAL (but before we sync). This fixes possible dataloss when two edits came in with same coordinates; we could order the edits in memstore differently to how they arrived in the WAL. Marked as an incompatible change because it breaks Distributed Log Replay, a feature we'd determined already was unreliable and to be removed (See http://search-hadoop.com/m/YGbbhTJpoal8GD1). was:See parent issue. > Backport 'Allow rowlock to be reader/write' to branch-1 > --- > > Key: HBASE-14465 > URL: https://issues.apache.org/jira/browse/HBASE-14465 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: stack >Assignee: stack > Fix For: 1.2.0, 1.3.0 > > Attachments: 14465.branch-1.txt, 14465.branch-1.v2.txt, > 14465.branch-1.v2.txt, 14465.branch-1.v2.txt, 14465.branch-1.v3.txt, > 14465.branch-1.v3.txt, 14465.branch-1.v4.txt, 14465.branch-1.v4.txt, > 14465.branch-1.v4.txt, 14465.branch-1.v5.txt, 14465.branch-1.v5.txt, > 14465.branch-1.v9.txt, 14465.branch-1.v9.txt, 14465.branch-1.v9.txt, 14465.txt > > > Backport to branch-1. > You want this in 1.2 [~busbey]? Its cleanup and fixes a possible dataloss. On > other hand, its a bit of refactoring. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15082) Fix merge of MVCC and SequenceID performance regression
[ https://issues.apache.org/jira/browse/HBASE-15082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15092814#comment-15092814 ] stack commented on HBASE-15082: --- bq. could we port the sibling issue to branch-1.2? The write pipeline changed a load with HBASE-12751. Porting the 'hack' from branch-1.1 is mostly done as part of this attached patch. I could spend time distilling and testing just forward porting the 'hack' for a 1.2. Preferred trying to do the proper 'fix' given HBASE-12751 opens up new vistas allowing reordering of when edits are sync'd and when they can go to memstore. > Fix merge of MVCC and SequenceID performance regression > --- > > Key: HBASE-15082 > URL: https://issues.apache.org/jira/browse/HBASE-15082 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: 15082.patch, 15082v2.patch, 15082v2.txt, 15082v3.txt, > 15082v4.patch, 15082v5.patch, 15082v6.patch, 15082v7.patch > > > This is general fix for increments (appends, checkAnd*) perf-regression > identified in the parent issue. HBASE-15031 has a narrow fix for branch-1.1 > and branch-1.0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14465) Backport 'Allow rowlock to be reader/write' to branch-1
[ https://issues.apache.org/jira/browse/HBASE-14465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15092828#comment-15092828 ] Sean Busbey commented on HBASE-14465: - copied release note from parent issue so it'll show up in generated release notes. > Backport 'Allow rowlock to be reader/write' to branch-1 > --- > > Key: HBASE-14465 > URL: https://issues.apache.org/jira/browse/HBASE-14465 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: stack >Assignee: stack > Fix For: 1.2.0, 1.3.0 > > Attachments: 14465.branch-1.txt, 14465.branch-1.v2.txt, > 14465.branch-1.v2.txt, 14465.branch-1.v2.txt, 14465.branch-1.v3.txt, > 14465.branch-1.v3.txt, 14465.branch-1.v4.txt, 14465.branch-1.v4.txt, > 14465.branch-1.v4.txt, 14465.branch-1.v5.txt, 14465.branch-1.v5.txt, > 14465.branch-1.v9.txt, 14465.branch-1.v9.txt, 14465.branch-1.v9.txt, 14465.txt > > > Backport to branch-1. > You want this in 1.2 [~busbey]? Its cleanup and fixes a possible dataloss. On > other hand, its a bit of refactoring. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14738) Backport HBASE-11927 (Use Native Hadoop Library for HFile checksum) to 0.98
[ https://issues.apache.org/jira/browse/HBASE-14738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-14738: --- Status: Open (was: Patch Available) > Backport HBASE-11927 (Use Native Hadoop Library for HFile checksum) to 0.98 > --- > > Key: HBASE-14738 > URL: https://issues.apache.org/jira/browse/HBASE-14738 > Project: HBase > Issue Type: Task >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Attachments: HBASE-14738-0.98.patch > > > Profiling 0.98.15 I see 20-30% of CPU time spent in Hadoop's PureJavaCrc32. > Not surprising given previous results described on HBASE-11927. Backport. > There are two issues with the backport: > # The patch on 11927 changes the default CRC type from CRC32 to CRC32C. > Although the changes are backwards compatible -files with either CRC type > will be handled correctly in a transparent manner - we should probably leave > the default alone in 0.98 and advise users on a site configuration change to > use CRC32C if desired, for potential hardware acceleration. > # Need a shim for differences between Hadoop's DataChecksum type. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15075) Allow region split request to carry metadata
[ https://issues.apache.org/jira/browse/HBASE-15075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-15075: --- Attachment: HBASE-15075.v2.patch > Allow region split request to carry metadata > > > Key: HBASE-15075 > URL: https://issues.apache.org/jira/browse/HBASE-15075 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 15075-v0.txt, 15075-v1.txt, 15075-v2.txt, > HBASE-15075.v2.patch > > > During the process of improving region normalization feature, I found that if > region split request triggered by the execution of SplitNormalizationPlan > fails, there is no way of knowing whether the failed split originated from > region normalization. > The association of particular split request with outcome of split would give > RegionNormalizer information so that it can make better normalization > decisions in the subsequent invocations. > One approach is to embed metadata in SplitRequest which gets passed through > RegionStateTransitionContext when > RegionServerServices#reportRegionStateTransition() is called. > This way, RegionStateListener can be notified with the metadata (id of the > requester). > See discussion on dev mailing list > http://search-hadoop.com/m/YGbbCXdkivihp2 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15082) Fix merge of MVCC and SequenceID performance regression
[ https://issues.apache.org/jira/browse/HBASE-15082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15092791#comment-15092791 ] Sean Busbey commented on HBASE-15082: - could we port the sibling issue to branch-1.2? > Fix merge of MVCC and SequenceID performance regression > --- > > Key: HBASE-15082 > URL: https://issues.apache.org/jira/browse/HBASE-15082 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: 15082.patch, 15082v2.patch, 15082v2.txt, 15082v3.txt, > 15082v4.patch, 15082v5.patch, 15082v6.patch, 15082v7.patch > > > This is general fix for increments (appends, checkAnd*) perf-regression > identified in the parent issue. HBASE-15031 has a narrow fix for branch-1.1 > and branch-1.0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15083) Gets from Multiactions are not counted in metrics for gets.
[
https://issues.apache.org/jira/browse/HBASE-15083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15092787#comment-15092787
]
Hadoop QA commented on HBASE-15083:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
31s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s
{color} | {color:green} master passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 54s
{color} | {color:red} hbase-server in master has 84 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} master passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
22m 37s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 113m 31s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0_66.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 106m 10s
{color} | {color:red} hbase-server in the patch failed with JDK v1.7.0_91.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 264m 32s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Timed out junit tests |
org.apache.hadoop.hbase.client.replication.TestReplicationAdminWithClusters |
| | org.apache.hadoop.hbase.filter.TestFuzzyRowFilterEndToEnd |
| JDK v1.7.0_91 Failed junit tests | hadoop.hbase.mapreduce.TestSyncTable |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.9.1 Server=1.9.1 Image:yetus/hbase:date2016-01-11 |
| JIRA
[jira] [Commented] (HBASE-15077) Support OffheapKV write in compaction with out copying data on heap
[
https://issues.apache.org/jira/browse/HBASE-15077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15092929#comment-15092929
]
Hadoop QA commented on HBASE-15077:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
51s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 41s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 59s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 5m
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 57s
{color} | {color:red} hbase-common in master has 14 extant Findbugs warnings.
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 15s
{color} | {color:red} hbase-server in master has 83 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 14s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 2s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 36s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 7m 21s {color}
| {color:red} hbase-common-jdk1.8.0 with JDK v1.8.0 generated 3 new issues (was
25, now 25). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 58s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 8m 20s {color}
| {color:red} hbase-common-jdk1.7.0_79 with JDK v1.7.0_79 generated 3 new
issues (was 25, now 25). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 58s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 21s
{color} | {color:red} Patch generated 1 new checkstyle issues in hbase-server
(total was 52, now 52). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 19s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 28s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 7s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 41s
{color} | {color:green} hbase-common in the patch passed with JDK v1.8.0.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 138m 37s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0. {color}
|
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 13s
{color} | {color:green} hbase-common in the patch passed with JDK v1.7.0_79.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 104m 56s
{color} | {color:red} hbase-server in the patch failed with JDK v1.7.0_79.
{color} |
| {color:green}+1{color} |
[jira] [Commented] (HBASE-15052) Use EnvironmentEdgeManager in ReplicationSource
[
https://issues.apache.org/jira/browse/HBASE-15052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15092975#comment-15092975
]
Hudson commented on HBASE-15052:
FAILURE: Integrated in HBase-1.1-JDK7 #1637 (See
[https://builds.apache.org/job/HBase-1.1-JDK7/1637/])
HBASE-15052 Use EnvironmentEdgeManager in ReplicationSource (matteo.bertozzi:
rev 893dff8bd39a6dbc7242fe10ed9328e46d3d5a71)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
> Use EnvironmentEdgeManager in ReplicationSource
>
>
> Key: HBASE-15052
> URL: https://issues.apache.org/jira/browse/HBASE-15052
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 2.0.0, 1.2.0, 1.1.2, 1.0.3, 0.98.16.1
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
>Priority: Trivial
> Fix For: 2.0.0, 1.2.1, 1.1.3, 0.98.17, 1.0.4
>
> Attachments: HBASE-15052-v0.patch, HBASE-15052-v00.patch
>
>
> ReplicationSource is passing System.currentTimeMillis() to
> MetricsSource.setAgeOfLastShippedOp() which is subtracting that from
> EnvironmentEdgeManager.currentTime().
> {code}
> // if there was nothing to ship and it's not an error
> // set "ageOfLastShippedOp" to to indicate that we're current
> metrics.setAgeOfLastShippedOp(System.currentTimeMillis(), walGroupId);
> public void setAgeOfLastShippedOp(long timestamp, String walGroup) {
> long age = EnvironmentEdgeManager.currentTime() - timestamp;
> {code}
> we should just use EnvironmentEdgeManager.currentTime() in ReplicationSource
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15085) IllegalStateException was thrown when scanning on bulkloaded HFiles
[
https://issues.apache.org/jira/browse/HBASE-15085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15091598#comment-15091598
]
ramkrishna.s.vasudevan commented on HBASE-15085:
[~chenheng]
Ya I agree with Victor Xu's reply here. There were previously some cases where
the HFiles that were created were not directly from a flat file instead it was
an HFile created under another cluster and just move it to the new cluster
(something like migration).
> IllegalStateException was thrown when scanning on bulkloaded HFiles
> ---
>
> Key: HBASE-15085
> URL: https://issues.apache.org/jira/browse/HBASE-15085
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.12, 1.1.2
> Environment: HBase-0.98.12 & Hadoop-2.6.0 & JDK1.7
> HBase-1.1.2 & Hadoop-2.6.0 & JDK1.7
>Reporter: Victor Xu
>Assignee: Victor Xu
> Labels: hfile
> Attachments: HBASE-15085-0.98-v1.patch, HBASE-15085-0.98-v2.patch,
> HBASE-15085-0.98-v3.patch, HBASE-15085-0.98-v4.patch, HBASE-15085-v1.patch,
> HBASE-15085-v2.patch, HBASE-15085-v3.patch, HBASE-15085-v4.patch
>
>
> IllegalStateException was thrown when we scanned from an HFile which was bulk
> loaded several minutes ago, as shown below:
> {code}
> 2015-12-16 22:20:54,456 ERROR
> com.taobao.kart.coprocessor.server.KartCoprocessor:
> icbu_ae_ws_product,/0055,1450275490479.6a6a700f465ad074287fed720c950f7c.
> batchNotify exception
> java.lang.IllegalStateException: EncodedScanner works only on encoded data
> blocks
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.updateCurrentBlock(HFileReaderV2.java:1042)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.seekTo(HFileReaderV2.java:1093)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:244)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:152)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:329)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:188)
> at
> org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:1879)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.(HRegion.java:4068)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2029)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2015)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1992)
> {code}
> I used 'hbase hfile' command to analyse the meta and block info of the hfile,
> finding that even through the DATA_BLOCK_ENCODING was 'DIFF' in FileInfo, the
> actual data blocks was written without any encoding algorithms(BlockType was
> 'DATA', not 'ENCODED_DATA'):
> {code}
> Fileinfo:
> BLOOM_FILTER_TYPE = ROW
> BULKLOAD_SOURCE_TASK = attempt_1442077249005_606706_r_12_0
> BULKLOAD_TIMESTAMP = \x00\x00\x01R\x12$\x13\x12
> DATA_BLOCK_ENCODING = DIFF
> ...
> DataBlock Header:
> HFileBlock [ fileOffset=0 headerSize()=33 blockType=DATA
> onDiskSizeWithoutHeader=65591 uncompressedSizeWithoutHeader=65571
> prevBlockOffset=-1 isUseHBaseChecksum()=true checksumType=CRC32
> bytesPerChecksum=16384 onDiskDataSizeWithHeader=65604
> getOnDiskSizeWithHeader()=65624 totalChecksumBytes()=20 isUnpacked()=true
> buf=[ java.nio.HeapByteBuffer[pos=0 lim=65624 cap=65657],
> array().length=65657, arrayOffset()=0 ]
> dataBeginsWith=\x00\x00\x003\x00\x00\x00\x0A\x00\x10/0008:18\x01dprod
> fileContext=HFileContext [ usesHBaseChecksum=true checksumType=CRC32
> bytesPerChecksum=16384 blocksize=65536 encoding=NONE includesMvcc=true
> includesTags=false compressAlgo=NONE compressTags=false cryptoContext=[
> cipher=NONE keyHash=NONE ] ] ]
> {code}
> The data block encoding in file info was not consistent with the one in data
> block, which means there must be something wrong with the bulkload process.
> After debugging on each step of bulkload, I found that LoadIncrementalHFiles
> had a bug when loading hfile into a splitted region.
> {code}
> /**
>* Copy half of an HFile into a new HFile.
>*/
> private static void copyHFileHalf(
> Configuration conf, Path inFile, Path outFile, Reference reference,
> HColumnDescriptor familyDescriptor)
> throws IOException {
> FileSystem fs = inFile.getFileSystem(conf);
> CacheConfig cacheConf = new CacheConfig(conf);
> HalfStoreFileReader halfReader = null;
> StoreFile.Writer halfWriter = null;
> try {
> halfReader = new HalfStoreFileReader(fs, inFile, cacheConf, reference,
> conf);
> Map fileInfo
[jira] [Commented] (HBASE-15085) IllegalStateException was thrown when scanning on bulkloaded HFiles
[
https://issues.apache.org/jira/browse/HBASE-15085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15091638#comment-15091638
]
Heng Chen commented on HBASE-15085:
---
{quote}
Actually, there are some scenarios where we had different encoding
configurations between table DATA_BLOCK_ENCODING and HFile loaded. ...
{quote}
OK. So we ignore encoding setting in HFile when writing into new Table. Patch
looks good to me. After QA passes, i will +1.
> IllegalStateException was thrown when scanning on bulkloaded HFiles
> ---
>
> Key: HBASE-15085
> URL: https://issues.apache.org/jira/browse/HBASE-15085
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.12, 1.1.2
> Environment: HBase-0.98.12 & Hadoop-2.6.0 & JDK1.7
> HBase-1.1.2 & Hadoop-2.6.0 & JDK1.7
>Reporter: Victor Xu
>Assignee: Victor Xu
> Labels: hfile
> Attachments: HBASE-15085-0.98-v1.patch, HBASE-15085-0.98-v2.patch,
> HBASE-15085-0.98-v3.patch, HBASE-15085-0.98-v4.patch, HBASE-15085-v1.patch,
> HBASE-15085-v2.patch, HBASE-15085-v3.patch, HBASE-15085-v4.patch
>
>
> IllegalStateException was thrown when we scanned from an HFile which was bulk
> loaded several minutes ago, as shown below:
> {code}
> 2015-12-16 22:20:54,456 ERROR
> com.taobao.kart.coprocessor.server.KartCoprocessor:
> icbu_ae_ws_product,/0055,1450275490479.6a6a700f465ad074287fed720c950f7c.
> batchNotify exception
> java.lang.IllegalStateException: EncodedScanner works only on encoded data
> blocks
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.updateCurrentBlock(HFileReaderV2.java:1042)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.seekTo(HFileReaderV2.java:1093)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:244)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:152)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:329)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:188)
> at
> org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:1879)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.(HRegion.java:4068)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2029)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2015)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1992)
> {code}
> I used 'hbase hfile' command to analyse the meta and block info of the hfile,
> finding that even through the DATA_BLOCK_ENCODING was 'DIFF' in FileInfo, the
> actual data blocks was written without any encoding algorithms(BlockType was
> 'DATA', not 'ENCODED_DATA'):
> {code}
> Fileinfo:
> BLOOM_FILTER_TYPE = ROW
> BULKLOAD_SOURCE_TASK = attempt_1442077249005_606706_r_12_0
> BULKLOAD_TIMESTAMP = \x00\x00\x01R\x12$\x13\x12
> DATA_BLOCK_ENCODING = DIFF
> ...
> DataBlock Header:
> HFileBlock [ fileOffset=0 headerSize()=33 blockType=DATA
> onDiskSizeWithoutHeader=65591 uncompressedSizeWithoutHeader=65571
> prevBlockOffset=-1 isUseHBaseChecksum()=true checksumType=CRC32
> bytesPerChecksum=16384 onDiskDataSizeWithHeader=65604
> getOnDiskSizeWithHeader()=65624 totalChecksumBytes()=20 isUnpacked()=true
> buf=[ java.nio.HeapByteBuffer[pos=0 lim=65624 cap=65657],
> array().length=65657, arrayOffset()=0 ]
> dataBeginsWith=\x00\x00\x003\x00\x00\x00\x0A\x00\x10/0008:18\x01dprod
> fileContext=HFileContext [ usesHBaseChecksum=true checksumType=CRC32
> bytesPerChecksum=16384 blocksize=65536 encoding=NONE includesMvcc=true
> includesTags=false compressAlgo=NONE compressTags=false cryptoContext=[
> cipher=NONE keyHash=NONE ] ] ]
> {code}
> The data block encoding in file info was not consistent with the one in data
> block, which means there must be something wrong with the bulkload process.
> After debugging on each step of bulkload, I found that LoadIncrementalHFiles
> had a bug when loading hfile into a splitted region.
> {code}
> /**
>* Copy half of an HFile into a new HFile.
>*/
> private static void copyHFileHalf(
> Configuration conf, Path inFile, Path outFile, Reference reference,
> HColumnDescriptor familyDescriptor)
> throws IOException {
> FileSystem fs = inFile.getFileSystem(conf);
> CacheConfig cacheConf = new CacheConfig(conf);
> HalfStoreFileReader halfReader = null;
> StoreFile.Writer halfWriter = null;
> try {
> halfReader = new HalfStoreFileReader(fs, inFile, cacheConf, reference,
> conf);
> Map fileInfo =
[jira] [Commented] (HBASE-15082) Fix merge of MVCC and SequenceID performance regression
[
https://issues.apache.org/jira/browse/HBASE-15082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15091760#comment-15091760
]
Anoop Sam John commented on HBASE-15082:
bq.this.setIsolationLevel(get.getIsolationLevel());
Good catch and fix.. Can we do the same in other constructor also? Scan(Scan
scan)
{code}
if (cell.getTagsLength() <= 0) return tags;
236 Iterator itr =
237 CellUtil.tagsIterator(cell.getTagsArray(),
cell.getTagsOffset(), cell.getTagsLength());
{code}
Suggest using CellUtil.tagsIterator(cell). No need to add the length check
also as the CellUtil API doing it and returning a static instance of Empty
iterator. So no unwanted object creation also.
bq.((HStore)store).getHRegion().getReadPoint(IsolationLevel.READ_COMMITTED));
getReadpoint() in Region interface which is not private. Is the name change
with out deprecation is right?
HRegion#doMinibatchOperation
What is the use of mvccNum now? I see this is being passed to
applyFamilyMapToMemstore but not assigned with any value now. Before sync()
call we wait for the seqId and that should be used to apply to cells while
adding to memstore ?
> Fix merge of MVCC and SequenceID performance regression
> ---
>
> Key: HBASE-15082
> URL: https://issues.apache.org/jira/browse/HBASE-15082
> Project: HBase
> Issue Type: Sub-task
> Components: Performance
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 15082.patch, 15082v2.patch, 15082v2.txt, 15082v3.txt,
> 15082v4.patch, 15082v5.patch, 15082v6.patch, 15082v7.patch
>
>
> This is general fix for increments (appends, checkAnd*) perf-regression
> identified in the parent issue. HBASE-15031 has a narrow fix for branch-1.1
> and branch-1.0.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15085) IllegalStateException was thrown when scanning on bulkloaded HFiles
[
https://issues.apache.org/jira/browse/HBASE-15085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15091590#comment-15091590
]
Victor Xu commented on HBASE-15085:
---
Good question. Actually, there are some scenarios where we had different
encoding configurations between table DATA_BLOCK_ENCODING and HFile loaded. For
example, we sometimes need to migrate HFiles from a table with DIFF encoding to
a NONE encoding table in another cluster. Thus, I think HFile should be
self-descriptive and everyone can bulk load and read the HFile freely as long
as the file version is compatible. Besides, there's another very common
situation: while the HTable descriptor is defined by the DBAs or Devs that know
how to optimize the HBase cluster, the HFiles are usually generated by
application clients who know little about DATA_BLOCK_ENCODING(and some other
parameters) on HTable, and the clients usually use one single
configuration(DIFF) for all different scenarios(some are DIFF tables, and
others are NONE tables).
> IllegalStateException was thrown when scanning on bulkloaded HFiles
> ---
>
> Key: HBASE-15085
> URL: https://issues.apache.org/jira/browse/HBASE-15085
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.12, 1.1.2
> Environment: HBase-0.98.12 & Hadoop-2.6.0 & JDK1.7
> HBase-1.1.2 & Hadoop-2.6.0 & JDK1.7
>Reporter: Victor Xu
>Assignee: Victor Xu
> Labels: hfile
> Attachments: HBASE-15085-0.98-v1.patch, HBASE-15085-0.98-v2.patch,
> HBASE-15085-0.98-v3.patch, HBASE-15085-0.98-v4.patch, HBASE-15085-v1.patch,
> HBASE-15085-v2.patch, HBASE-15085-v3.patch, HBASE-15085-v4.patch
>
>
> IllegalStateException was thrown when we scanned from an HFile which was bulk
> loaded several minutes ago, as shown below:
> {code}
> 2015-12-16 22:20:54,456 ERROR
> com.taobao.kart.coprocessor.server.KartCoprocessor:
> icbu_ae_ws_product,/0055,1450275490479.6a6a700f465ad074287fed720c950f7c.
> batchNotify exception
> java.lang.IllegalStateException: EncodedScanner works only on encoded data
> blocks
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.updateCurrentBlock(HFileReaderV2.java:1042)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.seekTo(HFileReaderV2.java:1093)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:244)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:152)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:329)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:188)
> at
> org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:1879)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.(HRegion.java:4068)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2029)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2015)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1992)
> {code}
> I used 'hbase hfile' command to analyse the meta and block info of the hfile,
> finding that even through the DATA_BLOCK_ENCODING was 'DIFF' in FileInfo, the
> actual data blocks was written without any encoding algorithms(BlockType was
> 'DATA', not 'ENCODED_DATA'):
> {code}
> Fileinfo:
> BLOOM_FILTER_TYPE = ROW
> BULKLOAD_SOURCE_TASK = attempt_1442077249005_606706_r_12_0
> BULKLOAD_TIMESTAMP = \x00\x00\x01R\x12$\x13\x12
> DATA_BLOCK_ENCODING = DIFF
> ...
> DataBlock Header:
> HFileBlock [ fileOffset=0 headerSize()=33 blockType=DATA
> onDiskSizeWithoutHeader=65591 uncompressedSizeWithoutHeader=65571
> prevBlockOffset=-1 isUseHBaseChecksum()=true checksumType=CRC32
> bytesPerChecksum=16384 onDiskDataSizeWithHeader=65604
> getOnDiskSizeWithHeader()=65624 totalChecksumBytes()=20 isUnpacked()=true
> buf=[ java.nio.HeapByteBuffer[pos=0 lim=65624 cap=65657],
> array().length=65657, arrayOffset()=0 ]
> dataBeginsWith=\x00\x00\x003\x00\x00\x00\x0A\x00\x10/0008:18\x01dprod
> fileContext=HFileContext [ usesHBaseChecksum=true checksumType=CRC32
> bytesPerChecksum=16384 blocksize=65536 encoding=NONE includesMvcc=true
> includesTags=false compressAlgo=NONE compressTags=false cryptoContext=[
> cipher=NONE keyHash=NONE ] ] ]
> {code}
> The data block encoding in file info was not consistent with the one in data
> block, which means there must be something wrong with the bulkload process.
> After debugging on each step of bulkload, I found that LoadIncrementalHFiles
> had a bug when loading hfile into a splitted region.
> {code}
> /**
>* Copy
[jira] [Commented] (HBASE-15087) Fix hbase-common findbugs complaints
[
https://issues.apache.org/jira/browse/HBASE-15087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15091591#comment-15091591
]
Anoop Sam John commented on HBASE-15087:
{code}
return
ByteBufferUtils.compareTo(((ByteBufferedCell)left).getFamilyByteBuffer(),
179 ((ByteBufferedCell)left).getFamilyPosition(),
left.getFamilyLength(),
180 right.getFamilyArray(), right.getFamilyOffset(),
right.getFamilyLength());
{code}
here right is a BBCell and left is normal Cell. So we should be calling
getXXXByteBuffer() method on right and getXXXArray() method on left and here it
is just reverse. Not correct for the performance reasons.
To avoid the negation, we should add method in BBUtil
compareTo(byte[] buf1, int o1, int l1, ByteBuffer buf2, int o2, int l2)
We already have compareTo(ByteBuffer buf1, int o1, int l1, byte[] buf2, int o2,
int l2)
> Fix hbase-common findbugs complaints
>
>
> Key: HBASE-15087
> URL: https://issues.apache.org/jira/browse/HBASE-15087
> Project: HBase
> Issue Type: Sub-task
> Components: build
>Reporter: stack
>Assignee: Stack
> Attachments: 15087.patch, 15087v2.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15083) Gets from Multiactions are not counted in metrics for gets.
[ https://issues.apache.org/jira/browse/HBASE-15083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heng Chen updated HBASE-15083: -- Attachment: HBASE-15083.patch > Gets from Multiactions are not counted in metrics for gets. > --- > > Key: HBASE-15083 > URL: https://issues.apache.org/jira/browse/HBASE-15083 > Project: HBase > Issue Type: Bug >Reporter: Elliott Clark > Fix For: 2.0.0 > > Attachments: HBASE-15083.patch, HBASE-15083.patch > > > RSRpcServices#get updates the get metrics. However Multiactions do not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15027) Refactor the way the CompactedHFileDischarger threads are created
[ https://issues.apache.org/jira/browse/HBASE-15027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15027: --- Attachment: HBASE-15027_findbug.patch Fixing the findbug warning introduced by this patch. > Refactor the way the CompactedHFileDischarger threads are created > - > > Key: HBASE-15027 > URL: https://issues.apache.org/jira/browse/HBASE-15027 > Project: HBase > Issue Type: Bug >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-15027.patch, HBASE-15027_1.patch, > HBASE-15027_2.patch, HBASE-15027_3.patch, HBASE-15027_3.patch, > HBASE-15027_4.patch, HBASE-15027_5.patch, HBASE-15027_6.patch, > HBASE-15027_7.patch, HBASE-15027_8.patch, HBASE-15027_findbug.patch > > > As per suggestion given over in HBASE-14970, if we need to create a single > thread pool service for the CompactionHFileDischarger we need to create an > exectuor service in the RegionServer level and create discharger handler > threads (Event handlers) and pass the Event to the new Exectuor service that > we create for the compaction hfiles discharger. What should be the default > number of threads here? If a HRS holds 100 of regions - will 10 threads be > enough? This issue will try to resolve this with tests and discussions and > suitable patch will be updated in HBASE-14970 for branch-1 once this is > committed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15087) Fix hbase-common findbugs complaints
[ https://issues.apache.org/jira/browse/HBASE-15087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15093195#comment-15093195 ] Anoop Sam John commented on HBASE-15087: +1 > Fix hbase-common findbugs complaints > > > Key: HBASE-15087 > URL: https://issues.apache.org/jira/browse/HBASE-15087 > Project: HBase > Issue Type: Sub-task > Components: build >Reporter: stack >Assignee: Stack > Attachments: 15087.patch, 15087v2.patch, 15087v3.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15085) IllegalStateException was thrown when scanning on bulkloaded HFiles
[
https://issues.apache.org/jira/browse/HBASE-15085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victor Xu updated HBASE-15085:
--
Attachment: HBASE-15085-v4.patch
HBASE-15085-0.98-v4.patch
Resubmit latest patch for all branches.
> IllegalStateException was thrown when scanning on bulkloaded HFiles
> ---
>
> Key: HBASE-15085
> URL: https://issues.apache.org/jira/browse/HBASE-15085
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.12, 1.1.2
> Environment: HBase-0.98.12 & Hadoop-2.6.0 & JDK1.7
> HBase-1.1.2 & Hadoop-2.6.0 & JDK1.7
>Reporter: Victor Xu
>Assignee: Victor Xu
> Labels: hfile
> Attachments: HBASE-15085-0.98-v1.patch, HBASE-15085-0.98-v2.patch,
> HBASE-15085-0.98-v3.patch, HBASE-15085-0.98-v4.patch,
> HBASE-15085-0.98-v4.patch, HBASE-15085-v1.patch, HBASE-15085-v2.patch,
> HBASE-15085-v3.patch, HBASE-15085-v4.patch, HBASE-15085-v4.patch,
> HBASE-15085-v4.patch
>
>
> IllegalStateException was thrown when we scanned from an HFile which was bulk
> loaded several minutes ago, as shown below:
> {code}
> 2015-12-16 22:20:54,456 ERROR
> com.taobao.kart.coprocessor.server.KartCoprocessor:
> icbu_ae_ws_product,/0055,1450275490479.6a6a700f465ad074287fed720c950f7c.
> batchNotify exception
> java.lang.IllegalStateException: EncodedScanner works only on encoded data
> blocks
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.updateCurrentBlock(HFileReaderV2.java:1042)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.seekTo(HFileReaderV2.java:1093)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:244)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:152)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:329)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:188)
> at
> org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:1879)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.(HRegion.java:4068)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2029)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2015)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1992)
> {code}
> I used 'hbase hfile' command to analyse the meta and block info of the hfile,
> finding that even through the DATA_BLOCK_ENCODING was 'DIFF' in FileInfo, the
> actual data blocks was written without any encoding algorithms(BlockType was
> 'DATA', not 'ENCODED_DATA'):
> {code}
> Fileinfo:
> BLOOM_FILTER_TYPE = ROW
> BULKLOAD_SOURCE_TASK = attempt_1442077249005_606706_r_12_0
> BULKLOAD_TIMESTAMP = \x00\x00\x01R\x12$\x13\x12
> DATA_BLOCK_ENCODING = DIFF
> ...
> DataBlock Header:
> HFileBlock [ fileOffset=0 headerSize()=33 blockType=DATA
> onDiskSizeWithoutHeader=65591 uncompressedSizeWithoutHeader=65571
> prevBlockOffset=-1 isUseHBaseChecksum()=true checksumType=CRC32
> bytesPerChecksum=16384 onDiskDataSizeWithHeader=65604
> getOnDiskSizeWithHeader()=65624 totalChecksumBytes()=20 isUnpacked()=true
> buf=[ java.nio.HeapByteBuffer[pos=0 lim=65624 cap=65657],
> array().length=65657, arrayOffset()=0 ]
> dataBeginsWith=\x00\x00\x003\x00\x00\x00\x0A\x00\x10/0008:18\x01dprod
> fileContext=HFileContext [ usesHBaseChecksum=true checksumType=CRC32
> bytesPerChecksum=16384 blocksize=65536 encoding=NONE includesMvcc=true
> includesTags=false compressAlgo=NONE compressTags=false cryptoContext=[
> cipher=NONE keyHash=NONE ] ] ]
> {code}
> The data block encoding in file info was not consistent with the one in data
> block, which means there must be something wrong with the bulkload process.
> After debugging on each step of bulkload, I found that LoadIncrementalHFiles
> had a bug when loading hfile into a splitted region.
> {code}
> /**
>* Copy half of an HFile into a new HFile.
>*/
> private static void copyHFileHalf(
> Configuration conf, Path inFile, Path outFile, Reference reference,
> HColumnDescriptor familyDescriptor)
> throws IOException {
> FileSystem fs = inFile.getFileSystem(conf);
> CacheConfig cacheConf = new CacheConfig(conf);
> HalfStoreFileReader halfReader = null;
> StoreFile.Writer halfWriter = null;
> try {
> halfReader = new HalfStoreFileReader(fs, inFile, cacheConf, reference,
> conf);
> Map fileInfo = halfReader.loadFileInfo();
> int blocksize = familyDescriptor.getBlocksize();
> Algorithm compression =
[jira] [Assigned] (HBASE-13590) TestEnableTableHandler.testEnableTableWithNoRegionServers is flakey
[ https://issues.apache.org/jira/browse/HBASE-13590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li reassigned HBASE-13590: - Assignee: Yu Li > TestEnableTableHandler.testEnableTableWithNoRegionServers is flakey > --- > > Key: HBASE-13590 > URL: https://issues.apache.org/jira/browse/HBASE-13590 > Project: HBase > Issue Type: Test > Components: master >Reporter: Nick Dimiduk >Assignee: Yu Li > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4 > > Attachments: HBASE-13590.branch-1.patch > > > Looking at our [build > history|https://builds.apache.org/job/HBase-1.1/buildTimeTrend], it seems > this test is flakey. See builds 429, 431, 439. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15089) Compatibility issue on HTable#flushCommits
[ https://issues.apache.org/jira/browse/HBASE-15089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li updated HBASE-15089: -- Status: Patch Available (was: Open) Submit patch for HadoopQA > Compatibility issue on HTable#flushCommits > -- > > Key: HBASE-15089 > URL: https://issues.apache.org/jira/browse/HBASE-15089 > Project: HBase > Issue Type: Bug >Affects Versions: 1.1.2 >Reporter: Yu Li >Assignee: Yu Li >Priority: Minor > Attachments: HBASE-15089.patch > > > Previously in 0.98 HTable#flushCommits throws InterruptedIOException and > RetriesExhaustedWithDetailsException, but now in 1.1.2 this method signature > has been changed to throw IOException, which will force application code > changes for exception handling (previous catch on InterruptedIOException and > RetriesExhaustedWithDetailsException become invalid). > In this JIRA we propose to recover the compatibility. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15077) Support OffheapKV write in compaction with out copying data on heap
[
https://issues.apache.org/jira/browse/HBASE-15077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15093478#comment-15093478
]
Hudson commented on HBASE-15077:
FAILURE: Integrated in HBase-Trunk_matrix #626 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/626/])
HBASE-15077 Support OffheapKV write in compaction with out copying data
(anoopsamjohn: rev da932ee38d29a13acb3e3ff6653402d4c8141d04)
*
hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileDataBlockEncoder.java
*
hbase-common/src/main/java/org/apache/hadoop/hbase/io/ByteBufferSupportOutputStream.java
*
hbase-common/src/main/java/org/apache/hadoop/hbase/io/ByteBufferSupportDataOutputStream.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/io/encoding/TestDataBlockEncoders.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/OffheapKeyValue.java
*
hbase-common/src/main/java/org/apache/hadoop/hbase/io/ByteArrayOutputStream.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/util/ByteBufferUtils.java
*
hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java
*
hbase-common/src/main/java/org/apache/hadoop/hbase/io/ByteBufferOutputStream.java
> Support OffheapKV write in compaction with out copying data on heap
> ---
>
> Key: HBASE-15077
> URL: https://issues.apache.org/jira/browse/HBASE-15077
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver, Scanners
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
> Attachments: HBASE-15077.patch, HBASE-15077_V2.patch,
> HBASE-15077_V2.patch, HBASE-15077_V3.patch
>
>
> HBASE-14832 is not enough to handle this. Doing the remaining needed here.
> {code}
> if (cell instanceof ByteBufferedCell) {
> 890 out.writeShort(rowLen);
> 891 ByteBufferUtils.copyBufferToStream(out, ((ByteBufferedCell)
> cell).getRowByteBuffer(),
> 892 ((ByteBufferedCell) cell).getRowPosition(), rowLen);
> 893 out.writeByte(fLen);
> 894 ByteBufferUtils.copyBufferToStream(out, ((ByteBufferedCell)
> cell).getFamilyByteBuffer(),
> 895 ((ByteBufferedCell) cell).getFamilyPosition(), fLen);
> 896 ByteBufferUtils.copyBufferToStream(out, ((ByteBufferedCell)
> cell).getQualifierByteBuffer(),
> 897 ((ByteBufferedCell) cell).getQualifierPosition(), qLen);
> {code}
> We have done this but it is not really helping us!
> In ByteBufferUtils#copyBufferToStream
> {code}
> public static void copyBufferToStream(OutputStream out, ByteBuffer in,
> int offset, int length) throws IOException {
> if (in.hasArray()) {
> out.write(in.array(), in.arrayOffset() + offset,
> length);
> } else {
> for (int i = 0; i < length; ++i) {
> out.write(toByte(in, offset + i));
> }
> }
> }
> {code}
> So for DBB it is so costly op writing byte by byte reading each to on heap.
> Even if we use writeByteBuffer(OutputStream out, ByteBuffer b, int offset,
> int length), it won't help us as the underlying stream is a
> ByteArrayOutputStream and so we will end up in copying.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[ https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15093125#comment-15093125 ] Anoop Sam John commented on HBASE-15055: bq.In the current code, it seems the skipLargeFiles() is possibly not called if 'forceMajor' is true. Yes. But this flag is on for explicit major compaction calls from user. I mean the system made major compaction after the configured interval. That check for whether this can be a major one is after this step of skipLargeFiles. Regarding ExploringCompactionPolicy, I too noticed and also saw this issue already been raised. See HBASE-14409. We need some cleanup. > Major compaction is not triggered when both of TTL and > hbase.hstore.compaction.max.size are set > --- > > Key: HBASE-15055 > URL: https://issues.apache.org/jira/browse/HBASE-15055 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Assignee: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15055-v1.patch, HBASE-15055-v2.patch, > HBASE-15055-v3.patch, HBASE-15055-v4.patch, HBASE-15055-v5.patch, > HBASE-15055-v6.patch, HBASE-15055-v7.patch, HBASE-15055-v8.patch, > HBASE-15055-v9.patch, HBASE-15055.patch > > > Some large files may be skipped by hbase.hstore.compaction.max.size in > candidate selection. It causes skipping of major compaction. So the TTL > expired records are still remained in the disks and keep consuming disks. > To resolve this issue, I suggest that to skip large files only if there is no > TTL expired record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[ https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15055: Status: Open (was: Patch Available) > Major compaction is not triggered when both of TTL and > hbase.hstore.compaction.max.size are set > --- > > Key: HBASE-15055 > URL: https://issues.apache.org/jira/browse/HBASE-15055 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Assignee: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15055-v1.patch, HBASE-15055-v2.patch, > HBASE-15055-v3.patch, HBASE-15055-v4.patch, HBASE-15055-v5.patch, > HBASE-15055-v6.patch, HBASE-15055-v7.patch, HBASE-15055-v8.patch, > HBASE-15055-v9.patch, HBASE-15055.patch > > > Some large files may be skipped by hbase.hstore.compaction.max.size in > candidate selection. It causes skipping of major compaction. So the TTL > expired records are still remained in the disks and keep consuming disks. > To resolve this issue, I suggest that to skip large files only if there is no > TTL expired record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15085) IllegalStateException was thrown when scanning on bulkloaded HFiles
[
https://issues.apache.org/jira/browse/HBASE-15085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15093289#comment-15093289
]
Hadoop QA commented on HBASE-15085:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 9m 49s
{color} | {color:red} Docker failed to build yetus/hbase:date2016-01-12.
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12781730/HBASE-15085-v4.patch |
| JIRA Issue | HBASE-15085 |
| Powered by | Apache Yetus 0.1.0 http://yetus.apache.org |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/71/console |
This message was automatically generated.
> IllegalStateException was thrown when scanning on bulkloaded HFiles
> ---
>
> Key: HBASE-15085
> URL: https://issues.apache.org/jira/browse/HBASE-15085
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.12, 1.1.2
> Environment: HBase-0.98.12 & Hadoop-2.6.0 & JDK1.7
> HBase-1.1.2 & Hadoop-2.6.0 & JDK1.7
>Reporter: Victor Xu
>Assignee: Victor Xu
> Labels: hfile
> Attachments: HBASE-15085-0.98-v1.patch, HBASE-15085-0.98-v2.patch,
> HBASE-15085-0.98-v3.patch, HBASE-15085-0.98-v4.patch, HBASE-15085-v1.patch,
> HBASE-15085-v2.patch, HBASE-15085-v3.patch, HBASE-15085-v4.patch,
> HBASE-15085-v4.patch
>
>
> IllegalStateException was thrown when we scanned from an HFile which was bulk
> loaded several minutes ago, as shown below:
> {code}
> 2015-12-16 22:20:54,456 ERROR
> com.taobao.kart.coprocessor.server.KartCoprocessor:
> icbu_ae_ws_product,/0055,1450275490479.6a6a700f465ad074287fed720c950f7c.
> batchNotify exception
> java.lang.IllegalStateException: EncodedScanner works only on encoded data
> blocks
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.updateCurrentBlock(HFileReaderV2.java:1042)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.seekTo(HFileReaderV2.java:1093)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:244)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:152)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:329)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:188)
> at
> org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:1879)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.(HRegion.java:4068)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2029)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2015)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1992)
> {code}
> I used 'hbase hfile' command to analyse the meta and block info of the hfile,
> finding that even through the DATA_BLOCK_ENCODING was 'DIFF' in FileInfo, the
> actual data blocks was written without any encoding algorithms(BlockType was
> 'DATA', not 'ENCODED_DATA'):
> {code}
> Fileinfo:
> BLOOM_FILTER_TYPE = ROW
> BULKLOAD_SOURCE_TASK = attempt_1442077249005_606706_r_12_0
> BULKLOAD_TIMESTAMP = \x00\x00\x01R\x12$\x13\x12
> DATA_BLOCK_ENCODING = DIFF
> ...
> DataBlock Header:
> HFileBlock [ fileOffset=0 headerSize()=33 blockType=DATA
> onDiskSizeWithoutHeader=65591 uncompressedSizeWithoutHeader=65571
> prevBlockOffset=-1 isUseHBaseChecksum()=true checksumType=CRC32
> bytesPerChecksum=16384 onDiskDataSizeWithHeader=65604
> getOnDiskSizeWithHeader()=65624 totalChecksumBytes()=20 isUnpacked()=true
> buf=[ java.nio.HeapByteBuffer[pos=0 lim=65624 cap=65657],
> array().length=65657, arrayOffset()=0 ]
> dataBeginsWith=\x00\x00\x003\x00\x00\x00\x0A\x00\x10/0008:18\x01dprod
> fileContext=HFileContext [ usesHBaseChecksum=true checksumType=CRC32
> bytesPerChecksum=16384 blocksize=65536 encoding=NONE includesMvcc=true
> includesTags=false compressAlgo=NONE compressTags=false cryptoContext=[
> cipher=NONE keyHash=NONE ] ] ]
> {code}
> The data block encoding in file info was not consistent with the one in data
> block, which means there must be something wrong with the bulkload process.
> After debugging on each step of bulkload, I found that LoadIncrementalHFiles
> had a bug when loading hfile into a splitted region.
> {code}
> /**
>* Copy half of an HFile into a new HFile.
>*/
> private static void copyHFileHalf(
> Configuration conf, Path inFile, Path outFile, Reference reference,
> HColumnDescriptor familyDescriptor)
> throws
[jira] [Updated] (HBASE-15085) IllegalStateException was thrown when scanning on bulkloaded HFiles
[
https://issues.apache.org/jira/browse/HBASE-15085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victor Xu updated HBASE-15085:
--
Attachment: HBASE-15085-branch-1.2-v1.patch
HBASE-15085-branch-1.1-v1.patch
HBASE-15085-branch-1.0-v1.patch
Attach patches for branch-1.0, 1.1, 1.2
> IllegalStateException was thrown when scanning on bulkloaded HFiles
> ---
>
> Key: HBASE-15085
> URL: https://issues.apache.org/jira/browse/HBASE-15085
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.12, 1.1.2
> Environment: HBase-0.98.12 & Hadoop-2.6.0 & JDK1.7
> HBase-1.1.2 & Hadoop-2.6.0 & JDK1.7
>Reporter: Victor Xu
>Assignee: Victor Xu
> Labels: hfile
> Attachments: HBASE-15085-0.98-v1.patch, HBASE-15085-0.98-v2.patch,
> HBASE-15085-0.98-v3.patch, HBASE-15085-0.98-v4.patch,
> HBASE-15085-0.98-v4.patch, HBASE-15085-branch-1.0-v1.patch,
> HBASE-15085-branch-1.1-v1.patch, HBASE-15085-branch-1.2-v1.patch,
> HBASE-15085-v1.patch, HBASE-15085-v2.patch, HBASE-15085-v3.patch,
> HBASE-15085-v4.patch, HBASE-15085-v4.patch, HBASE-15085-v4.patch
>
>
> IllegalStateException was thrown when we scanned from an HFile which was bulk
> loaded several minutes ago, as shown below:
> {code}
> 2015-12-16 22:20:54,456 ERROR
> com.taobao.kart.coprocessor.server.KartCoprocessor:
> icbu_ae_ws_product,/0055,1450275490479.6a6a700f465ad074287fed720c950f7c.
> batchNotify exception
> java.lang.IllegalStateException: EncodedScanner works only on encoded data
> blocks
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.updateCurrentBlock(HFileReaderV2.java:1042)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.seekTo(HFileReaderV2.java:1093)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:244)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:152)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:329)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:188)
> at
> org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:1879)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.(HRegion.java:4068)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2029)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2015)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1992)
> {code}
> I used 'hbase hfile' command to analyse the meta and block info of the hfile,
> finding that even through the DATA_BLOCK_ENCODING was 'DIFF' in FileInfo, the
> actual data blocks was written without any encoding algorithms(BlockType was
> 'DATA', not 'ENCODED_DATA'):
> {code}
> Fileinfo:
> BLOOM_FILTER_TYPE = ROW
> BULKLOAD_SOURCE_TASK = attempt_1442077249005_606706_r_12_0
> BULKLOAD_TIMESTAMP = \x00\x00\x01R\x12$\x13\x12
> DATA_BLOCK_ENCODING = DIFF
> ...
> DataBlock Header:
> HFileBlock [ fileOffset=0 headerSize()=33 blockType=DATA
> onDiskSizeWithoutHeader=65591 uncompressedSizeWithoutHeader=65571
> prevBlockOffset=-1 isUseHBaseChecksum()=true checksumType=CRC32
> bytesPerChecksum=16384 onDiskDataSizeWithHeader=65604
> getOnDiskSizeWithHeader()=65624 totalChecksumBytes()=20 isUnpacked()=true
> buf=[ java.nio.HeapByteBuffer[pos=0 lim=65624 cap=65657],
> array().length=65657, arrayOffset()=0 ]
> dataBeginsWith=\x00\x00\x003\x00\x00\x00\x0A\x00\x10/0008:18\x01dprod
> fileContext=HFileContext [ usesHBaseChecksum=true checksumType=CRC32
> bytesPerChecksum=16384 blocksize=65536 encoding=NONE includesMvcc=true
> includesTags=false compressAlgo=NONE compressTags=false cryptoContext=[
> cipher=NONE keyHash=NONE ] ] ]
> {code}
> The data block encoding in file info was not consistent with the one in data
> block, which means there must be something wrong with the bulkload process.
> After debugging on each step of bulkload, I found that LoadIncrementalHFiles
> had a bug when loading hfile into a splitted region.
> {code}
> /**
>* Copy half of an HFile into a new HFile.
>*/
> private static void copyHFileHalf(
> Configuration conf, Path inFile, Path outFile, Reference reference,
> HColumnDescriptor familyDescriptor)
> throws IOException {
> FileSystem fs = inFile.getFileSystem(conf);
> CacheConfig cacheConf = new CacheConfig(conf);
> HalfStoreFileReader halfReader = null;
> StoreFile.Writer halfWriter = null;
> try {
> halfReader = new HalfStoreFileReader(fs, inFile, cacheConf, reference,
> conf);
>
[jira] [Commented] (HBASE-15083) Gets from Multiactions are not counted in metrics for gets.
[ https://issues.apache.org/jira/browse/HBASE-15083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15093389#comment-15093389 ] Heng Chen commented on HBASE-15083: --- OK. Let me backport to branch-1+ > Gets from Multiactions are not counted in metrics for gets. > --- > > Key: HBASE-15083 > URL: https://issues.apache.org/jira/browse/HBASE-15083 > Project: HBase > Issue Type: Bug >Reporter: Elliott Clark >Assignee: Heng Chen > Fix For: 2.0.0 > > Attachments: HBASE-15083.patch, HBASE-15083.patch > > > RSRpcServices#get updates the get metrics. However Multiactions do not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14376) move hbase spark integration examples into their own module
[ https://issues.apache.org/jira/browse/HBASE-14376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15093277#comment-15093277 ] Appy commented on HBASE-14376: -- I believe it will be easier for users if they can find all examples in one place. So lets keep them in hbase-examples. > move hbase spark integration examples into their own module > --- > > Key: HBASE-14376 > URL: https://issues.apache.org/jira/browse/HBASE-14376 > Project: HBase > Issue Type: Task > Components: spark >Reporter: Sean Busbey >Assignee: Gabor Liptak > Labels: beginner > Attachments: HBASE-14376.1.patch, warnings.diff > > > take the examples that are currently in the hbase-spark module and move them > into a hbase-spark-examples module. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15085) IllegalStateException was thrown when scanning on bulkloaded HFiles
[
https://issues.apache.org/jira/browse/HBASE-15085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15093338#comment-15093338
]
Hadoop QA commented on HBASE-15085:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 3m 35s
{color} | {color:red} Docker failed to build yetus/hbase:date2016-01-12.
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12781742/HBASE-15085-branch-1.2-v1.patch
|
| JIRA Issue | HBASE-15085 |
| Powered by | Apache Yetus 0.1.0 http://yetus.apache.org |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/73/console |
This message was automatically generated.
> IllegalStateException was thrown when scanning on bulkloaded HFiles
> ---
>
> Key: HBASE-15085
> URL: https://issues.apache.org/jira/browse/HBASE-15085
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.12, 1.1.2
> Environment: HBase-0.98.12 & Hadoop-2.6.0 & JDK1.7
> HBase-1.1.2 & Hadoop-2.6.0 & JDK1.7
>Reporter: Victor Xu
>Assignee: Victor Xu
> Labels: hfile
> Attachments: HBASE-15085-0.98-v1.patch, HBASE-15085-0.98-v2.patch,
> HBASE-15085-0.98-v3.patch, HBASE-15085-0.98-v4.patch,
> HBASE-15085-0.98-v4.patch, HBASE-15085-branch-1.0-v1.patch,
> HBASE-15085-branch-1.1-v1.patch, HBASE-15085-branch-1.2-v1.patch,
> HBASE-15085-v1.patch, HBASE-15085-v2.patch, HBASE-15085-v3.patch,
> HBASE-15085-v4.patch, HBASE-15085-v4.patch, HBASE-15085-v4.patch
>
>
> IllegalStateException was thrown when we scanned from an HFile which was bulk
> loaded several minutes ago, as shown below:
> {code}
> 2015-12-16 22:20:54,456 ERROR
> com.taobao.kart.coprocessor.server.KartCoprocessor:
> icbu_ae_ws_product,/0055,1450275490479.6a6a700f465ad074287fed720c950f7c.
> batchNotify exception
> java.lang.IllegalStateException: EncodedScanner works only on encoded data
> blocks
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.updateCurrentBlock(HFileReaderV2.java:1042)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.seekTo(HFileReaderV2.java:1093)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:244)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:152)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:329)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:188)
> at
> org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:1879)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.(HRegion.java:4068)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2029)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2015)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1992)
> {code}
> I used 'hbase hfile' command to analyse the meta and block info of the hfile,
> finding that even through the DATA_BLOCK_ENCODING was 'DIFF' in FileInfo, the
> actual data blocks was written without any encoding algorithms(BlockType was
> 'DATA', not 'ENCODED_DATA'):
> {code}
> Fileinfo:
> BLOOM_FILTER_TYPE = ROW
> BULKLOAD_SOURCE_TASK = attempt_1442077249005_606706_r_12_0
> BULKLOAD_TIMESTAMP = \x00\x00\x01R\x12$\x13\x12
> DATA_BLOCK_ENCODING = DIFF
> ...
> DataBlock Header:
> HFileBlock [ fileOffset=0 headerSize()=33 blockType=DATA
> onDiskSizeWithoutHeader=65591 uncompressedSizeWithoutHeader=65571
> prevBlockOffset=-1 isUseHBaseChecksum()=true checksumType=CRC32
> bytesPerChecksum=16384 onDiskDataSizeWithHeader=65604
> getOnDiskSizeWithHeader()=65624 totalChecksumBytes()=20 isUnpacked()=true
> buf=[ java.nio.HeapByteBuffer[pos=0 lim=65624 cap=65657],
> array().length=65657, arrayOffset()=0 ]
> dataBeginsWith=\x00\x00\x003\x00\x00\x00\x0A\x00\x10/0008:18\x01dprod
> fileContext=HFileContext [ usesHBaseChecksum=true checksumType=CRC32
> bytesPerChecksum=16384 blocksize=65536 encoding=NONE includesMvcc=true
> includesTags=false compressAlgo=NONE compressTags=false cryptoContext=[
> cipher=NONE keyHash=NONE ] ] ]
> {code}
> The data block encoding in file info was not consistent with the one in data
> block, which means there must be something wrong with the bulkload process.
> After debugging on each step of bulkload, I found that LoadIncrementalHFiles
> had a bug when loading hfile into a splitted region.
> {code}
> /**
>* Copy half of an HFile into a new HFile.
>*/
> private
[jira] [Commented] (HBASE-15085) IllegalStateException was thrown when scanning on bulkloaded HFiles
[
https://issues.apache.org/jira/browse/HBASE-15085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15093349#comment-15093349
]
Victor Xu commented on HBASE-15085:
---
[~ram_krish]
> IllegalStateException was thrown when scanning on bulkloaded HFiles
> ---
>
> Key: HBASE-15085
> URL: https://issues.apache.org/jira/browse/HBASE-15085
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.12, 1.1.2
> Environment: HBase-0.98.12 & Hadoop-2.6.0 & JDK1.7
> HBase-1.1.2 & Hadoop-2.6.0 & JDK1.7
>Reporter: Victor Xu
>Assignee: Victor Xu
> Labels: hfile
> Attachments: HBASE-15085-0.98-v1.patch, HBASE-15085-0.98-v2.patch,
> HBASE-15085-0.98-v3.patch, HBASE-15085-0.98-v4.patch,
> HBASE-15085-0.98-v4.patch, HBASE-15085-branch-1.0-v1.patch,
> HBASE-15085-branch-1.1-v1.patch, HBASE-15085-branch-1.2-v1.patch,
> HBASE-15085-v1.patch, HBASE-15085-v2.patch, HBASE-15085-v3.patch,
> HBASE-15085-v4.patch, HBASE-15085-v4.patch, HBASE-15085-v4.patch
>
>
> IllegalStateException was thrown when we scanned from an HFile which was bulk
> loaded several minutes ago, as shown below:
> {code}
> 2015-12-16 22:20:54,456 ERROR
> com.taobao.kart.coprocessor.server.KartCoprocessor:
> icbu_ae_ws_product,/0055,1450275490479.6a6a700f465ad074287fed720c950f7c.
> batchNotify exception
> java.lang.IllegalStateException: EncodedScanner works only on encoded data
> blocks
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.updateCurrentBlock(HFileReaderV2.java:1042)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.seekTo(HFileReaderV2.java:1093)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:244)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:152)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:329)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:188)
> at
> org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:1879)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.(HRegion.java:4068)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2029)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2015)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1992)
> {code}
> I used 'hbase hfile' command to analyse the meta and block info of the hfile,
> finding that even through the DATA_BLOCK_ENCODING was 'DIFF' in FileInfo, the
> actual data blocks was written without any encoding algorithms(BlockType was
> 'DATA', not 'ENCODED_DATA'):
> {code}
> Fileinfo:
> BLOOM_FILTER_TYPE = ROW
> BULKLOAD_SOURCE_TASK = attempt_1442077249005_606706_r_12_0
> BULKLOAD_TIMESTAMP = \x00\x00\x01R\x12$\x13\x12
> DATA_BLOCK_ENCODING = DIFF
> ...
> DataBlock Header:
> HFileBlock [ fileOffset=0 headerSize()=33 blockType=DATA
> onDiskSizeWithoutHeader=65591 uncompressedSizeWithoutHeader=65571
> prevBlockOffset=-1 isUseHBaseChecksum()=true checksumType=CRC32
> bytesPerChecksum=16384 onDiskDataSizeWithHeader=65604
> getOnDiskSizeWithHeader()=65624 totalChecksumBytes()=20 isUnpacked()=true
> buf=[ java.nio.HeapByteBuffer[pos=0 lim=65624 cap=65657],
> array().length=65657, arrayOffset()=0 ]
> dataBeginsWith=\x00\x00\x003\x00\x00\x00\x0A\x00\x10/0008:18\x01dprod
> fileContext=HFileContext [ usesHBaseChecksum=true checksumType=CRC32
> bytesPerChecksum=16384 blocksize=65536 encoding=NONE includesMvcc=true
> includesTags=false compressAlgo=NONE compressTags=false cryptoContext=[
> cipher=NONE keyHash=NONE ] ] ]
> {code}
> The data block encoding in file info was not consistent with the one in data
> block, which means there must be something wrong with the bulkload process.
> After debugging on each step of bulkload, I found that LoadIncrementalHFiles
> had a bug when loading hfile into a splitted region.
> {code}
> /**
>* Copy half of an HFile into a new HFile.
>*/
> private static void copyHFileHalf(
> Configuration conf, Path inFile, Path outFile, Reference reference,
> HColumnDescriptor familyDescriptor)
> throws IOException {
> FileSystem fs = inFile.getFileSystem(conf);
> CacheConfig cacheConf = new CacheConfig(conf);
> HalfStoreFileReader halfReader = null;
> StoreFile.Writer halfWriter = null;
> try {
> halfReader = new HalfStoreFileReader(fs, inFile, cacheConf, reference,
> conf);
> Map fileInfo = halfReader.loadFileInfo();
> int blocksize = familyDescriptor.getBlocksize();
> Algorithm
[jira] [Commented] (HBASE-15085) IllegalStateException was thrown when scanning on bulkloaded HFiles
[
https://issues.apache.org/jira/browse/HBASE-15085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15093367#comment-15093367
]
ramkrishna.s.vasudevan commented on HBASE-15085:
[~victorunique], not sure what is the problem here. Let us check with
[~busbey]? Do you know why this failed?
> IllegalStateException was thrown when scanning on bulkloaded HFiles
> ---
>
> Key: HBASE-15085
> URL: https://issues.apache.org/jira/browse/HBASE-15085
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.12, 1.1.2
> Environment: HBase-0.98.12 & Hadoop-2.6.0 & JDK1.7
> HBase-1.1.2 & Hadoop-2.6.0 & JDK1.7
>Reporter: Victor Xu
>Assignee: Victor Xu
> Labels: hfile
> Attachments: HBASE-15085-0.98-v1.patch, HBASE-15085-0.98-v2.patch,
> HBASE-15085-0.98-v3.patch, HBASE-15085-0.98-v4.patch,
> HBASE-15085-0.98-v4.patch, HBASE-15085-branch-1.0-v1.patch,
> HBASE-15085-branch-1.1-v1.patch, HBASE-15085-branch-1.2-v1.patch,
> HBASE-15085-v1.patch, HBASE-15085-v2.patch, HBASE-15085-v3.patch,
> HBASE-15085-v4.patch, HBASE-15085-v4.patch, HBASE-15085-v4.patch
>
>
> IllegalStateException was thrown when we scanned from an HFile which was bulk
> loaded several minutes ago, as shown below:
> {code}
> 2015-12-16 22:20:54,456 ERROR
> com.taobao.kart.coprocessor.server.KartCoprocessor:
> icbu_ae_ws_product,/0055,1450275490479.6a6a700f465ad074287fed720c950f7c.
> batchNotify exception
> java.lang.IllegalStateException: EncodedScanner works only on encoded data
> blocks
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.updateCurrentBlock(HFileReaderV2.java:1042)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.seekTo(HFileReaderV2.java:1093)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:244)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:152)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:329)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:188)
> at
> org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:1879)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.(HRegion.java:4068)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2029)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2015)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1992)
> {code}
> I used 'hbase hfile' command to analyse the meta and block info of the hfile,
> finding that even through the DATA_BLOCK_ENCODING was 'DIFF' in FileInfo, the
> actual data blocks was written without any encoding algorithms(BlockType was
> 'DATA', not 'ENCODED_DATA'):
> {code}
> Fileinfo:
> BLOOM_FILTER_TYPE = ROW
> BULKLOAD_SOURCE_TASK = attempt_1442077249005_606706_r_12_0
> BULKLOAD_TIMESTAMP = \x00\x00\x01R\x12$\x13\x12
> DATA_BLOCK_ENCODING = DIFF
> ...
> DataBlock Header:
> HFileBlock [ fileOffset=0 headerSize()=33 blockType=DATA
> onDiskSizeWithoutHeader=65591 uncompressedSizeWithoutHeader=65571
> prevBlockOffset=-1 isUseHBaseChecksum()=true checksumType=CRC32
> bytesPerChecksum=16384 onDiskDataSizeWithHeader=65604
> getOnDiskSizeWithHeader()=65624 totalChecksumBytes()=20 isUnpacked()=true
> buf=[ java.nio.HeapByteBuffer[pos=0 lim=65624 cap=65657],
> array().length=65657, arrayOffset()=0 ]
> dataBeginsWith=\x00\x00\x003\x00\x00\x00\x0A\x00\x10/0008:18\x01dprod
> fileContext=HFileContext [ usesHBaseChecksum=true checksumType=CRC32
> bytesPerChecksum=16384 blocksize=65536 encoding=NONE includesMvcc=true
> includesTags=false compressAlgo=NONE compressTags=false cryptoContext=[
> cipher=NONE keyHash=NONE ] ] ]
> {code}
> The data block encoding in file info was not consistent with the one in data
> block, which means there must be something wrong with the bulkload process.
> After debugging on each step of bulkload, I found that LoadIncrementalHFiles
> had a bug when loading hfile into a splitted region.
> {code}
> /**
>* Copy half of an HFile into a new HFile.
>*/
> private static void copyHFileHalf(
> Configuration conf, Path inFile, Path outFile, Reference reference,
> HColumnDescriptor familyDescriptor)
> throws IOException {
> FileSystem fs = inFile.getFileSystem(conf);
> CacheConfig cacheConf = new CacheConfig(conf);
> HalfStoreFileReader halfReader = null;
> StoreFile.Writer halfWriter = null;
> try {
> halfReader = new HalfStoreFileReader(fs, inFile, cacheConf, reference,
> conf);
> Map
[jira] [Updated] (HBASE-15083) Gets from Multiactions are not counted in metrics for gets.
[ https://issues.apache.org/jira/browse/HBASE-15083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heng Chen updated HBASE-15083: -- Resolution: Fixed Assignee: Heng Chen Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) > Gets from Multiactions are not counted in metrics for gets. > --- > > Key: HBASE-15083 > URL: https://issues.apache.org/jira/browse/HBASE-15083 > Project: HBase > Issue Type: Bug >Reporter: Elliott Clark >Assignee: Heng Chen > Fix For: 2.0.0 > > Attachments: HBASE-15083.patch, HBASE-15083.patch > > > RSRpcServices#get updates the get metrics. However Multiactions do not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15083) Gets from Multiactions are not counted in metrics for gets.
[ https://issues.apache.org/jira/browse/HBASE-15083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heng Chen updated HBASE-15083: -- Fix Version/s: 1.0.0 1.1.0 1.3.0 1.2.0 > Gets from Multiactions are not counted in metrics for gets. > --- > > Key: HBASE-15083 > URL: https://issues.apache.org/jira/browse/HBASE-15083 > Project: HBase > Issue Type: Bug >Reporter: Elliott Clark >Assignee: Heng Chen > Fix For: 1.0.0, 2.0.0, 1.1.0, 1.2.0, 1.3.0 > > Attachments: HBASE-15083.patch, HBASE-15083.patch > > > RSRpcServices#get updates the get metrics. However Multiactions do not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13590) TestEnableTableHandler.testEnableTableWithNoRegionServers is flakey
[ https://issues.apache.org/jira/browse/HBASE-13590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li updated HBASE-13590: -- Status: Patch Available (was: Open) Submit the patch to see what HadoopQA will say > TestEnableTableHandler.testEnableTableWithNoRegionServers is flakey > --- > > Key: HBASE-13590 > URL: https://issues.apache.org/jira/browse/HBASE-13590 > Project: HBase > Issue Type: Test > Components: master >Reporter: Nick Dimiduk >Assignee: Yu Li > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4 > > Attachments: HBASE-13590.branch-1.patch > > > Looking at our [build > history|https://builds.apache.org/job/HBase-1.1/buildTimeTrend], it seems > this test is flakey. See builds 429, 431, 439. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15089) Compatibility issue on HTable#flushCommits
[ https://issues.apache.org/jira/browse/HBASE-15089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li updated HBASE-15089: -- Attachment: HBASE-15089.patch Attaching a straight-forward patch. After the change, new codes which catch IOException will still works, while old 0.98 codes which catch InterruptedIOException and RetriesExhaustedWithDetailsException separately will also work. > Compatibility issue on HTable#flushCommits > -- > > Key: HBASE-15089 > URL: https://issues.apache.org/jira/browse/HBASE-15089 > Project: HBase > Issue Type: Bug >Affects Versions: 1.1.2 >Reporter: Yu Li >Assignee: Yu Li >Priority: Minor > Attachments: HBASE-15089.patch > > > Previously in 0.98 HTable#flushCommits throws InterruptedIOException and > RetriesExhaustedWithDetailsException, but now in 1.1.2 this method signature > has been changed to throw IOException, which will force application code > changes for exception handling (previous catch on InterruptedIOException and > RetriesExhaustedWithDetailsException become invalid). > In this JIRA we propose to recover the compatibility. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15075) Allow region split request to carry identification information
[ https://issues.apache.org/jira/browse/HBASE-15075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-15075: --- Summary: Allow region split request to carry identification information (was: Allow region split request to carry metadata) > Allow region split request to carry identification information > -- > > Key: HBASE-15075 > URL: https://issues.apache.org/jira/browse/HBASE-15075 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 15075-v0.txt, 15075-v1.txt, 15075-v2.txt, > HBASE-15075.v2.patch, HBASE-15075.v3.patch > > > During the process of improving region normalization feature, I found that if > region split request triggered by the execution of SplitNormalizationPlan > fails, there is no way of knowing whether the failed split originated from > region normalization. > The association of particular split request with outcome of split would give > RegionNormalizer information so that it can make better normalization > decisions in the subsequent invocations. > One approach is to embed metadata, such as a UUID, in SplitRequest which gets > passed through RegionStateTransitionContext when > RegionServerServices#reportRegionStateTransition() is called. > This way, RegionStateListener can be notified with the metadata (id of the > requester). > See discussion on dev mailing list > http://search-hadoop.com/m/YGbbCXdkivihp2 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14962) TestSplitWalDataLoss fails on all branches
[
https://issues.apache.org/jira/browse/HBASE-14962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15093163#comment-15093163
]
Elliott Clark commented on HBASE-14962:
---
{code}
T E S T S
---
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m;
support was removed in 8.0
Running org.apache.hadoop.hbase.regionserver.TestSplitWalDataLoss
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 12.483 sec <<<
FAILURE! - in org.apache.hadoop.hbase.regionserver.TestSplitWalDataLoss
test(org.apache.hadoop.hbase.regionserver.TestSplitWalDataLoss) Time elapsed:
12.058 sec <<< ERROR!
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1
action: TestSplitWalDataLoss:dataloss: 1 time,
at
org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:246)
at
org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1800(AsyncProcess.java:226)
at
org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1753)
at
org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:240)
at
org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:190)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1430)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1021)
at
org.apache.hadoop.hbase.regionserver.TestSplitWalDataLoss.test(TestSplitWalDataLoss.java:124)
Results :
Tests in error:
TestSplitWalDataLoss.test:124 » RetriesExhaustedWithDetails Failed 1 action:
T...
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
[INFO]
[INFO] Reactor Summary:
[INFO]
[INFO] Apache HBase ... SUCCESS [ 3.042 s]
[INFO] Apache HBase - Checkstyle .. SUCCESS [ 0.547 s]
[INFO] Apache HBase - Resource Bundle . SUCCESS [ 0.111 s]
[INFO] Apache HBase - Annotations . SUCCESS [ 0.792 s]
[INFO] Apache HBase - Protocol SUCCESS [ 9.352 s]
[INFO] Apache HBase - Common .. SUCCESS [ 6.884 s]
[INFO] Apache HBase - Procedure ... SUCCESS [ 1.510 s]
[INFO] Apache HBase - Client .. SUCCESS [ 4.878 s]
[INFO] Apache HBase - Hadoop Compatibility SUCCESS [ 0.249 s]
[INFO] Apache HBase - Hadoop Two Compatibility SUCCESS [ 2.312 s]
[INFO] Apache HBase - Prefix Tree . SUCCESS [ 2.010 s]
[INFO] Apache HBase - Server .. FAILURE [ 36.195 s]
[INFO] Apache HBase - Testing Util SKIPPED
[INFO] Apache HBase - Thrift .. SKIPPED
[INFO] Apache HBase - Rest SKIPPED
[INFO] Apache HBase - Shell ... SKIPPED
[INFO] Apache HBase - Integration Tests ... SKIPPED
[INFO] Apache HBase - Examples SKIPPED
[INFO] Apache HBase - External Block Cache SKIPPED
[INFO] Apache HBase - Assembly SKIPPED
[INFO] Apache HBase - Shaded .. SKIPPED
[INFO] Apache HBase - Shaded - Client . SKIPPED
[INFO] Apache HBase - Shaded - Server . SKIPPED
[INFO]
[INFO] BUILD FAILURE
[INFO]
[INFO] Total time: 01:09 min
[INFO] Finished at: 2016-01-11T18:17:54-08:00
[INFO] Final Memory: 108M/2061M
[INFO]
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-surefire-plugin:2.18.1:test (default-test) on
project hbase-server: There are test failures.
[ERROR]
[ERROR] Please refer to
/data/users/elliott/hbase/hbase-server/target/surefire-reports for the
individual test results.
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :hbase-server
TestSplitWalDataLoss failed, iteration: 2
{code}
> TestSplitWalDataLoss fails on all branches
> --
>
> Key: HBASE-14962
>
[jira] [Updated] (HBASE-15077) Support OffheapKV write in compaction with out copying data on heap
[
https://issues.apache.org/jira/browse/HBASE-15077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anoop Sam John updated HBASE-15077:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
> Support OffheapKV write in compaction with out copying data on heap
> ---
>
> Key: HBASE-15077
> URL: https://issues.apache.org/jira/browse/HBASE-15077
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver, Scanners
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
> Attachments: HBASE-15077.patch, HBASE-15077_V2.patch,
> HBASE-15077_V2.patch, HBASE-15077_V3.patch
>
>
> HBASE-14832 is not enough to handle this. Doing the remaining needed here.
> {code}
> if (cell instanceof ByteBufferedCell) {
> 890 out.writeShort(rowLen);
> 891 ByteBufferUtils.copyBufferToStream(out, ((ByteBufferedCell)
> cell).getRowByteBuffer(),
> 892 ((ByteBufferedCell) cell).getRowPosition(), rowLen);
> 893 out.writeByte(fLen);
> 894 ByteBufferUtils.copyBufferToStream(out, ((ByteBufferedCell)
> cell).getFamilyByteBuffer(),
> 895 ((ByteBufferedCell) cell).getFamilyPosition(), fLen);
> 896 ByteBufferUtils.copyBufferToStream(out, ((ByteBufferedCell)
> cell).getQualifierByteBuffer(),
> 897 ((ByteBufferedCell) cell).getQualifierPosition(), qLen);
> {code}
> We have done this but it is not really helping us!
> In ByteBufferUtils#copyBufferToStream
> {code}
> public static void copyBufferToStream(OutputStream out, ByteBuffer in,
> int offset, int length) throws IOException {
> if (in.hasArray()) {
> out.write(in.array(), in.arrayOffset() + offset,
> length);
> } else {
> for (int i = 0; i < length; ++i) {
> out.write(toByte(in, offset + i));
> }
> }
> }
> {code}
> So for DBB it is so costly op writing byte by byte reading each to on heap.
> Even if we use writeByteBuffer(OutputStream out, ByteBuffer b, int offset,
> int length), it won't help us as the underlying stream is a
> ByteArrayOutputStream and so we will end up in copying.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15085) IllegalStateException was thrown when scanning on bulkloaded HFiles
[
https://issues.apache.org/jira/browse/HBASE-15085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15093348#comment-15093348
]
Victor Xu commented on HBASE-15085:
---
[~ramkrishna.s.vasudevan]
> IllegalStateException was thrown when scanning on bulkloaded HFiles
> ---
>
> Key: HBASE-15085
> URL: https://issues.apache.org/jira/browse/HBASE-15085
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.12, 1.1.2
> Environment: HBase-0.98.12 & Hadoop-2.6.0 & JDK1.7
> HBase-1.1.2 & Hadoop-2.6.0 & JDK1.7
>Reporter: Victor Xu
>Assignee: Victor Xu
> Labels: hfile
> Attachments: HBASE-15085-0.98-v1.patch, HBASE-15085-0.98-v2.patch,
> HBASE-15085-0.98-v3.patch, HBASE-15085-0.98-v4.patch,
> HBASE-15085-0.98-v4.patch, HBASE-15085-branch-1.0-v1.patch,
> HBASE-15085-branch-1.1-v1.patch, HBASE-15085-branch-1.2-v1.patch,
> HBASE-15085-v1.patch, HBASE-15085-v2.patch, HBASE-15085-v3.patch,
> HBASE-15085-v4.patch, HBASE-15085-v4.patch, HBASE-15085-v4.patch
>
>
> IllegalStateException was thrown when we scanned from an HFile which was bulk
> loaded several minutes ago, as shown below:
> {code}
> 2015-12-16 22:20:54,456 ERROR
> com.taobao.kart.coprocessor.server.KartCoprocessor:
> icbu_ae_ws_product,/0055,1450275490479.6a6a700f465ad074287fed720c950f7c.
> batchNotify exception
> java.lang.IllegalStateException: EncodedScanner works only on encoded data
> blocks
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.updateCurrentBlock(HFileReaderV2.java:1042)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.seekTo(HFileReaderV2.java:1093)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:244)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:152)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:329)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:188)
> at
> org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:1879)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.(HRegion.java:4068)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2029)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2015)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1992)
> {code}
> I used 'hbase hfile' command to analyse the meta and block info of the hfile,
> finding that even through the DATA_BLOCK_ENCODING was 'DIFF' in FileInfo, the
> actual data blocks was written without any encoding algorithms(BlockType was
> 'DATA', not 'ENCODED_DATA'):
> {code}
> Fileinfo:
> BLOOM_FILTER_TYPE = ROW
> BULKLOAD_SOURCE_TASK = attempt_1442077249005_606706_r_12_0
> BULKLOAD_TIMESTAMP = \x00\x00\x01R\x12$\x13\x12
> DATA_BLOCK_ENCODING = DIFF
> ...
> DataBlock Header:
> HFileBlock [ fileOffset=0 headerSize()=33 blockType=DATA
> onDiskSizeWithoutHeader=65591 uncompressedSizeWithoutHeader=65571
> prevBlockOffset=-1 isUseHBaseChecksum()=true checksumType=CRC32
> bytesPerChecksum=16384 onDiskDataSizeWithHeader=65604
> getOnDiskSizeWithHeader()=65624 totalChecksumBytes()=20 isUnpacked()=true
> buf=[ java.nio.HeapByteBuffer[pos=0 lim=65624 cap=65657],
> array().length=65657, arrayOffset()=0 ]
> dataBeginsWith=\x00\x00\x003\x00\x00\x00\x0A\x00\x10/0008:18\x01dprod
> fileContext=HFileContext [ usesHBaseChecksum=true checksumType=CRC32
> bytesPerChecksum=16384 blocksize=65536 encoding=NONE includesMvcc=true
> includesTags=false compressAlgo=NONE compressTags=false cryptoContext=[
> cipher=NONE keyHash=NONE ] ] ]
> {code}
> The data block encoding in file info was not consistent with the one in data
> block, which means there must be something wrong with the bulkload process.
> After debugging on each step of bulkload, I found that LoadIncrementalHFiles
> had a bug when loading hfile into a splitted region.
> {code}
> /**
>* Copy half of an HFile into a new HFile.
>*/
> private static void copyHFileHalf(
> Configuration conf, Path inFile, Path outFile, Reference reference,
> HColumnDescriptor familyDescriptor)
> throws IOException {
> FileSystem fs = inFile.getFileSystem(conf);
> CacheConfig cacheConf = new CacheConfig(conf);
> HalfStoreFileReader halfReader = null;
> StoreFile.Writer halfWriter = null;
> try {
> halfReader = new HalfStoreFileReader(fs, inFile, cacheConf, reference,
> conf);
> Map fileInfo = halfReader.loadFileInfo();
> int blocksize = familyDescriptor.getBlocksize();
>
[jira] [Commented] (HBASE-14161) Add hbase-spark integration tests to IT jenkins job
[ https://issues.apache.org/jira/browse/HBASE-14161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15093427#comment-15093427 ] Appy commented on HBASE-14161: -- The test suites under hbase-spark/src/test/scala/org/apache/hadoop/hbase/spark/... are the ones which are being referred here? They collectively run in <100 sec, should we consider adding them as large tests? Or is it because they use minicluster that we want them as ITs? [~busbey] > Add hbase-spark integration tests to IT jenkins job > --- > > Key: HBASE-14161 > URL: https://issues.apache.org/jira/browse/HBASE-14161 > Project: HBase > Issue Type: Task > Components: build >Reporter: Sean Busbey >Assignee: Sean Busbey > Fix For: 2.0.0 > > > expand the set of ITs we run to include the new hbase-spark tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15087) Fix hbase-common findbugs complaints
[
https://issues.apache.org/jira/browse/HBASE-15087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15093132#comment-15093132
]
Hadoop QA commented on HBASE-15087:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
1s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
26s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s
{color} | {color:green} master passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 5m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 43s
{color} | {color:red} hbase-common in master has 14 extant Findbugs warnings.
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 51s
{color} | {color:red} hbase-server in master has 84 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 38s
{color} | {color:green} master passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 49s
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
1s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 5m 5s {color}
| {color:red} hbase-common-jdk1.8.0_66 with JDK v1.8.0_66 generated 19 new
issues (was 25, now 26). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 43s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 5m 55s {color}
| {color:red} hbase-common-jdk1.7.0_91 with JDK v1.7.0_91 generated 19 new
issues (was 25, now 26). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 5m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 1m 29s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.4.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 2m 58s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.4.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 4m 29s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.5.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 5m 59s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.5.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 7m 27s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.5.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 8m 57s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.6.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 10m 26s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.6.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 11m 56s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.6.3. {color} |
|
[jira] [Updated] (HBASE-14962) TestSplitWalDataLoss fails on all branches
[
https://issues.apache.org/jira/browse/HBASE-14962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elliott Clark updated HBASE-14962:
--
Attachment:
org.apache.hadoop.hbase.regionserver.TestSplitWalDataLoss-output.txt
Here's what I just saw on mac.
The paste above was on my linux machine.
So I'm still seeing it pretty consistently
> TestSplitWalDataLoss fails on all branches
> --
>
> Key: HBASE-14962
> URL: https://issues.apache.org/jira/browse/HBASE-14962
> Project: HBase
> Issue Type: Sub-task
>Reporter: Elliott Clark
>Assignee: stack
>Priority: Blocker
> Fix For: 1.2.0
>
> Attachments:
> org.apache.hadoop.hbase.regionserver.TestSplitWalDataLoss-output.txt
>
>
> With some regularity I am seeing:
> {code}
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1
> action: TestSplitWalDataLoss:dataloss: 1 time,
> at
> org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:228)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1800(AsyncProcess.java:208)
> at
> org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1712)
> at
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:240)
> at
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:190)
> at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1430)
> at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1021)
> at
> org.apache.hadoop.hbase.regionserver.TestSplitWalDataLoss.test(TestSplitWalDataLoss.java:121)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15077) Support OffheapKV write in compaction with out copying data on heap
[
https://issues.apache.org/jira/browse/HBASE-15077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15093271#comment-15093271
]
Anoop Sam John commented on HBASE-15077:
Test failures seems not related.
No new findbugs with this patch.. Other complaints of javac etc seems some
issue with calculation in the reporting.
Will commit now.. Thanks for the reviews.
> Support OffheapKV write in compaction with out copying data on heap
> ---
>
> Key: HBASE-15077
> URL: https://issues.apache.org/jira/browse/HBASE-15077
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver, Scanners
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
> Attachments: HBASE-15077.patch, HBASE-15077_V2.patch,
> HBASE-15077_V2.patch, HBASE-15077_V3.patch
>
>
> HBASE-14832 is not enough to handle this. Doing the remaining needed here.
> {code}
> if (cell instanceof ByteBufferedCell) {
> 890 out.writeShort(rowLen);
> 891 ByteBufferUtils.copyBufferToStream(out, ((ByteBufferedCell)
> cell).getRowByteBuffer(),
> 892 ((ByteBufferedCell) cell).getRowPosition(), rowLen);
> 893 out.writeByte(fLen);
> 894 ByteBufferUtils.copyBufferToStream(out, ((ByteBufferedCell)
> cell).getFamilyByteBuffer(),
> 895 ((ByteBufferedCell) cell).getFamilyPosition(), fLen);
> 896 ByteBufferUtils.copyBufferToStream(out, ((ByteBufferedCell)
> cell).getQualifierByteBuffer(),
> 897 ((ByteBufferedCell) cell).getQualifierPosition(), qLen);
> {code}
> We have done this but it is not really helping us!
> In ByteBufferUtils#copyBufferToStream
> {code}
> public static void copyBufferToStream(OutputStream out, ByteBuffer in,
> int offset, int length) throws IOException {
> if (in.hasArray()) {
> out.write(in.array(), in.arrayOffset() + offset,
> length);
> } else {
> for (int i = 0; i < length; ++i) {
> out.write(toByte(in, offset + i));
> }
> }
> }
> {code}
> So for DBB it is so costly op writing byte by byte reading each to on heap.
> Even if we use writeByteBuffer(OutputStream out, ByteBuffer b, int offset,
> int length), it won't help us as the underlying stream is a
> ByteArrayOutputStream and so we will end up in copying.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14376) move hbase spark integration examples into their own module
[ https://issues.apache.org/jira/browse/HBASE-14376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-14376: - Status: Patch Available (was: Open) resubmitting patch. > move hbase spark integration examples into their own module > --- > > Key: HBASE-14376 > URL: https://issues.apache.org/jira/browse/HBASE-14376 > Project: HBase > Issue Type: Task > Components: spark >Reporter: Sean Busbey >Assignee: Gabor Liptak > Labels: beginner > Attachments: HBASE-14376.1.patch, warnings.diff > > > take the examples that are currently in the hbase-spark module and move them > into a hbase-spark-examples module. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14376) move hbase spark integration examples into their own module
[ https://issues.apache.org/jira/browse/HBASE-14376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-14376: - Status: Open (was: Patch Available) > move hbase spark integration examples into their own module > --- > > Key: HBASE-14376 > URL: https://issues.apache.org/jira/browse/HBASE-14376 > Project: HBase > Issue Type: Task > Components: spark >Reporter: Sean Busbey >Assignee: Gabor Liptak > Labels: beginner > Attachments: HBASE-14376.1.patch, warnings.diff > > > take the examples that are currently in the hbase-spark module and move them > into a hbase-spark-examples module. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14962) TestSplitWalDataLoss fails on all branches
[
https://issues.apache.org/jira/browse/HBASE-14962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15093330#comment-15093330
]
Elliott Clark commented on HBASE-14962:
---
So I can get this pretty easily to fail on server hardware.
Sometimes on my laptop I can get 10 runs to pass but never 20. It seems like
this fails more often when my laptop is plugged in vs on battery. Suggesting
that something is happening too fast and the slowed down on battery makes tests
pass.
> TestSplitWalDataLoss fails on all branches
> --
>
> Key: HBASE-14962
> URL: https://issues.apache.org/jira/browse/HBASE-14962
> Project: HBase
> Issue Type: Sub-task
>Reporter: Elliott Clark
>Assignee: stack
>Priority: Blocker
> Fix For: 1.2.0
>
> Attachments:
> org.apache.hadoop.hbase.regionserver.TestSplitWalDataLoss-output.txt
>
>
> With some regularity I am seeing:
> {code}
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1
> action: TestSplitWalDataLoss:dataloss: 1 time,
> at
> org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:228)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1800(AsyncProcess.java:208)
> at
> org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1712)
> at
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:240)
> at
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:190)
> at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1430)
> at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1021)
> at
> org.apache.hadoop.hbase.regionserver.TestSplitWalDataLoss.test(TestSplitWalDataLoss.java:121)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15085) IllegalStateException was thrown when scanning on bulkloaded HFiles
[
https://issues.apache.org/jira/browse/HBASE-15085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15093346#comment-15093346
]
Victor Xu commented on HBASE-15085:
---
~ramkrishna.s.vasudevan
The log in console output seems not to be hbase compilation error, any idea on
the 'Docker failed to build'?
> IllegalStateException was thrown when scanning on bulkloaded HFiles
> ---
>
> Key: HBASE-15085
> URL: https://issues.apache.org/jira/browse/HBASE-15085
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.12, 1.1.2
> Environment: HBase-0.98.12 & Hadoop-2.6.0 & JDK1.7
> HBase-1.1.2 & Hadoop-2.6.0 & JDK1.7
>Reporter: Victor Xu
>Assignee: Victor Xu
> Labels: hfile
> Attachments: HBASE-15085-0.98-v1.patch, HBASE-15085-0.98-v2.patch,
> HBASE-15085-0.98-v3.patch, HBASE-15085-0.98-v4.patch,
> HBASE-15085-0.98-v4.patch, HBASE-15085-branch-1.0-v1.patch,
> HBASE-15085-branch-1.1-v1.patch, HBASE-15085-branch-1.2-v1.patch,
> HBASE-15085-v1.patch, HBASE-15085-v2.patch, HBASE-15085-v3.patch,
> HBASE-15085-v4.patch, HBASE-15085-v4.patch, HBASE-15085-v4.patch
>
>
> IllegalStateException was thrown when we scanned from an HFile which was bulk
> loaded several minutes ago, as shown below:
> {code}
> 2015-12-16 22:20:54,456 ERROR
> com.taobao.kart.coprocessor.server.KartCoprocessor:
> icbu_ae_ws_product,/0055,1450275490479.6a6a700f465ad074287fed720c950f7c.
> batchNotify exception
> java.lang.IllegalStateException: EncodedScanner works only on encoded data
> blocks
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.updateCurrentBlock(HFileReaderV2.java:1042)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.seekTo(HFileReaderV2.java:1093)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:244)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:152)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:329)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:188)
> at
> org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:1879)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.(HRegion.java:4068)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2029)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2015)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1992)
> {code}
> I used 'hbase hfile' command to analyse the meta and block info of the hfile,
> finding that even through the DATA_BLOCK_ENCODING was 'DIFF' in FileInfo, the
> actual data blocks was written without any encoding algorithms(BlockType was
> 'DATA', not 'ENCODED_DATA'):
> {code}
> Fileinfo:
> BLOOM_FILTER_TYPE = ROW
> BULKLOAD_SOURCE_TASK = attempt_1442077249005_606706_r_12_0
> BULKLOAD_TIMESTAMP = \x00\x00\x01R\x12$\x13\x12
> DATA_BLOCK_ENCODING = DIFF
> ...
> DataBlock Header:
> HFileBlock [ fileOffset=0 headerSize()=33 blockType=DATA
> onDiskSizeWithoutHeader=65591 uncompressedSizeWithoutHeader=65571
> prevBlockOffset=-1 isUseHBaseChecksum()=true checksumType=CRC32
> bytesPerChecksum=16384 onDiskDataSizeWithHeader=65604
> getOnDiskSizeWithHeader()=65624 totalChecksumBytes()=20 isUnpacked()=true
> buf=[ java.nio.HeapByteBuffer[pos=0 lim=65624 cap=65657],
> array().length=65657, arrayOffset()=0 ]
> dataBeginsWith=\x00\x00\x003\x00\x00\x00\x0A\x00\x10/0008:18\x01dprod
> fileContext=HFileContext [ usesHBaseChecksum=true checksumType=CRC32
> bytesPerChecksum=16384 blocksize=65536 encoding=NONE includesMvcc=true
> includesTags=false compressAlgo=NONE compressTags=false cryptoContext=[
> cipher=NONE keyHash=NONE ] ] ]
> {code}
> The data block encoding in file info was not consistent with the one in data
> block, which means there must be something wrong with the bulkload process.
> After debugging on each step of bulkload, I found that LoadIncrementalHFiles
> had a bug when loading hfile into a splitted region.
> {code}
> /**
>* Copy half of an HFile into a new HFile.
>*/
> private static void copyHFileHalf(
> Configuration conf, Path inFile, Path outFile, Reference reference,
> HColumnDescriptor familyDescriptor)
> throws IOException {
> FileSystem fs = inFile.getFileSystem(conf);
> CacheConfig cacheConf = new CacheConfig(conf);
> HalfStoreFileReader halfReader = null;
> StoreFile.Writer halfWriter = null;
> try {
> halfReader = new HalfStoreFileReader(fs, inFile, cacheConf, reference,
> conf);
> Map
[jira] [Commented] (HBASE-15085) IllegalStateException was thrown when scanning on bulkloaded HFiles
[
https://issues.apache.org/jira/browse/HBASE-15085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15093384#comment-15093384
]
Victor Xu commented on HBASE-15085:
---
Thanks. Do I need to resubmit the patch for all branches?
> IllegalStateException was thrown when scanning on bulkloaded HFiles
> ---
>
> Key: HBASE-15085
> URL: https://issues.apache.org/jira/browse/HBASE-15085
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.12, 1.1.2
> Environment: HBase-0.98.12 & Hadoop-2.6.0 & JDK1.7
> HBase-1.1.2 & Hadoop-2.6.0 & JDK1.7
>Reporter: Victor Xu
>Assignee: Victor Xu
> Labels: hfile
> Attachments: HBASE-15085-0.98-v1.patch, HBASE-15085-0.98-v2.patch,
> HBASE-15085-0.98-v3.patch, HBASE-15085-0.98-v4.patch,
> HBASE-15085-0.98-v4.patch, HBASE-15085-branch-1.0-v1.patch,
> HBASE-15085-branch-1.1-v1.patch, HBASE-15085-branch-1.2-v1.patch,
> HBASE-15085-v1.patch, HBASE-15085-v2.patch, HBASE-15085-v3.patch,
> HBASE-15085-v4.patch, HBASE-15085-v4.patch, HBASE-15085-v4.patch
>
>
> IllegalStateException was thrown when we scanned from an HFile which was bulk
> loaded several minutes ago, as shown below:
> {code}
> 2015-12-16 22:20:54,456 ERROR
> com.taobao.kart.coprocessor.server.KartCoprocessor:
> icbu_ae_ws_product,/0055,1450275490479.6a6a700f465ad074287fed720c950f7c.
> batchNotify exception
> java.lang.IllegalStateException: EncodedScanner works only on encoded data
> blocks
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.updateCurrentBlock(HFileReaderV2.java:1042)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.seekTo(HFileReaderV2.java:1093)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:244)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:152)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:329)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:188)
> at
> org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:1879)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.(HRegion.java:4068)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2029)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2015)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1992)
> {code}
> I used 'hbase hfile' command to analyse the meta and block info of the hfile,
> finding that even through the DATA_BLOCK_ENCODING was 'DIFF' in FileInfo, the
> actual data blocks was written without any encoding algorithms(BlockType was
> 'DATA', not 'ENCODED_DATA'):
> {code}
> Fileinfo:
> BLOOM_FILTER_TYPE = ROW
> BULKLOAD_SOURCE_TASK = attempt_1442077249005_606706_r_12_0
> BULKLOAD_TIMESTAMP = \x00\x00\x01R\x12$\x13\x12
> DATA_BLOCK_ENCODING = DIFF
> ...
> DataBlock Header:
> HFileBlock [ fileOffset=0 headerSize()=33 blockType=DATA
> onDiskSizeWithoutHeader=65591 uncompressedSizeWithoutHeader=65571
> prevBlockOffset=-1 isUseHBaseChecksum()=true checksumType=CRC32
> bytesPerChecksum=16384 onDiskDataSizeWithHeader=65604
> getOnDiskSizeWithHeader()=65624 totalChecksumBytes()=20 isUnpacked()=true
> buf=[ java.nio.HeapByteBuffer[pos=0 lim=65624 cap=65657],
> array().length=65657, arrayOffset()=0 ]
> dataBeginsWith=\x00\x00\x003\x00\x00\x00\x0A\x00\x10/0008:18\x01dprod
> fileContext=HFileContext [ usesHBaseChecksum=true checksumType=CRC32
> bytesPerChecksum=16384 blocksize=65536 encoding=NONE includesMvcc=true
> includesTags=false compressAlgo=NONE compressTags=false cryptoContext=[
> cipher=NONE keyHash=NONE ] ] ]
> {code}
> The data block encoding in file info was not consistent with the one in data
> block, which means there must be something wrong with the bulkload process.
> After debugging on each step of bulkload, I found that LoadIncrementalHFiles
> had a bug when loading hfile into a splitted region.
> {code}
> /**
>* Copy half of an HFile into a new HFile.
>*/
> private static void copyHFileHalf(
> Configuration conf, Path inFile, Path outFile, Reference reference,
> HColumnDescriptor familyDescriptor)
> throws IOException {
> FileSystem fs = inFile.getFileSystem(conf);
> CacheConfig cacheConf = new CacheConfig(conf);
> HalfStoreFileReader halfReader = null;
> StoreFile.Writer halfWriter = null;
> try {
> halfReader = new HalfStoreFileReader(fs, inFile, cacheConf, reference,
> conf);
> Map fileInfo = halfReader.loadFileInfo();
> int blocksize =
[jira] [Commented] (HBASE-15083) Gets from Multiactions are not counted in metrics for gets.
[ https://issues.apache.org/jira/browse/HBASE-15083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15093386#comment-15093386 ] Elliott Clark commented on HBASE-15083: --- We should push this to other banches too if you are up for a backport. If not just let me know and I will. > Gets from Multiactions are not counted in metrics for gets. > --- > > Key: HBASE-15083 > URL: https://issues.apache.org/jira/browse/HBASE-15083 > Project: HBase > Issue Type: Bug >Reporter: Elliott Clark >Assignee: Heng Chen > Fix For: 2.0.0 > > Attachments: HBASE-15083.patch, HBASE-15083.patch > > > RSRpcServices#get updates the get metrics. However Multiactions do not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15083) Gets from Multiactions are not counted in metrics for gets.
[ https://issues.apache.org/jira/browse/HBASE-15083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15093406#comment-15093406 ] Elliott Clark commented on HBASE-15083: --- Thanks! > Gets from Multiactions are not counted in metrics for gets. > --- > > Key: HBASE-15083 > URL: https://issues.apache.org/jira/browse/HBASE-15083 > Project: HBase > Issue Type: Bug >Reporter: Elliott Clark >Assignee: Heng Chen > Fix For: 1.0.0, 2.0.0, 1.1.0, 1.2.0, 1.3.0 > > Attachments: HBASE-15083.patch, HBASE-15083.patch > > > RSRpcServices#get updates the get metrics. However Multiactions do not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15083) Gets from Multiactions are not counted in metrics for gets.
[ https://issues.apache.org/jira/browse/HBASE-15083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heng Chen updated HBASE-15083: -- Attachment: HBASE-15083-branch-1.patch > Gets from Multiactions are not counted in metrics for gets. > --- > > Key: HBASE-15083 > URL: https://issues.apache.org/jira/browse/HBASE-15083 > Project: HBase > Issue Type: Bug >Reporter: Elliott Clark >Assignee: Heng Chen > Fix For: 1.0.0, 2.0.0, 1.1.0, 1.2.0, 1.3.0 > > Attachments: HBASE-15083-branch-1.patch, HBASE-15083.patch, > HBASE-15083.patch > > > RSRpcServices#get updates the get metrics. However Multiactions do not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15089) Compatibility issue on HTable#flushCommits
Yu Li created HBASE-15089: - Summary: Compatibility issue on HTable#flushCommits Key: HBASE-15089 URL: https://issues.apache.org/jira/browse/HBASE-15089 Project: HBase Issue Type: Bug Affects Versions: 1.1.2 Reporter: Yu Li Assignee: Yu Li Priority: Minor Previously in 0.98 HTable#flushCommits throws InterruptedIOException and RetriesExhaustedWithDetailsException, but now in 1.1.2 this method signature has been changed to throw IOException, which will force application code changes for exception handling (previous catch on InterruptedIOException and RetriesExhaustedWithDetailsException become invalid). In this JIRA we propose to recover the compatibility. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15087) Fix hbase-common findbugs complaints
[ https://issues.apache.org/jira/browse/HBASE-15087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15093198#comment-15093198 ] ramkrishna.s.vasudevan commented on HBASE-15087: +1 > Fix hbase-common findbugs complaints > > > Key: HBASE-15087 > URL: https://issues.apache.org/jira/browse/HBASE-15087 > Project: HBase > Issue Type: Sub-task > Components: build >Reporter: stack >Assignee: Stack > Attachments: 15087.patch, 15087v2.patch, 15087v3.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15085) IllegalStateException was thrown when scanning on bulkloaded HFiles
[
https://issues.apache.org/jira/browse/HBASE-15085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-15085:
---
Hadoop Flags: Reviewed
Status: Patch Available (was: Open)
> IllegalStateException was thrown when scanning on bulkloaded HFiles
> ---
>
> Key: HBASE-15085
> URL: https://issues.apache.org/jira/browse/HBASE-15085
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.1.2, 0.98.12
> Environment: HBase-0.98.12 & Hadoop-2.6.0 & JDK1.7
> HBase-1.1.2 & Hadoop-2.6.0 & JDK1.7
>Reporter: Victor Xu
>Assignee: Victor Xu
> Labels: hfile
> Attachments: HBASE-15085-0.98-v1.patch, HBASE-15085-0.98-v2.patch,
> HBASE-15085-0.98-v3.patch, HBASE-15085-0.98-v4.patch, HBASE-15085-v1.patch,
> HBASE-15085-v2.patch, HBASE-15085-v3.patch, HBASE-15085-v4.patch,
> HBASE-15085-v4.patch
>
>
> IllegalStateException was thrown when we scanned from an HFile which was bulk
> loaded several minutes ago, as shown below:
> {code}
> 2015-12-16 22:20:54,456 ERROR
> com.taobao.kart.coprocessor.server.KartCoprocessor:
> icbu_ae_ws_product,/0055,1450275490479.6a6a700f465ad074287fed720c950f7c.
> batchNotify exception
> java.lang.IllegalStateException: EncodedScanner works only on encoded data
> blocks
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.updateCurrentBlock(HFileReaderV2.java:1042)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.seekTo(HFileReaderV2.java:1093)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:244)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:152)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:329)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:188)
> at
> org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:1879)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.(HRegion.java:4068)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2029)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2015)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1992)
> {code}
> I used 'hbase hfile' command to analyse the meta and block info of the hfile,
> finding that even through the DATA_BLOCK_ENCODING was 'DIFF' in FileInfo, the
> actual data blocks was written without any encoding algorithms(BlockType was
> 'DATA', not 'ENCODED_DATA'):
> {code}
> Fileinfo:
> BLOOM_FILTER_TYPE = ROW
> BULKLOAD_SOURCE_TASK = attempt_1442077249005_606706_r_12_0
> BULKLOAD_TIMESTAMP = \x00\x00\x01R\x12$\x13\x12
> DATA_BLOCK_ENCODING = DIFF
> ...
> DataBlock Header:
> HFileBlock [ fileOffset=0 headerSize()=33 blockType=DATA
> onDiskSizeWithoutHeader=65591 uncompressedSizeWithoutHeader=65571
> prevBlockOffset=-1 isUseHBaseChecksum()=true checksumType=CRC32
> bytesPerChecksum=16384 onDiskDataSizeWithHeader=65604
> getOnDiskSizeWithHeader()=65624 totalChecksumBytes()=20 isUnpacked()=true
> buf=[ java.nio.HeapByteBuffer[pos=0 lim=65624 cap=65657],
> array().length=65657, arrayOffset()=0 ]
> dataBeginsWith=\x00\x00\x003\x00\x00\x00\x0A\x00\x10/0008:18\x01dprod
> fileContext=HFileContext [ usesHBaseChecksum=true checksumType=CRC32
> bytesPerChecksum=16384 blocksize=65536 encoding=NONE includesMvcc=true
> includesTags=false compressAlgo=NONE compressTags=false cryptoContext=[
> cipher=NONE keyHash=NONE ] ] ]
> {code}
> The data block encoding in file info was not consistent with the one in data
> block, which means there must be something wrong with the bulkload process.
> After debugging on each step of bulkload, I found that LoadIncrementalHFiles
> had a bug when loading hfile into a splitted region.
> {code}
> /**
>* Copy half of an HFile into a new HFile.
>*/
> private static void copyHFileHalf(
> Configuration conf, Path inFile, Path outFile, Reference reference,
> HColumnDescriptor familyDescriptor)
> throws IOException {
> FileSystem fs = inFile.getFileSystem(conf);
> CacheConfig cacheConf = new CacheConfig(conf);
> HalfStoreFileReader halfReader = null;
> StoreFile.Writer halfWriter = null;
> try {
> halfReader = new HalfStoreFileReader(fs, inFile, cacheConf, reference,
> conf);
> Map fileInfo = halfReader.loadFileInfo();
> int blocksize = familyDescriptor.getBlocksize();
> Algorithm compression = familyDescriptor.getCompression();
> BloomType bloomFilterType =
[jira] [Updated] (HBASE-15085) IllegalStateException was thrown when scanning on bulkloaded HFiles
[
https://issues.apache.org/jira/browse/HBASE-15085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-15085:
---
Attachment: HBASE-15085-v4.patch
Resubmitting for QA to pick up so that it can be commited.
[~victorunique]
See if the patch can be applied 1.0 branches also. If not pls provided patches
for those branches also.
> IllegalStateException was thrown when scanning on bulkloaded HFiles
> ---
>
> Key: HBASE-15085
> URL: https://issues.apache.org/jira/browse/HBASE-15085
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.12, 1.1.2
> Environment: HBase-0.98.12 & Hadoop-2.6.0 & JDK1.7
> HBase-1.1.2 & Hadoop-2.6.0 & JDK1.7
>Reporter: Victor Xu
>Assignee: Victor Xu
> Labels: hfile
> Attachments: HBASE-15085-0.98-v1.patch, HBASE-15085-0.98-v2.patch,
> HBASE-15085-0.98-v3.patch, HBASE-15085-0.98-v4.patch, HBASE-15085-v1.patch,
> HBASE-15085-v2.patch, HBASE-15085-v3.patch, HBASE-15085-v4.patch,
> HBASE-15085-v4.patch
>
>
> IllegalStateException was thrown when we scanned from an HFile which was bulk
> loaded several minutes ago, as shown below:
> {code}
> 2015-12-16 22:20:54,456 ERROR
> com.taobao.kart.coprocessor.server.KartCoprocessor:
> icbu_ae_ws_product,/0055,1450275490479.6a6a700f465ad074287fed720c950f7c.
> batchNotify exception
> java.lang.IllegalStateException: EncodedScanner works only on encoded data
> blocks
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.updateCurrentBlock(HFileReaderV2.java:1042)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.seekTo(HFileReaderV2.java:1093)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:244)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:152)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:329)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:188)
> at
> org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:1879)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.(HRegion.java:4068)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2029)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2015)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1992)
> {code}
> I used 'hbase hfile' command to analyse the meta and block info of the hfile,
> finding that even through the DATA_BLOCK_ENCODING was 'DIFF' in FileInfo, the
> actual data blocks was written without any encoding algorithms(BlockType was
> 'DATA', not 'ENCODED_DATA'):
> {code}
> Fileinfo:
> BLOOM_FILTER_TYPE = ROW
> BULKLOAD_SOURCE_TASK = attempt_1442077249005_606706_r_12_0
> BULKLOAD_TIMESTAMP = \x00\x00\x01R\x12$\x13\x12
> DATA_BLOCK_ENCODING = DIFF
> ...
> DataBlock Header:
> HFileBlock [ fileOffset=0 headerSize()=33 blockType=DATA
> onDiskSizeWithoutHeader=65591 uncompressedSizeWithoutHeader=65571
> prevBlockOffset=-1 isUseHBaseChecksum()=true checksumType=CRC32
> bytesPerChecksum=16384 onDiskDataSizeWithHeader=65604
> getOnDiskSizeWithHeader()=65624 totalChecksumBytes()=20 isUnpacked()=true
> buf=[ java.nio.HeapByteBuffer[pos=0 lim=65624 cap=65657],
> array().length=65657, arrayOffset()=0 ]
> dataBeginsWith=\x00\x00\x003\x00\x00\x00\x0A\x00\x10/0008:18\x01dprod
> fileContext=HFileContext [ usesHBaseChecksum=true checksumType=CRC32
> bytesPerChecksum=16384 blocksize=65536 encoding=NONE includesMvcc=true
> includesTags=false compressAlgo=NONE compressTags=false cryptoContext=[
> cipher=NONE keyHash=NONE ] ] ]
> {code}
> The data block encoding in file info was not consistent with the one in data
> block, which means there must be something wrong with the bulkload process.
> After debugging on each step of bulkload, I found that LoadIncrementalHFiles
> had a bug when loading hfile into a splitted region.
> {code}
> /**
>* Copy half of an HFile into a new HFile.
>*/
> private static void copyHFileHalf(
> Configuration conf, Path inFile, Path outFile, Reference reference,
> HColumnDescriptor familyDescriptor)
> throws IOException {
> FileSystem fs = inFile.getFileSystem(conf);
> CacheConfig cacheConf = new CacheConfig(conf);
> HalfStoreFileReader halfReader = null;
> StoreFile.Writer halfWriter = null;
> try {
> halfReader = new HalfStoreFileReader(fs, inFile, cacheConf, reference,
> conf);
> Map fileInfo = halfReader.loadFileInfo();
> int blocksize =
[jira] [Commented] (HBASE-15085) IllegalStateException was thrown when scanning on bulkloaded HFiles
[
https://issues.apache.org/jira/browse/HBASE-15085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15093380#comment-15093380
]
Sean Busbey commented on HBASE-15085:
-
Docker has been having trouble installing shellcheck. I've switched to not
defaulting to Docker until I can investigate.
> IllegalStateException was thrown when scanning on bulkloaded HFiles
> ---
>
> Key: HBASE-15085
> URL: https://issues.apache.org/jira/browse/HBASE-15085
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.12, 1.1.2
> Environment: HBase-0.98.12 & Hadoop-2.6.0 & JDK1.7
> HBase-1.1.2 & Hadoop-2.6.0 & JDK1.7
>Reporter: Victor Xu
>Assignee: Victor Xu
> Labels: hfile
> Attachments: HBASE-15085-0.98-v1.patch, HBASE-15085-0.98-v2.patch,
> HBASE-15085-0.98-v3.patch, HBASE-15085-0.98-v4.patch,
> HBASE-15085-0.98-v4.patch, HBASE-15085-branch-1.0-v1.patch,
> HBASE-15085-branch-1.1-v1.patch, HBASE-15085-branch-1.2-v1.patch,
> HBASE-15085-v1.patch, HBASE-15085-v2.patch, HBASE-15085-v3.patch,
> HBASE-15085-v4.patch, HBASE-15085-v4.patch, HBASE-15085-v4.patch
>
>
> IllegalStateException was thrown when we scanned from an HFile which was bulk
> loaded several minutes ago, as shown below:
> {code}
> 2015-12-16 22:20:54,456 ERROR
> com.taobao.kart.coprocessor.server.KartCoprocessor:
> icbu_ae_ws_product,/0055,1450275490479.6a6a700f465ad074287fed720c950f7c.
> batchNotify exception
> java.lang.IllegalStateException: EncodedScanner works only on encoded data
> blocks
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.updateCurrentBlock(HFileReaderV2.java:1042)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.seekTo(HFileReaderV2.java:1093)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:244)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:152)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:329)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:188)
> at
> org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:1879)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.(HRegion.java:4068)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2029)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2015)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1992)
> {code}
> I used 'hbase hfile' command to analyse the meta and block info of the hfile,
> finding that even through the DATA_BLOCK_ENCODING was 'DIFF' in FileInfo, the
> actual data blocks was written without any encoding algorithms(BlockType was
> 'DATA', not 'ENCODED_DATA'):
> {code}
> Fileinfo:
> BLOOM_FILTER_TYPE = ROW
> BULKLOAD_SOURCE_TASK = attempt_1442077249005_606706_r_12_0
> BULKLOAD_TIMESTAMP = \x00\x00\x01R\x12$\x13\x12
> DATA_BLOCK_ENCODING = DIFF
> ...
> DataBlock Header:
> HFileBlock [ fileOffset=0 headerSize()=33 blockType=DATA
> onDiskSizeWithoutHeader=65591 uncompressedSizeWithoutHeader=65571
> prevBlockOffset=-1 isUseHBaseChecksum()=true checksumType=CRC32
> bytesPerChecksum=16384 onDiskDataSizeWithHeader=65604
> getOnDiskSizeWithHeader()=65624 totalChecksumBytes()=20 isUnpacked()=true
> buf=[ java.nio.HeapByteBuffer[pos=0 lim=65624 cap=65657],
> array().length=65657, arrayOffset()=0 ]
> dataBeginsWith=\x00\x00\x003\x00\x00\x00\x0A\x00\x10/0008:18\x01dprod
> fileContext=HFileContext [ usesHBaseChecksum=true checksumType=CRC32
> bytesPerChecksum=16384 blocksize=65536 encoding=NONE includesMvcc=true
> includesTags=false compressAlgo=NONE compressTags=false cryptoContext=[
> cipher=NONE keyHash=NONE ] ] ]
> {code}
> The data block encoding in file info was not consistent with the one in data
> block, which means there must be something wrong with the bulkload process.
> After debugging on each step of bulkload, I found that LoadIncrementalHFiles
> had a bug when loading hfile into a splitted region.
> {code}
> /**
>* Copy half of an HFile into a new HFile.
>*/
> private static void copyHFileHalf(
> Configuration conf, Path inFile, Path outFile, Reference reference,
> HColumnDescriptor familyDescriptor)
> throws IOException {
> FileSystem fs = inFile.getFileSystem(conf);
> CacheConfig cacheConf = new CacheConfig(conf);
> HalfStoreFileReader halfReader = null;
> StoreFile.Writer halfWriter = null;
> try {
> halfReader = new HalfStoreFileReader(fs, inFile, cacheConf, reference,
> conf);
> Map
[jira] [Commented] (HBASE-15083) Gets from Multiactions are not counted in metrics for gets.
[ https://issues.apache.org/jira/browse/HBASE-15083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15093382#comment-15093382 ] Heng Chen commented on HBASE-15083: --- push to master. Thanks [~eclark] for your review > Gets from Multiactions are not counted in metrics for gets. > --- > > Key: HBASE-15083 > URL: https://issues.apache.org/jira/browse/HBASE-15083 > Project: HBase > Issue Type: Bug >Reporter: Elliott Clark > Fix For: 2.0.0 > > Attachments: HBASE-15083.patch, HBASE-15083.patch > > > RSRpcServices#get updates the get metrics. However Multiactions do not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15085) IllegalStateException was thrown when scanning on bulkloaded HFiles
[
https://issues.apache.org/jira/browse/HBASE-15085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victor Xu updated HBASE-15085:
--
Attachment: HBASE-15085-v4.patch
Submit patch for master branch.
> IllegalStateException was thrown when scanning on bulkloaded HFiles
> ---
>
> Key: HBASE-15085
> URL: https://issues.apache.org/jira/browse/HBASE-15085
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.12, 1.1.2
> Environment: HBase-0.98.12 & Hadoop-2.6.0 & JDK1.7
> HBase-1.1.2 & Hadoop-2.6.0 & JDK1.7
>Reporter: Victor Xu
>Assignee: Victor Xu
> Labels: hfile
> Attachments: HBASE-15085-0.98-v1.patch, HBASE-15085-0.98-v2.patch,
> HBASE-15085-0.98-v3.patch, HBASE-15085-0.98-v4.patch,
> HBASE-15085-0.98-v4.patch, HBASE-15085-branch-1.0-v1.patch,
> HBASE-15085-branch-1.1-v1.patch, HBASE-15085-branch-1.2-v1.patch,
> HBASE-15085-v1.patch, HBASE-15085-v2.patch, HBASE-15085-v3.patch,
> HBASE-15085-v4.patch, HBASE-15085-v4.patch, HBASE-15085-v4.patch,
> HBASE-15085-v4.patch
>
>
> IllegalStateException was thrown when we scanned from an HFile which was bulk
> loaded several minutes ago, as shown below:
> {code}
> 2015-12-16 22:20:54,456 ERROR
> com.taobao.kart.coprocessor.server.KartCoprocessor:
> icbu_ae_ws_product,/0055,1450275490479.6a6a700f465ad074287fed720c950f7c.
> batchNotify exception
> java.lang.IllegalStateException: EncodedScanner works only on encoded data
> blocks
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.updateCurrentBlock(HFileReaderV2.java:1042)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.seekTo(HFileReaderV2.java:1093)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:244)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:152)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:329)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:188)
> at
> org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:1879)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.(HRegion.java:4068)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2029)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2015)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1992)
> {code}
> I used 'hbase hfile' command to analyse the meta and block info of the hfile,
> finding that even through the DATA_BLOCK_ENCODING was 'DIFF' in FileInfo, the
> actual data blocks was written without any encoding algorithms(BlockType was
> 'DATA', not 'ENCODED_DATA'):
> {code}
> Fileinfo:
> BLOOM_FILTER_TYPE = ROW
> BULKLOAD_SOURCE_TASK = attempt_1442077249005_606706_r_12_0
> BULKLOAD_TIMESTAMP = \x00\x00\x01R\x12$\x13\x12
> DATA_BLOCK_ENCODING = DIFF
> ...
> DataBlock Header:
> HFileBlock [ fileOffset=0 headerSize()=33 blockType=DATA
> onDiskSizeWithoutHeader=65591 uncompressedSizeWithoutHeader=65571
> prevBlockOffset=-1 isUseHBaseChecksum()=true checksumType=CRC32
> bytesPerChecksum=16384 onDiskDataSizeWithHeader=65604
> getOnDiskSizeWithHeader()=65624 totalChecksumBytes()=20 isUnpacked()=true
> buf=[ java.nio.HeapByteBuffer[pos=0 lim=65624 cap=65657],
> array().length=65657, arrayOffset()=0 ]
> dataBeginsWith=\x00\x00\x003\x00\x00\x00\x0A\x00\x10/0008:18\x01dprod
> fileContext=HFileContext [ usesHBaseChecksum=true checksumType=CRC32
> bytesPerChecksum=16384 blocksize=65536 encoding=NONE includesMvcc=true
> includesTags=false compressAlgo=NONE compressTags=false cryptoContext=[
> cipher=NONE keyHash=NONE ] ] ]
> {code}
> The data block encoding in file info was not consistent with the one in data
> block, which means there must be something wrong with the bulkload process.
> After debugging on each step of bulkload, I found that LoadIncrementalHFiles
> had a bug when loading hfile into a splitted region.
> {code}
> /**
>* Copy half of an HFile into a new HFile.
>*/
> private static void copyHFileHalf(
> Configuration conf, Path inFile, Path outFile, Reference reference,
> HColumnDescriptor familyDescriptor)
> throws IOException {
> FileSystem fs = inFile.getFileSystem(conf);
> CacheConfig cacheConf = new CacheConfig(conf);
> HalfStoreFileReader halfReader = null;
> StoreFile.Writer halfWriter = null;
> try {
> halfReader = new HalfStoreFileReader(fs, inFile, cacheConf, reference,
> conf);
> Map fileInfo = halfReader.loadFileInfo();
> int blocksize =
[jira] [Updated] (HBASE-14159) Resolve warning introduced by HBase-Spark module
[ https://issues.apache.org/jira/browse/HBASE-14159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-14159: - Status: Patch Available (was: Open) > Resolve warning introduced by HBase-Spark module > > > Key: HBASE-14159 > URL: https://issues.apache.org/jira/browse/HBASE-14159 > Project: HBase > Issue Type: Improvement > Components: build, spark >Reporter: Ted Malaska >Assignee: Sean Busbey >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-14159-master-v1.patch > > > Fix the following warning that is a result of something in the modules pom > file > [WARNING] warning: Class org.apache.hadoop.mapred.MiniMRCluster not found - > continuing with a stub. > [WARNING] one warning found -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15075) Allow region split request to carry metadata
[ https://issues.apache.org/jira/browse/HBASE-15075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15093126#comment-15093126 ] Ted Yu commented on HBASE-15075: bq. The description on this issue implies something generic When I logged this JIRA, that was the intention. However, working out a patch, I realized that the metadata has to go through RPC layer using protobuf. This narrows the choice for the data type used. Subject of the JIRA can be modified to reflect the actual approach. bq. Why use UUID? Why not a random long? UUID gives us wider scope for the purpose of identification of split requests. Meaning, for long running region servers, the chance of collision between UUIDs is very low. As for type long, the chance of collision is higher. bq. Should we add API so split requests can pass arbitrary metadata as issue description implies? I would be willing to see how the arbitrary metadata should be formulated. Keep in mind that such metadata needs to be protobuf'ed. For the use cases I am aware of, UUID suffices. bq. Should every request type be extended like this? I.e. merge transactions? Depending on the outcome of discussion for this JIRA, I think the adopted solution can be applied for merge request as well (maybe in another JIRA). > Allow region split request to carry metadata > > > Key: HBASE-15075 > URL: https://issues.apache.org/jira/browse/HBASE-15075 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 15075-v0.txt, 15075-v1.txt, 15075-v2.txt, > HBASE-15075.v2.patch, HBASE-15075.v3.patch > > > During the process of improving region normalization feature, I found that if > region split request triggered by the execution of SplitNormalizationPlan > fails, there is no way of knowing whether the failed split originated from > region normalization. > The association of particular split request with outcome of split would give > RegionNormalizer information so that it can make better normalization > decisions in the subsequent invocations. > One approach is to embed metadata in SplitRequest which gets passed through > RegionStateTransitionContext when > RegionServerServices#reportRegionStateTransition() is called. > This way, RegionStateListener can be notified with the metadata (id of the > requester). > See discussion on dev mailing list > http://search-hadoop.com/m/YGbbCXdkivihp2 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15088) Running mvn package creates ${project.basedir} directory
Gabor Liptak created HBASE-15088:
Summary: Running mvn package creates ${project.basedir} directory
Key: HBASE-15088
URL: https://issues.apache.org/jira/browse/HBASE-15088
Project: HBase
Issue Type: Bug
Reporter: Gabor Liptak
Priority: Minor
The substitution is not triggered in pom.xml at:
file:${project.basedir}/src/main/site/resources/repo
There is also another reference at:
${project.basedir}/../dev-support/findbugs-exclude.xml
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14962) TestSplitWalDataLoss fails on all branches
[
https://issues.apache.org/jira/browse/HBASE-14962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15093175#comment-15093175
]
Elliott Clark commented on HBASE-14962:
---
Let me get a log up.
> TestSplitWalDataLoss fails on all branches
> --
>
> Key: HBASE-14962
> URL: https://issues.apache.org/jira/browse/HBASE-14962
> Project: HBase
> Issue Type: Sub-task
>Reporter: Elliott Clark
>Assignee: stack
>Priority: Blocker
> Fix For: 1.2.0
>
>
> With some regularity I am seeing:
> {code}
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1
> action: TestSplitWalDataLoss:dataloss: 1 time,
> at
> org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:228)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1800(AsyncProcess.java:208)
> at
> org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1712)
> at
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:240)
> at
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:190)
> at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1430)
> at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1021)
> at
> org.apache.hadoop.hbase.regionserver.TestSplitWalDataLoss.test(TestSplitWalDataLoss.java:121)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[ https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15055: Status: Patch Available (was: Open) > Major compaction is not triggered when both of TTL and > hbase.hstore.compaction.max.size are set > --- > > Key: HBASE-15055 > URL: https://issues.apache.org/jira/browse/HBASE-15055 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Assignee: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15055-v1.patch, HBASE-15055-v10.patch, > HBASE-15055-v2.patch, HBASE-15055-v3.patch, HBASE-15055-v4.patch, > HBASE-15055-v5.patch, HBASE-15055-v6.patch, HBASE-15055-v7.patch, > HBASE-15055-v8.patch, HBASE-15055-v9.patch, HBASE-15055.patch > > > Some large files may be skipped by hbase.hstore.compaction.max.size in > candidate selection. It causes skipping of major compaction. So the TTL > expired records are still remained in the disks and keep consuming disks. > To resolve this issue, I suggest that to skip large files only if there is no > TTL expired record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[ https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15055: Attachment: HBASE-15055-v10.patch To address Anoop's comment, the new patch is submitted. In this patch, RatioBasedCompactionPolicy keeps all candidates and runs a major compaction irrespective of hbase.hstore.compaction.max.size if the major compaction period has elapsed. > Major compaction is not triggered when both of TTL and > hbase.hstore.compaction.max.size are set > --- > > Key: HBASE-15055 > URL: https://issues.apache.org/jira/browse/HBASE-15055 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Assignee: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15055-v1.patch, HBASE-15055-v10.patch, > HBASE-15055-v2.patch, HBASE-15055-v3.patch, HBASE-15055-v4.patch, > HBASE-15055-v5.patch, HBASE-15055-v6.patch, HBASE-15055-v7.patch, > HBASE-15055-v8.patch, HBASE-15055-v9.patch, HBASE-15055.patch > > > Some large files may be skipped by hbase.hstore.compaction.max.size in > candidate selection. It causes skipping of major compaction. So the TTL > expired records are still remained in the disks and keep consuming disks. > To resolve this issue, I suggest that to skip large files only if there is no > TTL expired record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14167) hbase-spark integration tests do not respect -DskipIntegrationTests
[
https://issues.apache.org/jira/browse/HBASE-14167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Appy updated HBASE-14167:
-
Component/s: pom
> hbase-spark integration tests do not respect -DskipIntegrationTests
> ---
>
> Key: HBASE-14167
> URL: https://issues.apache.org/jira/browse/HBASE-14167
> Project: HBase
> Issue Type: Improvement
> Components: pom, spark
>Affects Versions: 2.0.0
>Reporter: Andrew Purtell
>Assignee: Gabor Liptak
>Priority: Minor
> Attachments: HBASE-14167-master-v2.patch, HBASE-14167.11.patch
>
>
> When running a build with {{mvn ... -DskipIntegrationTests}}, the hbase-spark
> module's integration tests do not respect the flag and run anyway. Fix.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14376) move hbase spark integration examples into their own module
[ https://issues.apache.org/jira/browse/HBASE-14376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-14376: - Component/s: pom > move hbase spark integration examples into their own module > --- > > Key: HBASE-14376 > URL: https://issues.apache.org/jira/browse/HBASE-14376 > Project: HBase > Issue Type: Task > Components: pom, spark >Reporter: Sean Busbey >Assignee: Gabor Liptak > Labels: beginner > Attachments: HBASE-14376.1.patch, warnings.diff > > > take the examples that are currently in the hbase-spark module and move them > into a hbase-spark-examples module. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[
https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15093280#comment-15093280
]
Hadoop QA commented on HBASE-15055:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 10m 5s
{color} | {color:red} Docker failed to build yetus/hbase:date2016-01-12.
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12781728/HBASE-15055-v10.patch
|
| JIRA Issue | HBASE-15055 |
| Powered by | Apache Yetus 0.1.0 http://yetus.apache.org |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/70/console |
This message was automatically generated.
> Major compaction is not triggered when both of TTL and
> hbase.hstore.compaction.max.size are set
> ---
>
> Key: HBASE-15055
> URL: https://issues.apache.org/jira/browse/HBASE-15055
> Project: HBase
> Issue Type: Bug
>Reporter: Eungsop Yoo
>Assignee: Eungsop Yoo
>Priority: Minor
> Attachments: HBASE-15055-v1.patch, HBASE-15055-v10.patch,
> HBASE-15055-v2.patch, HBASE-15055-v3.patch, HBASE-15055-v4.patch,
> HBASE-15055-v5.patch, HBASE-15055-v6.patch, HBASE-15055-v7.patch,
> HBASE-15055-v8.patch, HBASE-15055-v9.patch, HBASE-15055.patch
>
>
> Some large files may be skipped by hbase.hstore.compaction.max.size in
> candidate selection. It causes skipping of major compaction. So the TTL
> expired records are still remained in the disks and keep consuming disks.
> To resolve this issue, I suggest that to skip large files only if there is no
> TTL expired record.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15085) IllegalStateException was thrown when scanning on bulkloaded HFiles
[
https://issues.apache.org/jira/browse/HBASE-15085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15093391#comment-15093391
]
ramkrishna.s.vasudevan commented on HBASE-15085:
Thanks [~busbey]. [~victorunique] just submit for master branch now. Lets see
if we get a clean run from that.
> IllegalStateException was thrown when scanning on bulkloaded HFiles
> ---
>
> Key: HBASE-15085
> URL: https://issues.apache.org/jira/browse/HBASE-15085
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.12, 1.1.2
> Environment: HBase-0.98.12 & Hadoop-2.6.0 & JDK1.7
> HBase-1.1.2 & Hadoop-2.6.0 & JDK1.7
>Reporter: Victor Xu
>Assignee: Victor Xu
> Labels: hfile
> Attachments: HBASE-15085-0.98-v1.patch, HBASE-15085-0.98-v2.patch,
> HBASE-15085-0.98-v3.patch, HBASE-15085-0.98-v4.patch,
> HBASE-15085-0.98-v4.patch, HBASE-15085-branch-1.0-v1.patch,
> HBASE-15085-branch-1.1-v1.patch, HBASE-15085-branch-1.2-v1.patch,
> HBASE-15085-v1.patch, HBASE-15085-v2.patch, HBASE-15085-v3.patch,
> HBASE-15085-v4.patch, HBASE-15085-v4.patch, HBASE-15085-v4.patch
>
>
> IllegalStateException was thrown when we scanned from an HFile which was bulk
> loaded several minutes ago, as shown below:
> {code}
> 2015-12-16 22:20:54,456 ERROR
> com.taobao.kart.coprocessor.server.KartCoprocessor:
> icbu_ae_ws_product,/0055,1450275490479.6a6a700f465ad074287fed720c950f7c.
> batchNotify exception
> java.lang.IllegalStateException: EncodedScanner works only on encoded data
> blocks
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.updateCurrentBlock(HFileReaderV2.java:1042)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.seekTo(HFileReaderV2.java:1093)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:244)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:152)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:329)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:188)
> at
> org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:1879)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.(HRegion.java:4068)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2029)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2015)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1992)
> {code}
> I used 'hbase hfile' command to analyse the meta and block info of the hfile,
> finding that even through the DATA_BLOCK_ENCODING was 'DIFF' in FileInfo, the
> actual data blocks was written without any encoding algorithms(BlockType was
> 'DATA', not 'ENCODED_DATA'):
> {code}
> Fileinfo:
> BLOOM_FILTER_TYPE = ROW
> BULKLOAD_SOURCE_TASK = attempt_1442077249005_606706_r_12_0
> BULKLOAD_TIMESTAMP = \x00\x00\x01R\x12$\x13\x12
> DATA_BLOCK_ENCODING = DIFF
> ...
> DataBlock Header:
> HFileBlock [ fileOffset=0 headerSize()=33 blockType=DATA
> onDiskSizeWithoutHeader=65591 uncompressedSizeWithoutHeader=65571
> prevBlockOffset=-1 isUseHBaseChecksum()=true checksumType=CRC32
> bytesPerChecksum=16384 onDiskDataSizeWithHeader=65604
> getOnDiskSizeWithHeader()=65624 totalChecksumBytes()=20 isUnpacked()=true
> buf=[ java.nio.HeapByteBuffer[pos=0 lim=65624 cap=65657],
> array().length=65657, arrayOffset()=0 ]
> dataBeginsWith=\x00\x00\x003\x00\x00\x00\x0A\x00\x10/0008:18\x01dprod
> fileContext=HFileContext [ usesHBaseChecksum=true checksumType=CRC32
> bytesPerChecksum=16384 blocksize=65536 encoding=NONE includesMvcc=true
> includesTags=false compressAlgo=NONE compressTags=false cryptoContext=[
> cipher=NONE keyHash=NONE ] ] ]
> {code}
> The data block encoding in file info was not consistent with the one in data
> block, which means there must be something wrong with the bulkload process.
> After debugging on each step of bulkload, I found that LoadIncrementalHFiles
> had a bug when loading hfile into a splitted region.
> {code}
> /**
>* Copy half of an HFile into a new HFile.
>*/
> private static void copyHFileHalf(
> Configuration conf, Path inFile, Path outFile, Reference reference,
> HColumnDescriptor familyDescriptor)
> throws IOException {
> FileSystem fs = inFile.getFileSystem(conf);
> CacheConfig cacheConf = new CacheConfig(conf);
> HalfStoreFileReader halfReader = null;
> StoreFile.Writer halfWriter = null;
> try {
> halfReader = new HalfStoreFileReader(fs, inFile, cacheConf, reference,
> conf);
> Map
[jira] [Updated] (HBASE-13590) TestEnableTableHandler.testEnableTableWithNoRegionServers is flakey
[
https://issues.apache.org/jira/browse/HBASE-13590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated HBASE-13590:
--
Attachment: HBASE-13590.branch-1.patch
[~ndimiduk] we encountered exactly the same problem with 1.1.2, and after a
carefully debug we've made sure it's the same as reported in HBASE-13041, and
since patch for HBASE-13041 only goes into trunk, branch-1 still has the
problem.
We were able to see this UT failure stably at a one-tenth chance by running
below cases together:
{{TestMasterOperationsForRegionReplicas,TestEnableTableHandler,TestRateLimiter,TestSplitTransactionOnCluster}},
and confirmed no failure happened in a 100-times round after the patch.
> TestEnableTableHandler.testEnableTableWithNoRegionServers is flakey
> ---
>
> Key: HBASE-13590
> URL: https://issues.apache.org/jira/browse/HBASE-13590
> Project: HBase
> Issue Type: Test
> Components: master
>Reporter: Nick Dimiduk
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4
>
> Attachments: HBASE-13590.branch-1.patch
>
>
> Looking at our [build
> history|https://builds.apache.org/job/HBase-1.1/buildTimeTrend], it seems
> this test is flakey. See builds 429, 431, 439.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15075) Allow region split request to carry metadata
[ https://issues.apache.org/jira/browse/HBASE-15075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-15075: --- Description: During the process of improving region normalization feature, I found that if region split request triggered by the execution of SplitNormalizationPlan fails, there is no way of knowing whether the failed split originated from region normalization. The association of particular split request with outcome of split would give RegionNormalizer information so that it can make better normalization decisions in the subsequent invocations. One approach is to embed metadata, such as a UUID, in SplitRequest which gets passed through RegionStateTransitionContext when RegionServerServices#reportRegionStateTransition() is called. This way, RegionStateListener can be notified with the metadata (id of the requester). See discussion on dev mailing list http://search-hadoop.com/m/YGbbCXdkivihp2 was: During the process of improving region normalization feature, I found that if region split request triggered by the execution of SplitNormalizationPlan fails, there is no way of knowing whether the failed split originated from region normalization. The association of particular split request with outcome of split would give RegionNormalizer information so that it can make better normalization decisions in the subsequent invocations. One approach is to embed metadata in SplitRequest which gets passed through RegionStateTransitionContext when RegionServerServices#reportRegionStateTransition() is called. This way, RegionStateListener can be notified with the metadata (id of the requester). See discussion on dev mailing list http://search-hadoop.com/m/YGbbCXdkivihp2 > Allow region split request to carry metadata > > > Key: HBASE-15075 > URL: https://issues.apache.org/jira/browse/HBASE-15075 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 15075-v0.txt, 15075-v1.txt, 15075-v2.txt, > HBASE-15075.v2.patch, HBASE-15075.v3.patch > > > During the process of improving region normalization feature, I found that if > region split request triggered by the execution of SplitNormalizationPlan > fails, there is no way of knowing whether the failed split originated from > region normalization. > The association of particular split request with outcome of split would give > RegionNormalizer information so that it can make better normalization > decisions in the subsequent invocations. > One approach is to embed metadata, such as a UUID, in SplitRequest which gets > passed through RegionStateTransitionContext when > RegionServerServices#reportRegionStateTransition() is called. > This way, RegionStateListener can be notified with the metadata (id of the > requester). > See discussion on dev mailing list > http://search-hadoop.com/m/YGbbCXdkivihp2 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14962) TestSplitWalDataLoss fails on all branches
[
https://issues.apache.org/jira/browse/HBASE-14962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15093240#comment-15093240
]
Elliott Clark commented on HBASE-14962:
---
Hmmm I just had a run of 10 pass. So this might be a little less frequent than
I first thought. Trying on linux more.
> TestSplitWalDataLoss fails on all branches
> --
>
> Key: HBASE-14962
> URL: https://issues.apache.org/jira/browse/HBASE-14962
> Project: HBase
> Issue Type: Sub-task
>Reporter: Elliott Clark
>Assignee: stack
>Priority: Blocker
> Fix For: 1.2.0
>
> Attachments:
> org.apache.hadoop.hbase.regionserver.TestSplitWalDataLoss-output.txt
>
>
> With some regularity I am seeing:
> {code}
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1
> action: TestSplitWalDataLoss:dataloss: 1 time,
> at
> org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:228)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1800(AsyncProcess.java:208)
> at
> org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1712)
> at
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:240)
> at
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:190)
> at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1430)
> at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1021)
> at
> org.apache.hadoop.hbase.regionserver.TestSplitWalDataLoss.test(TestSplitWalDataLoss.java:121)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14375) define public API for spark integration module
[ https://issues.apache.org/jira/browse/HBASE-14375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15093286#comment-15093286 ] Appy commented on HBASE-14375: -- Do we already know which classes are going to be public. Hopefully there aren't any gray areas which need discussion. :) > define public API for spark integration module > -- > > Key: HBASE-14375 > URL: https://issues.apache.org/jira/browse/HBASE-14375 > Project: HBase > Issue Type: Task > Components: spark >Reporter: Sean Busbey >Priority: Critical > > before we can put the spark integration module into a release, we need to > annotate its public api surface. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14962) TestSplitWalDataLoss fails on all branches
[
https://issues.apache.org/jira/browse/HBASE-14962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elliott Clark updated HBASE-14962:
--
Attachment:
org.apache.hadoop.hbase.regionserver.TestSplitWalDataLoss-output.txt
Another log.
> TestSplitWalDataLoss fails on all branches
> --
>
> Key: HBASE-14962
> URL: https://issues.apache.org/jira/browse/HBASE-14962
> Project: HBase
> Issue Type: Sub-task
>Reporter: Elliott Clark
>Assignee: stack
>Priority: Blocker
> Fix For: 1.2.0
>
> Attachments:
> org.apache.hadoop.hbase.regionserver.TestSplitWalDataLoss-output.txt,
> org.apache.hadoop.hbase.regionserver.TestSplitWalDataLoss-output.txt
>
>
> With some regularity I am seeing:
> {code}
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1
> action: TestSplitWalDataLoss:dataloss: 1 time,
> at
> org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:228)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1800(AsyncProcess.java:208)
> at
> org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1712)
> at
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:240)
> at
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:190)
> at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1430)
> at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1021)
> at
> org.apache.hadoop.hbase.regionserver.TestSplitWalDataLoss.test(TestSplitWalDataLoss.java:121)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14167) hbase-spark integration tests do not respect -DskipIntegrationTests
[
https://issues.apache.org/jira/browse/HBASE-14167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15092685#comment-15092685
]
Hadoop QA commented on HBASE-14167:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
24s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 56s
{color} | {color:green} master passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 6s
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s
{color} | {color:green} master passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
8s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 6s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 6s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
24m 32s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 55s
{color} | {color:green} hbase-spark in the patch passed with JDK v1.8.0_66.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 7s
{color} | {color:green} hbase-spark in the patch passed with JDK v1.7.0_91.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 39m 23s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.9.1 Server=1.9.1 Image:yetus/hbase:date2016-01-11 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12781636/HBASE-14167-master-v2.patch
|
| JIRA Issue | HBASE-14167 |
| Optional Tests | asflicense javac javadoc unit xml compile |
| uname | Linux 6049a94a59c9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / ec47a81 |
| JDK v1.7.0_91 Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/65/testReport/ |
| modules | C: hbase-spark U: hbase-spark |
| Max memory used | 174MB |
| Powered by | Apache Yetus 0.1.0 http://yetus.apache.org |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/65/console |
This message was
[jira] [Commented] (HBASE-15075) Allow region split request to carry metadata
[
https://issues.apache.org/jira/browse/HBASE-15075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15092708#comment-15092708
]
Hadoop QA commented on HBASE-15075:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 2s
{color} | {color:blue} The patch file was not named according to hbase's naming
conventions. Please see
https://yetus.apache.org/documentation/latest/precommit-patchnames for
instructions. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 9m 1s {color}
| {color:red} Docker failed to build yetus/hbase:date2016-01-11. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12781637/15075-v2.txt |
| JIRA Issue | HBASE-15075 |
| Powered by | Apache Yetus 0.1.0 http://yetus.apache.org |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/66/console |
This message was automatically generated.
> Allow region split request to carry metadata
>
>
> Key: HBASE-15075
> URL: https://issues.apache.org/jira/browse/HBASE-15075
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 15075-v0.txt, 15075-v1.txt, 15075-v2.txt
>
>
> During the process of improving region normalization feature, I found that if
> region split request triggered by the execution of SplitNormalizationPlan
> fails, there is no way of knowing whether the failed split originated from
> region normalization.
> The association of particular split request with outcome of split would give
> RegionNormalizer information so that it can make better normalization
> decisions in the subsequent invocations.
> One approach is to embed metadata in SplitRequest which gets passed through
> RegionStateTransitionContext when
> RegionServerServices#reportRegionStateTransition() is called.
> This way, RegionStateListener can be notified with the metadata (id of the
> requester).
> See discussion on dev mailing list
> http://search-hadoop.com/m/YGbbCXdkivihp2
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15011) turn off the jdk8 javadoc linter. :(
[
https://issues.apache.org/jira/browse/HBASE-15011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-15011:
---
Fix Version/s: 0.98.17
> turn off the jdk8 javadoc linter. :(
>
>
> Key: HBASE-15011
> URL: https://issues.apache.org/jira/browse/HBASE-15011
> Project: HBase
> Issue Type: Bug
> Components: build, documentation
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Blocker
> Labels: beginner
> Fix For: 2.0.0, 1.2.0, 1.3.0, 0.98.17
>
> Attachments: HBASE-15011.1.patch, HBASE-15011.2.patch,
> HBASE-15011.2.patch, HBASE-15011.3.patch
>
>
> there's a new javadoc warning that causes warnings on all of our new patches
> and (I believe) breaks us on jdk8.
> Thanks to [~saint@gmail.com] for chasing it down on HBASE-14849.
> {code}
> 1 warning
> [WARNING] Javadoc Warnings
> [WARNING]
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java:430:
> warning - @param argument "hfilesDir" is not a parameter name.
> [INFO]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[ https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15092601#comment-15092601 ] Eungsop Yoo commented on HBASE-15055: - I see. I will update the patch to trigger MC even if max size is specified. It is expecred that will resolve the problem related to TTL expirarion too. > Major compaction is not triggered when both of TTL and > hbase.hstore.compaction.max.size are set > --- > > Key: HBASE-15055 > URL: https://issues.apache.org/jira/browse/HBASE-15055 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Assignee: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15055-v1.patch, HBASE-15055-v2.patch, > HBASE-15055-v3.patch, HBASE-15055-v4.patch, HBASE-15055-v5.patch, > HBASE-15055-v6.patch, HBASE-15055-v7.patch, HBASE-15055-v8.patch, > HBASE-15055-v9.patch, HBASE-15055.patch > > > Some large files may be skipped by hbase.hstore.compaction.max.size in > candidate selection. It causes skipping of major compaction. So the TTL > expired records are still remained in the disks and keep consuming disks. > To resolve this issue, I suggest that to skip large files only if there is no > TTL expired record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14167) hbase-spark integration tests do not respect -DskipIntegrationTests
[
https://issues.apache.org/jira/browse/HBASE-14167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Appy updated HBASE-14167:
-
Assignee: Gabor Liptak
> hbase-spark integration tests do not respect -DskipIntegrationTests
> ---
>
> Key: HBASE-14167
> URL: https://issues.apache.org/jira/browse/HBASE-14167
> Project: HBase
> Issue Type: Improvement
> Components: spark
>Affects Versions: 2.0.0
>Reporter: Andrew Purtell
>Assignee: Gabor Liptak
>Priority: Minor
> Attachments: HBASE-14167-master-v2.patch, HBASE-14167.11.patch
>
>
> When running a build with {{mvn ... -DskipIntegrationTests}}, the hbase-spark
> module's integration tests do not respect the flag and run anyway. Fix.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14167) hbase-spark integration tests do not respect -DskipIntegrationTests
[
https://issues.apache.org/jira/browse/HBASE-14167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Appy updated HBASE-14167:
-
Attachment: HBASE-14167-master-v2.patch
Just corrected commit message.
> hbase-spark integration tests do not respect -DskipIntegrationTests
> ---
>
> Key: HBASE-14167
> URL: https://issues.apache.org/jira/browse/HBASE-14167
> Project: HBase
> Issue Type: Improvement
> Components: spark
>Affects Versions: 2.0.0
>Reporter: Andrew Purtell
>Priority: Minor
> Attachments: HBASE-14167-master-v2.patch, HBASE-14167.11.patch
>
>
> When running a build with {{mvn ... -DskipIntegrationTests}}, the hbase-spark
> module's integration tests do not respect the flag and run anyway. Fix.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14167) hbase-spark integration tests do not respect -DskipIntegrationTests
[
https://issues.apache.org/jira/browse/HBASE-14167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15092633#comment-15092633
]
Appy commented on HBASE-14167:
--
[~apurtell] should be good to go in now?
> hbase-spark integration tests do not respect -DskipIntegrationTests
> ---
>
> Key: HBASE-14167
> URL: https://issues.apache.org/jira/browse/HBASE-14167
> Project: HBase
> Issue Type: Improvement
> Components: spark
>Affects Versions: 2.0.0
>Reporter: Andrew Purtell
>Assignee: Gabor Liptak
>Priority: Minor
> Attachments: HBASE-14167-master-v2.patch, HBASE-14167.11.patch
>
>
> When running a build with {{mvn ... -DskipIntegrationTests}}, the hbase-spark
> module's integration tests do not respect the flag and run anyway. Fix.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15075) Allow region split request to carry metadata
[ https://issues.apache.org/jira/browse/HBASE-15075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-15075: --- Attachment: 15075-v2.txt > Allow region split request to carry metadata > > > Key: HBASE-15075 > URL: https://issues.apache.org/jira/browse/HBASE-15075 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 15075-v0.txt, 15075-v1.txt, 15075-v2.txt > > > During the process of improving region normalization feature, I found that if > region split request triggered by the execution of SplitNormalizationPlan > fails, there is no way of knowing whether the failed split originated from > region normalization. > The association of particular split request with outcome of split would give > RegionNormalizer information so that it can make better normalization > decisions in the subsequent invocations. > One approach is to embed metadata in SplitRequest which gets passed through > RegionStateTransitionContext when > RegionServerServices#reportRegionStateTransition() is called. > This way, RegionStateListener can be notified with the metadata (id of the > requester). > See discussion on dev mailing list > http://search-hadoop.com/m/YGbbCXdkivihp2 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
