[jira] [Commented] (HBASE-17088) Refactor RWQueueRpcExecutor/BalancedQueueRpcExecutor/RpcExecutor
[
https://issues.apache.org/jira/browse/HBASE-17088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15676046#comment-15676046
]
Duo Zhang commented on HBASE-17088:
---
[~zghaobac] Could you please prepare a patch for branch-1? Be careful that the
{{BalancedQueueRpcExecutor}} and {{RWQueueRpcExecutor}} and marked as
LimitedPrivate to coprocessor and phoenix, so you'd better keep the
constructors which is removed in the patch for master in the patch for branch-1
and mark them as deprecated.
Thanks.
> Refactor RWQueueRpcExecutor/BalancedQueueRpcExecutor/RpcExecutor
>
>
> Key: HBASE-17088
> URL: https://issues.apache.org/jira/browse/HBASE-17088
> Project: HBase
> Issue Type: Improvement
> Components: rpc
>Affects Versions: 2.0.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-17088-v1.patch, HBASE-17088-v2.patch,
> HBASE-17088-v3.patch, HBASE-17088-v3.patch, HBASE-17088-v4.patch,
> HBASE-17088-v4.patch
>
>
> 1. The RWQueueRpcExecutor has eight constructor method and the longest one
> has ten parameters. But It is only used in SimpleRpcScheduler and easy to
> confused when read the code.
> 2. There are duplicate method implement in RWQueueRpcExecutor and
> BalancedQueueRpcExecutor. They can be implemented in their parent class
> RpcExecutor.
> 3. SimpleRpcScheduler read many configs to new RpcExecutor. But the
> CALL_QUEUE_SCAN_SHARE_CONF_KEY is only needed by RWQueueRpcExecutor. And
> CALL_QUEUE_CODEL_TARGET_DELAY, CALL_QUEUE_CODEL_INTERVAL and
> CALL_QUEUE_CODEL_LIFO_THRESHOLD are only needed by AdaptiveLifoCoDelCallQueue.
> So I thought we can refactor it. Suggestions are welcome.
> Review board: https://reviews.apache.org/r/53726/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16820) BulkLoad mvcc visibility only works accidentally
[
https://issues.apache.org/jira/browse/HBASE-16820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675968#comment-15675968
]
Hadoop QA commented on HBASE-16820:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 27s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
2s {color} | {color:green} branch-1.1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s
{color} | {color:green} branch-1.1 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} branch-1.1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s {color} | {color:green} branch-1.1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s {color} | {color:green} branch-1.1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 48s
{color} | {color:red} hbase-server in branch-1.1 has 80 extant Findbugs
warnings. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 33s
{color} | {color:red} hbase-server in branch-1.1 failed with JDK v1.8.0_111.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s
{color} | {color:green} branch-1.1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
11m 52s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 6s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 24s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0_111.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 88m 37s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 120m 33s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestSplitWalDataLoss |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8012383 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12833147/HBASE-16820-branch-1.1-v0.patch
|
| JIRA Issue | HBASE-16820 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux a2c91f751adf 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05
[jira] [Commented] (HBASE-17124) HFileInputFormat can help user read hbase table with HFile way,more effective
[ https://issues.apache.org/jira/browse/HBASE-17124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675956#comment-15675956 ] Anoop Sam John commented on HBASE-17124: We do have MapReduce over snapshot files.. Eager to hear more abt this... Mind adding a design doc or so? > HFileInputFormat can help user read hbase table with HFile way,more effective > - > > Key: HBASE-17124 > URL: https://issues.apache.org/jira/browse/HBASE-17124 > Project: HBase > Issue Type: New Feature > Components: HFile >Reporter: Yuan Kang >Assignee: Yuan Kang > > When we need to read large amount of data from hbase,usually we use > tableinputformat to query hbase table. but hfileInputFormat way is more > effective -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16989) RowProcess#postBatchMutate doesn’t be executed before the mvcc transaction completion
[
https://issues.apache.org/jira/browse/HBASE-16989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ChiaPing Tsai updated HBASE-16989:
--
Status: Patch Available (was: Open)
> RowProcess#postBatchMutate doesn’t be executed before the mvcc transaction
> completion
> -
>
> Key: HBASE-16989
> URL: https://issues.apache.org/jira/browse/HBASE-16989
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
> Fix For: 2.0.0
>
> Attachments: HBASE-16989.v0.patch, HBASE-16989.v1.patch,
> HBASE-16989.v2.patch, HBASE-16989.v3.patch, HBASE-16989.v4.patch
>
>
> After the [HBASE-15158|https://issues.apache.org/jira/browse/HBASE-15158],
> RowProcess#postBatchMutate will be executed “after” the mvcc transaction
> completion.
> {code:title=HRegion#processRowsWithLocks}
> // STEP 8. Complete mvcc.
> mvcc.completeAndWait(writeEntry);
> writeEntry = null;
>
> // STEP 9. Release region lock
> if (locked) {
> this.updatesLock.readLock().unlock();
> locked = false;
> }
>
> // STEP 10. Release row lock(s)
> releaseRowLocks(acquiredRowLocks);
>
> // STEP 11. call postBatchMutate hook
> processor.postBatchMutate(this);
> {code}
> {code:title=RowProcess#postBatchMutate}
> /**
>* The hook to be executed after the process() and applying the Mutations
> to region. The
>* difference of this one with {@link #postProcess(HRegion, WALEdit,
> boolean)} is this hook will
>* be executed before the mvcc transaction completion.
>*/
> void postBatchMutate(HRegion region) throws IOException;
> {code}
> Do we ought to revamp the comment of RowProcess#postBatchMutate or change the
> call order?
> I prefer the former, because the HRegion#doMiniBatchMutate() also call
> postBatchMutate() after the mvcc transaction completion.
> Any comment? Thanks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16989) RowProcess#postBatchMutate doesn’t be executed before the mvcc transaction completion
[
https://issues.apache.org/jira/browse/HBASE-16989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ChiaPing Tsai updated HBASE-16989:
--
Attachment: HBASE-16989.v4.patch
TestMemStoreLAB pass locally. Retry
> RowProcess#postBatchMutate doesn’t be executed before the mvcc transaction
> completion
> -
>
> Key: HBASE-16989
> URL: https://issues.apache.org/jira/browse/HBASE-16989
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
> Fix For: 2.0.0
>
> Attachments: HBASE-16989.v0.patch, HBASE-16989.v1.patch,
> HBASE-16989.v2.patch, HBASE-16989.v3.patch, HBASE-16989.v4.patch
>
>
> After the [HBASE-15158|https://issues.apache.org/jira/browse/HBASE-15158],
> RowProcess#postBatchMutate will be executed “after” the mvcc transaction
> completion.
> {code:title=HRegion#processRowsWithLocks}
> // STEP 8. Complete mvcc.
> mvcc.completeAndWait(writeEntry);
> writeEntry = null;
>
> // STEP 9. Release region lock
> if (locked) {
> this.updatesLock.readLock().unlock();
> locked = false;
> }
>
> // STEP 10. Release row lock(s)
> releaseRowLocks(acquiredRowLocks);
>
> // STEP 11. call postBatchMutate hook
> processor.postBatchMutate(this);
> {code}
> {code:title=RowProcess#postBatchMutate}
> /**
>* The hook to be executed after the process() and applying the Mutations
> to region. The
>* difference of this one with {@link #postProcess(HRegion, WALEdit,
> boolean)} is this hook will
>* be executed before the mvcc transaction completion.
>*/
> void postBatchMutate(HRegion region) throws IOException;
> {code}
> Do we ought to revamp the comment of RowProcess#postBatchMutate or change the
> call order?
> I prefer the former, because the HRegion#doMiniBatchMutate() also call
> postBatchMutate() after the mvcc transaction completion.
> Any comment? Thanks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16989) RowProcess#postBatchMutate doesn’t be executed before the mvcc transaction completion
[
https://issues.apache.org/jira/browse/HBASE-16989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ChiaPing Tsai updated HBASE-16989:
--
Status: Open (was: Patch Available)
> RowProcess#postBatchMutate doesn’t be executed before the mvcc transaction
> completion
> -
>
> Key: HBASE-16989
> URL: https://issues.apache.org/jira/browse/HBASE-16989
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
> Fix For: 2.0.0
>
> Attachments: HBASE-16989.v0.patch, HBASE-16989.v1.patch,
> HBASE-16989.v2.patch, HBASE-16989.v3.patch
>
>
> After the [HBASE-15158|https://issues.apache.org/jira/browse/HBASE-15158],
> RowProcess#postBatchMutate will be executed “after” the mvcc transaction
> completion.
> {code:title=HRegion#processRowsWithLocks}
> // STEP 8. Complete mvcc.
> mvcc.completeAndWait(writeEntry);
> writeEntry = null;
>
> // STEP 9. Release region lock
> if (locked) {
> this.updatesLock.readLock().unlock();
> locked = false;
> }
>
> // STEP 10. Release row lock(s)
> releaseRowLocks(acquiredRowLocks);
>
> // STEP 11. call postBatchMutate hook
> processor.postBatchMutate(this);
> {code}
> {code:title=RowProcess#postBatchMutate}
> /**
>* The hook to be executed after the process() and applying the Mutations
> to region. The
>* difference of this one with {@link #postProcess(HRegion, WALEdit,
> boolean)} is this hook will
>* be executed before the mvcc transaction completion.
>*/
> void postBatchMutate(HRegion region) throws IOException;
> {code}
> Do we ought to revamp the comment of RowProcess#postBatchMutate or change the
> call order?
> I prefer the former, because the HRegion#doMiniBatchMutate() also call
> postBatchMutate() after the mvcc transaction completion.
> Any comment? Thanks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17080) rest.TestTableResource fails in master branch
[
https://issues.apache.org/jira/browse/HBASE-17080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675918#comment-15675918
]
Hadoop QA commented on HBASE-17080:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 9s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
11s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
32s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 15s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
37s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 3s
{color} | {color:green} hbase-rest in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
7s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 35m 43s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12839504/HBASE-17080.v0.patch |
| JIRA Issue | HBASE-17080 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 101a4f46fbd5 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 046d1a5 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4525/testReport/ |
| modules | C: hbase-rest U: hbase-rest |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4525/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> rest.TestTableResource fails in master branch
> -
>
> Key: HBASE-17080
> URL: https://issues.apache.org/jira/browse/HBASE-17080
> Project: HBase
> Issue Type: Test
>Affects Versions: 2.0.0
>Reporter: Ted Yu
>Assignee: ChiaPing Tsai
> Fix For: 2.0.0
>
> Attachments: HBASE-17080.v0.patch
>
>
> The test fails consistently.
> {code}
>
[jira] [Assigned] (HBASE-17080) rest.TestTableResource fails in master branch
[
https://issues.apache.org/jira/browse/HBASE-17080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ChiaPing Tsai reassigned HBASE-17080:
-
Assignee: ChiaPing Tsai
> rest.TestTableResource fails in master branch
> -
>
> Key: HBASE-17080
> URL: https://issues.apache.org/jira/browse/HBASE-17080
> Project: HBase
> Issue Type: Test
>Affects Versions: 2.0.0
>Reporter: Ted Yu
>Assignee: ChiaPing Tsai
> Fix For: 2.0.0
>
> Attachments: HBASE-17080.v0.patch
>
>
> The test fails consistently.
> {code}
> testTableInfoPB(org.apache.hadoop.hbase.rest.TestTableResource) Time
> elapsed: 0.118 sec <<< FAILURE!
> org.junit.ComparisonFailure: expected:<192.168.0.14:5055[8]> but
> was:<192.168.0.14:5055[5]>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.hbase.rest.TestTableResource.checkTableInfo(TestTableResource.java:191)
> at
> org.apache.hadoop.hbase.rest.TestTableResource.testTableInfoPB(TestTableResource.java:272)
> testTableInfoXML(org.apache.hadoop.hbase.rest.TestTableResource) Time
> elapsed: 0.084 sec <<< FAILURE!
> org.junit.ComparisonFailure: expected:<192.168.0.14:5055[8]> but
> was:<192.168.0.14:5055[5]>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.hbase.rest.TestTableResource.checkTableInfo(TestTableResource.java:191)
> at
> org.apache.hadoop.hbase.rest.TestTableResource.testTableInfoXML(TestTableResource.java:255)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17080) rest.TestTableResource fails in master branch
[
https://issues.apache.org/jira/browse/HBASE-17080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ChiaPing Tsai updated HBASE-17080:
--
Fix Version/s: 2.0.0
Affects Version/s: 2.0.0
Status: Patch Available (was: Open)
> rest.TestTableResource fails in master branch
> -
>
> Key: HBASE-17080
> URL: https://issues.apache.org/jira/browse/HBASE-17080
> Project: HBase
> Issue Type: Test
>Affects Versions: 2.0.0
>Reporter: Ted Yu
> Fix For: 2.0.0
>
> Attachments: HBASE-17080.v0.patch
>
>
> The test fails consistently.
> {code}
> testTableInfoPB(org.apache.hadoop.hbase.rest.TestTableResource) Time
> elapsed: 0.118 sec <<< FAILURE!
> org.junit.ComparisonFailure: expected:<192.168.0.14:5055[8]> but
> was:<192.168.0.14:5055[5]>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.hbase.rest.TestTableResource.checkTableInfo(TestTableResource.java:191)
> at
> org.apache.hadoop.hbase.rest.TestTableResource.testTableInfoPB(TestTableResource.java:272)
> testTableInfoXML(org.apache.hadoop.hbase.rest.TestTableResource) Time
> elapsed: 0.084 sec <<< FAILURE!
> org.junit.ComparisonFailure: expected:<192.168.0.14:5055[8]> but
> was:<192.168.0.14:5055[5]>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.hbase.rest.TestTableResource.checkTableInfo(TestTableResource.java:191)
> at
> org.apache.hadoop.hbase.rest.TestTableResource.testTableInfoXML(TestTableResource.java:255)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17080) rest.TestTableResource fails in master branch
[
https://issues.apache.org/jira/browse/HBASE-17080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ChiaPing Tsai updated HBASE-17080:
--
Attachment: HBASE-17080.v0.patch
We need to wait for the open of daughter regions; otherwise, we will retrieve
the transition information.
> rest.TestTableResource fails in master branch
> -
>
> Key: HBASE-17080
> URL: https://issues.apache.org/jira/browse/HBASE-17080
> Project: HBase
> Issue Type: Test
>Affects Versions: 2.0.0
>Reporter: Ted Yu
> Fix For: 2.0.0
>
> Attachments: HBASE-17080.v0.patch
>
>
> The test fails consistently.
> {code}
> testTableInfoPB(org.apache.hadoop.hbase.rest.TestTableResource) Time
> elapsed: 0.118 sec <<< FAILURE!
> org.junit.ComparisonFailure: expected:<192.168.0.14:5055[8]> but
> was:<192.168.0.14:5055[5]>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.hbase.rest.TestTableResource.checkTableInfo(TestTableResource.java:191)
> at
> org.apache.hadoop.hbase.rest.TestTableResource.testTableInfoPB(TestTableResource.java:272)
> testTableInfoXML(org.apache.hadoop.hbase.rest.TestTableResource) Time
> elapsed: 0.084 sec <<< FAILURE!
> org.junit.ComparisonFailure: expected:<192.168.0.14:5055[8]> but
> was:<192.168.0.14:5055[5]>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.hbase.rest.TestTableResource.checkTableInfo(TestTableResource.java:191)
> at
> org.apache.hadoop.hbase.rest.TestTableResource.testTableInfoXML(TestTableResource.java:255)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17123) Add postBulkLoadHFile variant that notifies the final paths for the hfiles
[
https://issues.apache.org/jira/browse/HBASE-17123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-17123:
---
Attachment: 17123.v1.txt
Draft showing what I meant.
> Add postBulkLoadHFile variant that notifies the final paths for the hfiles
> --
>
> Key: HBASE-17123
> URL: https://issues.apache.org/jira/browse/HBASE-17123
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
> Attachments: 17123.v1.txt
>
>
> Currently the postBulkLoadHFile() hook passes the same familyPaths parameter
> which it receives as method parameter.
> See code in SecureBulkLoadManager :
> {code}
>loaded = region.getCoprocessorHost().postBulkLoadHFile(familyPaths,
> loaded);
> {code}
> Meaning, the paths are not final, moved paths of the loaded hfiles.
> This issue is to add a variant which notifies the final paths of the loaded
> hfiles.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17072) CPU usage starts to climb up to 90-100% when using G1GC
[
https://issues.apache.org/jira/browse/HBASE-17072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675813#comment-15675813
]
stack commented on HBASE-17072:
---
I am in awe of the digging done here (even after second read). Seems like only
time this 'optimization' could cut in is when long, streaming read --
compaction as said above or a long scan with lots of data skipping. We take in
size as a param for case where an intermediate block came from cache; in that
case, we can get the next blocks size from the previous block we got from
cache. Let me see if I can use this param to preserve the long-scan/compaction
benefit w/o threadlocal
> CPU usage starts to climb up to 90-100% when using G1GC
> ---
>
> Key: HBASE-17072
> URL: https://issues.apache.org/jira/browse/HBASE-17072
> Project: HBase
> Issue Type: Bug
> Components: Performance, regionserver
>Affects Versions: 1.0.0, 1.2.0
>Reporter: Eiichi Sato
> Attachments: disable-block-header-cache.patch, mat-threadlocals.png,
> mat-threads.png, metrics.png, slave1.svg, slave2.svg, slave3.svg, slave4.svg
>
>
> h5. Problem
> CPU usage of a region server in our CDH 5.4.5 cluster, at some point, starts
> to gradually get higher up to nearly 90-100% when using G1GC. We've also run
> into this problem on CDH 5.7.3 and CDH 5.8.2.
> In our production cluster, it normally takes a few weeks for this to happen
> after restarting a RS. We reproduced this on our test cluster and attached
> the results. Please note that, to make it easy to reproduce, we did some
> "anti-tuning" on a table when running tests.
> In metrics.png, soon after we started running some workloads against a test
> cluster (CDH 5.8.2) at about 7 p.m. CPU usage of the two RSs started to rise.
> Flame Graphs (slave1.svg to slave4.svg) are generated from jstack dumps of
> each RS process around 10:30 a.m. the next day.
> After investigating heapdumps from another occurrence on a test cluster
> running CDH 5.7.3, we found that the ThreadLocalMap contain a lot of
> contiguous entries of {{HFileBlock$PrefetchedHeader}} probably due to primary
> clustering. This caused more loops in
> {{ThreadLocalMap#expungeStaleEntries()}}, consuming a certain amount of CPU
> time. What is worse is that the method is called from RPC metrics code,
> which means even a small amount of per-RPC time soon adds up to a huge amount
> of CPU time.
> This is very similar to the issue in HBASE-16616, but we have many
> {{HFileBlock$PrefetchedHeader}} not only {{Counter$IndexHolder}} instances.
> Here are some OQL counts from Eclipse Memory Analyzer (MAT). This shows a
> number of ThreadLocal instances in the ThreadLocalMap of a single handler
> thread.
> {code}
> SELECT *
> FROM OBJECTS (SELECT AS RETAINED SET OBJECTS value
> FROM OBJECTS 0x4ee380430) obj
> WHERE obj.@clazz.@name =
> "org.apache.hadoop.hbase.io.hfile.HFileBlock$PrefetchedHeader"
> #=> 10980 instances
> {code}
> {code}
> SELECT *
> FROM OBJECTS (SELECT AS RETAINED SET OBJECTS value
> FROM OBJECTS 0x4ee380430) obj
> WHERE obj.@clazz.@name = "org.apache.hadoop.hbase.util.Counter$IndexHolder"
> #=> 2052 instances
> {code}
> Although as described in HBASE-16616 this somewhat seems to be an issue in
> G1GC side regarding weakly-reachable objects, we should keep ThreadLocal
> usage minimal and avoid creating an indefinite number (in this case, a number
> of HFiles) of ThreadLocal instances.
> HBASE-16146 removes ThreadLocals from the RPC metrics code. That may solve
> the issue (I just saw the patch, never tested it at all), but the
> {{HFileBlock$PrefetchedHeader}} are still there in the ThreadLocalMap, which
> may cause issues in the future again.
> h5. Our Solution
> We simply removed the whole {{HFileBlock$PrefetchedHeader}} caching and
> fortunately we didn't notice any performance degradation for our production
> workloads.
> Because the PrefetchedHeader caching uses ThreadLocal and because RPCs are
> handled randomly in any of the handlers, small Get or small Scan RPCs do not
> benefit from the caching (See HBASE-10676 and HBASE-11402 for the details).
> Probably, we need to see how well reads are saved by the caching for large
> Scan or Get RPCs and especially for compactions if we really remove the
> caching. It's probably better if we can remove ThreadLocals without breaking
> the current caching behavior.
> FWIW, I'm attaching the patch we applied. It's for CDH 5.4.5.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17080) rest.TestTableResource fails in master branch
[
https://issues.apache.org/jira/browse/HBASE-17080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675800#comment-15675800
]
Ted Yu commented on HBASE-17080:
Please.
Thanks
> rest.TestTableResource fails in master branch
> -
>
> Key: HBASE-17080
> URL: https://issues.apache.org/jira/browse/HBASE-17080
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>
> The test fails consistently.
> {code}
> testTableInfoPB(org.apache.hadoop.hbase.rest.TestTableResource) Time
> elapsed: 0.118 sec <<< FAILURE!
> org.junit.ComparisonFailure: expected:<192.168.0.14:5055[8]> but
> was:<192.168.0.14:5055[5]>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.hbase.rest.TestTableResource.checkTableInfo(TestTableResource.java:191)
> at
> org.apache.hadoop.hbase.rest.TestTableResource.testTableInfoPB(TestTableResource.java:272)
> testTableInfoXML(org.apache.hadoop.hbase.rest.TestTableResource) Time
> elapsed: 0.084 sec <<< FAILURE!
> org.junit.ComparisonFailure: expected:<192.168.0.14:5055[8]> but
> was:<192.168.0.14:5055[5]>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.hbase.rest.TestTableResource.checkTableInfo(TestTableResource.java:191)
> at
> org.apache.hadoop.hbase.rest.TestTableResource.testTableInfoXML(TestTableResource.java:255)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17080) rest.TestTableResource fails in master branch
[
https://issues.apache.org/jira/browse/HBASE-17080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675787#comment-15675787
]
ChiaPing Tsai commented on HBASE-17080:
---
[~yuzhih...@gmail.com]
Can i give a patch? Thanks.
> rest.TestTableResource fails in master branch
> -
>
> Key: HBASE-17080
> URL: https://issues.apache.org/jira/browse/HBASE-17080
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>
> The test fails consistently.
> {code}
> testTableInfoPB(org.apache.hadoop.hbase.rest.TestTableResource) Time
> elapsed: 0.118 sec <<< FAILURE!
> org.junit.ComparisonFailure: expected:<192.168.0.14:5055[8]> but
> was:<192.168.0.14:5055[5]>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.hbase.rest.TestTableResource.checkTableInfo(TestTableResource.java:191)
> at
> org.apache.hadoop.hbase.rest.TestTableResource.testTableInfoPB(TestTableResource.java:272)
> testTableInfoXML(org.apache.hadoop.hbase.rest.TestTableResource) Time
> elapsed: 0.084 sec <<< FAILURE!
> org.junit.ComparisonFailure: expected:<192.168.0.14:5055[8]> but
> was:<192.168.0.14:5055[5]>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.hbase.rest.TestTableResource.checkTableInfo(TestTableResource.java:191)
> at
> org.apache.hadoop.hbase.rest.TestTableResource.testTableInfoXML(TestTableResource.java:255)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16708) Expose endpoint Coprocessor name in "responseTooSlow" log messages
[ https://issues.apache.org/jira/browse/HBASE-16708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675775#comment-15675775 ] Nick Dimiduk commented on HBASE-16708: -- Makes sense -- thanks for humoring me [~easyliangjob]. This is great! > Expose endpoint Coprocessor name in "responseTooSlow" log messages > -- > > Key: HBASE-16708 > URL: https://issues.apache.org/jira/browse/HBASE-16708 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Nick Dimiduk >Assignee: Yi Liang > Fix For: 2.0.0 > > Attachments: HBASE-16708-V1.patch, HBASE-16708-V2.patch, > HBASE-ShortName.patch > > > Operational diagnostics of a Phoenix install would be easier if we included > which endpoint coprocessor was being called in this responseTooSlow WARN > message. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15983) Replication improperly discards data from end-of-wal in some cases.
[ https://issues.apache.org/jira/browse/HBASE-15983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-15983: - Fix Version/s: (was: 1.1.8) 1.1.9 > Replication improperly discards data from end-of-wal in some cases. > --- > > Key: HBASE-15983 > URL: https://issues.apache.org/jira/browse/HBASE-15983 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 0.98.0, 1.0.0, 1.1.0, 1.2.0 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Critical > Fix For: 2.0.0, 1.4.0, 1.3.1, 0.98.23, 1.2.5, 1.1.9 > > > In some particular deployments, the Replication code believes it has > reached EOF for a WAL prior to successfully parsing all bytes known to > exist in a cleanly closed file. > The underlying issue is that several different underlying problems with a WAL > reader are all treated as end-of-file by the code in ReplicationSource that > decides if a given WAL is completed or needs to be retried. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17075) Old regionStates are not clearing for truncated tables
[ https://issues.apache.org/jira/browse/HBASE-17075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675756#comment-15675756 ] Nick Dimiduk commented on HBASE-17075: -- Can we close this as a dupe of HBASE-16649? > Old regionStates are not clearing for truncated tables > --- > > Key: HBASE-17075 > URL: https://issues.apache.org/jira/browse/HBASE-17075 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Affects Versions: 1.1.8 >Reporter: Y. SREENIVASULU REDDY >Priority: Minor > Fix For: 1.1.8 > > Attachments: HBASE-17075-branch-1.1.patch, HBASE-17075.001.patch > > > For truncated tables, not clearing the region states from master in-memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16820) BulkLoad mvcc visibility only works accidentally
[ https://issues.apache.org/jira/browse/HBASE-16820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675750#comment-15675750 ] Nick Dimiduk commented on HBASE-16820: -- Marking as blocker for 1.1.8. > BulkLoad mvcc visibility only works accidentally > - > > Key: HBASE-16820 > URL: https://issues.apache.org/jira/browse/HBASE-16820 > Project: HBase > Issue Type: Bug >Affects Versions: 1.1.8 >Reporter: Enis Soztutar >Assignee: Enis Soztutar >Priority: Blocker > Fix For: 1.1.8 > > Attachments: HBASE-16820-branch-1.1-v0.patch > > > [~sergey.soldatov] has been debugging an issue with a 1.1 code base where the > commit for HBASE-16721 broke the bulk load visibility. After bulk load, the > bulk load files is not visible because the sequence id assigned to the bulk > load is not advanced in mvcc. > Debugging further, we have noticed that bulk load behavior is wrong, but it > works "accidentally" in all code bases (but broken in 1.1 after HBASE-16721). > Let me explain: > - BL request can optionally request a flush before hand (this should be the > default) which causes the flush to happen with some sequenceId. The flush > sequence id is one past all the cells' sequenceids. This flush sequence id is > returned as a result to the flush operation. > - BL then uses this particular sequenceId to mark the files, but itself does > not get a new sequenceid of its own, or advance the mvcc number. > - BL completes WITHOUT making sure that the sequence id is visible. > - BL itself though writes entries to the WAL for the BL event, which in 1.2 > code bases goes through the whole mvcc + seqId paths, which makes sure that > earlier sequenceIds (the flush sequenceId) are visible via mvcc. > The problem with 1.1 is that the WAL entries only get sequence ids, but do > not touch mvcc. With the patch for HBASE-16721, we have made it so that the > flushedSequenceId is not used in mvcc as the highest read point (although all > the data is still visible). > BL relying on the flush sequence id is wrong for two reasons: > - BL files are loaded with the flush sequence id from the memstore. This > particular sequence id is used twice for two different things and ends up > being the sequence id for flushed file as well as BL'ed files. > - BL should make sure that it gets a new sequence id and that sequence id is > visible before returning the results. > [~ndimiduk] FYI. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16820) BulkLoad mvcc visibility only works accidentally
[ https://issues.apache.org/jira/browse/HBASE-16820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-16820: - Affects Version/s: 1.1.8 > BulkLoad mvcc visibility only works accidentally > - > > Key: HBASE-16820 > URL: https://issues.apache.org/jira/browse/HBASE-16820 > Project: HBase > Issue Type: Bug >Affects Versions: 1.1.8 >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 1.1.8 > > Attachments: HBASE-16820-branch-1.1-v0.patch > > > [~sergey.soldatov] has been debugging an issue with a 1.1 code base where the > commit for HBASE-16721 broke the bulk load visibility. After bulk load, the > bulk load files is not visible because the sequence id assigned to the bulk > load is not advanced in mvcc. > Debugging further, we have noticed that bulk load behavior is wrong, but it > works "accidentally" in all code bases (but broken in 1.1 after HBASE-16721). > Let me explain: > - BL request can optionally request a flush before hand (this should be the > default) which causes the flush to happen with some sequenceId. The flush > sequence id is one past all the cells' sequenceids. This flush sequence id is > returned as a result to the flush operation. > - BL then uses this particular sequenceId to mark the files, but itself does > not get a new sequenceid of its own, or advance the mvcc number. > - BL completes WITHOUT making sure that the sequence id is visible. > - BL itself though writes entries to the WAL for the BL event, which in 1.2 > code bases goes through the whole mvcc + seqId paths, which makes sure that > earlier sequenceIds (the flush sequenceId) are visible via mvcc. > The problem with 1.1 is that the WAL entries only get sequence ids, but do > not touch mvcc. With the patch for HBASE-16721, we have made it so that the > flushedSequenceId is not used in mvcc as the highest read point (although all > the data is still visible). > BL relying on the flush sequence id is wrong for two reasons: > - BL files are loaded with the flush sequence id from the memstore. This > particular sequence id is used twice for two different things and ends up > being the sequence id for flushed file as well as BL'ed files. > - BL should make sure that it gets a new sequence id and that sequence id is > visible before returning the results. > [~ndimiduk] FYI. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16820) BulkLoad mvcc visibility only works accidentally
[ https://issues.apache.org/jira/browse/HBASE-16820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-16820: - Priority: Blocker (was: Major) > BulkLoad mvcc visibility only works accidentally > - > > Key: HBASE-16820 > URL: https://issues.apache.org/jira/browse/HBASE-16820 > Project: HBase > Issue Type: Bug >Affects Versions: 1.1.8 >Reporter: Enis Soztutar >Assignee: Enis Soztutar >Priority: Blocker > Fix For: 1.1.8 > > Attachments: HBASE-16820-branch-1.1-v0.patch > > > [~sergey.soldatov] has been debugging an issue with a 1.1 code base where the > commit for HBASE-16721 broke the bulk load visibility. After bulk load, the > bulk load files is not visible because the sequence id assigned to the bulk > load is not advanced in mvcc. > Debugging further, we have noticed that bulk load behavior is wrong, but it > works "accidentally" in all code bases (but broken in 1.1 after HBASE-16721). > Let me explain: > - BL request can optionally request a flush before hand (this should be the > default) which causes the flush to happen with some sequenceId. The flush > sequence id is one past all the cells' sequenceids. This flush sequence id is > returned as a result to the flush operation. > - BL then uses this particular sequenceId to mark the files, but itself does > not get a new sequenceid of its own, or advance the mvcc number. > - BL completes WITHOUT making sure that the sequence id is visible. > - BL itself though writes entries to the WAL for the BL event, which in 1.2 > code bases goes through the whole mvcc + seqId paths, which makes sure that > earlier sequenceIds (the flush sequenceId) are visible via mvcc. > The problem with 1.1 is that the WAL entries only get sequence ids, but do > not touch mvcc. With the patch for HBASE-16721, we have made it so that the > flushedSequenceId is not used in mvcc as the highest read point (although all > the data is still visible). > BL relying on the flush sequence id is wrong for two reasons: > - BL files are loaded with the flush sequence id from the memstore. This > particular sequence id is used twice for two different things and ends up > being the sequence id for flushed file as well as BL'ed files. > - BL should make sure that it gets a new sequence id and that sequence id is > visible before returning the results. > [~ndimiduk] FYI. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16820) BulkLoad mvcc visibility only works accidentally
[ https://issues.apache.org/jira/browse/HBASE-16820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-16820: - Fix Version/s: 1.1.8 > BulkLoad mvcc visibility only works accidentally > - > > Key: HBASE-16820 > URL: https://issues.apache.org/jira/browse/HBASE-16820 > Project: HBase > Issue Type: Bug >Affects Versions: 1.1.8 >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 1.1.8 > > Attachments: HBASE-16820-branch-1.1-v0.patch > > > [~sergey.soldatov] has been debugging an issue with a 1.1 code base where the > commit for HBASE-16721 broke the bulk load visibility. After bulk load, the > bulk load files is not visible because the sequence id assigned to the bulk > load is not advanced in mvcc. > Debugging further, we have noticed that bulk load behavior is wrong, but it > works "accidentally" in all code bases (but broken in 1.1 after HBASE-16721). > Let me explain: > - BL request can optionally request a flush before hand (this should be the > default) which causes the flush to happen with some sequenceId. The flush > sequence id is one past all the cells' sequenceids. This flush sequence id is > returned as a result to the flush operation. > - BL then uses this particular sequenceId to mark the files, but itself does > not get a new sequenceid of its own, or advance the mvcc number. > - BL completes WITHOUT making sure that the sequence id is visible. > - BL itself though writes entries to the WAL for the BL event, which in 1.2 > code bases goes through the whole mvcc + seqId paths, which makes sure that > earlier sequenceIds (the flush sequenceId) are visible via mvcc. > The problem with 1.1 is that the WAL entries only get sequence ids, but do > not touch mvcc. With the patch for HBASE-16721, we have made it so that the > flushedSequenceId is not used in mvcc as the highest read point (although all > the data is still visible). > BL relying on the flush sequence id is wrong for two reasons: > - BL files are loaded with the flush sequence id from the memstore. This > particular sequence id is used twice for two different things and ends up > being the sequence id for flushed file as well as BL'ed files. > - BL should make sure that it gets a new sequence id and that sequence id is > visible before returning the results. > [~ndimiduk] FYI. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16820) BulkLoad mvcc visibility only works accidentally
[ https://issues.apache.org/jira/browse/HBASE-16820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-16820: - Status: Patch Available (was: Open) > BulkLoad mvcc visibility only works accidentally > - > > Key: HBASE-16820 > URL: https://issues.apache.org/jira/browse/HBASE-16820 > Project: HBase > Issue Type: Bug >Affects Versions: 1.1.8 >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 1.1.8 > > Attachments: HBASE-16820-branch-1.1-v0.patch > > > [~sergey.soldatov] has been debugging an issue with a 1.1 code base where the > commit for HBASE-16721 broke the bulk load visibility. After bulk load, the > bulk load files is not visible because the sequence id assigned to the bulk > load is not advanced in mvcc. > Debugging further, we have noticed that bulk load behavior is wrong, but it > works "accidentally" in all code bases (but broken in 1.1 after HBASE-16721). > Let me explain: > - BL request can optionally request a flush before hand (this should be the > default) which causes the flush to happen with some sequenceId. The flush > sequence id is one past all the cells' sequenceids. This flush sequence id is > returned as a result to the flush operation. > - BL then uses this particular sequenceId to mark the files, but itself does > not get a new sequenceid of its own, or advance the mvcc number. > - BL completes WITHOUT making sure that the sequence id is visible. > - BL itself though writes entries to the WAL for the BL event, which in 1.2 > code bases goes through the whole mvcc + seqId paths, which makes sure that > earlier sequenceIds (the flush sequenceId) are visible via mvcc. > The problem with 1.1 is that the WAL entries only get sequence ids, but do > not touch mvcc. With the patch for HBASE-16721, we have made it so that the > flushedSequenceId is not used in mvcc as the highest read point (although all > the data is still visible). > BL relying on the flush sequence id is wrong for two reasons: > - BL files are loaded with the flush sequence id from the memstore. This > particular sequence id is used twice for two different things and ends up > being the sequence id for flushed file as well as BL'ed files. > - BL should make sure that it gets a new sequence id and that sequence id is > visible before returning the results. > [~ndimiduk] FYI. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16663) JMX ConnectorServer stopped when unauthorized user try to stop HM/RS/cluster
[
https://issues.apache.org/jira/browse/HBASE-16663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675744#comment-15675744
]
Hadoop QA commented on HBASE-16663:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s {color}
| {color:red} HBASE-16663 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.3.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12839500/16663-branch-1.1.00.patch
|
| JIRA Issue | HBASE-16663 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4523/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> JMX ConnectorServer stopped when unauthorized user try to stop HM/RS/cluster
>
>
> Key: HBASE-16663
> URL: https://issues.apache.org/jira/browse/HBASE-16663
> Project: HBase
> Issue Type: Bug
> Components: metrics, security
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.4, 0.98.24, 1.1.8
>
> Attachments: 16663-branch-1.1.00.patch, HBASE-16663-0.98-V4.patch,
> HBASE-16663-0.98.patch, HBASE-16663-V2.patch, HBASE-16663-V3.patch,
> HBASE-16663-V4.patch, HBASE-16663-branch-1.patch, HBASE-16663.patch
>
>
> After HBASE-16284, unauthorized user will not able allowed to stop
> HM/RS/cluster, but while executing "cpHost.preStopMaster()", ConnectorServer
> will be stopped before AccessController validation.
> hbase-site.xml,
> {noformat}
>
> hbase.coprocessor.master.classes
>
> org.apache.hadoop.hbase.JMXListener,org.apache.hadoop.hbase.security.access.AccessController
>
>
> hbase.coprocessor.regionserver.classes
>
> org.apache.hadoop.hbase.JMXListener,org.apache.hadoop.hbase.security.access.AccessController
>
> {noformat}
> HBaseAdmin.stopMaster(),
> {noformat}
> 2016-09-20 21:12:26,796 INFO
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
> hbase.JMXListener: ConnectorServer stopped!
> 2016-09-20 21:13:55,380 WARN
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
> security.ShellBasedUnixGroupsMapping: got exception trying to get groups for
> user P72981
> ExitCodeException exitCode=1: id: P72981: No such user
> 2016-09-20 21:14:00,495 ERROR
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
> master.MasterRpcServices: Exception occurred while stopping master
> org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
> permissions for user 'P72981' (global, action=ADMIN)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requireGlobalPermission(AccessController.java:546)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requirePermission(AccessController.java:522)
> at
> org.apache.hadoop.hbase.security.access.AccessController.preStopMaster(AccessController.java:1297)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost$68.call(MasterCoprocessorHost.java:821)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.execOperation(MasterCoprocessorHost.java:1188)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.preStopMaster(MasterCoprocessorHost.java:817)
> at org.apache.hadoop.hbase.master.HMaster.stopMaster(HMaster.java:2352)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.stopMaster(MasterRpcServices.java:1364)
> {noformat}
> HBaseAdmin.stopRegionServer(rs-host-port),
> {noformat}
> 2016-09-20 20:59:01,234 INFO
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
> hbase.JMXListener: ConnectorServer stopped!
> 2016-09-20 20:59:01,250 WARN
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
> security.ShellBasedUnixGroupsMapping: got exception trying to get groups for
> user P72981
> ExitCodeException exitCode=1: id: P72981: No such user
> 2016-09-20 20:59:01,253 WARN
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
> regionserver.HRegionServer: The region server did not stop
> org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
> permissions for user 'P72981' (global, action=ADMIN)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requireGlobalPermission(AccessController.java:546)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requirePermission(AccessController.java:522)
> at
>
[jira] [Commented] (HBASE-16820) BulkLoad mvcc visibility only works accidentally
[ https://issues.apache.org/jira/browse/HBASE-16820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675743#comment-15675743 ] Nick Dimiduk commented on HBASE-16820: -- Thanks for the explanation. Where are you guys with this one? We can at least see what QA says about the patch. Do we have a mixed online + bulkload ITBLL variant we can use to test it? > BulkLoad mvcc visibility only works accidentally > - > > Key: HBASE-16820 > URL: https://issues.apache.org/jira/browse/HBASE-16820 > Project: HBase > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Attachments: HBASE-16820-branch-1.1-v0.patch > > > [~sergey.soldatov] has been debugging an issue with a 1.1 code base where the > commit for HBASE-16721 broke the bulk load visibility. After bulk load, the > bulk load files is not visible because the sequence id assigned to the bulk > load is not advanced in mvcc. > Debugging further, we have noticed that bulk load behavior is wrong, but it > works "accidentally" in all code bases (but broken in 1.1 after HBASE-16721). > Let me explain: > - BL request can optionally request a flush before hand (this should be the > default) which causes the flush to happen with some sequenceId. The flush > sequence id is one past all the cells' sequenceids. This flush sequence id is > returned as a result to the flush operation. > - BL then uses this particular sequenceId to mark the files, but itself does > not get a new sequenceid of its own, or advance the mvcc number. > - BL completes WITHOUT making sure that the sequence id is visible. > - BL itself though writes entries to the WAL for the BL event, which in 1.2 > code bases goes through the whole mvcc + seqId paths, which makes sure that > earlier sequenceIds (the flush sequenceId) are visible via mvcc. > The problem with 1.1 is that the WAL entries only get sequence ids, but do > not touch mvcc. With the patch for HBASE-16721, we have made it so that the > flushedSequenceId is not used in mvcc as the highest read point (although all > the data is still visible). > BL relying on the flush sequence id is wrong for two reasons: > - BL files are loaded with the flush sequence id from the memstore. This > particular sequence id is used twice for two different things and ends up > being the sequence id for flushed file as well as BL'ed files. > - BL should make sure that it gets a new sequence id and that sequence id is > visible before returning the results. > [~ndimiduk] FYI. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HBASE-16663) JMX ConnectorServer stopped when unauthorized user try to stop HM/RS/cluster
[
https://issues.apache.org/jira/browse/HBASE-16663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk reopened HBASE-16663:
--
Reopening for 1.1
> JMX ConnectorServer stopped when unauthorized user try to stop HM/RS/cluster
>
>
> Key: HBASE-16663
> URL: https://issues.apache.org/jira/browse/HBASE-16663
> Project: HBase
> Issue Type: Bug
> Components: metrics, security
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.4, 0.98.24, 1.1.8
>
> Attachments: 16663-branch-1.1.00.patch, HBASE-16663-0.98-V4.patch,
> HBASE-16663-0.98.patch, HBASE-16663-V2.patch, HBASE-16663-V3.patch,
> HBASE-16663-V4.patch, HBASE-16663-branch-1.patch, HBASE-16663.patch
>
>
> After HBASE-16284, unauthorized user will not able allowed to stop
> HM/RS/cluster, but while executing "cpHost.preStopMaster()", ConnectorServer
> will be stopped before AccessController validation.
> hbase-site.xml,
> {noformat}
>
> hbase.coprocessor.master.classes
>
> org.apache.hadoop.hbase.JMXListener,org.apache.hadoop.hbase.security.access.AccessController
>
>
> hbase.coprocessor.regionserver.classes
>
> org.apache.hadoop.hbase.JMXListener,org.apache.hadoop.hbase.security.access.AccessController
>
> {noformat}
> HBaseAdmin.stopMaster(),
> {noformat}
> 2016-09-20 21:12:26,796 INFO
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
> hbase.JMXListener: ConnectorServer stopped!
> 2016-09-20 21:13:55,380 WARN
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
> security.ShellBasedUnixGroupsMapping: got exception trying to get groups for
> user P72981
> ExitCodeException exitCode=1: id: P72981: No such user
> 2016-09-20 21:14:00,495 ERROR
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
> master.MasterRpcServices: Exception occurred while stopping master
> org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
> permissions for user 'P72981' (global, action=ADMIN)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requireGlobalPermission(AccessController.java:546)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requirePermission(AccessController.java:522)
> at
> org.apache.hadoop.hbase.security.access.AccessController.preStopMaster(AccessController.java:1297)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost$68.call(MasterCoprocessorHost.java:821)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.execOperation(MasterCoprocessorHost.java:1188)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.preStopMaster(MasterCoprocessorHost.java:817)
> at org.apache.hadoop.hbase.master.HMaster.stopMaster(HMaster.java:2352)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.stopMaster(MasterRpcServices.java:1364)
> {noformat}
> HBaseAdmin.stopRegionServer(rs-host-port),
> {noformat}
> 2016-09-20 20:59:01,234 INFO
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
> hbase.JMXListener: ConnectorServer stopped!
> 2016-09-20 20:59:01,250 WARN
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
> security.ShellBasedUnixGroupsMapping: got exception trying to get groups for
> user P72981
> ExitCodeException exitCode=1: id: P72981: No such user
> 2016-09-20 20:59:01,253 WARN
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
> regionserver.HRegionServer: The region server did not stop
> org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
> permissions for user 'P72981' (global, action=ADMIN)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requireGlobalPermission(AccessController.java:546)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requirePermission(AccessController.java:522)
> at
> org.apache.hadoop.hbase.security.access.AccessController.preStopRegionServer(AccessController.java:2501)
> at
> org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost$1.call(RegionServerCoprocessorHost.java:84)
> at
> org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost.execOperation(RegionServerCoprocessorHost.java:256)
> at
> org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost.preStop(RegionServerCoprocessorHost.java:80)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.stop(HRegionServer.java:1905)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.stopServer(RSRpcServices.java:1961)
> {noformat}
> HBaseAdmin.shutdown(),
> {noformat}
> 2016-09-21 12:09:08,259 INFO
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
> master.MasterRpcServices:
[jira] [Updated] (HBASE-16663) JMX ConnectorServer stopped when unauthorized user try to stop HM/RS/cluster
[
https://issues.apache.org/jira/browse/HBASE-16663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-16663:
-
Attachment: 16663-branch-1.1.00.patch
> JMX ConnectorServer stopped when unauthorized user try to stop HM/RS/cluster
>
>
> Key: HBASE-16663
> URL: https://issues.apache.org/jira/browse/HBASE-16663
> Project: HBase
> Issue Type: Bug
> Components: metrics, security
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.4, 0.98.24, 1.1.8
>
> Attachments: 16663-branch-1.1.00.patch, HBASE-16663-0.98-V4.patch,
> HBASE-16663-0.98.patch, HBASE-16663-V2.patch, HBASE-16663-V3.patch,
> HBASE-16663-V4.patch, HBASE-16663-branch-1.patch, HBASE-16663.patch
>
>
> After HBASE-16284, unauthorized user will not able allowed to stop
> HM/RS/cluster, but while executing "cpHost.preStopMaster()", ConnectorServer
> will be stopped before AccessController validation.
> hbase-site.xml,
> {noformat}
>
> hbase.coprocessor.master.classes
>
> org.apache.hadoop.hbase.JMXListener,org.apache.hadoop.hbase.security.access.AccessController
>
>
> hbase.coprocessor.regionserver.classes
>
> org.apache.hadoop.hbase.JMXListener,org.apache.hadoop.hbase.security.access.AccessController
>
> {noformat}
> HBaseAdmin.stopMaster(),
> {noformat}
> 2016-09-20 21:12:26,796 INFO
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
> hbase.JMXListener: ConnectorServer stopped!
> 2016-09-20 21:13:55,380 WARN
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
> security.ShellBasedUnixGroupsMapping: got exception trying to get groups for
> user P72981
> ExitCodeException exitCode=1: id: P72981: No such user
> 2016-09-20 21:14:00,495 ERROR
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
> master.MasterRpcServices: Exception occurred while stopping master
> org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
> permissions for user 'P72981' (global, action=ADMIN)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requireGlobalPermission(AccessController.java:546)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requirePermission(AccessController.java:522)
> at
> org.apache.hadoop.hbase.security.access.AccessController.preStopMaster(AccessController.java:1297)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost$68.call(MasterCoprocessorHost.java:821)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.execOperation(MasterCoprocessorHost.java:1188)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.preStopMaster(MasterCoprocessorHost.java:817)
> at org.apache.hadoop.hbase.master.HMaster.stopMaster(HMaster.java:2352)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.stopMaster(MasterRpcServices.java:1364)
> {noformat}
> HBaseAdmin.stopRegionServer(rs-host-port),
> {noformat}
> 2016-09-20 20:59:01,234 INFO
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
> hbase.JMXListener: ConnectorServer stopped!
> 2016-09-20 20:59:01,250 WARN
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
> security.ShellBasedUnixGroupsMapping: got exception trying to get groups for
> user P72981
> ExitCodeException exitCode=1: id: P72981: No such user
> 2016-09-20 20:59:01,253 WARN
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
> regionserver.HRegionServer: The region server did not stop
> org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
> permissions for user 'P72981' (global, action=ADMIN)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requireGlobalPermission(AccessController.java:546)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requirePermission(AccessController.java:522)
> at
> org.apache.hadoop.hbase.security.access.AccessController.preStopRegionServer(AccessController.java:2501)
> at
> org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost$1.call(RegionServerCoprocessorHost.java:84)
> at
> org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost.execOperation(RegionServerCoprocessorHost.java:256)
> at
> org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost.preStop(RegionServerCoprocessorHost.java:80)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.stop(HRegionServer.java:1905)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.stopServer(RSRpcServices.java:1961)
> {noformat}
> HBaseAdmin.shutdown(),
> {noformat}
> 2016-09-21 12:09:08,259 INFO
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
>
[jira] [Updated] (HBASE-16663) JMX ConnectorServer stopped when unauthorized user try to stop HM/RS/cluster
[
https://issues.apache.org/jira/browse/HBASE-16663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-16663:
-
Fix Version/s: 1.1.8
Status: Patch Available (was: Reopened)
> JMX ConnectorServer stopped when unauthorized user try to stop HM/RS/cluster
>
>
> Key: HBASE-16663
> URL: https://issues.apache.org/jira/browse/HBASE-16663
> Project: HBase
> Issue Type: Bug
> Components: metrics, security
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.24, 1.1.8, 1.2.4
>
> Attachments: 16663-branch-1.1.00.patch, HBASE-16663-0.98-V4.patch,
> HBASE-16663-0.98.patch, HBASE-16663-V2.patch, HBASE-16663-V3.patch,
> HBASE-16663-V4.patch, HBASE-16663-branch-1.patch, HBASE-16663.patch
>
>
> After HBASE-16284, unauthorized user will not able allowed to stop
> HM/RS/cluster, but while executing "cpHost.preStopMaster()", ConnectorServer
> will be stopped before AccessController validation.
> hbase-site.xml,
> {noformat}
>
> hbase.coprocessor.master.classes
>
> org.apache.hadoop.hbase.JMXListener,org.apache.hadoop.hbase.security.access.AccessController
>
>
> hbase.coprocessor.regionserver.classes
>
> org.apache.hadoop.hbase.JMXListener,org.apache.hadoop.hbase.security.access.AccessController
>
> {noformat}
> HBaseAdmin.stopMaster(),
> {noformat}
> 2016-09-20 21:12:26,796 INFO
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
> hbase.JMXListener: ConnectorServer stopped!
> 2016-09-20 21:13:55,380 WARN
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
> security.ShellBasedUnixGroupsMapping: got exception trying to get groups for
> user P72981
> ExitCodeException exitCode=1: id: P72981: No such user
> 2016-09-20 21:14:00,495 ERROR
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
> master.MasterRpcServices: Exception occurred while stopping master
> org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
> permissions for user 'P72981' (global, action=ADMIN)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requireGlobalPermission(AccessController.java:546)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requirePermission(AccessController.java:522)
> at
> org.apache.hadoop.hbase.security.access.AccessController.preStopMaster(AccessController.java:1297)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost$68.call(MasterCoprocessorHost.java:821)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.execOperation(MasterCoprocessorHost.java:1188)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.preStopMaster(MasterCoprocessorHost.java:817)
> at org.apache.hadoop.hbase.master.HMaster.stopMaster(HMaster.java:2352)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.stopMaster(MasterRpcServices.java:1364)
> {noformat}
> HBaseAdmin.stopRegionServer(rs-host-port),
> {noformat}
> 2016-09-20 20:59:01,234 INFO
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
> hbase.JMXListener: ConnectorServer stopped!
> 2016-09-20 20:59:01,250 WARN
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
> security.ShellBasedUnixGroupsMapping: got exception trying to get groups for
> user P72981
> ExitCodeException exitCode=1: id: P72981: No such user
> 2016-09-20 20:59:01,253 WARN
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
> regionserver.HRegionServer: The region server did not stop
> org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
> permissions for user 'P72981' (global, action=ADMIN)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requireGlobalPermission(AccessController.java:546)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requirePermission(AccessController.java:522)
> at
> org.apache.hadoop.hbase.security.access.AccessController.preStopRegionServer(AccessController.java:2501)
> at
> org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost$1.call(RegionServerCoprocessorHost.java:84)
> at
> org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost.execOperation(RegionServerCoprocessorHost.java:256)
> at
> org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost.preStop(RegionServerCoprocessorHost.java:80)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.stop(HRegionServer.java:1905)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.stopServer(RSRpcServices.java:1961)
> {noformat}
> HBaseAdmin.shutdown(),
> {noformat}
> 2016-09-21 12:09:08,259 INFO
>
[jira] [Commented] (HBASE-16663) JMX ConnectorServer stopped when unauthorized user try to stop HM/RS/cluster
[
https://issues.apache.org/jira/browse/HBASE-16663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675724#comment-15675724
]
Nick Dimiduk commented on HBASE-16663:
--
Sorry I missed this. Yes let's bring it back for 1.1 as well. {{6123106}} from
1.2 applies cleanly on 1.1. Let's see what QA says.
> JMX ConnectorServer stopped when unauthorized user try to stop HM/RS/cluster
>
>
> Key: HBASE-16663
> URL: https://issues.apache.org/jira/browse/HBASE-16663
> Project: HBase
> Issue Type: Bug
> Components: metrics, security
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.4, 0.98.24
>
> Attachments: HBASE-16663-0.98-V4.patch, HBASE-16663-0.98.patch,
> HBASE-16663-V2.patch, HBASE-16663-V3.patch, HBASE-16663-V4.patch,
> HBASE-16663-branch-1.patch, HBASE-16663.patch
>
>
> After HBASE-16284, unauthorized user will not able allowed to stop
> HM/RS/cluster, but while executing "cpHost.preStopMaster()", ConnectorServer
> will be stopped before AccessController validation.
> hbase-site.xml,
> {noformat}
>
> hbase.coprocessor.master.classes
>
> org.apache.hadoop.hbase.JMXListener,org.apache.hadoop.hbase.security.access.AccessController
>
>
> hbase.coprocessor.regionserver.classes
>
> org.apache.hadoop.hbase.JMXListener,org.apache.hadoop.hbase.security.access.AccessController
>
> {noformat}
> HBaseAdmin.stopMaster(),
> {noformat}
> 2016-09-20 21:12:26,796 INFO
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
> hbase.JMXListener: ConnectorServer stopped!
> 2016-09-20 21:13:55,380 WARN
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
> security.ShellBasedUnixGroupsMapping: got exception trying to get groups for
> user P72981
> ExitCodeException exitCode=1: id: P72981: No such user
> 2016-09-20 21:14:00,495 ERROR
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
> master.MasterRpcServices: Exception occurred while stopping master
> org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
> permissions for user 'P72981' (global, action=ADMIN)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requireGlobalPermission(AccessController.java:546)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requirePermission(AccessController.java:522)
> at
> org.apache.hadoop.hbase.security.access.AccessController.preStopMaster(AccessController.java:1297)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost$68.call(MasterCoprocessorHost.java:821)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.execOperation(MasterCoprocessorHost.java:1188)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.preStopMaster(MasterCoprocessorHost.java:817)
> at org.apache.hadoop.hbase.master.HMaster.stopMaster(HMaster.java:2352)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.stopMaster(MasterRpcServices.java:1364)
> {noformat}
> HBaseAdmin.stopRegionServer(rs-host-port),
> {noformat}
> 2016-09-20 20:59:01,234 INFO
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
> hbase.JMXListener: ConnectorServer stopped!
> 2016-09-20 20:59:01,250 WARN
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
> security.ShellBasedUnixGroupsMapping: got exception trying to get groups for
> user P72981
> ExitCodeException exitCode=1: id: P72981: No such user
> 2016-09-20 20:59:01,253 WARN
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
> regionserver.HRegionServer: The region server did not stop
> org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
> permissions for user 'P72981' (global, action=ADMIN)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requireGlobalPermission(AccessController.java:546)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requirePermission(AccessController.java:522)
> at
> org.apache.hadoop.hbase.security.access.AccessController.preStopRegionServer(AccessController.java:2501)
> at
> org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost$1.call(RegionServerCoprocessorHost.java:84)
> at
> org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost.execOperation(RegionServerCoprocessorHost.java:256)
> at
> org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost.preStop(RegionServerCoprocessorHost.java:80)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.stop(HRegionServer.java:1905)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.stopServer(RSRpcServices.java:1961)
> {noformat}
> HBaseAdmin.shutdown(),
> {noformat}
> 2016-09-21
[jira] [Commented] (HBASE-17124) HFileInputFormat can help user read hbase table with HFile way,more effective
[ https://issues.apache.org/jira/browse/HBASE-17124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675635#comment-15675635 ] Yu Li commented on HBASE-17124: --- Interesting, please tell us more :-) > HFileInputFormat can help user read hbase table with HFile way,more effective > - > > Key: HBASE-17124 > URL: https://issues.apache.org/jira/browse/HBASE-17124 > Project: HBase > Issue Type: New Feature > Components: HFile >Reporter: Yuan Kang >Assignee: Yuan Kang > > When we need to read large amount of data from hbase,usually we use > tableinputformat to query hbase table. but hfileInputFormat way is more > effective -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16335) update to latest apache parent pom
[ https://issues.apache.org/jira/browse/HBASE-16335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675622#comment-15675622 ] Hudson commented on HBASE-16335: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1972 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1972/]) HBASE-16335 Update to latest Apache parent pom (stack: rev 046d1a56b242d8586d8bcd81d26c11a859651972) * (edit) pom.xml > update to latest apache parent pom > -- > > Key: HBASE-16335 > URL: https://issues.apache.org/jira/browse/HBASE-16335 > Project: HBase > Issue Type: Task > Components: build, dependencies >Reporter: Sean Busbey >Assignee: Jan Hentschel > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16335.master.001.patch > > > Our apache parent pom is currently ~4 years old. update to latest. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17112) Prevent setting timestamp of delta operations the same as previous value's
[
https://issues.apache.org/jira/browse/HBASE-17112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675613#comment-15675613
]
Yu Li commented on HBASE-17112:
---
Since {{Affects Versions}} are set to {{1.1.7, 0.98.23, 1.2.4}} and JIRA type
to Bug, should this one also go into relative branches? Thanks.
> Prevent setting timestamp of delta operations the same as previous value's
> --
>
> Key: HBASE-17112
> URL: https://issues.apache.org/jira/browse/HBASE-17112
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.1.7, 0.98.23, 1.2.4
>Reporter: Phil Yang
>Assignee: Phil Yang
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17112-branch-1-v1.patch,
> HBASE-17112-branch-1-v1.patch, HBASE-17112-branch-1.1-v1.patch,
> HBASE-17112-branch-1.1-v1.patch, HBASE-17112-v1.patch, HBASE-17112-v2.patch,
> HBASE-17112-v2.patch
>
>
> In delta operations, Increment and Append. We will read current value first
> and then write the new whole result into WAL as the type of Put with current
> timestamp. If the previous ts is larger than current ts, we will use the
> previous ts.
> If we have two Puts with same TS, we will ignore the Put with lower sequence
> id. It is not friendly with versioning. And for replication we will drop
> sequence id while writing to peer cluster so in the slave we don't know what
> the order they are being written. If the pushing is disordered, the result
> will be wrong.
> We can set the new ts to previous+1 if the previous is not less than now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-17123) Add postBulkLoadHFile variant that notifies the final paths for the hfiles
[
https://issues.apache.org/jira/browse/HBASE-17123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675527#comment-15675527
]
Ted Yu edited comment on HBASE-17123 at 11/18/16 3:19 AM:
--
SecureBulkLoadManager calls bulkLoadHFiles() of Region:
{code}
return region.bulkLoadHFiles(familyPaths, true,
new SecureBulkLoadListener(fs, bulkToken, conf),
request.getCopyFile());
{code}
I plan to change the return value of bulkLoadHFiles() to Map, corresponding to paths of the final hfiles.
[~enis]:
What do you think ?
was (Author: yuzhih...@gmail.com):
SecureBulkLoadManager calls bulkLoadHFiles() of Region:
{code}
return region.bulkLoadHFiles(familyPaths, true,
new SecureBulkLoadListener(fs, bulkToken, conf),
request.getCopyFile());
{code}
I plan to change the return value of bulkLoadHFiles() to List>, corresponding to paths of the final hfiles.
[~enis]:
What do you think ?
> Add postBulkLoadHFile variant that notifies the final paths for the hfiles
> --
>
> Key: HBASE-17123
> URL: https://issues.apache.org/jira/browse/HBASE-17123
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>
> Currently the postBulkLoadHFile() hook passes the same familyPaths parameter
> which it receives as method parameter.
> See code in SecureBulkLoadManager :
> {code}
>loaded = region.getCoprocessorHost().postBulkLoadHFile(familyPaths,
> loaded);
> {code}
> Meaning, the paths are not final, moved paths of the loaded hfiles.
> This issue is to add a variant which notifies the final paths of the loaded
> hfiles.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17114) Add an option to set special retry pause when encountering CallQueueTooBigException
[
https://issues.apache.org/jira/browse/HBASE-17114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675588#comment-15675588
]
Yu Li commented on HBASE-17114:
---
bq. Another approach to this would be to allow the server to hint back to the
client how long it should back off
I guess the above statement about "back off" is the back off policy instead of
the exponential backoff array? So I checked the default value of
{{ClientBackoffPolicy}}, or could you please explain how to make server hint
back? [~ghelmling]
bq. If you want to make this overridable for some exception types, that seems
ok, but in that case the config property for overriding the value should be
more closely tied to the exception.
Well, if checking the uploaded patch, it's indeed tied to CQTBE only.
Introducing a new property is only for making things more flexible, and of
course we could use a hard-coded, like 5 times than the existing pause, for
CQTBE. But I'd say this is a trade-off, waiting longer for CQTBE could prevent
the vicious circle but is also causing a higher latency, and IMHO user should
be able to control such trade-off. If they don't want CQTBE to be special, they
could set {{hbase.client.pause.special}} to the same value as
{{hbase.client.pause}}, which gives them more options.
No offense but I'm even thinking of making CQTBE thrown optional, because for
some case dead-wait for the request to be executed in RpcServer until time-out
is preferable by user rather than receiving some exception and retry and fail
again, but obviously this is another topic (Smile).
bq. It's only special in the sense that it should not clear the client meta
cache
Sorry but I don't see any difference in "should not clear the client meta
cache" and "should not retry so frequently", both trying to resolve some
problem and make things better.
OTOH, we already have the {{RetryImmediatelyException}} just because in some
case retry w/o waiting is good, then why retry slower is not acceptable? Now
that the retry pause already split into immediately and wait, I think it's ok
to further split the wait case into quick and slow, wdyt?
Thanks.
> Add an option to set special retry pause when encountering
> CallQueueTooBigException
> ---
>
> Key: HBASE-17114
> URL: https://issues.apache.org/jira/browse/HBASE-17114
> Project: HBase
> Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-17114.patch
>
>
> As titled, after HBASE-15146 we will throw {{CallQueueTooBigException}}
> instead of dead-wait. This is good for performance for most cases but might
> cause a side-effect that if too many clients connect to the busy RS, that the
> retry requests may come over and over again and RS never got the chance for
> recovering, and the issue will become especially critical when the target
> region is META.
> So here in this JIRA we propose to supply some special retry pause for CQTBE
> in name of {{hbase.client.pause.special}}, and by default it will be 500ms (5
> times of {{hbase.client.pause}} default value)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17110) Add an "Overall Strategy" option(balanced both on table level and server level) to SimpleLoadBalancer
[
https://issues.apache.org/jira/browse/HBASE-17110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675548#comment-15675548
]
Enis Soztutar commented on HBASE-17110:
---
Agreed with Anoop that instead of creating a new mode, we can make the existing
modes just better. For example StochasticLB takes into account per-server and
per-table at the same time when it is invoked at the global mode. In similar
sense, we can make the SimpleLB also look at per-table balance in the global
mode.
bq. Since big batch of region moving will cause spike and is bad for online
performance/stability
Maybe we should introduce how many regions can be moved at once as a general LB
configuration. StochasticLB already has a "cost" for moving regions in its
model, so it should not end up doing a lot of moves concurrently. Maybe it is a
matter of auto-tuning these for smaller as well as bigger clusters.
> Add an "Overall Strategy" option(balanced both on table level and server
> level) to SimpleLoadBalancer
> -
>
> Key: HBASE-17110
> URL: https://issues.apache.org/jira/browse/HBASE-17110
> Project: HBase
> Issue Type: New Feature
> Components: Balancer
>Affects Versions: 2.0.0, 1.2.4
>Reporter: Charlie Qiangeng Xu
>Assignee: Charlie Qiangeng Xu
> Attachments: HBASE-17110-V2.patch, HBASE-17110-V3.patch,
> HBASE-17110.patch
>
>
> This jira is about an enhancement of simpleLoadBalancer. Here we introduce a
> new strategy: "bytableOverall" which could be controlled by adding:
> {noformat}
>
> hbase.master.loadbalance.bytableOverall
> true
>
> {noformat}
> We have been using the strategy on our largest cluster for several months.
> it's proven to be very helpful and stable, especially, the result is quite
> visible to the users.
> Here is the reason why it's helpful:
> When operating large scale clusters(our case), some companies still prefer to
> use {{SimpleLoadBalancer}} due to its simplicity, quick balance plan
> generation, etc. Current SimpleLoadBalancer has two modes:
> 1. byTable, which only guarantees that the regions of one table could be
> uniformly distributed.
> 2. byCluster, which ignores the distribution within tables and balance the
> regions all together.
> If the pressures on different tables are different, the first byTable option
> is the preferable one in most case. Yet, this choice sacrifice the cluster
> level balance and would cause some servers to have significantly higher load,
> e.g. 242 regions on server A but 417 regions on server B.(real world stats)
> Consider this case, a cluster has 3 tables and 4 servers:
> {noformat}
> server A has 3 regions: table1:1, table2:1, table3:1
> server B has 3 regions: table1:2, table2:2, table3:2
> server C has 3 regions: table1:3, table2:3, table3:3
> server D has 0 regions.
> {noformat}
> From the byTable strategy's perspective, the cluster has already been
> perfectly balanced on table level. But a perfect status should be like:
> {noformat}
> server A has 2 regions: table2:1, table3:1
> server B has 2 regions: table1:2, table3:2
> server C has 3 regions: table1:3, table2:3, table3:3
> server D has 2 regions: table1:1, table2:2
> {noformat}
> We can see the server loads change from 3,3,3,0 to 2,2,3,2, while the table1,
> table2 and table3 still keep balanced.
> And this is what the new mode "byTableOverall" can achieve.
> Two UTs have been added as well and the last one demonstrates the advantage
> of the new strategy.
> Also, a onConfigurationChange method has been implemented to hot control the
> "slop" variable.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17124) HFileInputFormat can help user read hbase table with HFile way,more effective
[ https://issues.apache.org/jira/browse/HBASE-17124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Kang updated HBASE-17124: -- Status: Open (was: Patch Available) > HFileInputFormat can help user read hbase table with HFile way,more effective > - > > Key: HBASE-17124 > URL: https://issues.apache.org/jira/browse/HBASE-17124 > Project: HBase > Issue Type: New Feature > Components: HFile >Reporter: Yuan Kang >Assignee: Yuan Kang > > When we need to read large amount of data from hbase,usually we use > tableinputformat to query hbase table. but hfileInputFormat way is more > effective -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17124) HFileInputFormat can help user read hbase table with HFile way,more effective
[ https://issues.apache.org/jira/browse/HBASE-17124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Kang updated HBASE-17124: -- Status: Patch Available (was: Open) > HFileInputFormat can help user read hbase table with HFile way,more effective > - > > Key: HBASE-17124 > URL: https://issues.apache.org/jira/browse/HBASE-17124 > Project: HBase > Issue Type: New Feature > Components: HFile >Reporter: Yuan Kang >Assignee: Yuan Kang > > When we need to read large amount of data from hbase,usually we use > tableinputformat to query hbase table. but hfileInputFormat way is more > effective -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-17124) HFileInputFormat can help user read hbase table with HFile way,more effective
Yuan Kang created HBASE-17124: - Summary: HFileInputFormat can help user read hbase table with HFile way,more effective Key: HBASE-17124 URL: https://issues.apache.org/jira/browse/HBASE-17124 Project: HBase Issue Type: New Feature Components: HFile Reporter: Yuan Kang Assignee: Yuan Kang When we need to read large amount of data from hbase,usually we use tableinputformat to query hbase table. but hfileInputFormat way is more effective -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17058) Lower epsilon used for jitter verification from HBASE-15324
[ https://issues.apache.org/jira/browse/HBASE-17058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675531#comment-15675531 ] Hudson commented on HBASE-17058: SUCCESS: Integrated in Jenkins build HBase-1.2-JDK7 #72 (See [https://builds.apache.org/job/HBase-1.2-JDK7/72/]) HBASE-17058 Lower epsilon used for jitter verification from HBASE-15324 (esteban: rev bb891c6834a0691302b958d78e0c009b3601c442) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java > Lower epsilon used for jitter verification from HBASE-15324 > --- > > Key: HBASE-17058 > URL: https://issues.apache.org/jira/browse/HBASE-17058 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4 >Reporter: Esteban Gutierrez >Assignee: Esteban Gutierrez > Fix For: 2.0.0, 1.4.0, 1.3.1, 1.2.5, 1.1.8 > > Attachments: HBASE-17058.master.001.patch > > > The current epsilon used is 1E-6 and its too big it might overflow the > desiredMaxFileSize. A trivial fix is to lower the epsilon to 2^-52 or even > 2^-53. An option to consider too is just to shift the jitter to always > decrement hbase.hregion.max.filesize (MAX_FILESIZE) instead of increase the > size of the region and having to deal with the round off. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15324) Jitter may cause desiredMaxFileSize overflow in ConstantSizeRegionSplitPolicy and trigger unexpected split
[
https://issues.apache.org/jira/browse/HBASE-15324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675530#comment-15675530
]
Hudson commented on HBASE-15324:
SUCCESS: Integrated in Jenkins build HBase-1.2-JDK7 #72 (See
[https://builds.apache.org/job/HBase-1.2-JDK7/72/])
HBASE-17058 Lower epsilon used for jitter verification from HBASE-15324
(esteban: rev bb891c6834a0691302b958d78e0c009b3601c442)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java
> Jitter may cause desiredMaxFileSize overflow in ConstantSizeRegionSplitPolicy
> and trigger unexpected split
> --
>
> Key: HBASE-15324
> URL: https://issues.apache.org/jira/browse/HBASE-15324
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.1.3
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.5, 1.1.8
>
> Attachments: HBASE-15324.patch, HBASE-15324_v2.patch,
> HBASE-15324_v3.patch, HBASE-15324_v3.patch
>
>
> We introduce jitter for region split decision in HBASE-13412, but the
> following line in {{ConstantSizeRegionSplitPolicy}} may cause long value
> overflow if MAX_FILESIZE is specified to Long.MAX_VALUE:
> {code}
> this.desiredMaxFileSize += (long)(desiredMaxFileSize * (RANDOM.nextFloat() -
> 0.5D) * jitter);
> {code}
> In our case we specify MAX_FILESIZE to Long.MAX_VALUE to prevent target
> region to split.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17123) Add postBulkLoadHFile variant that notifies the final paths for the hfiles
[
https://issues.apache.org/jira/browse/HBASE-17123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675527#comment-15675527
]
Ted Yu commented on HBASE-17123:
SecureBulkLoadManager calls bulkLoadHFiles() of Region:
{code}
return region.bulkLoadHFiles(familyPaths, true,
new SecureBulkLoadListener(fs, bulkToken, conf),
request.getCopyFile());
{code}
I plan to change the return value of bulkLoadHFiles() to List>, corresponding to paths of the final hfiles.
[~enis]:
What do you think ?
> Add postBulkLoadHFile variant that notifies the final paths for the hfiles
> --
>
> Key: HBASE-17123
> URL: https://issues.apache.org/jira/browse/HBASE-17123
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>
> Currently the postBulkLoadHFile() hook passes the same familyPaths parameter
> which it receives as method parameter.
> See code in SecureBulkLoadManager :
> {code}
>loaded = region.getCoprocessorHost().postBulkLoadHFile(familyPaths,
> loaded);
> {code}
> Meaning, the paths are not final, moved paths of the loaded hfiles.
> This issue is to add a variant which notifies the final paths of the loaded
> hfiles.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17123) Add postBulkLoadHFile variant that notifies the final paths for the hfiles
[
https://issues.apache.org/jira/browse/HBASE-17123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-17123:
---
Description:
Currently the postBulkLoadHFile() hook passes the same familyPaths parameter
which it receives as method parameter.
See code in SecureBulkLoadManager :
{code}
loaded = region.getCoprocessorHost().postBulkLoadHFile(familyPaths,
loaded);
{code}
Meaning, the paths are not final, moved paths of the loaded hfiles.
This issue is to add a variant which notifies the final paths of the loaded
hfiles.
was:
Currently the postBulkLoadHFile() hook passes the same familyPaths parameter
which it receives as method parameter:
{code}
loaded = region.getCoprocessorHost().postBulkLoadHFile(familyPaths,
loaded);
{code}
Meaning, the paths are not final, moved paths of the loaded hfiles.
This issue is to add a variant which notifies the final paths of the loaded
hfiles.
> Add postBulkLoadHFile variant that notifies the final paths for the hfiles
> --
>
> Key: HBASE-17123
> URL: https://issues.apache.org/jira/browse/HBASE-17123
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>
> Currently the postBulkLoadHFile() hook passes the same familyPaths parameter
> which it receives as method parameter.
> See code in SecureBulkLoadManager :
> {code}
>loaded = region.getCoprocessorHost().postBulkLoadHFile(familyPaths,
> loaded);
> {code}
> Meaning, the paths are not final, moved paths of the loaded hfiles.
> This issue is to add a variant which notifies the final paths of the loaded
> hfiles.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16708) Expose endpoint Coprocessor name in "responseTooSlow" log messages
[ https://issues.apache.org/jira/browse/HBASE-16708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675502#comment-15675502 ] Jerry He commented on HBASE-16708: -- I think the short service name looks ok, if we don't have to pull in all the extras into RpcServer. The service name to the implementation mapping is 1-to-1. We would loss much clarity. nit. bq. "coprocessor=" Maybe "coprocessorService="? > Expose endpoint Coprocessor name in "responseTooSlow" log messages > -- > > Key: HBASE-16708 > URL: https://issues.apache.org/jira/browse/HBASE-16708 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Nick Dimiduk >Assignee: Yi Liang > Fix For: 2.0.0 > > Attachments: HBASE-16708-V1.patch, HBASE-16708-V2.patch, > HBASE-ShortName.patch > > > Operational diagnostics of a Phoenix install would be easier if we included > which endpoint coprocessor was being called in this responseTooSlow WARN > message. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16335) update to latest apache parent pom
[ https://issues.apache.org/jira/browse/HBASE-16335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675493#comment-15675493 ] Hudson commented on HBASE-16335: SUCCESS: Integrated in Jenkins build HBase-1.4 #538 (See [https://builds.apache.org/job/HBase-1.4/538/]) HBASE-16335 Update to latest Apache parent pom (stack: rev 0a171717e7e1817d6af94510208081a193b18c0b) * (edit) pom.xml > update to latest apache parent pom > -- > > Key: HBASE-16335 > URL: https://issues.apache.org/jira/browse/HBASE-16335 > Project: HBase > Issue Type: Task > Components: build, dependencies >Reporter: Sean Busbey >Assignee: Jan Hentschel > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16335.master.001.patch > > > Our apache parent pom is currently ~4 years old. update to latest. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15324) Jitter may cause desiredMaxFileSize overflow in ConstantSizeRegionSplitPolicy and trigger unexpected split
[
https://issues.apache.org/jira/browse/HBASE-15324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675492#comment-15675492
]
Hudson commented on HBASE-15324:
SUCCESS: Integrated in Jenkins build HBase-1.4 #538 (See
[https://builds.apache.org/job/HBase-1.4/538/])
HBASE-17058 Lower epsilon used for jitter verification from HBASE-15324
(esteban: rev 19441937ea688b6798675993c6af4a961f931c3a)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java
> Jitter may cause desiredMaxFileSize overflow in ConstantSizeRegionSplitPolicy
> and trigger unexpected split
> --
>
> Key: HBASE-15324
> URL: https://issues.apache.org/jira/browse/HBASE-15324
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.1.3
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.5, 1.1.8
>
> Attachments: HBASE-15324.patch, HBASE-15324_v2.patch,
> HBASE-15324_v3.patch, HBASE-15324_v3.patch
>
>
> We introduce jitter for region split decision in HBASE-13412, but the
> following line in {{ConstantSizeRegionSplitPolicy}} may cause long value
> overflow if MAX_FILESIZE is specified to Long.MAX_VALUE:
> {code}
> this.desiredMaxFileSize += (long)(desiredMaxFileSize * (RANDOM.nextFloat() -
> 0.5D) * jitter);
> {code}
> In our case we specify MAX_FILESIZE to Long.MAX_VALUE to prevent target
> region to split.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17058) Lower epsilon used for jitter verification from HBASE-15324
[ https://issues.apache.org/jira/browse/HBASE-17058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675494#comment-15675494 ] Hudson commented on HBASE-17058: SUCCESS: Integrated in Jenkins build HBase-1.4 #538 (See [https://builds.apache.org/job/HBase-1.4/538/]) HBASE-17058 Lower epsilon used for jitter verification from HBASE-15324 (esteban: rev 19441937ea688b6798675993c6af4a961f931c3a) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java > Lower epsilon used for jitter verification from HBASE-15324 > --- > > Key: HBASE-17058 > URL: https://issues.apache.org/jira/browse/HBASE-17058 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4 >Reporter: Esteban Gutierrez >Assignee: Esteban Gutierrez > Fix For: 2.0.0, 1.4.0, 1.3.1, 1.2.5, 1.1.8 > > Attachments: HBASE-17058.master.001.patch > > > The current epsilon used is 1E-6 and its too big it might overflow the > desiredMaxFileSize. A trivial fix is to lower the epsilon to 2^-52 or even > 2^-53. An option to consider too is just to shift the jitter to always > decrement hbase.hregion.max.filesize (MAX_FILESIZE) instead of increase the > size of the region and having to deal with the round off. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17072) CPU usage starts to climb up to 90-100% when using G1GC
[
https://issues.apache.org/jira/browse/HBASE-17072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675448#comment-15675448
]
Tomu Tsuruhara commented on HBASE-17072:
ah, I see. Thanks.
> CPU usage starts to climb up to 90-100% when using G1GC
> ---
>
> Key: HBASE-17072
> URL: https://issues.apache.org/jira/browse/HBASE-17072
> Project: HBase
> Issue Type: Bug
> Components: Performance, regionserver
>Affects Versions: 1.0.0, 1.2.0
>Reporter: Eiichi Sato
> Attachments: disable-block-header-cache.patch, mat-threadlocals.png,
> mat-threads.png, metrics.png, slave1.svg, slave2.svg, slave3.svg, slave4.svg
>
>
> h5. Problem
> CPU usage of a region server in our CDH 5.4.5 cluster, at some point, starts
> to gradually get higher up to nearly 90-100% when using G1GC. We've also run
> into this problem on CDH 5.7.3 and CDH 5.8.2.
> In our production cluster, it normally takes a few weeks for this to happen
> after restarting a RS. We reproduced this on our test cluster and attached
> the results. Please note that, to make it easy to reproduce, we did some
> "anti-tuning" on a table when running tests.
> In metrics.png, soon after we started running some workloads against a test
> cluster (CDH 5.8.2) at about 7 p.m. CPU usage of the two RSs started to rise.
> Flame Graphs (slave1.svg to slave4.svg) are generated from jstack dumps of
> each RS process around 10:30 a.m. the next day.
> After investigating heapdumps from another occurrence on a test cluster
> running CDH 5.7.3, we found that the ThreadLocalMap contain a lot of
> contiguous entries of {{HFileBlock$PrefetchedHeader}} probably due to primary
> clustering. This caused more loops in
> {{ThreadLocalMap#expungeStaleEntries()}}, consuming a certain amount of CPU
> time. What is worse is that the method is called from RPC metrics code,
> which means even a small amount of per-RPC time soon adds up to a huge amount
> of CPU time.
> This is very similar to the issue in HBASE-16616, but we have many
> {{HFileBlock$PrefetchedHeader}} not only {{Counter$IndexHolder}} instances.
> Here are some OQL counts from Eclipse Memory Analyzer (MAT). This shows a
> number of ThreadLocal instances in the ThreadLocalMap of a single handler
> thread.
> {code}
> SELECT *
> FROM OBJECTS (SELECT AS RETAINED SET OBJECTS value
> FROM OBJECTS 0x4ee380430) obj
> WHERE obj.@clazz.@name =
> "org.apache.hadoop.hbase.io.hfile.HFileBlock$PrefetchedHeader"
> #=> 10980 instances
> {code}
> {code}
> SELECT *
> FROM OBJECTS (SELECT AS RETAINED SET OBJECTS value
> FROM OBJECTS 0x4ee380430) obj
> WHERE obj.@clazz.@name = "org.apache.hadoop.hbase.util.Counter$IndexHolder"
> #=> 2052 instances
> {code}
> Although as described in HBASE-16616 this somewhat seems to be an issue in
> G1GC side regarding weakly-reachable objects, we should keep ThreadLocal
> usage minimal and avoid creating an indefinite number (in this case, a number
> of HFiles) of ThreadLocal instances.
> HBASE-16146 removes ThreadLocals from the RPC metrics code. That may solve
> the issue (I just saw the patch, never tested it at all), but the
> {{HFileBlock$PrefetchedHeader}} are still there in the ThreadLocalMap, which
> may cause issues in the future again.
> h5. Our Solution
> We simply removed the whole {{HFileBlock$PrefetchedHeader}} caching and
> fortunately we didn't notice any performance degradation for our production
> workloads.
> Because the PrefetchedHeader caching uses ThreadLocal and because RPCs are
> handled randomly in any of the handlers, small Get or small Scan RPCs do not
> benefit from the caching (See HBASE-10676 and HBASE-11402 for the details).
> Probably, we need to see how well reads are saved by the caching for large
> Scan or Get RPCs and especially for compactions if we really remove the
> caching. It's probably better if we can remove ThreadLocals without breaking
> the current caching behavior.
> FWIW, I'm attaching the patch we applied. It's for CDH 5.4.5.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-17123) Add postBulkLoadHFile variant that notifies the final paths for the hfiles
Ted Yu created HBASE-17123:
--
Summary: Add postBulkLoadHFile variant that notifies the final
paths for the hfiles
Key: HBASE-17123
URL: https://issues.apache.org/jira/browse/HBASE-17123
Project: HBase
Issue Type: Improvement
Reporter: Ted Yu
Currently the postBulkLoadHFile() hook passes the same familyPaths parameter
which it receives as method parameter:
{code}
loaded = region.getCoprocessorHost().postBulkLoadHFile(familyPaths,
loaded);
{code}
Meaning, the paths are not final, moved paths of the loaded hfiles.
This issue is to add a variant which notifies the final paths of the loaded
hfiles.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17058) Lower epsilon used for jitter verification from HBASE-15324
[ https://issues.apache.org/jira/browse/HBASE-17058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675410#comment-15675410 ] Hudson commented on HBASE-17058: SUCCESS: Integrated in Jenkins build HBase-1.3-JDK8 #80 (See [https://builds.apache.org/job/HBase-1.3-JDK8/80/]) HBASE-17058 Lower epsilon used for jitter verification from HBASE-15324 (esteban: rev f4ed43e06108687488ebb161086b00274d172bc0) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java > Lower epsilon used for jitter verification from HBASE-15324 > --- > > Key: HBASE-17058 > URL: https://issues.apache.org/jira/browse/HBASE-17058 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4 >Reporter: Esteban Gutierrez >Assignee: Esteban Gutierrez > Fix For: 2.0.0, 1.4.0, 1.3.1, 1.2.5, 1.1.8 > > Attachments: HBASE-17058.master.001.patch > > > The current epsilon used is 1E-6 and its too big it might overflow the > desiredMaxFileSize. A trivial fix is to lower the epsilon to 2^-52 or even > 2^-53. An option to consider too is just to shift the jitter to always > decrement hbase.hregion.max.filesize (MAX_FILESIZE) instead of increase the > size of the region and having to deal with the round off. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15324) Jitter may cause desiredMaxFileSize overflow in ConstantSizeRegionSplitPolicy and trigger unexpected split
[
https://issues.apache.org/jira/browse/HBASE-15324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675409#comment-15675409
]
Hudson commented on HBASE-15324:
SUCCESS: Integrated in Jenkins build HBase-1.3-JDK8 #80 (See
[https://builds.apache.org/job/HBase-1.3-JDK8/80/])
HBASE-17058 Lower epsilon used for jitter verification from HBASE-15324
(esteban: rev f4ed43e06108687488ebb161086b00274d172bc0)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java
> Jitter may cause desiredMaxFileSize overflow in ConstantSizeRegionSplitPolicy
> and trigger unexpected split
> --
>
> Key: HBASE-15324
> URL: https://issues.apache.org/jira/browse/HBASE-15324
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.1.3
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.5, 1.1.8
>
> Attachments: HBASE-15324.patch, HBASE-15324_v2.patch,
> HBASE-15324_v3.patch, HBASE-15324_v3.patch
>
>
> We introduce jitter for region split decision in HBASE-13412, but the
> following line in {{ConstantSizeRegionSplitPolicy}} may cause long value
> overflow if MAX_FILESIZE is specified to Long.MAX_VALUE:
> {code}
> this.desiredMaxFileSize += (long)(desiredMaxFileSize * (RANDOM.nextFloat() -
> 0.5D) * jitter);
> {code}
> In our case we specify MAX_FILESIZE to Long.MAX_VALUE to prevent target
> region to split.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17058) Lower epsilon used for jitter verification from HBASE-15324
[ https://issues.apache.org/jira/browse/HBASE-17058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675399#comment-15675399 ] Hudson commented on HBASE-17058: SUCCESS: Integrated in Jenkins build HBase-1.2-JDK8 #66 (See [https://builds.apache.org/job/HBase-1.2-JDK8/66/]) HBASE-17058 Lower epsilon used for jitter verification from HBASE-15324 (esteban: rev bb891c6834a0691302b958d78e0c009b3601c442) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java > Lower epsilon used for jitter verification from HBASE-15324 > --- > > Key: HBASE-17058 > URL: https://issues.apache.org/jira/browse/HBASE-17058 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4 >Reporter: Esteban Gutierrez >Assignee: Esteban Gutierrez > Fix For: 2.0.0, 1.4.0, 1.3.1, 1.2.5, 1.1.8 > > Attachments: HBASE-17058.master.001.patch > > > The current epsilon used is 1E-6 and its too big it might overflow the > desiredMaxFileSize. A trivial fix is to lower the epsilon to 2^-52 or even > 2^-53. An option to consider too is just to shift the jitter to always > decrement hbase.hregion.max.filesize (MAX_FILESIZE) instead of increase the > size of the region and having to deal with the round off. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15324) Jitter may cause desiredMaxFileSize overflow in ConstantSizeRegionSplitPolicy and trigger unexpected split
[
https://issues.apache.org/jira/browse/HBASE-15324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675397#comment-15675397
]
Hudson commented on HBASE-15324:
SUCCESS: Integrated in Jenkins build HBase-1.2-JDK8 #66 (See
[https://builds.apache.org/job/HBase-1.2-JDK8/66/])
HBASE-17058 Lower epsilon used for jitter verification from HBASE-15324
(esteban: rev bb891c6834a0691302b958d78e0c009b3601c442)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java
> Jitter may cause desiredMaxFileSize overflow in ConstantSizeRegionSplitPolicy
> and trigger unexpected split
> --
>
> Key: HBASE-15324
> URL: https://issues.apache.org/jira/browse/HBASE-15324
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.1.3
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.5, 1.1.8
>
> Attachments: HBASE-15324.patch, HBASE-15324_v2.patch,
> HBASE-15324_v3.patch, HBASE-15324_v3.patch
>
>
> We introduce jitter for region split decision in HBASE-13412, but the
> following line in {{ConstantSizeRegionSplitPolicy}} may cause long value
> overflow if MAX_FILESIZE is specified to Long.MAX_VALUE:
> {code}
> this.desiredMaxFileSize += (long)(desiredMaxFileSize * (RANDOM.nextFloat() -
> 0.5D) * jitter);
> {code}
> In our case we specify MAX_FILESIZE to Long.MAX_VALUE to prevent target
> region to split.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17072) CPU usage starts to climb up to 90-100% when using G1GC
[
https://issues.apache.org/jira/browse/HBASE-17072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675394#comment-15675394
]
Eiichi Sato commented on HBASE-17072:
-
I think ThreadLocal#remove() needs to be called on every handler thread because
it only removes a thread-local value on the calling thread.
> CPU usage starts to climb up to 90-100% when using G1GC
> ---
>
> Key: HBASE-17072
> URL: https://issues.apache.org/jira/browse/HBASE-17072
> Project: HBase
> Issue Type: Bug
> Components: Performance, regionserver
>Affects Versions: 1.0.0, 1.2.0
>Reporter: Eiichi Sato
> Attachments: disable-block-header-cache.patch, mat-threadlocals.png,
> mat-threads.png, metrics.png, slave1.svg, slave2.svg, slave3.svg, slave4.svg
>
>
> h5. Problem
> CPU usage of a region server in our CDH 5.4.5 cluster, at some point, starts
> to gradually get higher up to nearly 90-100% when using G1GC. We've also run
> into this problem on CDH 5.7.3 and CDH 5.8.2.
> In our production cluster, it normally takes a few weeks for this to happen
> after restarting a RS. We reproduced this on our test cluster and attached
> the results. Please note that, to make it easy to reproduce, we did some
> "anti-tuning" on a table when running tests.
> In metrics.png, soon after we started running some workloads against a test
> cluster (CDH 5.8.2) at about 7 p.m. CPU usage of the two RSs started to rise.
> Flame Graphs (slave1.svg to slave4.svg) are generated from jstack dumps of
> each RS process around 10:30 a.m. the next day.
> After investigating heapdumps from another occurrence on a test cluster
> running CDH 5.7.3, we found that the ThreadLocalMap contain a lot of
> contiguous entries of {{HFileBlock$PrefetchedHeader}} probably due to primary
> clustering. This caused more loops in
> {{ThreadLocalMap#expungeStaleEntries()}}, consuming a certain amount of CPU
> time. What is worse is that the method is called from RPC metrics code,
> which means even a small amount of per-RPC time soon adds up to a huge amount
> of CPU time.
> This is very similar to the issue in HBASE-16616, but we have many
> {{HFileBlock$PrefetchedHeader}} not only {{Counter$IndexHolder}} instances.
> Here are some OQL counts from Eclipse Memory Analyzer (MAT). This shows a
> number of ThreadLocal instances in the ThreadLocalMap of a single handler
> thread.
> {code}
> SELECT *
> FROM OBJECTS (SELECT AS RETAINED SET OBJECTS value
> FROM OBJECTS 0x4ee380430) obj
> WHERE obj.@clazz.@name =
> "org.apache.hadoop.hbase.io.hfile.HFileBlock$PrefetchedHeader"
> #=> 10980 instances
> {code}
> {code}
> SELECT *
> FROM OBJECTS (SELECT AS RETAINED SET OBJECTS value
> FROM OBJECTS 0x4ee380430) obj
> WHERE obj.@clazz.@name = "org.apache.hadoop.hbase.util.Counter$IndexHolder"
> #=> 2052 instances
> {code}
> Although as described in HBASE-16616 this somewhat seems to be an issue in
> G1GC side regarding weakly-reachable objects, we should keep ThreadLocal
> usage minimal and avoid creating an indefinite number (in this case, a number
> of HFiles) of ThreadLocal instances.
> HBASE-16146 removes ThreadLocals from the RPC metrics code. That may solve
> the issue (I just saw the patch, never tested it at all), but the
> {{HFileBlock$PrefetchedHeader}} are still there in the ThreadLocalMap, which
> may cause issues in the future again.
> h5. Our Solution
> We simply removed the whole {{HFileBlock$PrefetchedHeader}} caching and
> fortunately we didn't notice any performance degradation for our production
> workloads.
> Because the PrefetchedHeader caching uses ThreadLocal and because RPCs are
> handled randomly in any of the handlers, small Get or small Scan RPCs do not
> benefit from the caching (See HBASE-10676 and HBASE-11402 for the details).
> Probably, we need to see how well reads are saved by the caching for large
> Scan or Get RPCs and especially for compactions if we really remove the
> caching. It's probably better if we can remove ThreadLocals without breaking
> the current caching behavior.
> FWIW, I'm attaching the patch we applied. It's for CDH 5.4.5.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14123) HBase Backup/Restore Phase 2
[
https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675373#comment-15675373
]
Hadoop QA commented on HBASE-14123:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 5s
{color} | {color:blue} The patch file was not named according to hbase's naming
conventions. Please see
https://yetus.apache.org/documentation/0.3.0/precommit-patchnames for
instructions. {color} |
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m 7s
{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 47 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
0s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 36s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
19s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
17s {color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 50s
{color} | {color:red} hbase-protocol-shaded in master has 24 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 13s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 38s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 4m 38s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
5s {color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 24s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:red}-1{color} | {color:red} hbaseprotoc {color} | {color:red} 0m 24s
{color} | {color:red} Patch generated 1 new protoc errors in hbase-server.
{color} |
| {color:red}-1{color} | {color:red} hbaseprotoc {color} | {color:red} 2m 15s
{color} | {color:red} Patch generated 1 new protoc errors in .. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 55s
{color} | {color:red} hbase-server generated 1 new + 0 unchanged - 0 fixed = 1
total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 16s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 41s
{color} |
[jira] [Commented] (HBASE-17122) Change in behavior when creating a scanner for a disabled table
[
https://issues.apache.org/jira/browse/HBASE-17122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675309#comment-15675309
]
Samarth Jain commented on HBASE-17122:
--
[~gjacoby]
> Change in behavior when creating a scanner for a disabled table
> ---
>
> Key: HBASE-17122
> URL: https://issues.apache.org/jira/browse/HBASE-17122
> Project: HBase
> Issue Type: Bug
>Reporter: Samarth Jain
>
> {code}
> @Test
> public void testQueryingDisabledTable() throws Exception {
> try (Connection conn = DriverManager.getConnection(getUrl())) {
> String tableName = generateUniqueName();
> conn.createStatement().execute(
> "CREATE TABLE " + tableName
> + " (k1 VARCHAR NOT NULL, k2 VARCHAR, CONSTRAINT PK
> PRIMARY KEY(K1,K2)) ");
> try (HBaseAdmin admin =
> conn.unwrap(PhoenixConnection.class).getQueryServices().getAdmin()) {
> admin.disableTable(Bytes.toBytes(tableName));
> }
> String query = "SELECT * FROM " + tableName + " WHERE 1=1";
> try (Connection conn2 = DriverManager.getConnection(getUrl())) {
> try (ResultSet rs =
> conn2.createStatement().executeQuery(query)) {
> assertFalse(rs.next());
> }
> }
> }
> }
> {code}
> This is a phoenix specific test case. I will try an come up with something
> using the HBase API. But the gist is that with HBase 0.98.21 and beyond, we
> are seeing that creating a scanner is throwing a NotServingRegionException.
> Stacktrace for NotServingRegionException
> {code}
> org.apache.phoenix.exception.PhoenixIOException:
> org.apache.phoenix.exception.PhoenixIOException: callTimeout=120,
> callDuration=9000104: row '' on table 'T01' at
> region=T01,,1479429739864.643dde31cc19b549192576eea7791a6f.,
> hostname=localhost,60022,1479429692090, seqNum=1
> at
> org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:113)
> at
> org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:752)
> at
> org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:696)
> at
> org.apache.phoenix.iterate.ConcatResultIterator.getIterators(ConcatResultIterator.java:50)
> at
> org.apache.phoenix.iterate.ConcatResultIterator.currentIterator(ConcatResultIterator.java:97)
> at
> org.apache.phoenix.iterate.ConcatResultIterator.next(ConcatResultIterator.java:117)
> at
> org.apache.phoenix.jdbc.PhoenixResultSet.next(PhoenixResultSet.java:778)
> at
> org.apache.phoenix.end2end.PhoenixRuntimeIT.testQueryingDisabledTable(PhoenixRuntimeIT.java:167)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
> at
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
>
[jira] [Created] (HBASE-17122) Change in behavior when creating a scanner for a disabled table
Samarth Jain created HBASE-17122:
Summary: Change in behavior when creating a scanner for a disabled
table
Key: HBASE-17122
URL: https://issues.apache.org/jira/browse/HBASE-17122
Project: HBase
Issue Type: Bug
Reporter: Samarth Jain
{code}
@Test
public void testQueryingDisabledTable() throws Exception {
try (Connection conn = DriverManager.getConnection(getUrl())) {
String tableName = generateUniqueName();
conn.createStatement().execute(
"CREATE TABLE " + tableName
+ " (k1 VARCHAR NOT NULL, k2 VARCHAR, CONSTRAINT PK PRIMARY
KEY(K1,K2)) ");
try (HBaseAdmin admin =
conn.unwrap(PhoenixConnection.class).getQueryServices().getAdmin()) {
admin.disableTable(Bytes.toBytes(tableName));
}
String query = "SELECT * FROM " + tableName + " WHERE 1=1";
try (Connection conn2 = DriverManager.getConnection(getUrl())) {
try (ResultSet rs =
conn2.createStatement().executeQuery(query)) {
assertFalse(rs.next());
}
}
}
}
{code}
This is a phoenix specific test case. I will try an come up with something
using the HBase API. But the gist is that with HBase 0.98.21 and beyond, we are
seeing that creating a scanner is throwing a NotServingRegionException.
Stacktrace for NotServingRegionException
{code}
org.apache.phoenix.exception.PhoenixIOException:
org.apache.phoenix.exception.PhoenixIOException: callTimeout=120,
callDuration=9000104: row '' on table 'T01' at
region=T01,,1479429739864.643dde31cc19b549192576eea7791a6f.,
hostname=localhost,60022,1479429692090, seqNum=1
at
org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:113)
at
org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:752)
at
org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:696)
at
org.apache.phoenix.iterate.ConcatResultIterator.getIterators(ConcatResultIterator.java:50)
at
org.apache.phoenix.iterate.ConcatResultIterator.currentIterator(ConcatResultIterator.java:97)
at
org.apache.phoenix.iterate.ConcatResultIterator.next(ConcatResultIterator.java:117)
at
org.apache.phoenix.jdbc.PhoenixResultSet.next(PhoenixResultSet.java:778)
at
org.apache.phoenix.end2end.PhoenixRuntimeIT.testQueryingDisabledTable(PhoenixRuntimeIT.java:167)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at
org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
at
org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
Caused by: java.util.concurrent.ExecutionException:
org.apache.phoenix.exception.PhoenixIOException: callTimeout=120,
[jira] [Commented] (HBASE-15324) Jitter may cause desiredMaxFileSize overflow in ConstantSizeRegionSplitPolicy and trigger unexpected split
[
https://issues.apache.org/jira/browse/HBASE-15324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675207#comment-15675207
]
Hudson commented on HBASE-15324:
SUCCESS: Integrated in Jenkins build HBase-1.1-JDK7 #1820 (See
[https://builds.apache.org/job/HBase-1.1-JDK7/1820/])
HBASE-17058 Lower epsilon used for jitter verification from HBASE-15324
(esteban: rev 7c58547a37c85a148f481398819badd7c26129bc)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java
> Jitter may cause desiredMaxFileSize overflow in ConstantSizeRegionSplitPolicy
> and trigger unexpected split
> --
>
> Key: HBASE-15324
> URL: https://issues.apache.org/jira/browse/HBASE-15324
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.1.3
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.5, 1.1.8
>
> Attachments: HBASE-15324.patch, HBASE-15324_v2.patch,
> HBASE-15324_v3.patch, HBASE-15324_v3.patch
>
>
> We introduce jitter for region split decision in HBASE-13412, but the
> following line in {{ConstantSizeRegionSplitPolicy}} may cause long value
> overflow if MAX_FILESIZE is specified to Long.MAX_VALUE:
> {code}
> this.desiredMaxFileSize += (long)(desiredMaxFileSize * (RANDOM.nextFloat() -
> 0.5D) * jitter);
> {code}
> In our case we specify MAX_FILESIZE to Long.MAX_VALUE to prevent target
> region to split.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17058) Lower epsilon used for jitter verification from HBASE-15324
[ https://issues.apache.org/jira/browse/HBASE-17058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675210#comment-15675210 ] Hudson commented on HBASE-17058: SUCCESS: Integrated in Jenkins build HBase-1.1-JDK7 #1820 (See [https://builds.apache.org/job/HBase-1.1-JDK7/1820/]) HBASE-17058 Lower epsilon used for jitter verification from HBASE-15324 (esteban: rev 7c58547a37c85a148f481398819badd7c26129bc) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java > Lower epsilon used for jitter verification from HBASE-15324 > --- > > Key: HBASE-17058 > URL: https://issues.apache.org/jira/browse/HBASE-17058 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4 >Reporter: Esteban Gutierrez >Assignee: Esteban Gutierrez > Fix For: 2.0.0, 1.4.0, 1.3.1, 1.2.5, 1.1.8 > > Attachments: HBASE-17058.master.001.patch > > > The current epsilon used is 1E-6 and its too big it might overflow the > desiredMaxFileSize. A trivial fix is to lower the epsilon to 2^-52 or even > 2^-53. An option to consider too is just to shift the jitter to always > decrement hbase.hregion.max.filesize (MAX_FILESIZE) instead of increase the > size of the region and having to deal with the round off. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15324) Jitter may cause desiredMaxFileSize overflow in ConstantSizeRegionSplitPolicy and trigger unexpected split
[
https://issues.apache.org/jira/browse/HBASE-15324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675184#comment-15675184
]
Hudson commented on HBASE-15324:
SUCCESS: Integrated in Jenkins build HBase-1.3-JDK7 #69 (See
[https://builds.apache.org/job/HBase-1.3-JDK7/69/])
HBASE-17058 Lower epsilon used for jitter verification from HBASE-15324
(esteban: rev f4ed43e06108687488ebb161086b00274d172bc0)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java
> Jitter may cause desiredMaxFileSize overflow in ConstantSizeRegionSplitPolicy
> and trigger unexpected split
> --
>
> Key: HBASE-15324
> URL: https://issues.apache.org/jira/browse/HBASE-15324
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.1.3
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.5, 1.1.8
>
> Attachments: HBASE-15324.patch, HBASE-15324_v2.patch,
> HBASE-15324_v3.patch, HBASE-15324_v3.patch
>
>
> We introduce jitter for region split decision in HBASE-13412, but the
> following line in {{ConstantSizeRegionSplitPolicy}} may cause long value
> overflow if MAX_FILESIZE is specified to Long.MAX_VALUE:
> {code}
> this.desiredMaxFileSize += (long)(desiredMaxFileSize * (RANDOM.nextFloat() -
> 0.5D) * jitter);
> {code}
> In our case we specify MAX_FILESIZE to Long.MAX_VALUE to prevent target
> region to split.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16956) Refactor FavoredNodePlan to use regionNames as keys
[ https://issues.apache.org/jira/browse/HBASE-16956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675187#comment-15675187 ] Hudson commented on HBASE-16956: SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #1971 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1971/]) HBASE-16956 Refactor FavoredNodePlan to use regionNames as keys (ddas: rev 5753d18c705d55e207ef56154ac5f4512d1b01dd) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/FavoredNodesPlan.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionPlacementMaintainer.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRegionPlacement.java > Refactor FavoredNodePlan to use regionNames as keys > --- > > Key: HBASE-16956 > URL: https://issues.apache.org/jira/browse/HBASE-16956 > Project: HBase > Issue Type: Sub-task >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16956.branch-1.001.patch, > HBASE-16956.master.001.patch, HBASE-16956.master.002.patch, > HBASE-16956.master.003.patch, HBASE-16956.master.004.patch, > HBASE-16956.master.005.patch, HBASE-16956.master.006.patch, > HBASE-16956.master.007.patch > > > We would like to rely on the FNPlan cache whether a region is offline or not. > Sticking to regionNames as keys makes that possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15324) Jitter may cause desiredMaxFileSize overflow in ConstantSizeRegionSplitPolicy and trigger unexpected split
[
https://issues.apache.org/jira/browse/HBASE-15324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675188#comment-15675188
]
Hudson commented on HBASE-15324:
SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #1971 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/1971/])
HBASE-17058 Lower epsilon used for jitter verification from HBASE-15324
(esteban: rev 7c6e839f6a98cf2c3ed37109318632db13b4a0df)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java
> Jitter may cause desiredMaxFileSize overflow in ConstantSizeRegionSplitPolicy
> and trigger unexpected split
> --
>
> Key: HBASE-15324
> URL: https://issues.apache.org/jira/browse/HBASE-15324
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.1.3
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.5, 1.1.8
>
> Attachments: HBASE-15324.patch, HBASE-15324_v2.patch,
> HBASE-15324_v3.patch, HBASE-15324_v3.patch
>
>
> We introduce jitter for region split decision in HBASE-13412, but the
> following line in {{ConstantSizeRegionSplitPolicy}} may cause long value
> overflow if MAX_FILESIZE is specified to Long.MAX_VALUE:
> {code}
> this.desiredMaxFileSize += (long)(desiredMaxFileSize * (RANDOM.nextFloat() -
> 0.5D) * jitter);
> {code}
> In our case we specify MAX_FILESIZE to Long.MAX_VALUE to prevent target
> region to split.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17058) Lower epsilon used for jitter verification from HBASE-15324
[ https://issues.apache.org/jira/browse/HBASE-17058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675189#comment-15675189 ] Hudson commented on HBASE-17058: SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #1971 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1971/]) HBASE-17058 Lower epsilon used for jitter verification from HBASE-15324 (esteban: rev 7c6e839f6a98cf2c3ed37109318632db13b4a0df) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java > Lower epsilon used for jitter verification from HBASE-15324 > --- > > Key: HBASE-17058 > URL: https://issues.apache.org/jira/browse/HBASE-17058 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4 >Reporter: Esteban Gutierrez >Assignee: Esteban Gutierrez > Fix For: 2.0.0, 1.4.0, 1.3.1, 1.2.5, 1.1.8 > > Attachments: HBASE-17058.master.001.patch > > > The current epsilon used is 1E-6 and its too big it might overflow the > desiredMaxFileSize. A trivial fix is to lower the epsilon to 2^-52 or even > 2^-53. An option to consider too is just to shift the jitter to always > decrement hbase.hregion.max.filesize (MAX_FILESIZE) instead of increase the > size of the region and having to deal with the round off. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17058) Lower epsilon used for jitter verification from HBASE-15324
[ https://issues.apache.org/jira/browse/HBASE-17058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675185#comment-15675185 ] Hudson commented on HBASE-17058: SUCCESS: Integrated in Jenkins build HBase-1.3-JDK7 #69 (See [https://builds.apache.org/job/HBase-1.3-JDK7/69/]) HBASE-17058 Lower epsilon used for jitter verification from HBASE-15324 (esteban: rev f4ed43e06108687488ebb161086b00274d172bc0) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java > Lower epsilon used for jitter verification from HBASE-15324 > --- > > Key: HBASE-17058 > URL: https://issues.apache.org/jira/browse/HBASE-17058 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4 >Reporter: Esteban Gutierrez >Assignee: Esteban Gutierrez > Fix For: 2.0.0, 1.4.0, 1.3.1, 1.2.5, 1.1.8 > > Attachments: HBASE-17058.master.001.patch > > > The current epsilon used is 1E-6 and its too big it might overflow the > desiredMaxFileSize. A trivial fix is to lower the epsilon to 2^-52 or even > 2^-53. An option to consider too is just to shift the jitter to always > decrement hbase.hregion.max.filesize (MAX_FILESIZE) instead of increase the > size of the region and having to deal with the round off. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15324) Jitter may cause desiredMaxFileSize overflow in ConstantSizeRegionSplitPolicy and trigger unexpected split
[
https://issues.apache.org/jira/browse/HBASE-15324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675176#comment-15675176
]
Hudson commented on HBASE-15324:
SUCCESS: Integrated in Jenkins build HBase-1.1-JDK8 #1904 (See
[https://builds.apache.org/job/HBase-1.1-JDK8/1904/])
HBASE-17058 Lower epsilon used for jitter verification from HBASE-15324
(esteban: rev 7c58547a37c85a148f481398819badd7c26129bc)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java
> Jitter may cause desiredMaxFileSize overflow in ConstantSizeRegionSplitPolicy
> and trigger unexpected split
> --
>
> Key: HBASE-15324
> URL: https://issues.apache.org/jira/browse/HBASE-15324
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.1.3
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.5, 1.1.8
>
> Attachments: HBASE-15324.patch, HBASE-15324_v2.patch,
> HBASE-15324_v3.patch, HBASE-15324_v3.patch
>
>
> We introduce jitter for region split decision in HBASE-13412, but the
> following line in {{ConstantSizeRegionSplitPolicy}} may cause long value
> overflow if MAX_FILESIZE is specified to Long.MAX_VALUE:
> {code}
> this.desiredMaxFileSize += (long)(desiredMaxFileSize * (RANDOM.nextFloat() -
> 0.5D) * jitter);
> {code}
> In our case we specify MAX_FILESIZE to Long.MAX_VALUE to prevent target
> region to split.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17058) Lower epsilon used for jitter verification from HBASE-15324
[ https://issues.apache.org/jira/browse/HBASE-17058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675177#comment-15675177 ] Hudson commented on HBASE-17058: SUCCESS: Integrated in Jenkins build HBase-1.1-JDK8 #1904 (See [https://builds.apache.org/job/HBase-1.1-JDK8/1904/]) HBASE-17058 Lower epsilon used for jitter verification from HBASE-15324 (esteban: rev 7c58547a37c85a148f481398819badd7c26129bc) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java > Lower epsilon used for jitter verification from HBASE-15324 > --- > > Key: HBASE-17058 > URL: https://issues.apache.org/jira/browse/HBASE-17058 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4 >Reporter: Esteban Gutierrez >Assignee: Esteban Gutierrez > Fix For: 2.0.0, 1.4.0, 1.3.1, 1.2.5, 1.1.8 > > Attachments: HBASE-17058.master.001.patch > > > The current epsilon used is 1E-6 and its too big it might overflow the > desiredMaxFileSize. A trivial fix is to lower the epsilon to 2^-52 or even > 2^-53. An option to consider too is just to shift the jitter to always > decrement hbase.hregion.max.filesize (MAX_FILESIZE) instead of increase the > size of the region and having to deal with the round off. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16708) Expose endpoint Coprocessor name in "responseTooSlow" log messages
[
https://issues.apache.org/jira/browse/HBASE-16708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675170#comment-15675170
]
Hadoop QA commented on HBASE-16708:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
9s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
46s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 54s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 3s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
7s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 37m 0s {color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12839456/HBASE-ShortName.patch
|
| JIRA Issue | HBASE-16708 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 21a59601b464 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 046d1a5 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4522/testReport/ |
| modules | C: hbase-client U: hbase-client |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4522/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Expose endpoint Coprocessor name in "responseTooSlow" log messages
> --
>
> Key: HBASE-16708
> URL: https://issues.apache.org/jira/browse/HBASE-16708
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Nick
[jira] [Updated] (HBASE-16708) Expose endpoint Coprocessor name in "responseTooSlow" log messages
[ https://issues.apache.org/jira/browse/HBASE-16708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Liang updated HBASE-16708: - Attachment: HBASE-ShortName.patch > Expose endpoint Coprocessor name in "responseTooSlow" log messages > -- > > Key: HBASE-16708 > URL: https://issues.apache.org/jira/browse/HBASE-16708 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Nick Dimiduk >Assignee: Yi Liang > Fix For: 2.0.0 > > Attachments: HBASE-16708-V1.patch, HBASE-16708-V2.patch, > HBASE-ShortName.patch > > > Operational diagnostics of a Phoenix install would be easier if we included > which endpoint coprocessor was being called in this responseTooSlow WARN > message. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14123) HBase Backup/Restore Phase 2
[
https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675035#comment-15675035
]
stack commented on HBASE-14123:
---
More feedback from usage.
So, in backup usage, the backup root dir is called backup_root. In restore, it
is called backup_path. I presume these are meant to refer to the same thing?
In restore, the id for the backup is the backup_id but in describe when I am to
pass a backup id, it is called backupId. Ditto on delete. In the dump of the
history, the backup id is called 'ID'.
Suggest you use same name for the arg everywhere. The inconsistency makes the
tool appear untrustworthy.
Are all backup ids prefixed with backup_? Seems superfluous.
Looking at the backup history. What is Progress? What is 100 w/l a percent on
it? (I asked about this previous I believe).
Want to fix this log message so it doesn't have java object ids in it?
2016-11-17 14:11:14,639 DEBUG [LoadIncrementalHFiles-2]
mapreduce.LoadIncrementalHFiles: Going to connect to server
region=x_1_restore,999adfe23d3bda876af50397a462f7d8-18028,1479420656653.15bd3f57fb7364c5d738f8bf7be7a32c.,
hostname=ve0540.halxg.cloudera.com,16020,1479274141369, seqNum=2 for row
999adfe23d3bda876af50397a462f7d8-18028 with hfile group
[{[B@68c87fc3,hdfs://ve0524.halxg.cloudera.com:8020/user/stack/hbase-staging/restore/fa3bff72113aee83c50c812c722cd8f4/test_cf/33a9cac5e3944e338005dc87e70f85fb},
{[B@68c87fc3,hdfs://ve0524.halxg.cloudera.com:8020/user/stack/hbase-staging/restore/fa3bff72113aee83c50c812c722cd8f4/test_cf/935f048447394f93856e917b8dc34a8f}]
I see this in the log:
{code}
19 2016-11-17 14:13:36,596 INFO [main] util.RestoreServerUtil: Creating
target table 'x_1_restore2'
20 2016-11-17 14:13:36,596 DEBUG [main] util.RestoreServerUtil: Parsing
region dir:
hdfs://ve0524.halxg.cloudera.com:8020/user/stack/backup/backup_1479419995738/default/x_1/archive/data/default/x_1/049560f124c122186f174168988f4eec
21 2016-11-17 14:13:36,598 DEBUG [main] util.RestoreServerUtil: Parsing
family dir
[hdfs://ve0524.halxg.cloudera.com:8020/user/stack/backup/backup_1479419995738/default/x_1/archive/data/default/x_1/049560f124c122186f174168988f4eec/test_cf
in region
[hdfs://ve0524.halxg.cloudera.com:8020/user/stack/backup/backup_1479419995738/default/x_1/archive/data/default/x_1/049560f124c122186f174168988f4eec]
22 2016-11-17 14:13:36,615 INFO [main] hfile.CacheConfig: Allocating
LruBlockCache size=5.29 GB, blockSize=64 KB
23 2016-11-17 14:13:36,628 DEBUG [main] hfile.CacheConfig: Trying to use
Internal l2 cache
{code}
Is util supposed to be using a block cache? Seems to be doing this a lot (Lots
of spew though restoring a one row table).
What does this mean?
253 2016-11-17 14:13:39,782 DEBUG [main] util.RestoreServerUtil: File
hdfs://ve0524.halxg.cloudera.com:8020/user/stack/backup/backup_1479419995738/default/x_1/archive/data/default/x_1
on local cluster, back it up before restore
Is this a full copy of the backup to elsewhere?
296 2016-11-17 14:13:47,907 DEBUG [main] util.RestoreServerUtil: Copied to
temporary path on local cluster: /user/stack/hbase-staging/restore
Why this, the 62minute timeout? Is it excessive?
298 2016-11-17 14:13:47,955 INFO [main] util.RestoreServerUtil: Setting
configuration for restore with LoadIncrementalHFile: hbase.rpc.timeout to 62
minutes, to handle the number of files in backup
/user/stack/hbase-staging/restore
Why when I run an incremental backup, I get summary from the
FullTableBackupClient? I'd think I'd get report from the incremental version.
I'd also expect a summary of what was in the incremental?
I'm trying to figure out what these tools are doing when they run from log
messages but not sure. Is there a writeup somewhere? Looks like the technical
background in user guide could be beefed up some. Later.
After reading the user doc., I am unclear on how to restore from a full backup
and two incrementals. I have to manually apply these in order manually? Is
there a way to ask the tool to do this for me?
> HBase Backup/Restore Phase 2
>
>
> Key: HBASE-14123
> URL: https://issues.apache.org/jira/browse/HBASE-14123
> Project: HBase
> Issue Type: Umbrella
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: 14123-master.v14.txt, 14123-master.v15.txt,
> 14123-master.v16.txt, 14123-master.v17.txt, 14123-master.v18.txt,
> 14123-master.v19.txt, 14123-master.v2.txt, 14123-master.v20.txt,
> 14123-master.v21.txt, 14123-master.v24.txt, 14123-master.v25.txt,
> 14123-master.v27.txt, 14123-master.v28.txt, 14123-master.v29.full.txt,
> 14123-master.v3.txt, 14123-master.v30.txt, 14123-master.v31.txt,
> 14123-master.v32.txt, 14123-master.v33.txt, 14123-master.v34.txt,
>
[jira] [Commented] (HBASE-14123) HBase Backup/Restore Phase 2
[ https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674996#comment-15674996 ] stack commented on HBASE-14123: --- What was addressed and how? Thanks. > HBase Backup/Restore Phase 2 > > > Key: HBASE-14123 > URL: https://issues.apache.org/jira/browse/HBASE-14123 > Project: HBase > Issue Type: Umbrella >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Blocker > Fix For: 2.0.0 > > Attachments: 14123-master.v14.txt, 14123-master.v15.txt, > 14123-master.v16.txt, 14123-master.v17.txt, 14123-master.v18.txt, > 14123-master.v19.txt, 14123-master.v2.txt, 14123-master.v20.txt, > 14123-master.v21.txt, 14123-master.v24.txt, 14123-master.v25.txt, > 14123-master.v27.txt, 14123-master.v28.txt, 14123-master.v29.full.txt, > 14123-master.v3.txt, 14123-master.v30.txt, 14123-master.v31.txt, > 14123-master.v32.txt, 14123-master.v33.txt, 14123-master.v34.txt, > 14123-master.v35.txt, 14123-master.v36.txt, 14123-master.v37.txt, > 14123-master.v38.txt, 14123-master.v5.txt, 14123-master.v6.txt, > 14123-master.v7.txt, 14123-master.v8.txt, 14123-master.v9.txt, 14123-v14.txt, > HBASE-14123-for-7912-v1.patch, HBASE-14123-for-7912-v6.patch, > HBASE-14123-v1.patch, HBASE-14123-v10.patch, HBASE-14123-v11.patch, > HBASE-14123-v12.patch, HBASE-14123-v13.patch, HBASE-14123-v15.patch, > HBASE-14123-v16.patch, HBASE-14123-v2.patch, HBASE-14123-v3.patch, > HBASE-14123-v4.patch, HBASE-14123-v5.patch, HBASE-14123-v6.patch, > HBASE-14123-v7.patch, HBASE-14123-v9.patch > > > Phase 2 umbrella JIRA. See HBASE-7912 for design document and description. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-7912) HBase Backup/Restore Based on HBase Snapshot
[ https://issues.apache.org/jira/browse/HBASE-7912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674988#comment-15674988 ] stack commented on HBASE-7912: -- Any follow up here? > HBase Backup/Restore Based on HBase Snapshot > > > Key: HBASE-7912 > URL: https://issues.apache.org/jira/browse/HBASE-7912 > Project: HBase > Issue Type: Sub-task >Reporter: Richard Ding >Assignee: Vladimir Rodionov > Labels: backup > Fix For: 2.0.0 > > Attachments: Backup-and-Restore-Apache_19Sep2016.pdf, > Backup-and-Restore-Apache_9Sep2016.pdf, HBaseBackupAndRestore - v0.8.pdf, > HBaseBackupAndRestore -0.91.pdf, HBaseBackupAndRestore-v0.9.pdf, > HBaseBackupAndRestore.pdf, HBaseBackupRestore-Jira-7912-DesignDoc-v1.pdf, > HBaseBackupRestore-Jira-7912-DesignDoc-v2.pdf, > HBaseBackupRestore-Jira-7912-v4.pdf, HBaseBackupRestore-Jira-7912-v5 .pdf, > HBaseBackupRestore-Jira-7912-v6.pdf, HBase_BackupRestore-Jira-7912-CLI-v1.pdf > > > Finally, we completed the implementation of our backup/restore solution, and > would like to share with community through this jira. > We are leveraging existing hbase snapshot feature, and provide a general > solution to common users. Our full backup is using snapshot to capture > metadata locally and using exportsnapshot to move data to another cluster; > the incremental backup is using offline-WALplayer to backup HLogs; we also > leverage global distribution rolllog and flush to improve performance; other > added-on values such as convert, merge, progress report, and CLI commands. So > that a common user can backup hbase data without in-depth knowledge of hbase. > Our solution also contains some usability features for enterprise users. > The detail design document and CLI command will be attached in this jira. We > plan to use 10~12 subtasks to share each of the following features, and > document the detail implement in the subtasks: > * *Full Backup* : provide local and remote back/restore for a list of tables > * *offline-WALPlayer* to convert HLog to HFiles offline (for incremental > backup) > * *distributed* Logroll and distributed flush > * Backup *Manifest* and history > * *Incremental* backup: to build on top of full backup as daily/weekly backup > * *Convert* incremental backup WAL files into hfiles > * *Merge* several backup images into one(like merge weekly into monthly) > * *add and remove* table to and from Backup image > * *Cancel* a backup process > * backup progress *status* > * full backup based on *existing snapshot* > *-* > *Below is the original description, to keep here as the history for the > design and discussion back in 2013* > There have been attempts in the past to come up with a viable HBase > backup/restore solution (e.g., HBASE-4618). Recently, there are many > advancements and new features in HBase, for example, FileLink, Snapshot, and > Distributed Barrier Procedure. This is a proposal for a backup/restore > solution that utilizes these new features to achieve better performance and > consistency. > > A common practice of backup and restore in database is to first take full > baseline backup, and then periodically take incremental backup that capture > the changes since the full baseline backup. HBase cluster can store massive > amount data. Combination of full backups with incremental backups has > tremendous benefit for HBase as well. The following is a typical scenario > for full and incremental backup. > # The user takes a full backup of a table or a set of tables in HBase. > # The user schedules periodical incremental backups to capture the changes > from the full backup, or from last incremental backup. > # The user needs to restore table data to a past point of time. > # The full backup is restored to the table(s) or to different table name(s). > Then the incremental backups that are up to the desired point in time are > applied on top of the full backup. > We would support the following key features and capabilities. > * Full backup uses HBase snapshot to capture HFiles. > * Use HBase WALs to capture incremental changes, but we use bulk load of > HFiles for fast incremental restore. > * Support single table or a set of tables, and column family level backup and > restore. > * Restore to different table names. > * Support adding additional tables or CF to backup set without interruption > of incremental backup schedule. > * Support rollup/combining of incremental backups into longer period and > bigger incremental backups. > * Unified command line interface for all the above. > The solution will support HBase backup to FileSystem, either on the same > cluster or across clusters. It has the
[jira] [Commented] (HBASE-17118) StoreScanner leaked in KeyValueHeap
[
https://issues.apache.org/jira/browse/HBASE-17118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674946#comment-15674946
]
Hudson commented on HBASE-17118:
SUCCESS: Integrated in Jenkins build HBase-1.4 #537 (See
[https://builds.apache.org/job/HBase-1.4/537/])
HBASE-17118 StoreScanner leaked in KeyValueHeap (binlijin) (tedyu: rev
d722b2aab72459272dc1bb05c118568c88c797ca)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueHeap.java
> StoreScanner leaked in KeyValueHeap
> ---
>
> Key: HBASE-17118
> URL: https://issues.apache.org/jira/browse/HBASE-17118
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: binlijin
>Assignee: binlijin
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17118-master_v1.patch,
> HBASE-17118-master_v2.patch, HBASE-17118-master_v3.patch,
> HBASE-17118-master_v4.patch, HBASE-17118-master_v5.patch,
> HBASE-17118.branch-1.v1.patch, StoreScanner.png, StoreScannerLeakHeap.png
>
>
> KeyValueHeap#generalizedSeek
> KeyValueScanner scanner = current;
> while (scanner != null) {
> Cell topKey = scanner.peek();
> ..
> boolean seekResult;
> if (isLazy && heap.size() > 0) {
> // If there is only one scanner left, we don't do lazy seek.
> seekResult = scanner.requestSeek(seekKey, forward, useBloom);
> } else {
> seekResult = NonLazyKeyValueScanner.doRealSeek(scanner, seekKey,
> forward);
> }
> ..
> scanner = heap.poll();
> }
> (1) scanner = heap.poll(); Retrieves and removes the head of this queue
> (2) scanner.requestSeek(seekKey, forward, useBloom); or
> NonLazyKeyValueScanner.doRealSeek(scanner, seekKey, forward);
> throw exception, and scanner will have no chance to close, so will cause the
> scanner leak.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17112) Prevent setting timestamp of delta operations the same as previous value's
[ https://issues.apache.org/jira/browse/HBASE-17112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674947#comment-15674947 ] Hudson commented on HBASE-17112: SUCCESS: Integrated in Jenkins build HBase-1.4 #537 (See [https://builds.apache.org/job/HBase-1.4/537/]) HBASE-17112 Prevent setting timestamp of delta operations the same as (tedyu: rev c6f1b6e62456a02ffe367c1574946a10262d5a03) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java > Prevent setting timestamp of delta operations the same as previous value's > -- > > Key: HBASE-17112 > URL: https://issues.apache.org/jira/browse/HBASE-17112 > Project: HBase > Issue Type: Bug >Affects Versions: 1.1.7, 0.98.23, 1.2.4 >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17112-branch-1-v1.patch, > HBASE-17112-branch-1-v1.patch, HBASE-17112-branch-1.1-v1.patch, > HBASE-17112-branch-1.1-v1.patch, HBASE-17112-v1.patch, HBASE-17112-v2.patch, > HBASE-17112-v2.patch > > > In delta operations, Increment and Append. We will read current value first > and then write the new whole result into WAL as the type of Put with current > timestamp. If the previous ts is larger than current ts, we will use the > previous ts. > If we have two Puts with same TS, we will ignore the Put with lower sequence > id. It is not friendly with versioning. And for replication we will drop > sequence id while writing to peer cluster so in the slave we don't know what > the order they are being written. If the pushing is disordered, the result > will be wrong. > We can set the new ts to previous+1 if the previous is not less than now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16940) Address review of "Backup/Restore (HBASE-7912, HBASE-14030, HBASE-14123) mega patch" posted on RB
[ https://issues.apache.org/jira/browse/HBASE-16940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674927#comment-15674927 ] stack commented on HBASE-16940: --- We want to have some accounting of what has been addressed from rb either here, in the patch commit message, or up on rb (Preferrably in all three places)? > Address review of "Backup/Restore (HBASE-7912, HBASE-14030, HBASE-14123) mega > patch" posted on RB > -- > > Key: HBASE-16940 > URL: https://issues.apache.org/jira/browse/HBASE-16940 > Project: HBase > Issue Type: Sub-task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > Attachments: HBASE-16940-v1.patch, HBASE-16940-v2.patch, > HBASE-16940.addendum, HBASE-16940.addendum.2 > > > Review 52748 remaining issues. > https://reviews.apache.org/r/52748 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-16335) update to latest apache parent pom
[ https://issues.apache.org/jira/browse/HBASE-16335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-16335. --- Resolution: Fixed Hadoop Flags: Reviewed Pushed to branch-1 and master. Thanks for the patch [~Jan Hentschel] > update to latest apache parent pom > -- > > Key: HBASE-16335 > URL: https://issues.apache.org/jira/browse/HBASE-16335 > Project: HBase > Issue Type: Task > Components: build, dependencies >Reporter: Sean Busbey >Assignee: Jan Hentschel > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16335.master.001.patch > > > Our apache parent pom is currently ~4 years old. update to latest. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16708) Expose endpoint Coprocessor name in "responseTooSlow" log messages
[ https://issues.apache.org/jira/browse/HBASE-16708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674874#comment-15674874 ] Yi Liang edited comment on HBASE-16708 at 11/17/16 9:37 PM: [~ndimiduk], As you can see in the patch, the RpcServer need go to region to get endpoint coprocessor implementation class name, since the coprocessor implementation class name is only registered in each region, master or regionserver. I doubt it is worthy to go that deeper to get those information, since I think the short service name(i.e SumServices) may be enough for us to figure out the what is the real coprocessor implement class(i.e org.myname.hbase.coprocessor.endpoint.SumEndPoint). If just print short service name, RpcServer do not need to get access to region, and can make code more beautiful. And I will provide patch for only printing short service, so you can see the difference. was (Author: easyliangjob): [~ndimiduk], As you can see in the patch, the RpcServer need go to region to get endpoint coprocessor implementation class name, since the coprocessor implementation class name is only registered in each region, master or regionserver. I doubt it is worthy to go that deeper to get those information, since I think the short service name(i.e SumServices) may be enough for us to figure out the what is the real coprocessor implement class(i.e org.myname.hbase.coprocessor.endpoint.SumEndPoint). If just print short service name, RpcServer do not need to get access to region, and can make code more beautiful. > Expose endpoint Coprocessor name in "responseTooSlow" log messages > -- > > Key: HBASE-16708 > URL: https://issues.apache.org/jira/browse/HBASE-16708 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Nick Dimiduk >Assignee: Yi Liang > Fix For: 2.0.0 > > Attachments: HBASE-16708-V1.patch, HBASE-16708-V2.patch > > > Operational diagnostics of a Phoenix install would be easier if we included > which endpoint coprocessor was being called in this responseTooSlow WARN > message. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16708) Expose endpoint Coprocessor name in "responseTooSlow" log messages
[ https://issues.apache.org/jira/browse/HBASE-16708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674874#comment-15674874 ] Yi Liang commented on HBASE-16708: -- [~ndimiduk], As you can see in the patch, the RpcServer need go to region to get endpoint coprocessor implementation class name, since the coprocessor implementation class name is only registered in each region, master or regionserver. I doubt it is worthy to go that deeper to get those information, since I think the short service name(i.e SumServices) may be enough for us to figure out the what is the real coprocessor implement class(i.e org.myname.hbase.coprocessor.endpoint.SumEndPoint). If just print short service name, RpcServer do not need to get access to region, and can make code more beautiful. > Expose endpoint Coprocessor name in "responseTooSlow" log messages > -- > > Key: HBASE-16708 > URL: https://issues.apache.org/jira/browse/HBASE-16708 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Nick Dimiduk >Assignee: Yi Liang > Fix For: 2.0.0 > > Attachments: HBASE-16708-V1.patch, HBASE-16708-V2.patch > > > Operational diagnostics of a Phoenix install would be easier if we included > which endpoint coprocessor was being called in this responseTooSlow WARN > message. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-16999) [Master] Inform RegionServers on size quota violations
[ https://issues.apache.org/jira/browse/HBASE-16999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser reassigned HBASE-16999: -- Assignee: Josh Elser > [Master] Inform RegionServers on size quota violations > -- > > Key: HBASE-16999 > URL: https://issues.apache.org/jira/browse/HBASE-16999 > Project: HBase > Issue Type: Sub-task >Reporter: Josh Elser >Assignee: Josh Elser > > The Master, after computing the total usage across all regionservers, needs > to inform the RegionServers about tables that need to change violation policy > enforcement (enable or disable enforcement). > RPC, ZK, or ProcV2's notification bus are some examples of ways this could be > implemented. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16489) Configuration parsing
[ https://issues.apache.org/jira/browse/HBASE-16489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674857#comment-15674857 ] Enis Soztutar commented on HBASE-16489: --- bq. The build/ directory is a temporary directory where we store object files, shared libs and other binaries on compilation. I can change the unit tests to create temporary xml configuration files under build/test-data Sounds good. bq. Also, We never delete from or write to /etc/hbase/conf. It is used only for reading. Ok, but we cannot even assume that the machine running the unit test will have /etc/hbase/conf directory created. Better to confine everything under temp directories in build/test-data. bq. The idea behind using loader.Load(conf); was that we can use loader to load files from different search paths and use the same configuration object with the updated properties instead of using conf = loader.Load(); where a new object will be returned every time It is fine to return a new object. In almost all of the usage, the user will be loding default file names from default search path. We do not need to complicate stuff for reloading using the same config object. Lets make it so that default usage is only 2 lines instead of 5 lines, similar to the HDFS Configuration case. > Configuration parsing > - > > Key: HBASE-16489 > URL: https://issues.apache.org/jira/browse/HBASE-16489 > Project: HBase > Issue Type: Sub-task >Reporter: Sudeep Sunthankar >Assignee: Sudeep Sunthankar > Attachments: HBASE-16489.HBASE-14850.v1.patch, > HBASE-16489.HBASE-14850.v2.patch, HBASE-16489.HBASE-14850.v3.patch, > HBASE-16489.HBASE-14850.v4.patch > > > Reading hbase-site.xml is required to read various properties viz. > zookeeper-quorum, client retires etc. We can either use Apache Xerces or > Boost libraries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16708) Expose endpoint Coprocessor name in "responseTooSlow" log messages
[
https://issues.apache.org/jira/browse/HBASE-16708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674812#comment-15674812
]
Hadoop QA commented on HBASE-16708:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
58s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
39s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
29m 28s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
58s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 32s
{color} | {color:red} hbase-server generated 1 new + 1 unchanged - 0 fixed = 2
total (was 1) {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 104m 34s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 146m 9s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.hbase.filter.TestFuzzyRowFilterEndToEnd |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12839417/HBASE-16708-V2.patch |
| JIRA Issue | HBASE-16708 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux a4d4810d0811 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 5753d18 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| javadoc |
https://builds.apache.org/job/PreCommit-HBASE-Build/4519/artifact/patchprocess/diff-javadoc-javadoc-hbase-server.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/4519/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/4519/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4519/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
[jira] [Updated] (HBASE-16335) update to latest apache parent pom
[ https://issues.apache.org/jira/browse/HBASE-16335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jan Hentschel updated HBASE-16335: -- Attachment: HBASE-16335.master.001.patch > update to latest apache parent pom > -- > > Key: HBASE-16335 > URL: https://issues.apache.org/jira/browse/HBASE-16335 > Project: HBase > Issue Type: Task > Components: build, dependencies >Reporter: Sean Busbey > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16335.master.001.patch > > > Our apache parent pom is currently ~4 years old. update to latest. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16335) update to latest apache parent pom
[ https://issues.apache.org/jira/browse/HBASE-16335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674798#comment-15674798 ] Jan Hentschel commented on HBASE-16335: --- Thanks [~stack], it did work. Already love this little tool. > update to latest apache parent pom > -- > > Key: HBASE-16335 > URL: https://issues.apache.org/jira/browse/HBASE-16335 > Project: HBase > Issue Type: Task > Components: build, dependencies >Reporter: Sean Busbey > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16335.master.001.patch > > > Our apache parent pom is currently ~4 years old. update to latest. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-16335) update to latest apache parent pom
[ https://issues.apache.org/jira/browse/HBASE-16335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jan Hentschel reassigned HBASE-16335: - Assignee: Jan Hentschel > update to latest apache parent pom > -- > > Key: HBASE-16335 > URL: https://issues.apache.org/jira/browse/HBASE-16335 > Project: HBase > Issue Type: Task > Components: build, dependencies >Reporter: Sean Busbey >Assignee: Jan Hentschel > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16335.master.001.patch > > > Our apache parent pom is currently ~4 years old. update to latest. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16335) update to latest apache parent pom
[ https://issues.apache.org/jira/browse/HBASE-16335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674778#comment-15674778 ] stack commented on HBASE-16335: --- Try now [~Jan Hentschel] (and thanks for using submit-jira tool) > update to latest apache parent pom > -- > > Key: HBASE-16335 > URL: https://issues.apache.org/jira/browse/HBASE-16335 > Project: HBase > Issue Type: Task > Components: build, dependencies >Reporter: Sean Busbey > Fix For: 2.0.0, 1.4.0 > > > Our apache parent pom is currently ~4 years old. update to latest. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HBASE-16940) Address review of "Backup/Restore (HBASE-7912, HBASE-14030, HBASE-14123) mega patch" posted on RB
[ https://issues.apache.org/jira/browse/HBASE-16940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu reopened HBASE-16940: > Address review of "Backup/Restore (HBASE-7912, HBASE-14030, HBASE-14123) mega > patch" posted on RB > -- > > Key: HBASE-16940 > URL: https://issues.apache.org/jira/browse/HBASE-16940 > Project: HBase > Issue Type: Sub-task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > Attachments: HBASE-16940-v1.patch, HBASE-16940-v2.patch, > HBASE-16940.addendum, HBASE-16940.addendum.2 > > > Review 52748 remaining issues. > https://reviews.apache.org/r/52748 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-16940) Address review of "Backup/Restore (HBASE-7912, HBASE-14030, HBASE-14123) mega patch" posted on RB
[ https://issues.apache.org/jira/browse/HBASE-16940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HBASE-16940. Resolution: Fixed > Address review of "Backup/Restore (HBASE-7912, HBASE-14030, HBASE-14123) mega > patch" posted on RB > -- > > Key: HBASE-16940 > URL: https://issues.apache.org/jira/browse/HBASE-16940 > Project: HBase > Issue Type: Sub-task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > Attachments: HBASE-16940-v1.patch, HBASE-16940-v2.patch, > HBASE-16940.addendum, HBASE-16940.addendum.2 > > > Review 52748 remaining issues. > https://reviews.apache.org/r/52748 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16335) update to latest apache parent pom
[ https://issues.apache.org/jira/browse/HBASE-16335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674762#comment-15674762 ] Jan Hentschel commented on HBASE-16335: --- I would like to pick up this one, but I get a 403 error when using dev-support/submit-patch.py and I'm not able to attach a patch. Could someone please give me the appropriate rights to do this? > update to latest apache parent pom > -- > > Key: HBASE-16335 > URL: https://issues.apache.org/jira/browse/HBASE-16335 > Project: HBase > Issue Type: Task > Components: build, dependencies >Reporter: Sean Busbey > Fix For: 2.0.0, 1.4.0 > > > Our apache parent pom is currently ~4 years old. update to latest. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16940) Address review of "Backup/Restore (HBASE-7912, HBASE-14030, HBASE-14123) mega patch" posted on RB
[ https://issues.apache.org/jira/browse/HBASE-16940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674758#comment-15674758 ] Vladimir Rodionov edited comment on HBASE-16940 at 11/17/16 8:46 PM: - [~te...@apache.org], lets commit addendum #2, but I will keep this JIRA open to address command-line formatting issues and some other stuff [~saint@gmail.com] has found during mega patch review. Here: https://issues.apache.org/jira/browse/HBASE-14123?focusedCommentId=15669647=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15669647 was (Author: vrodionov): [~te...@apache.org], lets commit addendum #2, but I will keep this JIRA open to address command-line formatting issues and some other stuff [~saint@gmail.com] has found during mega patch review. > Address review of "Backup/Restore (HBASE-7912, HBASE-14030, HBASE-14123) mega > patch" posted on RB > -- > > Key: HBASE-16940 > URL: https://issues.apache.org/jira/browse/HBASE-16940 > Project: HBase > Issue Type: Sub-task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > Attachments: HBASE-16940-v1.patch, HBASE-16940-v2.patch, > HBASE-16940.addendum, HBASE-16940.addendum.2 > > > Review 52748 remaining issues. > https://reviews.apache.org/r/52748 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16940) Address review of "Backup/Restore (HBASE-7912, HBASE-14030, HBASE-14123) mega patch" posted on RB
[ https://issues.apache.org/jira/browse/HBASE-16940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674758#comment-15674758 ] Vladimir Rodionov commented on HBASE-16940: --- [~te...@apache.org], lets commit addendum #2, but I will keep this JIRA open to address command-line formatting issues and some other stuff [~saint@gmail.com] has found during mega patch review. > Address review of "Backup/Restore (HBASE-7912, HBASE-14030, HBASE-14123) mega > patch" posted on RB > -- > > Key: HBASE-16940 > URL: https://issues.apache.org/jira/browse/HBASE-16940 > Project: HBase > Issue Type: Sub-task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > Attachments: HBASE-16940-v1.patch, HBASE-16940-v2.patch, > HBASE-16940.addendum, HBASE-16940.addendum.2 > > > Review 52748 remaining issues. > https://reviews.apache.org/r/52748 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16940) Address review of "Backup/Restore (HBASE-7912, HBASE-14030, HBASE-14123) mega patch" posted on RB
[ https://issues.apache.org/jira/browse/HBASE-16940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-16940: -- Attachment: HBASE-16940.addendum.2 Addendum #2. > Address review of "Backup/Restore (HBASE-7912, HBASE-14030, HBASE-14123) mega > patch" posted on RB > -- > > Key: HBASE-16940 > URL: https://issues.apache.org/jira/browse/HBASE-16940 > Project: HBase > Issue Type: Sub-task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > Attachments: HBASE-16940-v1.patch, HBASE-16940-v2.patch, > HBASE-16940.addendum, HBASE-16940.addendum.2 > > > Review 52748 remaining issues. > https://reviews.apache.org/r/52748 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17058) Lower epsilon used for jitter verification from HBASE-15324
[ https://issues.apache.org/jira/browse/HBASE-17058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esteban Gutierrez updated HBASE-17058: -- Fix Version/s: 1.2.5 > Lower epsilon used for jitter verification from HBASE-15324 > --- > > Key: HBASE-17058 > URL: https://issues.apache.org/jira/browse/HBASE-17058 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4 >Reporter: Esteban Gutierrez >Assignee: Esteban Gutierrez > Fix For: 2.0.0, 1.4.0, 1.3.1, 1.2.5, 1.1.8 > > Attachments: HBASE-17058.master.001.patch > > > The current epsilon used is 1E-6 and its too big it might overflow the > desiredMaxFileSize. A trivial fix is to lower the epsilon to 2^-52 or even > 2^-53. An option to consider too is just to shift the jitter to always > decrement hbase.hregion.max.filesize (MAX_FILESIZE) instead of increase the > size of the region and having to deal with the round off. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17058) Lower epsilon used for jitter verification from HBASE-15324
[ https://issues.apache.org/jira/browse/HBASE-17058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esteban Gutierrez updated HBASE-17058: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Lower epsilon used for jitter verification from HBASE-15324 > --- > > Key: HBASE-17058 > URL: https://issues.apache.org/jira/browse/HBASE-17058 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4 >Reporter: Esteban Gutierrez >Assignee: Esteban Gutierrez > Fix For: 2.0.0, 1.4.0, 1.3.1, 1.2.5, 1.1.8 > > Attachments: HBASE-17058.master.001.patch > > > The current epsilon used is 1E-6 and its too big it might overflow the > desiredMaxFileSize. A trivial fix is to lower the epsilon to 2^-52 or even > 2^-53. An option to consider too is just to shift the jitter to always > decrement hbase.hregion.max.filesize (MAX_FILESIZE) instead of increase the > size of the region and having to deal with the round off. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17058) Lower epsilon used for jitter verification from HBASE-15324
[ https://issues.apache.org/jira/browse/HBASE-17058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esteban Gutierrez updated HBASE-17058: -- Fix Version/s: 1.1.8 > Lower epsilon used for jitter verification from HBASE-15324 > --- > > Key: HBASE-17058 > URL: https://issues.apache.org/jira/browse/HBASE-17058 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4 >Reporter: Esteban Gutierrez >Assignee: Esteban Gutierrez > Fix For: 2.0.0, 1.4.0, 1.3.1, 1.1.8 > > Attachments: HBASE-17058.master.001.patch > > > The current epsilon used is 1E-6 and its too big it might overflow the > desiredMaxFileSize. A trivial fix is to lower the epsilon to 2^-52 or even > 2^-53. An option to consider too is just to shift the jitter to always > decrement hbase.hregion.max.filesize (MAX_FILESIZE) instead of increase the > size of the region and having to deal with the round off. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17058) Lower epsilon used for jitter verification from HBASE-15324
[ https://issues.apache.org/jira/browse/HBASE-17058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esteban Gutierrez updated HBASE-17058: -- Fix Version/s: 1.3.1 > Lower epsilon used for jitter verification from HBASE-15324 > --- > > Key: HBASE-17058 > URL: https://issues.apache.org/jira/browse/HBASE-17058 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4 >Reporter: Esteban Gutierrez >Assignee: Esteban Gutierrez > Fix For: 2.0.0, 1.4.0, 1.3.1, 1.1.8 > > Attachments: HBASE-17058.master.001.patch > > > The current epsilon used is 1E-6 and its too big it might overflow the > desiredMaxFileSize. A trivial fix is to lower the epsilon to 2^-52 or even > 2^-53. An option to consider too is just to shift the jitter to always > decrement hbase.hregion.max.filesize (MAX_FILESIZE) instead of increase the > size of the region and having to deal with the round off. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17058) Lower epsilon used for jitter verification from HBASE-15324
[ https://issues.apache.org/jira/browse/HBASE-17058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esteban Gutierrez updated HBASE-17058: -- Fix Version/s: 1.4.0 2.0.0 > Lower epsilon used for jitter verification from HBASE-15324 > --- > > Key: HBASE-17058 > URL: https://issues.apache.org/jira/browse/HBASE-17058 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4 >Reporter: Esteban Gutierrez >Assignee: Esteban Gutierrez > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17058.master.001.patch > > > The current epsilon used is 1E-6 and its too big it might overflow the > desiredMaxFileSize. A trivial fix is to lower the epsilon to 2^-52 or even > 2^-53. An option to consider too is just to shift the jitter to always > decrement hbase.hregion.max.filesize (MAX_FILESIZE) instead of increase the > size of the region and having to deal with the round off. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17069) RegionServer writes invalid META entries for split daughters in some circumstances
[
https://issues.apache.org/jira/browse/HBASE-17069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674691#comment-15674691
]
Andrew Purtell commented on HBASE-17069:
Data directories on all ephemeral volumes. Formatted XFS
> RegionServer writes invalid META entries for split daughters in some
> circumstances
> --
>
> Key: HBASE-17069
> URL: https://issues.apache.org/jira/browse/HBASE-17069
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.4
>Reporter: Andrew Purtell
>Priority: Critical
> Attachments: daughter_1_d55ef81c2f8299abbddfce0445067830.log,
> daughter_2_08629d59564726da2497f70451aafcdb.log, logs.tar.gz,
> parent-393d2bfd8b1c52ce08540306659624f2.log
>
>
> I have been seeing frequent ITBLL failures testing various versions of 1.2.x.
> Over the lifetime of 1.2.x the following issues have been fixed:
> - HBASE-15315 (Remove always set super user call as high priority)
> - HBASE-16093 (Fix splits failed before creating daughter regions leave meta
> inconsistent)
> And this one is pending:
> - HBASE-17044 (Fix merge failed before creating merged region leaves meta
> inconsistent)
> I can apply all of the above to branch-1.2 and still see this failure:
> *The life of stillborn region d55ef81c2f8299abbddfce0445067830*
> *Master sees SPLITTING_NEW*
> {noformat}
> 2016-11-08 04:23:21,186 INFO [AM.ZK.Worker-pool2-t82] master.RegionStates:
> Transition null to {d55ef81c2f8299abbddfce0445067830 state=SPLITTING_NEW,
> ts=1478579001186, server=node-3.cluster,16020,1478578389506}
> {noformat}
> *The RegionServer creates it*
> {noformat}
> 2016-11-08 04:23:26,035 INFO
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created
> cacheConfig for GomnU: blockCache=LruBlockCache{blockCount=34,
> currentSize=14996112, freeSize=12823716208, maxSize=12838712320,
> heapSize=14996112, minSize=12196776960, minFactor=0.95, multiSize=6098388480,
> multiFactor=0.5, singleSize=3049194240, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> 2016-11-08 04:23:26,038 INFO
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created
> cacheConfig for big: blockCache=LruBlockCache{blockCount=34,
> currentSize=14996112, freeSize=12823716208, maxSize=12838712320,
> heapSize=14996112, minSize=12196776960, minFactor=0.95, multiSize=6098388480,
> multiFactor=0.5, singleSize=3049194240, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> 2016-11-08 04:23:26,442 INFO
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created
> cacheConfig for meta: blockCache=LruBlockCache{blockCount=63,
> currentSize=17187656, freeSize=12821524664, maxSize=12838712320,
> heapSize=17187656, minSize=12196776960, minFactor=0.95, multiSize=6098388480,
> multiFactor=0.5, singleSize=3049194240, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> 2016-11-08 04:23:26,713 INFO
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created
> cacheConfig for nwmrW: blockCache=LruBlockCache{blockCount=96,
> currentSize=19178440, freeSize=12819533880, maxSize=12838712320,
> heapSize=19178440, minSize=12196776960, minFactor=0.95, multiSize=6098388480,
> multiFactor=0.5, singleSize=3049194240, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> 2016-11-08 04:23:26,715 INFO
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created
> cacheConfig for piwbr: blockCache=LruBlockCache{blockCount=96,
> currentSize=19178440, freeSize=12819533880, maxSize=12838712320,
> heapSize=19178440, minSize=12196776960, minFactor=0.95, multiSize=6098388480,
> multiFactor=0.5, singleSize=3049194240, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> 2016-11-08 04:23:26,717 INFO
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created
> cacheConfig for tiny: blockCache=LruBlockCache{blockCount=96,
> currentSize=19178440, freeSize=12819533880, maxSize=12838712320,
> heapSize=19178440, minSize=12196776960, minFactor=0.95, multiSize=6098388480,
>
[jira] [Commented] (HBASE-16981) Expand Mob Compaction Partition policy from daily to weekly, monthly and beyond
[ https://issues.apache.org/jira/browse/HBASE-16981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674688#comment-15674688 ] huaxiang sun commented on HBASE-16981: -- Thanks [~jingcheng...@intel.com] and [~anoop.hbase], very good points. I will create the first draft based on your comments. The one point I want to add is the major mob compact. Major compact will do something I implement today. Some use cases I am aware of is that mob compaction chore is disabled, the major mob compaction is manually scheduled to reduce the number of the mob files. > Expand Mob Compaction Partition policy from daily to weekly, monthly and > beyond > --- > > Key: HBASE-16981 > URL: https://issues.apache.org/jira/browse/HBASE-16981 > Project: HBase > Issue Type: New Feature > Components: mob >Affects Versions: 2.0.0 >Reporter: huaxiang sun >Assignee: huaxiang sun > Attachments: HBASE-16981.master.001.patch, > HBASE-16981.master.002.patch, > Supportingweeklyandmonthlymobcompactionpartitionpolicyinhbase.pdf > > > Today the mob region holds all mob files for all regions. With daily > partition mob compaction policy, after major mob compaction, there is still > one file per region daily. Given there is 365 days in one year, at least 365 > files per region. Since HDFS has limitation for number of files under one > folder, this is not going to scale if there are lots of regions. To reduce > mob file number, we want to introduce other partition policies such as > weekly, monthly to compact mob files within one week or month into one file. > This jira is create to track this effort. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17069) RegionServer writes invalid META entries for split daughters in some circumstances
[
https://issues.apache.org/jira/browse/HBASE-17069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674687#comment-15674687
]
Andrew Purtell commented on HBASE-17069:
The typical thing clusterdock does on a d2.4xlarge instance. Ubuntu 14 based.
8u102 64 bit JVM. Hadoop 2.7.3. Zookeeper 3.4.9. Native bits in place for
compression support.
NYC actually. :-)
> RegionServer writes invalid META entries for split daughters in some
> circumstances
> --
>
> Key: HBASE-17069
> URL: https://issues.apache.org/jira/browse/HBASE-17069
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.4
>Reporter: Andrew Purtell
>Priority: Critical
> Attachments: daughter_1_d55ef81c2f8299abbddfce0445067830.log,
> daughter_2_08629d59564726da2497f70451aafcdb.log, logs.tar.gz,
> parent-393d2bfd8b1c52ce08540306659624f2.log
>
>
> I have been seeing frequent ITBLL failures testing various versions of 1.2.x.
> Over the lifetime of 1.2.x the following issues have been fixed:
> - HBASE-15315 (Remove always set super user call as high priority)
> - HBASE-16093 (Fix splits failed before creating daughter regions leave meta
> inconsistent)
> And this one is pending:
> - HBASE-17044 (Fix merge failed before creating merged region leaves meta
> inconsistent)
> I can apply all of the above to branch-1.2 and still see this failure:
> *The life of stillborn region d55ef81c2f8299abbddfce0445067830*
> *Master sees SPLITTING_NEW*
> {noformat}
> 2016-11-08 04:23:21,186 INFO [AM.ZK.Worker-pool2-t82] master.RegionStates:
> Transition null to {d55ef81c2f8299abbddfce0445067830 state=SPLITTING_NEW,
> ts=1478579001186, server=node-3.cluster,16020,1478578389506}
> {noformat}
> *The RegionServer creates it*
> {noformat}
> 2016-11-08 04:23:26,035 INFO
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created
> cacheConfig for GomnU: blockCache=LruBlockCache{blockCount=34,
> currentSize=14996112, freeSize=12823716208, maxSize=12838712320,
> heapSize=14996112, minSize=12196776960, minFactor=0.95, multiSize=6098388480,
> multiFactor=0.5, singleSize=3049194240, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> 2016-11-08 04:23:26,038 INFO
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created
> cacheConfig for big: blockCache=LruBlockCache{blockCount=34,
> currentSize=14996112, freeSize=12823716208, maxSize=12838712320,
> heapSize=14996112, minSize=12196776960, minFactor=0.95, multiSize=6098388480,
> multiFactor=0.5, singleSize=3049194240, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> 2016-11-08 04:23:26,442 INFO
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created
> cacheConfig for meta: blockCache=LruBlockCache{blockCount=63,
> currentSize=17187656, freeSize=12821524664, maxSize=12838712320,
> heapSize=17187656, minSize=12196776960, minFactor=0.95, multiSize=6098388480,
> multiFactor=0.5, singleSize=3049194240, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> 2016-11-08 04:23:26,713 INFO
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created
> cacheConfig for nwmrW: blockCache=LruBlockCache{blockCount=96,
> currentSize=19178440, freeSize=12819533880, maxSize=12838712320,
> heapSize=19178440, minSize=12196776960, minFactor=0.95, multiSize=6098388480,
> multiFactor=0.5, singleSize=3049194240, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> 2016-11-08 04:23:26,715 INFO
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created
> cacheConfig for piwbr: blockCache=LruBlockCache{blockCount=96,
> currentSize=19178440, freeSize=12819533880, maxSize=12838712320,
> heapSize=19178440, minSize=12196776960, minFactor=0.95, multiSize=6098388480,
> multiFactor=0.5, singleSize=3049194240, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> 2016-11-08 04:23:26,717 INFO
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created
> cacheConfig for tiny: blockCache=LruBlockCache{blockCount=96,
>
[jira] [Commented] (HBASE-17114) Add an option to set special retry pause when encountering CallQueueTooBigException
[
https://issues.apache.org/jira/browse/HBASE-17114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674690#comment-15674690
]
Gary Helmling commented on HBASE-17114:
---
bq. AFAICS we're already doing this in
ClientExceptionsUtil#isMetaClearingException and treated
CQTBE/RegionTooBusyException etc. as special exceptions:
It's only special in the sense that it should not clear the client meta cache.
I don't think that implies it should use a different retry pause.
bq. Agree this is another good way to handle this, but by default we are still
using NoBackoffPolicy right?
No, a number of places use ConnectionUtils.getPauseTime() which uses an
exponential backoff. Maybe this has changed in master with consolidating use
of AsyncProcess, but that would be an unexpected change in behavior.
I'm -1 on using a special unique pause time for CQTBE by default. I think it
should use the configured pause time by default. If you want to make this
overridable for some exception types, that seems ok, but in that case the
config property for overriding the value should be more closely tied to the
exception. As a user of HBase, there's no way I would know what
"hbase.client.pause.special" means and why it is different.
> Add an option to set special retry pause when encountering
> CallQueueTooBigException
> ---
>
> Key: HBASE-17114
> URL: https://issues.apache.org/jira/browse/HBASE-17114
> Project: HBase
> Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-17114.patch
>
>
> As titled, after HBASE-15146 we will throw {{CallQueueTooBigException}}
> instead of dead-wait. This is good for performance for most cases but might
> cause a side-effect that if too many clients connect to the busy RS, that the
> retry requests may come over and over again and RS never got the chance for
> recovering, and the issue will become especially critical when the target
> region is META.
> So here in this JIRA we propose to supply some special retry pause for CQTBE
> in name of {{hbase.client.pause.special}}, and by default it will be 500ms (5
> times of {{hbase.client.pause}} default value)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17118) StoreScanner leaked in KeyValueHeap
[
https://issues.apache.org/jira/browse/HBASE-17118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674669#comment-15674669
]
Mikhail Antonov commented on HBASE-17118:
-
[~tedyu] let me have a look and come back in a bit
> StoreScanner leaked in KeyValueHeap
> ---
>
> Key: HBASE-17118
> URL: https://issues.apache.org/jira/browse/HBASE-17118
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: binlijin
>Assignee: binlijin
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17118-master_v1.patch,
> HBASE-17118-master_v2.patch, HBASE-17118-master_v3.patch,
> HBASE-17118-master_v4.patch, HBASE-17118-master_v5.patch,
> HBASE-17118.branch-1.v1.patch, StoreScanner.png, StoreScannerLeakHeap.png
>
>
> KeyValueHeap#generalizedSeek
> KeyValueScanner scanner = current;
> while (scanner != null) {
> Cell topKey = scanner.peek();
> ..
> boolean seekResult;
> if (isLazy && heap.size() > 0) {
> // If there is only one scanner left, we don't do lazy seek.
> seekResult = scanner.requestSeek(seekKey, forward, useBloom);
> } else {
> seekResult = NonLazyKeyValueScanner.doRealSeek(scanner, seekKey,
> forward);
> }
> ..
> scanner = heap.poll();
> }
> (1) scanner = heap.poll(); Retrieves and removes the head of this queue
> (2) scanner.requestSeek(seekKey, forward, useBloom); or
> NonLazyKeyValueScanner.doRealSeek(scanner, seekKey, forward);
> throw exception, and scanner will have no chance to close, so will cause the
> scanner leak.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14123) HBase Backup/Restore Phase 2
[ https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-14123: -- Attachment: 14123-master.v38.txt v38 address some of the recent comments on RB. Ready to merge version. cc: [~te...@apache.org] > HBase Backup/Restore Phase 2 > > > Key: HBASE-14123 > URL: https://issues.apache.org/jira/browse/HBASE-14123 > Project: HBase > Issue Type: Umbrella >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Blocker > Fix For: 2.0.0 > > Attachments: 14123-master.v14.txt, 14123-master.v15.txt, > 14123-master.v16.txt, 14123-master.v17.txt, 14123-master.v18.txt, > 14123-master.v19.txt, 14123-master.v2.txt, 14123-master.v20.txt, > 14123-master.v21.txt, 14123-master.v24.txt, 14123-master.v25.txt, > 14123-master.v27.txt, 14123-master.v28.txt, 14123-master.v29.full.txt, > 14123-master.v3.txt, 14123-master.v30.txt, 14123-master.v31.txt, > 14123-master.v32.txt, 14123-master.v33.txt, 14123-master.v34.txt, > 14123-master.v35.txt, 14123-master.v36.txt, 14123-master.v37.txt, > 14123-master.v38.txt, 14123-master.v5.txt, 14123-master.v6.txt, > 14123-master.v7.txt, 14123-master.v8.txt, 14123-master.v9.txt, 14123-v14.txt, > HBASE-14123-for-7912-v1.patch, HBASE-14123-for-7912-v6.patch, > HBASE-14123-v1.patch, HBASE-14123-v10.patch, HBASE-14123-v11.patch, > HBASE-14123-v12.patch, HBASE-14123-v13.patch, HBASE-14123-v15.patch, > HBASE-14123-v16.patch, HBASE-14123-v2.patch, HBASE-14123-v3.patch, > HBASE-14123-v4.patch, HBASE-14123-v5.patch, HBASE-14123-v6.patch, > HBASE-14123-v7.patch, HBASE-14123-v9.patch > > > Phase 2 umbrella JIRA. See HBASE-7912 for design document and description. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17118) StoreScanner leaked in KeyValueHeap
[
https://issues.apache.org/jira/browse/HBASE-17118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674651#comment-15674651
]
Hudson commented on HBASE-17118:
SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #1970 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/1970/])
HBASE-17118 StoreScanner leaked in KeyValueHeap (binlijin) (tedyu: rev
f6fc94ede999766d37f38828fe36b581924fcc30)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueHeap.java
> StoreScanner leaked in KeyValueHeap
> ---
>
> Key: HBASE-17118
> URL: https://issues.apache.org/jira/browse/HBASE-17118
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: binlijin
>Assignee: binlijin
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17118-master_v1.patch,
> HBASE-17118-master_v2.patch, HBASE-17118-master_v3.patch,
> HBASE-17118-master_v4.patch, HBASE-17118-master_v5.patch,
> HBASE-17118.branch-1.v1.patch, StoreScanner.png, StoreScannerLeakHeap.png
>
>
> KeyValueHeap#generalizedSeek
> KeyValueScanner scanner = current;
> while (scanner != null) {
> Cell topKey = scanner.peek();
> ..
> boolean seekResult;
> if (isLazy && heap.size() > 0) {
> // If there is only one scanner left, we don't do lazy seek.
> seekResult = scanner.requestSeek(seekKey, forward, useBloom);
> } else {
> seekResult = NonLazyKeyValueScanner.doRealSeek(scanner, seekKey,
> forward);
> }
> ..
> scanner = heap.poll();
> }
> (1) scanner = heap.poll(); Retrieves and removes the head of this queue
> (2) scanner.requestSeek(seekKey, forward, useBloom); or
> NonLazyKeyValueScanner.doRealSeek(scanner, seekKey, forward);
> throw exception, and scanner will have no chance to close, so will cause the
> scanner leak.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
