[jira] [Comment Edited] (HBASE-19290) Reduce zk request when doing split log
[
https://issues.apache.org/jira/browse/HBASE-19290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258912#comment-16258912
]
binlijin edited comment on HBASE-19290 at 11/20/17 7:58 AM:
bq. Please add a short commit message explaining 'idea' of the change (makes
reviewing easy then having to reverse engineer the 'idea')
Add the idea in description.
bq. grabTask only returns 0 or 1. Use boolean please.
OK.
bq. Add @return comment to grabTask.
OK.
bq. We probably don't need {{if (taskGrabed == 0 && !shouldStop) }},
taskReadySeq.wait is indefinite wait on availability of next task.
taskReadySeq.wait may not execute because it has condition.
bq. Can we pull the while loop one level outside and share the
if(calculateAvailableSplitters(..)) condition.
OK.
was (Author: aoxiang):
bq. Please add a short commit message explaining 'idea' of the change (makes
reviewing easy then having to reverse engineer the 'idea')
Add the idea in description.
bq. grabTask only returns 0 or 1. Use boolean please.
OK.
bq. Add @return comment to grabTask.
OK.
bq. We probably don't need {{if (taskGrabed == 0 && !shouldStop) }},
taskReadySeq.wait is indefinite wait on availability of next task.
taskReadySeq.wait may not execute because it have condition.
bq. Can we pull the while loop one level outside and share the
if(calculateAvailableSplitters(..)) condition.
OK.
> Reduce zk request when doing split log
> --
>
> Key: HBASE-19290
> URL: https://issues.apache.org/jira/browse/HBASE-19290
> Project: HBase
> Issue Type: Improvement
>Reporter: binlijin
>Assignee: binlijin
> Attachments: HBASE-19290.master.001.patch,
> HBASE-19290.master.002.patch
>
>
> We observe once the cluster has 1000+ nodes and when hundreds of nodes abort
> and doing split log, the split is very very slow, and we find the
> regionserver and master wait on the zookeeper response, so we need to reduce
> zookeeper request and pressure for big cluster.
> (1) Reduce request to rsZNode, every time calculateAvailableSplitters will
> get rsZNode's children from zookeeper, when cluster is huge, this is heavy.
> This patch reduce the request.
> (2) When the regionserver has max split tasks running, it may still trying to
> grab task and issue zookeeper request, we should sleep and wait until we can
> grab tasks again.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19290) Reduce zk request when doing split log
[ https://issues.apache.org/jira/browse/HBASE-19290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258913#comment-16258913 ] binlijin commented on HBASE-19290: -- bq. Add javadoc for availableRSs. OK. bq. Grabed -> Grabbed OK. bq. Consider using shorter sleep time. OK. > Reduce zk request when doing split log > -- > > Key: HBASE-19290 > URL: https://issues.apache.org/jira/browse/HBASE-19290 > Project: HBase > Issue Type: Improvement >Reporter: binlijin >Assignee: binlijin > Attachments: HBASE-19290.master.001.patch, > HBASE-19290.master.002.patch > > > We observe once the cluster has 1000+ nodes and when hundreds of nodes abort > and doing split log, the split is very very slow, and we find the > regionserver and master wait on the zookeeper response, so we need to reduce > zookeeper request and pressure for big cluster. > (1) Reduce request to rsZNode, every time calculateAvailableSplitters will > get rsZNode's children from zookeeper, when cluster is huge, this is heavy. > This patch reduce the request. > (2) When the regionserver has max split tasks running, it may still trying to > grab task and issue zookeeper request, we should sleep and wait until we can > grab tasks again. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19290) Reduce zk request when doing split log
[
https://issues.apache.org/jira/browse/HBASE-19290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258912#comment-16258912
]
binlijin commented on HBASE-19290:
--
bq. Please add a short commit message explaining 'idea' of the change (makes
reviewing easy then having to reverse engineer the 'idea')
Add the idea in description.
bq. grabTask only returns 0 or 1. Use boolean please.
OK.
bq. Add @return comment to grabTask.
OK.
bq. We probably don't need {{if (taskGrabed == 0 && !shouldStop) }},
taskReadySeq.wait is indefinite wait on availability of next task.
taskReadySeq.wait may not execute because it have condition.
bq. Can we pull the while loop one level outside and share the
if(calculateAvailableSplitters(..)) condition.
OK.
> Reduce zk request when doing split log
> --
>
> Key: HBASE-19290
> URL: https://issues.apache.org/jira/browse/HBASE-19290
> Project: HBase
> Issue Type: Improvement
>Reporter: binlijin
>Assignee: binlijin
> Attachments: HBASE-19290.master.001.patch,
> HBASE-19290.master.002.patch
>
>
> We observe once the cluster has 1000+ nodes and when hundreds of nodes abort
> and doing split log, the split is very very slow, and we find the
> regionserver and master wait on the zookeeper response, so we need to reduce
> zookeeper request and pressure for big cluster.
> (1) Reduce request to rsZNode, every time calculateAvailableSplitters will
> get rsZNode's children from zookeeper, when cluster is huge, this is heavy.
> This patch reduce the request.
> (2) When the regionserver has max split tasks running, it may still trying to
> grab task and issue zookeeper request, we should sleep and wait until we can
> grab tasks again.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19290) Reduce zk request when doing split log
[ https://issues.apache.org/jira/browse/HBASE-19290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] binlijin updated HBASE-19290: - Description: We observe once the cluster has 1000+ nodes and when hundreds of nodes abort and doing split log, the split is very very slow, and we find the regionserver and master wait on the zookeeper response, so we need to reduce zookeeper request and pressure for big cluster. (1) Reduce request to rsZNode, every time calculateAvailableSplitters will get rsZNode's children from zookeeper, when cluster is huge, this is heavy. This patch reduce the request. (2) When the regionserver has max split tasks running, it may still trying to grab task and issue zookeeper request, we should sleep and wait until we can grab tasks again. was:We observe once the cluster has 1000+ nodes and when hundreds of nodes abort and doing split log, the split is very very slow, and we find the regionserver and master wait on the zookeeper response, so we need to reduce zookeeper request and pressure for big cluster. > Reduce zk request when doing split log > -- > > Key: HBASE-19290 > URL: https://issues.apache.org/jira/browse/HBASE-19290 > Project: HBase > Issue Type: Improvement >Reporter: binlijin >Assignee: binlijin > Attachments: HBASE-19290.master.001.patch, > HBASE-19290.master.002.patch > > > We observe once the cluster has 1000+ nodes and when hundreds of nodes abort > and doing split log, the split is very very slow, and we find the > regionserver and master wait on the zookeeper response, so we need to reduce > zookeeper request and pressure for big cluster. > (1) Reduce request to rsZNode, every time calculateAvailableSplitters will > get rsZNode's children from zookeeper, when cluster is huge, this is heavy. > This patch reduce the request. > (2) When the regionserver has max split tasks running, it may still trying to > grab task and issue zookeeper request, we should sleep and wait until we can > grab tasks again. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18309) Support multi threads in CleanerChore
[
https://issues.apache.org/jira/browse/HBASE-18309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258899#comment-16258899
]
Hadoop QA commented on HBASE-18309:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
9s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
53s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s{color} | {color:green} hbase-server: The patch generated 0 new + 4
unchanged - 4 fixed = 4 total (was 8) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
47s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
51m 3s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 2.7.4 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 95m
1s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}164m 58s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-18309 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12898415/HBASE-18309.master.010.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux c33b1a26a831 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 9b7b83d862 |
| maven | version: Apache Maven 3.5.2
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/9920/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/9920/console |
| Powered by | Apache Yetus 0.6.0 http://yetus.apache.org |
This message was automatically generated.
> Support multi threads in CleanerChore
> -
>
> Key:
[jira] [Updated] (HBASE-19290) Reduce zk request when doing split log
[ https://issues.apache.org/jira/browse/HBASE-19290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] binlijin updated HBASE-19290: - Attachment: HBASE-19290.master.002.patch > Reduce zk request when doing split log > -- > > Key: HBASE-19290 > URL: https://issues.apache.org/jira/browse/HBASE-19290 > Project: HBase > Issue Type: Improvement >Reporter: binlijin >Assignee: binlijin > Attachments: HBASE-19290.master.001.patch, > HBASE-19290.master.002.patch > > > We observe once the cluster has 1000+ nodes and when hundreds of nodes abort > and doing split log, the split is very very slow, and we find the > regionserver and master wait on the zookeeper response, so we need to reduce > zookeeper request and pressure for big cluster. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19301) Provide way for CPs to create short circuited connection with custom configurations
[ https://issues.apache.org/jira/browse/HBASE-19301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258883#comment-16258883 ] Anoop Sam John commented on HBASE-19301: bq.A Connection accessing local Server will short-circuit (To be verified). Not all. Only the connection created within RS will use this way. A connection created at client side (Using ConnectionFactory#createConnection) will NOT use the short circuit path. Ya the short circuit removes the RPC things from call. It will be a direct stub call. But we have to do the PB convert. Say when mutation happens, all mutations has to be converted to PB and back to Pojo. (Some thing to see later how/whether we can avoid this) bq.But perhaps you are referring to difference between an invocation that goes via protobuf service but skips the rpc invocation vs a direct invocation, one that skips doing protobuf marshalling/unmarshaling AND rpc? This is what the short circuit connection doing right? Are there 2 short circuit stuff? I dont know. > Provide way for CPs to create short circuited connection with custom > configurations > --- > > Key: HBASE-19301 > URL: https://issues.apache.org/jira/browse/HBASE-19301 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19301.patch > > > Over in HBASE-18359 we have discussions for this. > Right now HBase provide getConnection() in RegionCPEnv, MasterCPEnv etc. But > this returns a pre created connection (per server). This uses the configs at > hbase-site.xml at that server. > Phoenix needs creating connection in CP with some custom configs. Having this > custom changes in hbase-site.xml is harmful as that will affect all > connections been created at that server. > This issue is for providing an overloaded getConnection(Configuration) API -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19299) Assert only one Connection is constructed when calculating splits in a MultiTableInputFormat
[ https://issues.apache.org/jira/browse/HBASE-19299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258878#comment-16258878 ] Duo Zhang commented on HBASE-19299: --- But no harm to add it to branch-2? :) > Assert only one Connection is constructed when calculating splits in a > MultiTableInputFormat > > > Key: HBASE-19299 > URL: https://issues.apache.org/jira/browse/HBASE-19299 > Project: HBase > Issue Type: Test > Components: test >Reporter: stack >Assignee: stack >Priority: Minor > Fix For: 3.0.0 > > Attachments: > 0002-HBASE-19299-Assert-only-one-Connection-is-constructe.patch > > > Small test that verifies only one Connection made when calculating splits. We > used to put up one per Table until recently and before that, a Connection per > table per split which put a heavy load on hbase;meta. Here is a test to > ensure we don't go back to the bad-old-days. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19092) Make Tag IA.LimitedPrivate and expose for CPs
[ https://issues.apache.org/jira/browse/HBASE-19092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258858#comment-16258858 ] Anoop Sam John commented on HBASE-19092: bq. we want Tags integral to Cell, not apart as they have been, so getting Tags from Cell/ExtendedCell is where we want to be Exactly. This is what I too considered. We want Tags to be part of Cell. When we do it for say 3.0, we need getTags in Cell. Not in ExtendedCell. We have to expose tags to client side also. So if we add to ExtendedCell today, it is bound to change again tomorrow right? To be to Cell. That is what my consideration was. Then having in a static util API way is not that bad as type casting calls on ExtendedCell. In 3.0, we would deprecate that static against Cell#getTags(). Any way take a call... Am ok for any thing decided here. > Make Tag IA.LimitedPrivate and expose for CPs > - > > Key: HBASE-19092 > URL: https://issues.apache.org/jira/browse/HBASE-19092 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19092-branch-2.patch, > HBASE-19092-branch-2_5.patch, HBASE-19092-branch-2_5.patch, > HBASE-19092.branch-2.0.02.patch, HBASE-19092_001-branch-2.patch, > HBASE-19092_001.patch, HBASE-19092_002-branch-2.patch, HBASE-19092_002.patch > > > We need to make tags as LimitedPrivate as some use cases are trying to use > tags like timeline server. The same topic was discussed in dev@ and also in > HBASE-18995. > Shall we target this for beta1 - cc [~saint@gmail.com]. > So once we do this all related Util methods and APIs should also move to > LimitedPrivate Util classes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-17436) Add facility to provide more information for Other Regions seen on Master UI
[ https://issues.apache.org/jira/browse/HBASE-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258843#comment-16258843 ] stack commented on HBASE-17436: --- [~gubjanos] thank you. Patch looks good then. How do folks discover this new means of consuming the Region list? I don't see a menu item added. Thanks. > Add facility to provide more information for Other Regions seen on Master UI > > > Key: HBASE-17436 > URL: https://issues.apache.org/jira/browse/HBASE-17436 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Janos Gub > Labels: ui, usability > Fix For: 2.0.0 > > Attachments: 17436.v10.txt, 17436.v9.txt, HBASE-17436-v2.patch, > HBASE-17436-v5.patch, HBASE-17436-v6.patch, HBASE-17436-v6.patch, > HBASE-17436-v7.patch, HBASE-17436-v8.patch, HBASE-17436.patch, > HBASE-17779-v3.patch, HBASE-17779-v4.patch, Screen Shot 2017-04-24 at 8.43.31 > PM.png, Screen Shot 2017-04-26 at 4.39.49 PM.png, Screen Shot 2017-11-08 at > 10.29.49 AM.png, Screen Shot 2017-11-08 at 10.30.09 AM.png, initial.patch, > regionstates.png, table-selected.png, tableregions.png > > > [~rpednekar] and I were looking at a case where the count displayed under > Other Regions was high (~1200). > Since the table page just maintains a count instead of List of region names, > it is very difficult for user to determine which regions belong to this > category. > We should provide facility to provide more details for this category (LOG, > JMX, etc). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19288) Intermittent test failure in TestHStore.testRunDoubleMemStoreCompactors
[
https://issues.apache.org/jira/browse/HBASE-19288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258842#comment-16258842
]
Hudson commented on HBASE-19288:
FAILURE: Integrated in Jenkins build HBase-2.0 #883 (See
[https://builds.apache.org/job/HBase-2.0/883/])
HBASE-19288 Intermittent test failure in (tedyu: rev
b554bbc65ec8e9482f3b00c87f8ecc39922e22dc)
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHStore.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactingMemStore.java
> Intermittent test failure in TestHStore.testRunDoubleMemStoreCompactors
> ---
>
> Key: HBASE-19288
> URL: https://issues.apache.org/jira/browse/HBASE-19288
> Project: HBase
> Issue Type: Test
> Components: test
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0-beta-1
>
> Attachments: 19288.v1.txt, 19288.v2.txt,
> testRunDoubleMemStoreCompactors.out
>
>
> Here was one of the test failures:
> https://builds.apache.org/job/PreCommit-HBASE-Build/9812/testReport/junit/org.apache.hadoop.hbase.regionserver/TestHStore/testRunDoubleMemStoreCompactors/
>
> {code}
> [ERROR]
> org.apache.hadoop.hbase.regionserver.TestHStore.testRunDoubleMemStoreCompactors(org.apache.hadoop.hbase.regionserver.TestHStore)
> [ERROR] Run 1: TestHStore.testRunDoubleMemStoreCompactors:1500 expected:<2>
> but was:<3>
> [ERROR] Run 2: TestHStore.testRunDoubleMemStoreCompactors:1481 expected:<1>
> but was:<4>
> [ERROR] Run 3: TestHStore.testRunDoubleMemStoreCompactors:1481 expected:<1>
> but was:<5>
> {code}
> From the counts for second and third runs, we know that RUNNER_COUNT was not
> cleared in between the reruns, leading to failure at the 1st assertion.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19299) Assert only one Connection is constructed when calculating splits in a MultiTableInputFormat
[ https://issues.apache.org/jira/browse/HBASE-19299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258839#comment-16258839 ] stack commented on HBASE-19299: --- bq. Why master only? Its trivial test addition only. Didn't think anyone else would care. > Assert only one Connection is constructed when calculating splits in a > MultiTableInputFormat > > > Key: HBASE-19299 > URL: https://issues.apache.org/jira/browse/HBASE-19299 > Project: HBase > Issue Type: Test > Components: test >Reporter: stack >Assignee: stack >Priority: Minor > Fix For: 3.0.0 > > Attachments: > 0002-HBASE-19299-Assert-only-one-Connection-is-constructe.patch > > > Small test that verifies only one Connection made when calculating splits. We > used to put up one per Table until recently and before that, a Connection per > table per split which put a heavy load on hbase;meta. Here is a test to > ensure we don't go back to the bad-old-days. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-16890) Analyze the performance of AsyncWAL and fix the same
[ https://issues.apache.org/jira/browse/HBASE-16890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258838#comment-16258838 ] stack commented on HBASE-16890: --- bq. Is this test enough? No. But I like the numbers you are getting. Can you paste config. and command you ran so can repro? Thanks [~ram_krish] > Analyze the performance of AsyncWAL and fix the same > > > Key: HBASE-16890 > URL: https://issues.apache.org/jira/browse/HBASE-16890 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Blocker > Fix For: 2.0.0-beta-1 > > Attachments: AsyncWAL_disruptor.patch, AsyncWAL_disruptor_1 > (2).patch, AsyncWAL_disruptor_3.patch, AsyncWAL_disruptor_3.patch, > AsyncWAL_disruptor_4.patch, AsyncWAL_disruptor_6.patch, > HBASE-16890-rc-v2.patch, HBASE-16890-rc-v3.patch, > HBASE-16890-remove-contention-v1.patch, HBASE-16890-remove-contention.patch, > Screen Shot 2016-10-25 at 7.34.47 PM.png, Screen Shot 2016-10-25 at 7.39.07 > PM.png, Screen Shot 2016-10-25 at 7.39.48 PM.png, Screen Shot 2016-11-04 at > 5.21.27 PM.png, Screen Shot 2016-11-04 at 5.30.18 PM.png, async.svg, > classic.svg, contention.png, contention_defaultWAL.png > > > Tests reveal that AsyncWAL under load in single node cluster performs slower > than the Default WAL. This task is to analyze and see if we could fix it. > See some discussions in the tail of JIRA HBASE-15536. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19301) Provide way for CPs to create short circuited connection with custom configurations
[ https://issues.apache.org/jira/browse/HBASE-19301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258836#comment-16258836 ] stack commented on HBASE-19301: --- bq. he only disadvantage will be that it will NOT be a short circuit enabled connection. A Connection accessing local Server will short-circuit (To be verified). But perhaps you are referring to difference between an invocation that goes via protobuf service but skips the rpc invocation vs a direct invocation, one that skips doing protobuf marshalling/unmarshaling AND rpc? If this is what you mean, can we make it so there is only one short-circuit type -- the direct-invocation -- everywhere? I've not looked. Perhaps you have? i.e. do what the Phoenix 'short-circuit' used do but native inside hbase when it notices it an invocation against a local server? Thanks [~anoop.hbase] > Provide way for CPs to create short circuited connection with custom > configurations > --- > > Key: HBASE-19301 > URL: https://issues.apache.org/jira/browse/HBASE-19301 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19301.patch > > > Over in HBASE-18359 we have discussions for this. > Right now HBase provide getConnection() in RegionCPEnv, MasterCPEnv etc. But > this returns a pre created connection (per server). This uses the configs at > hbase-site.xml at that server. > Phoenix needs creating connection in CP with some custom configs. Having this > custom changes in hbase-site.xml is harmful as that will affect all > connections been created at that server. > This issue is for providing an overloaded getConnection(Configuration) API -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19092) Make Tag IA.LimitedPrivate and expose for CPs
[ https://issues.apache.org/jira/browse/HBASE-19092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258835#comment-16258835 ] stack commented on HBASE-19092: --- [~anoop.hbase] says bq. This will look as a bad design IMHO. ... if CPs have to cast Cell to ExtendedCell if they want to pull Tags from a Cell. I agree. But it is late in the game. So, we ok allowing that getting Tags from Cell is a little messy in hbase2? That said, we want Tags integral to Cell, not apart as they have been, so getting Tags from Cell/ExtendedCell is where we want to be; its just not clean in hbase2. I think we can live w/ that (needs good doc). > Make Tag IA.LimitedPrivate and expose for CPs > - > > Key: HBASE-19092 > URL: https://issues.apache.org/jira/browse/HBASE-19092 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19092-branch-2.patch, > HBASE-19092-branch-2_5.patch, HBASE-19092-branch-2_5.patch, > HBASE-19092.branch-2.0.02.patch, HBASE-19092_001-branch-2.patch, > HBASE-19092_001.patch, HBASE-19092_002-branch-2.patch, HBASE-19092_002.patch > > > We need to make tags as LimitedPrivate as some use cases are trying to use > tags like timeline server. The same topic was discussed in dev@ and also in > HBASE-18995. > Shall we target this for beta1 - cc [~saint@gmail.com]. > So once we do this all related Util methods and APIs should also move to > LimitedPrivate Util classes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19301) Provide way for CPs to create short circuited connection with custom configurations
[ https://issues.apache.org/jira/browse/HBASE-19301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258832#comment-16258832 ] Anoop Sam John commented on HBASE-19301: CPs can create own connection with custom configs as how users do that at client side. The only disadvantage will be that it will NOT be a short circuit enabled connection. Phoenix use cases introduced this short circuit based connections and so they use that in their CPs. That is why thought of a CP based getConnection. > Provide way for CPs to create short circuited connection with custom > configurations > --- > > Key: HBASE-19301 > URL: https://issues.apache.org/jira/browse/HBASE-19301 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19301.patch > > > Over in HBASE-18359 we have discussions for this. > Right now HBase provide getConnection() in RegionCPEnv, MasterCPEnv etc. But > this returns a pre created connection (per server). This uses the configs at > hbase-site.xml at that server. > Phoenix needs creating connection in CP with some custom configs. Having this > custom changes in hbase-site.xml is harmful as that will affect all > connections been created at that server. > This issue is for providing an overloaded getConnection(Configuration) API -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19260) Add lock back to avoid parallel accessing meta to locate region
[
https://issues.apache.org/jira/browse/HBASE-19260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258828#comment-16258828
]
stack commented on HBASE-19260:
---
Any time [~carp84]
> Add lock back to avoid parallel accessing meta to locate region
> ---
>
> Key: HBASE-19260
> URL: https://issues.apache.org/jira/browse/HBASE-19260
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.1, 1.2.6, 2.0.0-alpha-3, 1.1.12
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 1.4.0, 1.3.2, 1.2.7, 2.0.0-beta-1
>
> Attachments: HBASE-19260.patch, HBASE-19260.v2.patch
>
>
> In branch-0.98 we have below codes to avoid accessing meta in parallel in
> {{HConnectionManager}}:
> {code}
> Result regionInfoRow;
> // This block guards against two threads trying to load the meta
> // region at the same time. The first will load the meta region and
> // the second will use the value that the first one found.
> if (useCache) {
> if (TableName.META_TABLE_NAME.equals(parentTable) && usePrefetch
> &&
> getRegionCachePrefetch(tableName)) {
> synchronized (regionLockObject) {
> // Check the cache again for a hit in case some other thread
> made the
> // same query while we were waiting on the lock.
> ...
> }
> }
> ...
> {code}
> while in HBASE-10018 we removed such logic along with
> region-location-prefetching.
> We regard this as an unexpected behavior change and observed below phenomenon
> in our product env:
> 1. Unnecessary connection setup to meta when multiple threads locating region
> in a client process
> 2. Priority handler of the RS holding meta region exhausted, application keep
> retrying and cause a vicious circle
> To resolve this problem, we propose to add the {{userRegionLock}} back and
> keep the behavior in accordance with 0.98
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19301) Provide way for CPs to create short circuited connection with custom configurations
[ https://issues.apache.org/jira/browse/HBASE-19301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258827#comment-16258827 ] stack commented on HBASE-19301: --- Thinking about this, do we need to add this to CpEnv [~anoop.hbase]? What is to prevent a CP from making its own Configuration and subsequently its own Connection? What is advantage getting it from CpEnv? > Provide way for CPs to create short circuited connection with custom > configurations > --- > > Key: HBASE-19301 > URL: https://issues.apache.org/jira/browse/HBASE-19301 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19301.patch > > > Over in HBASE-18359 we have discussions for this. > Right now HBase provide getConnection() in RegionCPEnv, MasterCPEnv etc. But > this returns a pre created connection (per server). This uses the configs at > hbase-site.xml at that server. > Phoenix needs creating connection in CP with some custom configs. Having this > custom changes in hbase-site.xml is harmful as that will affect all > connections been created at that server. > This issue is for providing an overloaded getConnection(Configuration) API -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19092) Make Tag IA.LimitedPrivate and expose for CPs
[ https://issues.apache.org/jira/browse/HBASE-19092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258818#comment-16258818 ] ramkrishna.s.vasudevan commented on HBASE-19092: bq.Tag#get(Cell) might be best for now at this late stage but I should be able to do Cell#getTags or at least ExtendedCell#getTags? Yes. We will have ExtendedCell#getTags which will return a list of tags. That was the point of discussion as I was feeling it is ok to do ExtendedCell#getTags though the CP needed a type cast. If you all agree I can update the patch accordingly. > Make Tag IA.LimitedPrivate and expose for CPs > - > > Key: HBASE-19092 > URL: https://issues.apache.org/jira/browse/HBASE-19092 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19092-branch-2.patch, > HBASE-19092-branch-2_5.patch, HBASE-19092-branch-2_5.patch, > HBASE-19092.branch-2.0.02.patch, HBASE-19092_001-branch-2.patch, > HBASE-19092_001.patch, HBASE-19092_002-branch-2.patch, HBASE-19092_002.patch > > > We need to make tags as LimitedPrivate as some use cases are trying to use > tags like timeline server. The same topic was discussed in dev@ and also in > HBASE-18995. > Shall we target this for beta1 - cc [~saint@gmail.com]. > So once we do this all related Util methods and APIs should also move to > LimitedPrivate Util classes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HBASE-18309) Support multi threads in CleanerChore
[ https://issues.apache.org/jira/browse/HBASE-18309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258814#comment-16258814 ] Reid Chan edited comment on HBASE-18309 at 11/20/17 4:48 AM: - Sorry for keeping you waiting [~carp84], i will file it later. was (Author: reidchan): Sorry to keep you waiting[~carp84], i will file it later. > Support multi threads in CleanerChore > - > > Key: HBASE-18309 > URL: https://issues.apache.org/jira/browse/HBASE-18309 > Project: HBase > Issue Type: Improvement > Components: wal >Reporter: binlijin >Assignee: Reid Chan > Attachments: HBASE-18309.master.001.patch, > HBASE-18309.master.002.patch, HBASE-18309.master.004.patch, > HBASE-18309.master.005.patch, HBASE-18309.master.006.patch, > HBASE-18309.master.007.patch, HBASE-18309.master.008.patch, > HBASE-18309.master.009.patch, HBASE-18309.master.010.patch, > space_consumption_in_archive.png > > > There is only one thread in LogCleaner to clean oldWALs and in our big > cluster we find this is not enough. The number of files under oldWALs reach > the max-directory-items limit of HDFS and cause region server crash, so we > use multi threads for LogCleaner and the crash not happened any more. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18309) Support multi threads in CleanerChore
[ https://issues.apache.org/jira/browse/HBASE-18309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258814#comment-16258814 ] Reid Chan commented on HBASE-18309: --- Sorry to keep you waiting[~carp84], i will file it later. > Support multi threads in CleanerChore > - > > Key: HBASE-18309 > URL: https://issues.apache.org/jira/browse/HBASE-18309 > Project: HBase > Issue Type: Improvement > Components: wal >Reporter: binlijin >Assignee: Reid Chan > Attachments: HBASE-18309.master.001.patch, > HBASE-18309.master.002.patch, HBASE-18309.master.004.patch, > HBASE-18309.master.005.patch, HBASE-18309.master.006.patch, > HBASE-18309.master.007.patch, HBASE-18309.master.008.patch, > HBASE-18309.master.009.patch, HBASE-18309.master.010.patch, > space_consumption_in_archive.png > > > There is only one thread in LogCleaner to clean oldWALs and in our big > cluster we find this is not enough. The number of files under oldWALs reach > the max-directory-items limit of HDFS and cause region server crash, so we > use multi threads for LogCleaner and the crash not happened any more. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18309) Support multi threads in CleanerChore
[ https://issues.apache.org/jira/browse/HBASE-18309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reid Chan updated HBASE-18309: -- Status: Patch Available (was: Open) > Support multi threads in CleanerChore > - > > Key: HBASE-18309 > URL: https://issues.apache.org/jira/browse/HBASE-18309 > Project: HBase > Issue Type: Improvement > Components: wal >Reporter: binlijin >Assignee: Reid Chan > Attachments: HBASE-18309.master.001.patch, > HBASE-18309.master.002.patch, HBASE-18309.master.004.patch, > HBASE-18309.master.005.patch, HBASE-18309.master.006.patch, > HBASE-18309.master.007.patch, HBASE-18309.master.008.patch, > HBASE-18309.master.009.patch, HBASE-18309.master.010.patch, > space_consumption_in_archive.png > > > There is only one thread in LogCleaner to clean oldWALs and in our big > cluster we find this is not enough. The number of files under oldWALs reach > the max-directory-items limit of HDFS and cause region server crash, so we > use multi threads for LogCleaner and the crash not happened any more. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HBASE-18309) Support multi threads in CleanerChore
[
https://issues.apache.org/jira/browse/HBASE-18309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258813#comment-16258813
]
Reid Chan edited comment on HBASE-18309 at 11/20/17 4:42 AM:
-
Rebase master branch, and remove this {{@Override public int compare(FileStatus
f1, FileStatus f2) {}}} an unnecessary change.
was (Author: reidchan):
Rebase master branch, and remove this {@Override public int compare(FileStatus
f1, FileStatus f2) {}} an unnecessary change.
> Support multi threads in CleanerChore
> -
>
> Key: HBASE-18309
> URL: https://issues.apache.org/jira/browse/HBASE-18309
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: binlijin
>Assignee: Reid Chan
> Attachments: HBASE-18309.master.001.patch,
> HBASE-18309.master.002.patch, HBASE-18309.master.004.patch,
> HBASE-18309.master.005.patch, HBASE-18309.master.006.patch,
> HBASE-18309.master.007.patch, HBASE-18309.master.008.patch,
> HBASE-18309.master.009.patch, HBASE-18309.master.010.patch,
> space_consumption_in_archive.png
>
>
> There is only one thread in LogCleaner to clean oldWALs and in our big
> cluster we find this is not enough. The number of files under oldWALs reach
> the max-directory-items limit of HDFS and cause region server crash, so we
> use multi threads for LogCleaner and the crash not happened any more.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18309) Support multi threads in CleanerChore
[
https://issues.apache.org/jira/browse/HBASE-18309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Reid Chan updated HBASE-18309:
--
Attachment: HBASE-18309.master.010.patch
Rebase master branch, and remove this {@Override public int compare(FileStatus
f1, FileStatus f2) {}} an unnecessary change.
> Support multi threads in CleanerChore
> -
>
> Key: HBASE-18309
> URL: https://issues.apache.org/jira/browse/HBASE-18309
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: binlijin
>Assignee: Reid Chan
> Attachments: HBASE-18309.master.001.patch,
> HBASE-18309.master.002.patch, HBASE-18309.master.004.patch,
> HBASE-18309.master.005.patch, HBASE-18309.master.006.patch,
> HBASE-18309.master.007.patch, HBASE-18309.master.008.patch,
> HBASE-18309.master.009.patch, HBASE-18309.master.010.patch,
> space_consumption_in_archive.png
>
>
> There is only one thread in LogCleaner to clean oldWALs and in our big
> cluster we find this is not enough. The number of files under oldWALs reach
> the max-directory-items limit of HDFS and cause region server crash, so we
> use multi threads for LogCleaner and the crash not happened any more.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18309) Support multi threads in CleanerChore
[ https://issues.apache.org/jira/browse/HBASE-18309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reid Chan updated HBASE-18309: -- Status: Open (was: Patch Available) > Support multi threads in CleanerChore > - > > Key: HBASE-18309 > URL: https://issues.apache.org/jira/browse/HBASE-18309 > Project: HBase > Issue Type: Improvement > Components: wal >Reporter: binlijin >Assignee: Reid Chan > Attachments: HBASE-18309.master.001.patch, > HBASE-18309.master.002.patch, HBASE-18309.master.004.patch, > HBASE-18309.master.005.patch, HBASE-18309.master.006.patch, > HBASE-18309.master.007.patch, HBASE-18309.master.008.patch, > HBASE-18309.master.009.patch, space_consumption_in_archive.png > > > There is only one thread in LogCleaner to clean oldWALs and in our big > cluster we find this is not enough. The number of files under oldWALs reach > the max-directory-items limit of HDFS and cause region server crash, so we > use multi threads for LogCleaner and the crash not happened any more. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18309) Support multi threads in CleanerChore
[ https://issues.apache.org/jira/browse/HBASE-18309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258806#comment-16258806 ] Yu Li commented on HBASE-18309: --- Patch v9 LGTM, +1 bq. As concerned, it's possible for thread flood happen and drag NN down. And i have an idea (not quite sure, and need some discussions), but would please allow me to file another jira for that, it's too squeezed to put into one patch. Still waiting for the new JIRA, and please add a link here (or please add a link directly and forgive my ignorance if already opened). Thanks. > Support multi threads in CleanerChore > - > > Key: HBASE-18309 > URL: https://issues.apache.org/jira/browse/HBASE-18309 > Project: HBase > Issue Type: Improvement > Components: wal >Reporter: binlijin >Assignee: Reid Chan > Attachments: HBASE-18309.master.001.patch, > HBASE-18309.master.002.patch, HBASE-18309.master.004.patch, > HBASE-18309.master.005.patch, HBASE-18309.master.006.patch, > HBASE-18309.master.007.patch, HBASE-18309.master.008.patch, > HBASE-18309.master.009.patch, space_consumption_in_archive.png > > > There is only one thread in LogCleaner to clean oldWALs and in our big > cluster we find this is not enough. The number of files under oldWALs reach > the max-directory-items limit of HDFS and cause region server crash, so we > use multi threads for LogCleaner and the crash not happened any more. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19260) Add lock back to avoid parallel accessing meta to locate region
[
https://issues.apache.org/jira/browse/HBASE-19260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258794#comment-16258794
]
Yu Li commented on HBASE-19260:
---
Thanks for help commit boss (and sorry that I was held by some online issue)
[~stack]
> Add lock back to avoid parallel accessing meta to locate region
> ---
>
> Key: HBASE-19260
> URL: https://issues.apache.org/jira/browse/HBASE-19260
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.1, 1.2.6, 2.0.0-alpha-3, 1.1.12
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 1.4.0, 1.3.2, 1.2.7, 2.0.0-beta-1
>
> Attachments: HBASE-19260.patch, HBASE-19260.v2.patch
>
>
> In branch-0.98 we have below codes to avoid accessing meta in parallel in
> {{HConnectionManager}}:
> {code}
> Result regionInfoRow;
> // This block guards against two threads trying to load the meta
> // region at the same time. The first will load the meta region and
> // the second will use the value that the first one found.
> if (useCache) {
> if (TableName.META_TABLE_NAME.equals(parentTable) && usePrefetch
> &&
> getRegionCachePrefetch(tableName)) {
> synchronized (regionLockObject) {
> // Check the cache again for a hit in case some other thread
> made the
> // same query while we were waiting on the lock.
> ...
> }
> }
> ...
> {code}
> while in HBASE-10018 we removed such logic along with
> region-location-prefetching.
> We regard this as an unexpected behavior change and observed below phenomenon
> in our product env:
> 1. Unnecessary connection setup to meta when multiple threads locating region
> in a client process
> 2. Priority handler of the RS holding meta region exhausted, application keep
> retrying and cause a vicious circle
> To resolve this problem, we propose to add the {{userRegionLock}} back and
> keep the behavior in accordance with 0.98
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19267) Eclipse project import issues on 2.0
[ https://issues.apache.org/jira/browse/HBASE-19267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258792#comment-16258792 ] Guanghao Zhang commented on HBASE-19267: What the error looks like? I still use eclipse and I can take a try for the patch. [~elserj] > Eclipse project import issues on 2.0 > > > Key: HBASE-19267 > URL: https://issues.apache.org/jira/browse/HBASE-19267 > Project: HBase > Issue Type: Task > Components: build >Reporter: Josh Elser >Assignee: Josh Elser > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19267.001.branch-2.patch > > > Trying to do a fresh import of branch-2 nets some errors.. > It seems like a previous change I made to clean up errors (HBASE-13236), > specifically adding the maven-compiler-plugin lifecycle mapping for > m2eclipse, is now causing Eclipse to not compile HBase as Java8. Removing the > lifecycle mapping fixes this. > I assume this only needs to happen for 2.0. > I keep having issues with the JavaNature being ignored. Not yet sure if this > is a result of something we're doing wrong (or just Eclipse being Eclipse). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18309) Support multi threads in CleanerChore
[ https://issues.apache.org/jira/browse/HBASE-18309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258785#comment-16258785 ] Reid Chan commented on HBASE-18309: --- Any other thoughts or comments? > Support multi threads in CleanerChore > - > > Key: HBASE-18309 > URL: https://issues.apache.org/jira/browse/HBASE-18309 > Project: HBase > Issue Type: Improvement > Components: wal >Reporter: binlijin >Assignee: Reid Chan > Attachments: HBASE-18309.master.001.patch, > HBASE-18309.master.002.patch, HBASE-18309.master.004.patch, > HBASE-18309.master.005.patch, HBASE-18309.master.006.patch, > HBASE-18309.master.007.patch, HBASE-18309.master.008.patch, > HBASE-18309.master.009.patch, space_consumption_in_archive.png > > > There is only one thread in LogCleaner to clean oldWALs and in our big > cluster we find this is not enough. The number of files under oldWALs reach > the max-directory-items limit of HDFS and cause region server crash, so we > use multi threads for LogCleaner and the crash not happened any more. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19293) Support add a disabled state replication peer directly
[ https://issues.apache.org/jira/browse/HBASE-19293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258771#comment-16258771 ] Guanghao Zhang commented on HBASE-19293: Attach a 003 patch to address the checkstyle warnings and add STATE default value to document and the ruby script help message. > Support add a disabled state replication peer directly > -- > > Key: HBASE-19293 > URL: https://issues.apache.org/jira/browse/HBASE-19293 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19293.master.001.patch, > HBASE-19293.master.002.patch, HBASE-19293.master.003.patch > > > Now when add a replication peer, the default state is enabled. If you want > add a disabled replication peer, you need add a peer first, then disable it. > It need two step to finish now. > Use case for add a disabled replication peer. When user want sync data from a > cluster A to a new peer cluster. > 1. Add a disabled replication peer. And config the table to peer config. > 2. Take a snapshot of table and export snapshot to peer cluster. > 3. Restore snapshot in peer cluster. > 4. Enable the peer and wait all stuck replication log replicated to peer > cluster. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19293) Support add a disabled state replication peer directly
[ https://issues.apache.org/jira/browse/HBASE-19293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-19293: --- Attachment: HBASE-19293.master.003.patch > Support add a disabled state replication peer directly > -- > > Key: HBASE-19293 > URL: https://issues.apache.org/jira/browse/HBASE-19293 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19293.master.001.patch, > HBASE-19293.master.002.patch, HBASE-19293.master.003.patch > > > Now when add a replication peer, the default state is enabled. If you want > add a disabled replication peer, you need add a peer first, then disable it. > It need two step to finish now. > Use case for add a disabled replication peer. When user want sync data from a > cluster A to a new peer cluster. > 1. Add a disabled replication peer. And config the table to peer config. > 2. Take a snapshot of table and export snapshot to peer cluster. > 3. Restore snapshot in peer cluster. > 4. Enable the peer and wait all stuck replication log replicated to peer > cluster. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HBASE-17165) Add retry to LoadIncrementalHFiles tool
[ https://issues.apache.org/jira/browse/HBASE-17165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258765#comment-16258765 ] Mike Grimes edited comment on HBASE-17165 at 11/20/17 2:51 AM: --- Sorry for the half-fixed patch uploaded before the weekend. I noticed a bug in the unit test after submitting. New ones should be good. was (Author: grimesmi): Sorry for the half-fixed patch uploaded before the weekend. I noticed a bug in the unit test after submitting. > Add retry to LoadIncrementalHFiles tool > --- > > Key: HBASE-17165 > URL: https://issues.apache.org/jira/browse/HBASE-17165 > Project: HBase > Issue Type: Improvement > Components: hbase, HFile, tooling >Affects Versions: 2.0.0, 1.2.3 >Reporter: Mike Grimes >Assignee: Mike Grimes >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17165.branch-1.001.patch, > HBASE-17165.branch-1.001.patch, HBASE-17165.branch-1.002.patch, > HBASE-17165.branch-1.2.001.patch, HBASE-17165.branch-1.2.002.patch, > HBASE-17165.branch-1.2.003.patch, HBASE-17165.branch-1.2.004.patch, > HBASE-17165.master.001.patch, HBASE-17165.master.002.patch, > HBASE-17165.master.003.patch, HBASE-17165.master.004.patch > > Original Estimate: 0h > Remaining Estimate: 0h > > As using the LoadIncrementalHFiles tool with S3 as the filesystem is prone to > failing due to FileNotFoundExceptions due to inconsistency, simple, > configurable retry logic was added. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-17165) Add retry to LoadIncrementalHFiles tool
[ https://issues.apache.org/jira/browse/HBASE-17165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Grimes updated HBASE-17165: Attachment: HBASE-17165.branch-1.002.patch HBASE-17165.master.004.patch Sorry for the half-fixed patch uploaded before the weekend. I noticed a bug in the unit test after submitting. > Add retry to LoadIncrementalHFiles tool > --- > > Key: HBASE-17165 > URL: https://issues.apache.org/jira/browse/HBASE-17165 > Project: HBase > Issue Type: Improvement > Components: hbase, HFile, tooling >Affects Versions: 2.0.0, 1.2.3 >Reporter: Mike Grimes >Assignee: Mike Grimes >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17165.branch-1.001.patch, > HBASE-17165.branch-1.001.patch, HBASE-17165.branch-1.002.patch, > HBASE-17165.branch-1.2.001.patch, HBASE-17165.branch-1.2.002.patch, > HBASE-17165.branch-1.2.003.patch, HBASE-17165.branch-1.2.004.patch, > HBASE-17165.master.001.patch, HBASE-17165.master.002.patch, > HBASE-17165.master.003.patch, HBASE-17165.master.004.patch > > Original Estimate: 0h > Remaining Estimate: 0h > > As using the LoadIncrementalHFiles tool with S3 as the filesystem is prone to > failing due to FileNotFoundExceptions due to inconsistency, simple, > configurable retry logic was added. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-19303) Cleanup the usage of deprecated ReplicationAdmin
Guanghao Zhang created HBASE-19303: -- Summary: Cleanup the usage of deprecated ReplicationAdmin Key: HBASE-19303 URL: https://issues.apache.org/jira/browse/HBASE-19303 Project: HBase Issue Type: Improvement Reporter: Guanghao Zhang Assignee: Guanghao Zhang -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-19302) document test infra
Sean Busbey created HBASE-19302: --- Summary: document test infra Key: HBASE-19302 URL: https://issues.apache.org/jira/browse/HBASE-19302 Project: HBase Issue Type: Task Components: community, test Reporter: Sean Busbey we should update the ref guide to provide guidance on the test infra we have built up on builds.a.o. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19293) Support add a disabled state replication peer directly
[ https://issues.apache.org/jira/browse/HBASE-19293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258749#comment-16258749 ] Guanghao Zhang commented on HBASE-19293: bq. could we keep removal of deprecated methods for 3.0 in a different patch? listReplicationPeers(regex) was marked as Deprecated. But it was introduced since 2.0. So I remove it. > Support add a disabled state replication peer directly > -- > > Key: HBASE-19293 > URL: https://issues.apache.org/jira/browse/HBASE-19293 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19293.master.001.patch, > HBASE-19293.master.002.patch > > > Now when add a replication peer, the default state is enabled. If you want > add a disabled replication peer, you need add a peer first, then disable it. > It need two step to finish now. > Use case for add a disabled replication peer. When user want sync data from a > cluster A to a new peer cluster. > 1. Add a disabled replication peer. And config the table to peer config. > 2. Take a snapshot of table and export snapshot to peer cluster. > 3. Restore snapshot in peer cluster. > 4. Enable the peer and wait all stuck replication log replicated to peer > cluster. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19293) Support add a disabled state replication peer directly
[ https://issues.apache.org/jira/browse/HBASE-19293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258747#comment-16258747 ] Guanghao Zhang commented on HBASE-19293: Yes, all the modify of Admin methods was introduced since 2.0. As we not release 2.0 now, so I thought it is ok to modify them. bq. Why we add the default modifier at beginning but then remove them in this patch? In the previous implementation, I plan to add them to branch-1, too. So I use default method when add them. But this was only introduced to 2.0, the "default" is not need any more. > Support add a disabled state replication peer directly > -- > > Key: HBASE-19293 > URL: https://issues.apache.org/jira/browse/HBASE-19293 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19293.master.001.patch, > HBASE-19293.master.002.patch > > > Now when add a replication peer, the default state is enabled. If you want > add a disabled replication peer, you need add a peer first, then disable it. > It need two step to finish now. > Use case for add a disabled replication peer. When user want sync data from a > cluster A to a new peer cluster. > 1. Add a disabled replication peer. And config the table to peer config. > 2. Take a snapshot of table and export snapshot to peer cluster. > 3. Restore snapshot in peer cluster. > 4. Enable the peer and wait all stuck replication log replicated to peer > cluster. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HBASE-19293) Support add a disabled state replication peer directly
[ https://issues.apache.org/jira/browse/HBASE-19293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258747#comment-16258747 ] Guanghao Zhang edited comment on HBASE-19293 at 11/20/17 2:17 AM: -- Yes, all the modify of Admin methods was introduced since 2.0. As we not release 2.0 now, so I thought it is ok to modify them. bq. Why we add the default modifier at beginning but then remove them in this patch? In the previous implementation, I plan to add them to branch-1, too. So I use default method when add them. But this was only introduced to 2.0, the "default" is not needed any more. was (Author: zghaobac): Yes, all the modify of Admin methods was introduced since 2.0. As we not release 2.0 now, so I thought it is ok to modify them. bq. Why we add the default modifier at beginning but then remove them in this patch? In the previous implementation, I plan to add them to branch-1, too. So I use default method when add them. But this was only introduced to 2.0, the "default" is not need any more. > Support add a disabled state replication peer directly > -- > > Key: HBASE-19293 > URL: https://issues.apache.org/jira/browse/HBASE-19293 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19293.master.001.patch, > HBASE-19293.master.002.patch > > > Now when add a replication peer, the default state is enabled. If you want > add a disabled replication peer, you need add a peer first, then disable it. > It need two step to finish now. > Use case for add a disabled replication peer. When user want sync data from a > cluster A to a new peer cluster. > 1. Add a disabled replication peer. And config the table to peer config. > 2. Take a snapshot of table and export snapshot to peer cluster. > 3. Restore snapshot in peer cluster. > 4. Enable the peer and wait all stuck replication log replicated to peer > cluster. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19293) Support add a disabled state replication peer directly
[ https://issues.apache.org/jira/browse/HBASE-19293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258726#comment-16258726 ] Duo Zhang commented on HBASE-19293: --- And yes, add default modifier for the new methods introduced in 2.0 may benefit users who implement their own Admin, but I'm still neutral on this. We should depend on the method itself to decide whether to make it default. If it is a critical method that must be presented then it should not be declared as default method, even if it may break the compatibility. I believe we already mentioned this in our ref guide. I'm not familiar with the background of the replication related methods in Admin, seems they are moved from ReplicationAdmin? [~zghaobac] Why we add the default modifier at beginning but then remove them in this patch? Thanks. > Support add a disabled state replication peer directly > -- > > Key: HBASE-19293 > URL: https://issues.apache.org/jira/browse/HBASE-19293 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19293.master.001.patch, > HBASE-19293.master.002.patch > > > Now when add a replication peer, the default state is enabled. If you want > add a disabled replication peer, you need add a peer first, then disable it. > It need two step to finish now. > Use case for add a disabled replication peer. When user want sync data from a > cluster A to a new peer cluster. > 1. Add a disabled replication peer. And config the table to peer config. > 2. Take a snapshot of table and export snapshot to peer cluster. > 3. Restore snapshot in peer cluster. > 4. Enable the peer and wait all stuck replication log replicated to peer > cluster. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19293) Support add a disabled state replication peer directly
[ https://issues.apache.org/jira/browse/HBASE-19293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258720#comment-16258720 ] Duo Zhang commented on HBASE-19293: --- [~busbey] I've already asked on rb. These methods are all added in 2.0 so it is safe to modify them, and also the required field. Thanks. > Support add a disabled state replication peer directly > -- > > Key: HBASE-19293 > URL: https://issues.apache.org/jira/browse/HBASE-19293 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19293.master.001.patch, > HBASE-19293.master.002.patch > > > Now when add a replication peer, the default state is enabled. If you want > add a disabled replication peer, you need add a peer first, then disable it. > It need two step to finish now. > Use case for add a disabled replication peer. When user want sync data from a > cluster A to a new peer cluster. > 1. Add a disabled replication peer. And config the table to peer config. > 2. Take a snapshot of table and export snapshot to peer cluster. > 3. Restore snapshot in peer cluster. > 4. Enable the peer and wait all stuck replication log replicated to peer > cluster. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19293) Support add a disabled state replication peer directly
[
https://issues.apache.org/jira/browse/HBASE-19293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258717#comment-16258717
]
Sean Busbey commented on HBASE-19293:
-
-1 on v1 and v2 patches.
1) please do not remove default methods from IA.Public interfaces (i.e. Admin).
That will break folks who rely on that default being present.
2) the current QA flags are from the patch. The checkstyle one won't show up if
you back out the removal of defaults from Admin.
3) could we keep removal of deprecated methods for 3.0 in a different patch?
It's easier to review the changes for this patch if they're just doing what it
says in the subject.
4)
{code}
diff --git a/hbase-protocol-shaded/src/main/protobuf/Replication.proto
b/hbase-protocol-shaded/src/main/protobuf/Replication.proto
index 7e78144..88efa00 100644
--- a/hbase-protocol-shaded/src/main/protobuf/Replication.proto
+++ b/hbase-protocol-shaded/src/main/protobuf/Replication.proto
@@ -77,6 +77,7 @@ message ReplicationHLogPosition {
message AddReplicationPeerRequest {
required string peer_id = 1;
required ReplicationPeer peer_config = 2;
+ required ReplicationState peer_state = 3;
}
{code}
This will make it so that we break admin clients further. Can we make the
peer_state optional and default to ENABLED?
> Support add a disabled state replication peer directly
> --
>
> Key: HBASE-19293
> URL: https://issues.apache.org/jira/browse/HBASE-19293
> Project: HBase
> Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Fix For: 2.0.0-beta-1
>
> Attachments: HBASE-19293.master.001.patch,
> HBASE-19293.master.002.patch
>
>
> Now when add a replication peer, the default state is enabled. If you want
> add a disabled replication peer, you need add a peer first, then disable it.
> It need two step to finish now.
> Use case for add a disabled replication peer. When user want sync data from a
> cluster A to a new peer cluster.
> 1. Add a disabled replication peer. And config the table to peer config.
> 2. Take a snapshot of table and export snapshot to peer cluster.
> 3. Restore snapshot in peer cluster.
> 4. Enable the peer and wait all stuck replication log replicated to peer
> cluster.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HBASE-14170) [HBase Rest] RESTServer is not shutting down if "hbase.rest.port" Address already in use.
[
https://issues.apache.org/jira/browse/HBASE-14170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey reassigned HBASE-14170:
---
Assignee: Bar Rotstein
all yours!
> [HBase Rest] RESTServer is not shutting down if "hbase.rest.port" Address
> already in use.
> -
>
> Key: HBASE-14170
> URL: https://issues.apache.org/jira/browse/HBASE-14170
> Project: HBase
> Issue Type: Bug
> Components: REST
>Reporter: Y. SREENIVASULU REDDY
>Assignee: Bar Rotstein
> Fix For: 2.0.0
>
>
> [HBase Rest] RESTServer is not shutting down if "hbase.rest.port" Address
> already in use.
> If "hbase.rest.port" Address already in use, RESTServer should shutdown,
> with out this "hbase.rest.port" we cant perform any operations on
> RESTServer. Then there is no use of running RESTServer process.
> {code}
> 2015-07-30 11:49:48,273 WARN [main] mortbay.log: failed
> SelectChannelConnector@0.0.0.0:8080: java.net.BindException: Address already
> in use
> 2015-07-30 11:49:48,274 WARN [main] mortbay.log: failed Server@563f38c4:
> java.net.BindException: Address already in use
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HBASE-14171) [HBase Thrift] ThriftServer is not shutting down if "hbase.regionserver.thrift.port" Address already in use.
[
https://issues.apache.org/jira/browse/HBASE-14171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey reassigned HBASE-14171:
---
Assignee: Bar Rotstein
[~barrotsteindev], all yours! If you need any help, let us know.
> [HBase Thrift] ThriftServer is not shutting down if
> "hbase.regionserver.thrift.port" Address already in use.
>
>
> Key: HBASE-14171
> URL: https://issues.apache.org/jira/browse/HBASE-14171
> Project: HBase
> Issue Type: Bug
> Components: Thrift
>Reporter: Y. SREENIVASULU REDDY
>Assignee: Bar Rotstein
> Fix For: 2.0.0
>
>
> If "hbase.regionserver.thrift.port" Address already in use, ThriftServer
> should shutdown,
> with out this "hbase.regionserver.thrift.port" we cant perform any
> operations on ThriftServer . Then there is no use of running ThriftServer
> process.
> If port is already used then exception or message also not throwing like
> address already in use.
> Only info port information is showing.
> {code}
> 2015-07-30 12:20:56,186 INFO [main] http.HttpServer: Jetty bound to port 9095
> 2015-07-30 12:20:56,186 INFO [main] mortbay.log: jetty-6.1.26
> 2015-07-30 12:20:56,227 WARN [main] mortbay.log: Can't reuse
> /tmp/Jetty_0_0_0_0_9095_thrift.vqpz9l, using
> /tmp/Jetty_0_0_0_0_9095_thrift.vqpz9l_4913486964252131199
> 2015-07-30 12:20:56,553 INFO [main] mortbay.log: Started
> HttpServer$SelectChannelConnectorWithSafeStartup@0.0.0.0:9095 {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19289) CommonFSUtils$StreamLacksCapabilityException: hflush when running test against hadoop3 beta1
[
https://issues.apache.org/jira/browse/HBASE-19289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258707#comment-16258707
]
Sean Busbey commented on HBASE-19289:
-
Thanks for weighing in [~ste...@apache.org]. This helps a ton; I had overlooked
that limitation of ChecksumFileSystem.
Would updating RawLocalFileSystem to create output streams that do implement
hflush/hsync be viable? I think long term HBase needs a FileSystem
implementation that can be used with the local disks on a single node. I'd
rather it get maintained in a place where folks other than us can use it and
help with maintenance.
> CommonFSUtils$StreamLacksCapabilityException: hflush when running test
> against hadoop3 beta1
>
>
> Key: HBASE-19289
> URL: https://issues.apache.org/jira/browse/HBASE-19289
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
> Attachments: 19289.v1.txt
>
>
> As of commit d8fb10c8329b19223c91d3cda6ef149382ad4ea0 , I encountered the
> following exception when running unit test against hadoop3 beta1:
> {code}
> testRefreshStoreFiles(org.apache.hadoop.hbase.regionserver.TestHStore) Time
> elapsed: 0.061 sec <<< ERROR!
> java.io.IOException: cannot get log writer
> at
> org.apache.hadoop.hbase.regionserver.TestHStore.initHRegion(TestHStore.java:215)
> at
> org.apache.hadoop.hbase.regionserver.TestHStore.init(TestHStore.java:220)
> at
> org.apache.hadoop.hbase.regionserver.TestHStore.init(TestHStore.java:195)
> at
> org.apache.hadoop.hbase.regionserver.TestHStore.init(TestHStore.java:190)
> at
> org.apache.hadoop.hbase.regionserver.TestHStore.init(TestHStore.java:185)
> at
> org.apache.hadoop.hbase.regionserver.TestHStore.init(TestHStore.java:179)
> at
> org.apache.hadoop.hbase.regionserver.TestHStore.init(TestHStore.java:173)
> at
> org.apache.hadoop.hbase.regionserver.TestHStore.testRefreshStoreFiles(TestHStore.java:962)
> Caused by:
> org.apache.hadoop.hbase.util.CommonFSUtils$StreamLacksCapabilityException:
> hflush
> at
> org.apache.hadoop.hbase.regionserver.TestHStore.initHRegion(TestHStore.java:215)
> at
> org.apache.hadoop.hbase.regionserver.TestHStore.init(TestHStore.java:220)
> at
> org.apache.hadoop.hbase.regionserver.TestHStore.init(TestHStore.java:195)
> at
> org.apache.hadoop.hbase.regionserver.TestHStore.init(TestHStore.java:190)
> at
> org.apache.hadoop.hbase.regionserver.TestHStore.init(TestHStore.java:185)
> at
> org.apache.hadoop.hbase.regionserver.TestHStore.init(TestHStore.java:179)
> at
> org.apache.hadoop.hbase.regionserver.TestHStore.init(TestHStore.java:173)
> at
> org.apache.hadoop.hbase.regionserver.TestHStore.testRefreshStoreFiles(TestHStore.java:962)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-15320) HBase connector for Kafka Connect
[ https://issues.apache.org/jira/browse/HBASE-15320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Wingert updated HBASE-15320: - Attachment: HBASE-15320.pdf Updated documentation. > HBase connector for Kafka Connect > - > > Key: HBASE-15320 > URL: https://issues.apache.org/jira/browse/HBASE-15320 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: Andrew Purtell >Assignee: Mike Wingert > Labels: beginner > Fix For: 3.0.0 > > Attachments: HBASE-15320.master.1.patch, HBASE-15320.master.2.patch, > HBASE-15320.master.3.patch, HBASE-15320.master.4.patch, > HBASE-15320.master.5.patch, HBASE-15320.master.6.patch, > HBASE-15320.master.7.patch, HBASE-15320.master.8.patch, HBASE-15320.pdf, > HBASE-15320.pdf > > > Implement an HBase connector with source and sink tasks for the Connect > framework (http://docs.confluent.io/2.0.0/connect/index.html) available in > Kafka 0.9 and later. > See also: > http://www.confluent.io/blog/announcing-kafka-connect-building-large-scale-low-latency-data-pipelines > An HBase source > (http://docs.confluent.io/2.0.0/connect/devguide.html#task-example-source-task) > could be implemented as a replication endpoint or WALObserver, publishing > cluster wide change streams from the WAL to one or more topics, with > configurable mapping and partitioning of table changes to topics. > An HBase sink task > (http://docs.confluent.io/2.0.0/connect/devguide.html#sink-tasks) would > persist, with optional transformation (JSON? Avro?, map fields to native > schema?), Kafka SinkRecords into HBase tables. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-15320) HBase connector for Kafka Connect
[ https://issues.apache.org/jira/browse/HBASE-15320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Wingert updated HBASE-15320: - Attachment: HBASE-15320.master.8.patch > HBase connector for Kafka Connect > - > > Key: HBASE-15320 > URL: https://issues.apache.org/jira/browse/HBASE-15320 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: Andrew Purtell >Assignee: Mike Wingert > Labels: beginner > Fix For: 3.0.0 > > Attachments: HBASE-15320.master.1.patch, HBASE-15320.master.2.patch, > HBASE-15320.master.3.patch, HBASE-15320.master.4.patch, > HBASE-15320.master.5.patch, HBASE-15320.master.6.patch, > HBASE-15320.master.7.patch, HBASE-15320.master.8.patch, HBASE-15320.pdf > > > Implement an HBase connector with source and sink tasks for the Connect > framework (http://docs.confluent.io/2.0.0/connect/index.html) available in > Kafka 0.9 and later. > See also: > http://www.confluent.io/blog/announcing-kafka-connect-building-large-scale-low-latency-data-pipelines > An HBase source > (http://docs.confluent.io/2.0.0/connect/devguide.html#task-example-source-task) > could be implemented as a replication endpoint or WALObserver, publishing > cluster wide change streams from the WAL to one or more topics, with > configurable mapping and partitioning of table changes to topics. > An HBase sink task > (http://docs.confluent.io/2.0.0/connect/devguide.html#sink-tasks) would > persist, with optional transformation (JSON? Avro?, map fields to native > schema?), Kafka SinkRecords into HBase tables. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-15320) HBase connector for Kafka Connect
[ https://issues.apache.org/jira/browse/HBASE-15320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258681#comment-16258681 ] Mike Wingert commented on HBASE-15320: -- Updated with Ted's suggestions. > HBase connector for Kafka Connect > - > > Key: HBASE-15320 > URL: https://issues.apache.org/jira/browse/HBASE-15320 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: Andrew Purtell >Assignee: Mike Wingert > Labels: beginner > Fix For: 3.0.0 > > Attachments: HBASE-15320.master.1.patch, HBASE-15320.master.2.patch, > HBASE-15320.master.3.patch, HBASE-15320.master.4.patch, > HBASE-15320.master.5.patch, HBASE-15320.master.6.patch, > HBASE-15320.master.7.patch, HBASE-15320.master.8.patch, HBASE-15320.pdf > > > Implement an HBase connector with source and sink tasks for the Connect > framework (http://docs.confluent.io/2.0.0/connect/index.html) available in > Kafka 0.9 and later. > See also: > http://www.confluent.io/blog/announcing-kafka-connect-building-large-scale-low-latency-data-pipelines > An HBase source > (http://docs.confluent.io/2.0.0/connect/devguide.html#task-example-source-task) > could be implemented as a replication endpoint or WALObserver, publishing > cluster wide change streams from the WAL to one or more topics, with > configurable mapping and partitioning of table changes to topics. > An HBase sink task > (http://docs.confluent.io/2.0.0/connect/devguide.html#sink-tasks) would > persist, with optional transformation (JSON? Avro?, map fields to native > schema?), Kafka SinkRecords into HBase tables. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-15320) HBase connector for Kafka Connect
[ https://issues.apache.org/jira/browse/HBASE-15320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258662#comment-16258662 ] Mike Wingert commented on HBASE-15320: -- It's there, sorry about all the mistakes. > HBase connector for Kafka Connect > - > > Key: HBASE-15320 > URL: https://issues.apache.org/jira/browse/HBASE-15320 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: Andrew Purtell >Assignee: Mike Wingert > Labels: beginner > Fix For: 3.0.0 > > Attachments: HBASE-15320.master.1.patch, HBASE-15320.master.2.patch, > HBASE-15320.master.3.patch, HBASE-15320.master.4.patch, > HBASE-15320.master.5.patch, HBASE-15320.master.6.patch, > HBASE-15320.master.7.patch, HBASE-15320.pdf > > > Implement an HBase connector with source and sink tasks for the Connect > framework (http://docs.confluent.io/2.0.0/connect/index.html) available in > Kafka 0.9 and later. > See also: > http://www.confluent.io/blog/announcing-kafka-connect-building-large-scale-low-latency-data-pipelines > An HBase source > (http://docs.confluent.io/2.0.0/connect/devguide.html#task-example-source-task) > could be implemented as a replication endpoint or WALObserver, publishing > cluster wide change streams from the WAL to one or more topics, with > configurable mapping and partitioning of table changes to topics. > An HBase sink task > (http://docs.confluent.io/2.0.0/connect/devguide.html#sink-tasks) would > persist, with optional transformation (JSON? Avro?, map fields to native > schema?), Kafka SinkRecords into HBase tables. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-15320) HBase connector for Kafka Connect
[ https://issues.apache.org/jira/browse/HBASE-15320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258660#comment-16258660 ] Ted Yu commented on HBASE-15320: For https://reviews.apache.org/r/63926/, I don't see button for downloading patch. And there is no button to show the changes. Can you double check ? > HBase connector for Kafka Connect > - > > Key: HBASE-15320 > URL: https://issues.apache.org/jira/browse/HBASE-15320 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: Andrew Purtell >Assignee: Mike Wingert > Labels: beginner > Fix For: 3.0.0 > > Attachments: HBASE-15320.master.1.patch, HBASE-15320.master.2.patch, > HBASE-15320.master.3.patch, HBASE-15320.master.4.patch, > HBASE-15320.master.5.patch, HBASE-15320.master.6.patch, > HBASE-15320.master.7.patch, HBASE-15320.pdf > > > Implement an HBase connector with source and sink tasks for the Connect > framework (http://docs.confluent.io/2.0.0/connect/index.html) available in > Kafka 0.9 and later. > See also: > http://www.confluent.io/blog/announcing-kafka-connect-building-large-scale-low-latency-data-pipelines > An HBase source > (http://docs.confluent.io/2.0.0/connect/devguide.html#task-example-source-task) > could be implemented as a replication endpoint or WALObserver, publishing > cluster wide change streams from the WAL to one or more topics, with > configurable mapping and partitioning of table changes to topics. > An HBase sink task > (http://docs.confluent.io/2.0.0/connect/devguide.html#sink-tasks) would > persist, with optional transformation (JSON? Avro?, map fields to native > schema?), Kafka SinkRecords into HBase tables. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19096) Add RowMutions batch support in AsyncTable
[ https://issues.apache.org/jira/browse/HBASE-19096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258600#comment-16258600 ] Jerry He commented on HBASE-19096: -- [~zghaobac] Thanks for chiming in. bq. Seems we will iterate RegionAction's actions twice. One is in buildReq. Another one is in buildNoDataRegionAction. Can we move this to one loop? That is a good idea. There is an old TODO there. Probably need some refracting around buildNoDataRegionAction. Let me see what I can do. bq. If a RegionAction's actions have a RowMutations and some put/delete, then there are 2 RegionAction will be added to MutliRequest? Yes, each RowMutations will be a separate RegionAction. bq. About buildReq, there are some code same with MultiServerCallable, can we do some refactor to avoid this? Again, need some refactoring. Let me see if it is too much. > Add RowMutions batch support in AsyncTable > -- > > Key: HBASE-19096 > URL: https://issues.apache.org/jira/browse/HBASE-19096 > Project: HBase > Issue Type: Sub-task >Reporter: Jerry He >Assignee: Jerry He > Fix For: 2.0.0 > > Attachments: HBASE-19096-master.patch > > > Batch support for RowMutations has been added in the Table interface, but is > not in AsyncTable. This JIRA will add it. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19096) Add RowMutions batch support in AsyncTable
[
https://issues.apache.org/jira/browse/HBASE-19096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258597#comment-16258597
]
Jerry He commented on HBASE-19096:
--
bq. We do this if (row instanceof RowMutations) { ... but if the row is
anything else, we do nothing? Could it be something else?
All other cases handled in the code below that.
I will add more comment.
> Add RowMutions batch support in AsyncTable
> --
>
> Key: HBASE-19096
> URL: https://issues.apache.org/jira/browse/HBASE-19096
> Project: HBase
> Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
> Fix For: 2.0.0
>
> Attachments: HBASE-19096-master.patch
>
>
> Batch support for RowMutations has been added in the Table interface, but is
> not in AsyncTable. This JIRA will add it.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19301) Provide way for CPs to create short circuited connection with custom configurations
[
https://issues.apache.org/jira/browse/HBASE-19301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258582#comment-16258582
]
Hadoop QA commented on HBASE-19301:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 2m
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 17 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 9m
32s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
54s{color} | {color:red} hbase-server: The patch generated 7 new + 340
unchanged - 1 fixed = 347 total (was 341) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 410 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
22s{color} | {color:red} The patch 384 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 8m
15s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
87m 4s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 2.7.4 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}111m
22s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}235m 22s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19301 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12898383/HBASE-19301.patch |
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 80b7f3fa2bac 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / b4fbf5fe18 |
| maven | version: Apache Maven 3.5.2
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HBASE-Build/9916/artifact/patchprocess/diff-checkstyle-hbase-server.txt
|
| whitespace |
[jira] [Commented] (HBASE-14171) [HBase Thrift] ThriftServer is not shutting down if "hbase.regionserver.thrift.port" Address already in use.
[
https://issues.apache.org/jira/browse/HBASE-14171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258578#comment-16258578
]
Bar Rotstein commented on HBASE-14171:
--
I would like to start working on this issue,
can it be assigned to me please?
> [HBase Thrift] ThriftServer is not shutting down if
> "hbase.regionserver.thrift.port" Address already in use.
>
>
> Key: HBASE-14171
> URL: https://issues.apache.org/jira/browse/HBASE-14171
> Project: HBase
> Issue Type: Bug
> Components: Thrift
>Reporter: Y. SREENIVASULU REDDY
> Fix For: 2.0.0
>
>
> If "hbase.regionserver.thrift.port" Address already in use, ThriftServer
> should shutdown,
> with out this "hbase.regionserver.thrift.port" we cant perform any
> operations on ThriftServer . Then there is no use of running ThriftServer
> process.
> If port is already used then exception or message also not throwing like
> address already in use.
> Only info port information is showing.
> {code}
> 2015-07-30 12:20:56,186 INFO [main] http.HttpServer: Jetty bound to port 9095
> 2015-07-30 12:20:56,186 INFO [main] mortbay.log: jetty-6.1.26
> 2015-07-30 12:20:56,227 WARN [main] mortbay.log: Can't reuse
> /tmp/Jetty_0_0_0_0_9095_thrift.vqpz9l, using
> /tmp/Jetty_0_0_0_0_9095_thrift.vqpz9l_4913486964252131199
> 2015-07-30 12:20:56,553 INFO [main] mortbay.log: Started
> HttpServer$SelectChannelConnectorWithSafeStartup@0.0.0.0:9095 {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (HBASE-14170) [HBase Rest] RESTServer is not shutting down if "hbase.rest.port" Address already in use.
[
https://issues.apache.org/jira/browse/HBASE-14170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258575#comment-16258575
]
Bar Rotstein edited comment on HBASE-14170 at 11/19/17 7:26 PM:
Hey,
if Andrew hasn't solved this issue yet,
I would like to start working on this issue.
I'll try and catch the exception to shutdown the REST server.
was (Author: barrotsteindev):
Hey,
I would like to start working on this issue.
I'll try and catch the exception to shutdown the REST server.
> [HBase Rest] RESTServer is not shutting down if "hbase.rest.port" Address
> already in use.
> -
>
> Key: HBASE-14170
> URL: https://issues.apache.org/jira/browse/HBASE-14170
> Project: HBase
> Issue Type: Bug
> Components: REST
>Reporter: Y. SREENIVASULU REDDY
> Fix For: 2.0.0
>
>
> [HBase Rest] RESTServer is not shutting down if "hbase.rest.port" Address
> already in use.
> If "hbase.rest.port" Address already in use, RESTServer should shutdown,
> with out this "hbase.rest.port" we cant perform any operations on
> RESTServer. Then there is no use of running RESTServer process.
> {code}
> 2015-07-30 11:49:48,273 WARN [main] mortbay.log: failed
> SelectChannelConnector@0.0.0.0:8080: java.net.BindException: Address already
> in use
> 2015-07-30 11:49:48,274 WARN [main] mortbay.log: failed Server@563f38c4:
> java.net.BindException: Address already in use
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-14170) [HBase Rest] RESTServer is not shutting down if "hbase.rest.port" Address already in use.
[
https://issues.apache.org/jira/browse/HBASE-14170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258575#comment-16258575
]
Bar Rotstein commented on HBASE-14170:
--
Hey,
I would like to start working on this issue.
I'll try and catch the exception to shutdown the REST server.
> [HBase Rest] RESTServer is not shutting down if "hbase.rest.port" Address
> already in use.
> -
>
> Key: HBASE-14170
> URL: https://issues.apache.org/jira/browse/HBASE-14170
> Project: HBase
> Issue Type: Bug
> Components: REST
>Reporter: Y. SREENIVASULU REDDY
> Fix For: 2.0.0
>
>
> [HBase Rest] RESTServer is not shutting down if "hbase.rest.port" Address
> already in use.
> If "hbase.rest.port" Address already in use, RESTServer should shutdown,
> with out this "hbase.rest.port" we cant perform any operations on
> RESTServer. Then there is no use of running RESTServer process.
> {code}
> 2015-07-30 11:49:48,273 WARN [main] mortbay.log: failed
> SelectChannelConnector@0.0.0.0:8080: java.net.BindException: Address already
> in use
> 2015-07-30 11:49:48,274 WARN [main] mortbay.log: failed Server@563f38c4:
> java.net.BindException: Address already in use
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-17436) Add facility to provide more information for Other Regions seen on Master UI
[ https://issues.apache.org/jira/browse/HBASE-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258572#comment-16258572 ] Janos Gub commented on HBASE-17436: --- @stack There is no further information on the JSON. I think it is just way more easier to consume programmatically than the UI. > Add facility to provide more information for Other Regions seen on Master UI > > > Key: HBASE-17436 > URL: https://issues.apache.org/jira/browse/HBASE-17436 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Janos Gub > Labels: ui, usability > Fix For: 2.0.0 > > Attachments: 17436.v10.txt, 17436.v9.txt, HBASE-17436-v2.patch, > HBASE-17436-v5.patch, HBASE-17436-v6.patch, HBASE-17436-v6.patch, > HBASE-17436-v7.patch, HBASE-17436-v8.patch, HBASE-17436.patch, > HBASE-17779-v3.patch, HBASE-17779-v4.patch, Screen Shot 2017-04-24 at 8.43.31 > PM.png, Screen Shot 2017-04-26 at 4.39.49 PM.png, Screen Shot 2017-11-08 at > 10.29.49 AM.png, Screen Shot 2017-11-08 at 10.30.09 AM.png, initial.patch, > regionstates.png, table-selected.png, tableregions.png > > > [~rpednekar] and I were looking at a case where the count displayed under > Other Regions was high (~1200). > Since the table page just maintains a count instead of List of region names, > it is very difficult for user to determine which regions belong to this > category. > We should provide facility to provide more details for this category (LOG, > JMX, etc). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18936) Backport HBASE-16870 to branch-1.3
[ https://issues.apache.org/jira/browse/HBASE-18936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258527#comment-16258527 ] Chia-Ping Tsai commented on HBASE-18936: Will commit it tomorrow. > Backport HBASE-16870 to branch-1.3 > -- > > Key: HBASE-18936 > URL: https://issues.apache.org/jira/browse/HBASE-18936 > Project: HBase > Issue Type: Sub-task >Affects Versions: 1.3.1 >Reporter: Ashish Singhi >Assignee: Peter Somogyi > Fix For: 1.3.2 > > Attachments: HBASE-18936.branch-1.3.001.patch, > HBASE-18936.branch-1.3.002.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19204) branch-1.2 times out and is taking 6-7 hours to complete
[ https://issues.apache.org/jira/browse/HBASE-19204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258524#comment-16258524 ] Sean Busbey commented on HBASE-19204: - bq. Possible to compile w/ 1.7 but run w/ 1.8 Sean Busbey? Our yetus probably doesn't allow this? I'd have to play with the multi-jdk stuff to see if I could get the unit test running JDK to differ from the corresponding compile. What do we think this will catch that we won't catch running JDK8 for unit tests with JDK8 compilation? With our current set up it'd be much easier for me to do a "make sure things compile with JDK7" and then just run unit tests in our existing "build and run with JDK8" test step. If we can't run the unit test suite with JDK7, what does that imply about the reliability for folks who run JDK7? should we call out concerns where we currently say running with JDK7 is supported? We could set up a jdk7 profile that disables some tests. Sounds like maybe that'll be too many to be useful? > branch-1.2 times out and is taking 6-7 hours to complete > > > Key: HBASE-19204 > URL: https://issues.apache.org/jira/browse/HBASE-19204 > Project: HBase > Issue Type: Umbrella > Components: test >Reporter: stack > > Sean has been looking at tooling and infra. This Umbrellas is about looking > at actual tests. For example, running locally on dedicated machine I picked a > random test, TestPerColumnFamilyFlush. In my test run, it wrote 16M lines. It > seems to be having zk issues but it is catching interrupts and ignoring them > ([~carp84] fixed this in later versions over in HBASE-18441). > Let me try and do some fixup under this umbrella so we can get a 1.2.7 out > the door. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18936) Backport HBASE-16870 to branch-1.3
[ https://issues.apache.org/jira/browse/HBASE-18936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258523#comment-16258523 ] Peter Somogyi commented on HBASE-18936: --- Thank you [~chia7712] for checking! > Backport HBASE-16870 to branch-1.3 > -- > > Key: HBASE-18936 > URL: https://issues.apache.org/jira/browse/HBASE-18936 > Project: HBase > Issue Type: Sub-task >Affects Versions: 1.3.1 >Reporter: Ashish Singhi >Assignee: Peter Somogyi > Fix For: 1.3.2 > > Attachments: HBASE-18936.branch-1.3.001.patch, > HBASE-18936.branch-1.3.002.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18936) Backport HBASE-16870 to branch-1.3
[
https://issues.apache.org/jira/browse/HBASE-18936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258522#comment-16258522
]
Chia-Ping Tsai commented on HBASE-18936:
{code}
23:33:36 | Vote | Subsystem | Runtime | Comment
23:33:36
23:33:36 | | || Prechecks
23:33:36 | +1 | hbaseanti | 0m 0s | Patch does not have any
anti-patterns.
23:33:36 | +1 |@author | 0m 0s | The patch does not contain any
@author
23:33:36 | | || tags.
23:33:36 | +1 | test4tests | 0m 0s | The patch appears to include 1
new or
23:33:36 | | || modified test files.
23:33:37 | | || branch-1.3 Compile Tests
23:33:37 | +1 | mvninstall | 0m 56s | branch-1.3 passed
23:33:37 | +1 |compile | 0m 17s | branch-1.3 passed
23:33:37 | +1 | checkstyle | 0m 56s | branch-1.3 passed
23:33:37 | +1 | findbugs | 0m 47s | branch-1.3 passed
23:33:37 | +1 |javadoc | 0m 51s | branch-1.3 passed
23:33:37 | | || Patch Compile Tests
23:33:37 | +1 | mvninstall | 0m 22s | the patch passed
23:33:37 | +1 |compile | 0m 16s | the patch passed
23:33:37 | +1 | javac | 0m 16s | the patch passed
23:33:37 | +1 | checkstyle | 0m 31s | the patch passed
23:33:37 | +1 | whitespace | 0m 0s | The patch has no whitespace
issues.
23:33:37 | +1 |hadoopcheck | 8m 26s | The patch does not cause any
errors
23:33:37 | | || with Hadoop 2.4.0 2.4.1 2.5.0
2.5.1
23:33:37 | | || 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1.
23:33:37 | +1 | findbugs | 0m 53s | the patch passed
23:33:37 | +1 |javadoc | 0m 13s | the patch passed
23:33:37 | | || Other Tests
23:33:37 | -1 | unit | 71m 14s | hbase-server in the patch
failed.
23:33:37 | +1 | asflicense | 0m 29s | The patch does not generate
ASF License
23:33:37 | | || warnings.
23:33:37 | | | 86m 45s |
{code}
The failed tests pass if I run them again. so +1
> Backport HBASE-16870 to branch-1.3
> --
>
> Key: HBASE-18936
> URL: https://issues.apache.org/jira/browse/HBASE-18936
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 1.3.1
>Reporter: Ashish Singhi
>Assignee: Peter Somogyi
> Fix For: 1.3.2
>
> Attachments: HBASE-18936.branch-1.3.001.patch,
> HBASE-18936.branch-1.3.002.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19266) TestAcidGuarantees should cover adaptive in-memory compaction
[
https://issues.apache.org/jira/browse/HBASE-19266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258520#comment-16258520
]
Chia-Ping Tsai commented on HBASE-19266:
It seems {{TestAcidGuarantees}} exceeds the given timeout 900s. We should
separate the {{TestAcidGuarantees}} by the policy. In my opinion, it can be
addressed in this issue. [~tedyu] I’d like to deal with this issue if you don't
work at it yet.
> TestAcidGuarantees should cover adaptive in-memory compaction
> -
>
> Key: HBASE-19266
> URL: https://issues.apache.org/jira/browse/HBASE-19266
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Priority: Minor
>
> Currently TestAcidGuarantees populates 3 policies of (in-memory) compaction.
> Adaptive in-memory compaction is new and should be added as 4th compaction
> policy.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19301) Provide way for CPs to create short circuited connection with custom configurations
[ https://issues.apache.org/jira/browse/HBASE-19301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-19301: --- Attachment: HBASE-19301.patch > Provide way for CPs to create short circuited connection with custom > configurations > --- > > Key: HBASE-19301 > URL: https://issues.apache.org/jira/browse/HBASE-19301 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19301.patch > > > Over in HBASE-18359 we have discussions for this. > Right now HBase provide getConnection() in RegionCPEnv, MasterCPEnv etc. But > this returns a pre created connection (per server). This uses the configs at > hbase-site.xml at that server. > Phoenix needs creating connection in CP with some custom configs. Having this > custom changes in hbase-site.xml is harmful as that will affect all > connections been created at that server. > This issue is for providing an overloaded getConnection(Configuration) API -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19301) Provide way for CPs to create short circuited connection with custom configurations
[ https://issues.apache.org/jira/browse/HBASE-19301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-19301: --- Status: Patch Available (was: Open) > Provide way for CPs to create short circuited connection with custom > configurations > --- > > Key: HBASE-19301 > URL: https://issues.apache.org/jira/browse/HBASE-19301 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19301.patch > > > Over in HBASE-18359 we have discussions for this. > Right now HBase provide getConnection() in RegionCPEnv, MasterCPEnv etc. But > this returns a pre created connection (per server). This uses the configs at > hbase-site.xml at that server. > Phoenix needs creating connection in CP with some custom configs. Having this > custom changes in hbase-site.xml is harmful as that will affect all > connections been created at that server. > This issue is for providing an overloaded getConnection(Configuration) API -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19266) TestAcidGuarantees should cover adaptive in-memory compaction
[ https://issues.apache.org/jira/browse/HBASE-19266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258505#comment-16258505 ] Ted Yu commented on HBASE-19266: Looking at flaky test dashboard, TestAcidGuarantees has been hanging. Addition of coverage should be done after making TestAcidGuarantees stable. > TestAcidGuarantees should cover adaptive in-memory compaction > - > > Key: HBASE-19266 > URL: https://issues.apache.org/jira/browse/HBASE-19266 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Priority: Minor > > Currently TestAcidGuarantees populates 3 policies of (in-memory) compaction. > Adaptive in-memory compaction is new and should be added as 4th compaction > policy. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18936) Backport HBASE-16870 to branch-1.3
[ https://issues.apache.org/jira/browse/HBASE-18936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi updated HBASE-18936: -- Status: Patch Available (was: In Progress) > Backport HBASE-16870 to branch-1.3 > -- > > Key: HBASE-18936 > URL: https://issues.apache.org/jira/browse/HBASE-18936 > Project: HBase > Issue Type: Sub-task >Affects Versions: 1.3.1 >Reporter: Ashish Singhi >Assignee: Peter Somogyi > Fix For: 1.3.2 > > Attachments: HBASE-18936.branch-1.3.001.patch, > HBASE-18936.branch-1.3.002.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18936) Backport HBASE-16870 to branch-1.3
[ https://issues.apache.org/jira/browse/HBASE-18936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi updated HBASE-18936: -- Attachment: HBASE-18936.branch-1.3.002.patch Jira ID was missing from commit message. Fixed. > Backport HBASE-16870 to branch-1.3 > -- > > Key: HBASE-18936 > URL: https://issues.apache.org/jira/browse/HBASE-18936 > Project: HBase > Issue Type: Sub-task >Affects Versions: 1.3.1 >Reporter: Ashish Singhi >Assignee: Peter Somogyi > Fix For: 1.3.2 > > Attachments: HBASE-18936.branch-1.3.001.patch, > HBASE-18936.branch-1.3.002.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HBASE-19293) Support add a disabled state replication peer directly
[ https://issues.apache.org/jira/browse/HBASE-19293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258495#comment-16258495 ] Ashish Singhi edited comment on HBASE-19293 at 11/19/17 2:02 PM: - No, +1. nit (can address on commit): Since you have mentioned STATE is optional it would be good to mention what is the default value also in the ruby script and document. Are -1s from QA related to the patch, if yes pls address them also. Thanks. was (Author: ashish singhi): No, +1. nit (can address on commit): Since you have mentioned STATE is optional it would be good to mention what is the default value also in the ruby script and document. > Support add a disabled state replication peer directly > -- > > Key: HBASE-19293 > URL: https://issues.apache.org/jira/browse/HBASE-19293 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19293.master.001.patch, > HBASE-19293.master.002.patch > > > Now when add a replication peer, the default state is enabled. If you want > add a disabled replication peer, you need add a peer first, then disable it. > It need two step to finish now. > Use case for add a disabled replication peer. When user want sync data from a > cluster A to a new peer cluster. > 1. Add a disabled replication peer. And config the table to peer config. > 2. Take a snapshot of table and export snapshot to peer cluster. > 3. Restore snapshot in peer cluster. > 4. Enable the peer and wait all stuck replication log replicated to peer > cluster. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19293) Support add a disabled state replication peer directly
[ https://issues.apache.org/jira/browse/HBASE-19293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258495#comment-16258495 ] Ashish Singhi commented on HBASE-19293: --- No, +1. nit (can address on commit): Since you have mentioned STATE is optional it would be good to mention what is the default value also in the ruby script and document. > Support add a disabled state replication peer directly > -- > > Key: HBASE-19293 > URL: https://issues.apache.org/jira/browse/HBASE-19293 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19293.master.001.patch, > HBASE-19293.master.002.patch > > > Now when add a replication peer, the default state is enabled. If you want > add a disabled replication peer, you need add a peer first, then disable it. > It need two step to finish now. > Use case for add a disabled replication peer. When user want sync data from a > cluster A to a new peer cluster. > 1. Add a disabled replication peer. And config the table to peer config. > 2. Take a snapshot of table and export snapshot to peer cluster. > 3. Restore snapshot in peer cluster. > 4. Enable the peer and wait all stuck replication log replicated to peer > cluster. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18936) Backport HBASE-16870 to branch-1.3
[ https://issues.apache.org/jira/browse/HBASE-18936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi updated HBASE-18936: -- Status: In Progress (was: Patch Available) > Backport HBASE-16870 to branch-1.3 > -- > > Key: HBASE-18936 > URL: https://issues.apache.org/jira/browse/HBASE-18936 > Project: HBase > Issue Type: Sub-task >Affects Versions: 1.3.1 >Reporter: Ashish Singhi >Assignee: Peter Somogyi > Fix For: 1.3.2 > > Attachments: HBASE-18936.branch-1.3.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18294) Reduce global heap pressure: flush based on heap occupancy
[
https://issues.apache.org/jira/browse/HBASE-18294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258494#comment-16258494
]
Hadoop QA commented on HBASE-18294:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s{color}
| {color:red} HBASE-18294 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.6.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HBASE-18294 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12898380/HBASE-18294.05.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/9913/console |
| Powered by | Apache Yetus 0.6.0 http://yetus.apache.org |
This message was automatically generated.
> Reduce global heap pressure: flush based on heap occupancy
> --
>
> Key: HBASE-18294
> URL: https://issues.apache.org/jira/browse/HBASE-18294
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Eshcar Hillel
>Assignee: Eshcar Hillel
> Attachments: HBASE-18294.01.patch, HBASE-18294.02.patch,
> HBASE-18294.03.patch, HBASE-18294.04.patch, HBASE-18294.05.patch
>
>
> A region is flushed if its memory component exceed a threshold (default size
> is 128MB).
> A flush policy decides whether to flush a store by comparing the size of the
> store to another threshold (that can be configured with
> hbase.hregion.percolumnfamilyflush.size.lower.bound).
> Currently the implementation (in both cases) compares the data size
> (key-value only) to the threshold where it should compare the heap size
> (which includes index size, and metadata).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18294) Reduce global heap pressure: flush based on heap occupancy
[ https://issues.apache.org/jira/browse/HBASE-18294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258493#comment-16258493 ] Eshcar Hillel commented on HBASE-18294: --- A new patch is available, also on RB. Added a new region level off-heap flush size configuration property. Updated code for RS-level flushes. > Reduce global heap pressure: flush based on heap occupancy > -- > > Key: HBASE-18294 > URL: https://issues.apache.org/jira/browse/HBASE-18294 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel > Attachments: HBASE-18294.01.patch, HBASE-18294.02.patch, > HBASE-18294.03.patch, HBASE-18294.04.patch, HBASE-18294.05.patch > > > A region is flushed if its memory component exceed a threshold (default size > is 128MB). > A flush policy decides whether to flush a store by comparing the size of the > store to another threshold (that can be configured with > hbase.hregion.percolumnfamilyflush.size.lower.bound). > Currently the implementation (in both cases) compares the data size > (key-value only) to the threshold where it should compare the heap size > (which includes index size, and metadata). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18294) Reduce global heap pressure: flush based on heap occupancy
[ https://issues.apache.org/jira/browse/HBASE-18294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel updated HBASE-18294: -- Attachment: HBASE-18294.05.patch > Reduce global heap pressure: flush based on heap occupancy > -- > > Key: HBASE-18294 > URL: https://issues.apache.org/jira/browse/HBASE-18294 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel > Attachments: HBASE-18294.01.patch, HBASE-18294.02.patch, > HBASE-18294.03.patch, HBASE-18294.04.patch, HBASE-18294.05.patch > > > A region is flushed if its memory component exceed a threshold (default size > is 128MB). > A flush policy decides whether to flush a store by comparing the size of the > store to another threshold (that can be configured with > hbase.hregion.percolumnfamilyflush.size.lower.bound). > Currently the implementation (in both cases) compares the data size > (key-value only) to the threshold where it should compare the heap size > (which includes index size, and metadata). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18936) Backport HBASE-16870 to branch-1.3
[ https://issues.apache.org/jira/browse/HBASE-18936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi updated HBASE-18936: -- Status: Patch Available (was: Open) > Backport HBASE-16870 to branch-1.3 > -- > > Key: HBASE-18936 > URL: https://issues.apache.org/jira/browse/HBASE-18936 > Project: HBase > Issue Type: Sub-task >Affects Versions: 1.3.1 >Reporter: Ashish Singhi >Assignee: Peter Somogyi > Fix For: 1.3.2 > > Attachments: HBASE-18936.branch-1.3.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18936) Backport HBASE-16870 to branch-1.3
[ https://issues.apache.org/jira/browse/HBASE-18936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi updated HBASE-18936: -- Attachment: HBASE-18936.branch-1.3.001.patch > Backport HBASE-16870 to branch-1.3 > -- > > Key: HBASE-18936 > URL: https://issues.apache.org/jira/browse/HBASE-18936 > Project: HBase > Issue Type: Sub-task >Affects Versions: 1.3.1 >Reporter: Ashish Singhi >Assignee: Peter Somogyi > Fix For: 1.3.2 > > Attachments: HBASE-18936.branch-1.3.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18600) Provide a method to disable the Short-Circuit HTable in coprocessor
[ https://issues.apache.org/jira/browse/HBASE-18600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-18600: --- Hadoop Flags: (was: Incompatible change) > Provide a method to disable the Short-Circuit HTable in coprocessor > --- > > Key: HBASE-18600 > URL: https://issues.apache.org/jira/browse/HBASE-18600 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-18600.master.001.patch > > > HBASE-9534: Allow coprocessors accessing an HTable to short-circuit access to > the local HRegionServer, rather than requiring the usual RPC path. And it > only worked for RegionCoprocessorEnvironment. This issue add a new method for > RegionCoprocessorEnvironment. getShortCircuitTable method is used to get a > short-circuit access HTable. And getTable method is used to get a normal > table. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18600) Provide a method to disable the Short-Circuit HTable in coprocessor
[ https://issues.apache.org/jira/browse/HBASE-18600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-18600: --- Resolution: Duplicate Status: Resolved (was: Patch Available) Duplicate with HBASE-19301. > Provide a method to disable the Short-Circuit HTable in coprocessor > --- > > Key: HBASE-18600 > URL: https://issues.apache.org/jira/browse/HBASE-18600 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-18600.master.001.patch > > > HBASE-9534: Allow coprocessors accessing an HTable to short-circuit access to > the local HRegionServer, rather than requiring the usual RPC path. And it > only worked for RegionCoprocessorEnvironment. This issue add a new method for > RegionCoprocessorEnvironment. getShortCircuitTable method is used to get a > short-circuit access HTable. And getTable method is used to get a normal > table. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19293) Support add a disabled state replication peer directly
[ https://issues.apache.org/jira/browse/HBASE-19293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258481#comment-16258481 ] Guanghao Zhang commented on HBASE-19293: [~ashish singhi] Any more concerns? > Support add a disabled state replication peer directly > -- > > Key: HBASE-19293 > URL: https://issues.apache.org/jira/browse/HBASE-19293 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19293.master.001.patch, > HBASE-19293.master.002.patch > > > Now when add a replication peer, the default state is enabled. If you want > add a disabled replication peer, you need add a peer first, then disable it. > It need two step to finish now. > Use case for add a disabled replication peer. When user want sync data from a > cluster A to a new peer cluster. > 1. Add a disabled replication peer. And config the table to peer config. > 2. Take a snapshot of table and export snapshot to peer cluster. > 3. Restore snapshot in peer cluster. > 4. Enable the peer and wait all stuck replication log replicated to peer > cluster. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HBASE-18936) Backport HBASE-16870 to branch-1.3
[ https://issues.apache.org/jira/browse/HBASE-18936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi reassigned HBASE-18936: - Assignee: Peter Somogyi > Backport HBASE-16870 to branch-1.3 > -- > > Key: HBASE-18936 > URL: https://issues.apache.org/jira/browse/HBASE-18936 > Project: HBase > Issue Type: Sub-task >Affects Versions: 1.3.1 >Reporter: Ashish Singhi >Assignee: Peter Somogyi > Fix For: 1.3.2 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18936) Backport HBASE-16870 to branch-1.3
[ https://issues.apache.org/jira/browse/HBASE-18936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258465#comment-16258465 ] Peter Somogyi commented on HBASE-18936: --- Sure, no problem! > Backport HBASE-16870 to branch-1.3 > -- > > Key: HBASE-18936 > URL: https://issues.apache.org/jira/browse/HBASE-18936 > Project: HBase > Issue Type: Sub-task >Affects Versions: 1.3.1 >Reporter: Ashish Singhi > Fix For: 1.3.2 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-19301) Provide way for CPs to create short circuited connection with custom configurations
Anoop Sam John created HBASE-19301: -- Summary: Provide way for CPs to create short circuited connection with custom configurations Key: HBASE-19301 URL: https://issues.apache.org/jira/browse/HBASE-19301 Project: HBase Issue Type: Sub-task Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0-beta-1 Over in HBASE-18359 we have discussions for this. Right now HBase provide getConnection() in RegionCPEnv, MasterCPEnv etc. But this returns a pre created connection (per server). This uses the configs at hbase-site.xml at that server. Phoenix needs creating connection in CP with some custom configs. Having this custom changes in hbase-site.xml is harmful as that will affect all connections been created at that server. This issue is for providing an overloaded getConnection(Configuration) API -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19289) CommonFSUtils$StreamLacksCapabilityException: hflush when running test against hadoop3 beta1
[
https://issues.apache.org/jira/browse/HBASE-19289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258446#comment-16258446
]
Steve Loughran commented on HBASE-19289:
Closed HADOOP-15051 as wontfix. LocalFS output streams don't declare their
support for hflush/sync for the following reason, as covered in HADOOP-13327
(oustanding, reviews welcome)
h3. Output streams which do not implement the flush/persitence semantics of
hflush/hsync MUST NOT declare that their streams have that capability.
LocalFileSystem is a subclass of ChecksumFileSystem; ChecksumFileSystem output
streams don't implement hflush/hsync, therefore it's the correct behaviour in
the Hadoop code.
If HBase requires the methods for the correct persistence of its data, then it
cannot safely use localFS as destination of its output. It's check is therefore
also the correct behavior
In which case, "expressly tell folks not to run HBase on top of
LocalFileSystem," is the correct action on your part. People must not be using
the local FS as a direct destination of HDFS output.
> CommonFSUtils$StreamLacksCapabilityException: hflush when running test
> against hadoop3 beta1
>
>
> Key: HBASE-19289
> URL: https://issues.apache.org/jira/browse/HBASE-19289
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
> Attachments: 19289.v1.txt
>
>
> As of commit d8fb10c8329b19223c91d3cda6ef149382ad4ea0 , I encountered the
> following exception when running unit test against hadoop3 beta1:
> {code}
> testRefreshStoreFiles(org.apache.hadoop.hbase.regionserver.TestHStore) Time
> elapsed: 0.061 sec <<< ERROR!
> java.io.IOException: cannot get log writer
> at
> org.apache.hadoop.hbase.regionserver.TestHStore.initHRegion(TestHStore.java:215)
> at
> org.apache.hadoop.hbase.regionserver.TestHStore.init(TestHStore.java:220)
> at
> org.apache.hadoop.hbase.regionserver.TestHStore.init(TestHStore.java:195)
> at
> org.apache.hadoop.hbase.regionserver.TestHStore.init(TestHStore.java:190)
> at
> org.apache.hadoop.hbase.regionserver.TestHStore.init(TestHStore.java:185)
> at
> org.apache.hadoop.hbase.regionserver.TestHStore.init(TestHStore.java:179)
> at
> org.apache.hadoop.hbase.regionserver.TestHStore.init(TestHStore.java:173)
> at
> org.apache.hadoop.hbase.regionserver.TestHStore.testRefreshStoreFiles(TestHStore.java:962)
> Caused by:
> org.apache.hadoop.hbase.util.CommonFSUtils$StreamLacksCapabilityException:
> hflush
> at
> org.apache.hadoop.hbase.regionserver.TestHStore.initHRegion(TestHStore.java:215)
> at
> org.apache.hadoop.hbase.regionserver.TestHStore.init(TestHStore.java:220)
> at
> org.apache.hadoop.hbase.regionserver.TestHStore.init(TestHStore.java:195)
> at
> org.apache.hadoop.hbase.regionserver.TestHStore.init(TestHStore.java:190)
> at
> org.apache.hadoop.hbase.regionserver.TestHStore.init(TestHStore.java:185)
> at
> org.apache.hadoop.hbase.regionserver.TestHStore.init(TestHStore.java:179)
> at
> org.apache.hadoop.hbase.regionserver.TestHStore.init(TestHStore.java:173)
> at
> org.apache.hadoop.hbase.regionserver.TestHStore.testRefreshStoreFiles(TestHStore.java:962)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19288) Intermittent test failure in TestHStore.testRunDoubleMemStoreCompactors
[
https://issues.apache.org/jira/browse/HBASE-19288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258430#comment-16258430
]
Eshcar Hillel commented on HBASE-19288:
---
I agree setting the in-memory flush factor to 0.25 in this test should fix the
problem.
With flush size 500, when the factor is 0.25 then in-memory flush is invoked at
125, with the new default it is invoked at 10 and hence the first assertion
failure.
Setting the counter to 0 at the beginning of the test avoids the assertions
upon retries.
+1
> Intermittent test failure in TestHStore.testRunDoubleMemStoreCompactors
> ---
>
> Key: HBASE-19288
> URL: https://issues.apache.org/jira/browse/HBASE-19288
> Project: HBase
> Issue Type: Test
> Components: test
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0-beta-1
>
> Attachments: 19288.v1.txt, 19288.v2.txt,
> testRunDoubleMemStoreCompactors.out
>
>
> Here was one of the test failures:
> https://builds.apache.org/job/PreCommit-HBASE-Build/9812/testReport/junit/org.apache.hadoop.hbase.regionserver/TestHStore/testRunDoubleMemStoreCompactors/
>
> {code}
> [ERROR]
> org.apache.hadoop.hbase.regionserver.TestHStore.testRunDoubleMemStoreCompactors(org.apache.hadoop.hbase.regionserver.TestHStore)
> [ERROR] Run 1: TestHStore.testRunDoubleMemStoreCompactors:1500 expected:<2>
> but was:<3>
> [ERROR] Run 2: TestHStore.testRunDoubleMemStoreCompactors:1481 expected:<1>
> but was:<4>
> [ERROR] Run 3: TestHStore.testRunDoubleMemStoreCompactors:1481 expected:<1>
> but was:<5>
> {code}
> From the counts for second and third runs, we know that RUNNER_COUNT was not
> cleared in between the reruns, leading to failure at the 1st assertion.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19295) The Configuration returned by CPEnv should be read-only.
[ https://issues.apache.org/jira/browse/HBASE-19295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258429#comment-16258429 ] Anoop Sam John commented on HBASE-19295: Moved as a sub task under HBASE-18169 (cp cleanup issue) > The Configuration returned by CPEnv should be read-only. > > > Key: HBASE-19295 > URL: https://issues.apache.org/jira/browse/HBASE-19295 > Project: HBase > Issue Type: Sub-task >Reporter: stack > > The Configuration a CP gets when it does a getConfiguration on the > environment is that of the RegionServer. The CP should not be able to modify > this config. We should throw exception if they try to write us. > Ditto w/ the Connection they can get from the env. They should not be able to > close it at min. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19295) The Configuration returned by CPEnv should be read-only.
[ https://issues.apache.org/jira/browse/HBASE-19295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-19295: --- Issue Type: Sub-task (was: Task) Parent: HBASE-18169 > The Configuration returned by CPEnv should be read-only. > > > Key: HBASE-19295 > URL: https://issues.apache.org/jira/browse/HBASE-19295 > Project: HBase > Issue Type: Sub-task >Reporter: stack > > The Configuration a CP gets when it does a getConfiguration on the > environment is that of the RegionServer. The CP should not be able to modify > this config. We should throw exception if they try to write us. > Ditto w/ the Connection they can get from the env. They should not be able to > close it at min. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18359) CoprocessorHConnection#getConnectionForEnvironment should read config from CoprocessorEnvironment
[
https://issues.apache.org/jira/browse/HBASE-18359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258425#comment-16258425
]
Anoop Sam John commented on HBASE-18359:
Yes Stack as Samarth Jain said.
The connection we return from CPEnv.getConnection() is a single connection
created for this RS. This will have short circuit provision. But it will use
all config options as in hbase-site.xml at RS side. Phoenix need
customizations on the configs and connection they use within CPs to have these
custom configs. The new API also should have short circuit ways. But create a
new connection using the config been passed. Will raise an issue and provide a
patch. This issue as such been created for 2.0 . Can we close this as 'Wont
fix'?
> CoprocessorHConnection#getConnectionForEnvironment should read config from
> CoprocessorEnvironment
> -
>
> Key: HBASE-18359
> URL: https://issues.apache.org/jira/browse/HBASE-18359
> Project: HBase
> Issue Type: Bug
>Reporter: Samarth Jain
> Fix For: 2.0.0
>
>
> It seems like the method getConnectionForEnvironment isn't doing the right
> thing when it is creating a CoprocessorHConnection by reading the config from
> HRegionServer and not from the env passed in.
> If coprocessors want to use a CoprocessorHConnection with some custom config
> settings, then they have no option but to configure it in the hbase-site.xml
> of the region servers. This isn't ideal as a lot of times these "global"
> level configs can have side effects. See PHOENIX-3974 as an example where
> configuring ServerRpcControllerFactory (a Phoenix implementation of
> RpcControllerFactory) could result in deadlocks. Or PHOENIX-3983 where
> presence of this global config causes our index rebuild code to incorrectly
> use handlers it shouldn't.
> If the CoprocessorHConnection created through getConnectionForEnvironment API
> used the CoprocessorEnvironment config, then it would allow co-processors to
> pass in their own config without needing to configure them in hbase-site.xml.
> The change would be simple. Basically change the below
> {code}
> if (services instanceof HRegionServer) {
> return new CoprocessorHConnection((HRegionServer) services);
> }
> {code}
> to
> {code}
> if (services instanceof HRegionServer) {
> return new CoprocessorHConnection(env.getConfiguration(),
> (HRegionServer) services);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-17204) Make Bucket Cache off heap cache default ON
[ https://issues.apache.org/jira/browse/HBASE-17204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-17204: --- Summary: Make Bucket Cache off heap cache default ON (was: Make L2 off heap cache default ON) > Make Bucket Cache off heap cache default ON > --- > > Key: HBASE-17204 > URL: https://issues.apache.org/jira/browse/HBASE-17204 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Anoop Sam John >Assignee: Anoop Sam John >Priority: Blocker > Fix For: 2.0.0-beta-1 > > > L2 cache can be used for data blocks. By default it is off now. After > HBASE-11425 work, L2 off heap cache can equally perform with L1 on heap > cache. On heavy loaded workload, this can even out perform L1 cache. Pls see > recently published report by Alibaba. Also this work was backported by > Rocketfuel and similar perf improvement report from them too. > Let us turn L2 off heap cache ON. As it is off heap, we can have much larger > sized L2 BC. What should be the default size? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-17204) Make Bucket Cache off heap cache default ON
[ https://issues.apache.org/jira/browse/HBASE-17204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258420#comment-16258420 ] Anoop Sam John commented on HBASE-17204: We fixed critical issues around BucketCache including the fragmentation one. Still the open one is we maintain at least one free bucket for every size. This gives some wastage of space when the bucket sizes are not properly tuned. That sub jira is still open ! May be we should just allow all the buckets of a size to be converted to other size as needed? No reserve of one block at least. U asked this Q some where. On the route to make off heap BC def ON , we should 1. Remove the L1, L2 concept fully. By default yes this is the way it works now also. When BC is ON, by default we will keep data blocks in BC only and index blocks in LRU cache. We have a config to make the BC a strict L2 cache (By L2 as a victim handler for L1) 2. Dont keep BC as a victim handler for LRU cache at all. Right now this is done when combined mode is true/false. Lets keep it simple. Index blocks will go to LRU and data blocks to BC. When LRU is not enough, we will see evictions there. 3. Reduce the heap % for LRU. Right now it degfaults to 40%. But when it is going to keep index blocks alone, lets make the default to be 10%? 4. Off heap mode BC is ON by def. We need a def off heap size. Remember the off heap size is not given as a % of some thing. It is exact value. May be we should give a def value of 8 GB? (or much lesser or greater?) > Make Bucket Cache off heap cache default ON > --- > > Key: HBASE-17204 > URL: https://issues.apache.org/jira/browse/HBASE-17204 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Anoop Sam John >Assignee: Anoop Sam John >Priority: Blocker > Fix For: 2.0.0-beta-1 > > > L2 cache can be used for data blocks. By default it is off now. After > HBASE-11425 work, L2 off heap cache can equally perform with L1 on heap > cache. On heavy loaded workload, this can even out perform L1 cache. Pls see > recently published report by Alibaba. Also this work was backported by > Rocketfuel and similar perf improvement report from them too. > Let us turn L2 off heap cache ON. As it is off heap, we can have much larger > sized L2 BC. What should be the default size? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19299) Assert only one Connection is constructed when calculating splits in a MultiTableInputFormat
[ https://issues.apache.org/jira/browse/HBASE-19299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258405#comment-16258405 ] Duo Zhang commented on HBASE-19299: --- Why master only? > Assert only one Connection is constructed when calculating splits in a > MultiTableInputFormat > > > Key: HBASE-19299 > URL: https://issues.apache.org/jira/browse/HBASE-19299 > Project: HBase > Issue Type: Test > Components: test >Reporter: stack >Assignee: stack >Priority: Minor > Fix For: 3.0.0 > > Attachments: > 0002-HBASE-19299-Assert-only-one-Connection-is-constructe.patch > > > Small test that verifies only one Connection made when calculating splits. We > used to put up one per Table until recently and before that, a Connection per > table per split which put a heavy load on hbase;meta. Here is a test to > ensure we don't go back to the bad-old-days. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
