[jira] [Commented] (HBASE-14490) [RpcServer] reuse request read buffer
[
https://issues.apache.org/jira/browse/HBASE-14490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14998701#comment-14998701
]
Zephyr Guo commented on HBASE-14490:
[~ikeda]
{quote}
you must have used one or two connection(s) from the client.
{quote}
I use YCSB with 25 threads.Because read request is small usually, do once
{{readAndProcess()}} can get full data from socket. Then Reader will put buffer
back immediately. Reader is single thread that maintain many Connections.If
most of the Connection do read request, it may need a buffer only.
{quote}
BoundedByteBufferPool uses locks inside but this benchmark might not expose its
disadvantage.
{quote}
Both {{ConcurrentLinkedDeque}} and {{ReentrantLock}} base on CAS.They will lead
to waiting in fact.
https://en.wikipedia.org/wiki/Non-blocking_algorithm
{quote}
I just noticed that your patch is not equivalent to the original code when
using SASL. Zephyr Guo have you confirmed that is OK?
{quote}
Yes. What is wrong i missed?
> [RpcServer] reuse request read buffer
> -
>
> Key: HBASE-14490
> URL: https://issues.apache.org/jira/browse/HBASE-14490
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC
>Affects Versions: 2.0.0, 1.0.2
>Reporter: Zephyr Guo
>Assignee: Zephyr Guo
> Labels: performance
> Fix For: 2.0.0, 1.0.2
>
> Attachments: ByteBufferPool.java, HBASE-14490-v1.patch,
> HBASE-14490-v10.patch, HBASE-14490-v11.patch, HBASE-14490-v12.patch,
> HBASE-14490-v2.patch, HBASE-14490-v3.patch, HBASE-14490-v4.patch,
> HBASE-14490-v5.patch, HBASE-14490-v6.patch, HBASE-14490-v7.patch,
> HBASE-14490-v8.patch, HBASE-14490-v9.patch, test-v12-patch
>
>
> Reusing buffer to read request.It's not necessary to every request free
> buffer.The idea of optimization is to reduce the times that allocate
> ByteBuffer.
> *Modification*
> 1. {{saslReadAndProcess}} ,{{processOneRpc}}, {{processUnwrappedData}},
> {{processConnectionHeader}} accept a ByteBuffer instead of byte[].They can
> move {{ByteBuffer.position}} correctly when we have read the data.
> 2. {{processUnwrappedData}} no longer use any extra memory.
> 3. Maintaining a buffer pool in each {{Connection}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14750) HBaseConfiguration.create() doesn't load properties
[
https://issues.apache.org/jira/browse/HBASE-14750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14998739#comment-14998739
]
Hadoop QA commented on HBASE-14750:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12771538/HBASE-14750-20151110-1323.patch
against master branch at commit 112900d0425a8157b89041f0e353ebf5cc259c69.
ATTACHMENT ID: 12771538
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1 release
audit warnings (more than the master's current 0 warnings).
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16477//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16477//artifact/patchprocess/patchReleaseAuditWarnings.txt
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16477//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16477//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16477//console
This message is automatically generated.
> HBaseConfiguration.create() doesn't load properties
> ---
>
> Key: HBASE-14750
> URL: https://issues.apache.org/jira/browse/HBASE-14750
> Project: HBase
> Issue Type: Bug
>Reporter: Niels Basjes
>Assignee: Niels Basjes
> Attachments: HBASE-14750-20151110-1323.patch
>
>
> While writing an application that uses HBase I ran into the problem that
> although I have setup the hbase-site.xml in the {{HBASE_CONF_DIR}}; the
> settings in this file are not picked up in my application.
> In several places I find instructions to include things like the
> {{hbase.zookeeper.quorum}} in a properties file and the explicitly set these
> value from within the application (I have actually done this in the past
> quite a few times).
> Although this works I expect the system to automatically pickup the settings
> I have installed already which are picked up by tools like the hbase shell
> and pig.
> So I ended up writing this helper method:
> {code:java}
> public static Configuration createConfiguration() {
> String hbaseConfDir = System.getenv("HBASE_CONF_DIR");
> if (hbaseConfDir == null) {
> hbaseConfDir = "/etc/hbase/conf";
> }
> Configuration conf = HBaseConfiguration.create();
> conf.addResource(new Path(hbaseConfDir + "/hbase-site.xml"));
> return conf;
> }
> {code}
> I expect HBaseConfiguration.create() to give me a working config in an
> environment where everything for all the other HBase clients has already been
> setup correctly.
> My proposal is to change the HBaseConfiguration.create() to effectively
> include what my helper method does.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14766) WALEntryFilter's filter implement, cell.getFamily() needs to be replaced with the new low-cost implementation.
[ https://issues.apache.org/jira/browse/HBASE-14766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14998724#comment-14998724 ] Ted Yu commented on HBASE-14766: Mind re-attaching patch to get a clean QA run ? > WALEntryFilter's filter implement, cell.getFamily() needs to be replaced with > the new low-cost implementation. > -- > > Key: HBASE-14766 > URL: https://issues.apache.org/jira/browse/HBASE-14766 > Project: HBase > Issue Type: Improvement >Reporter: huaxiang sun >Assignee: huaxiang sun > Attachments: HBASE-14766-v001.patch, HBASE-14766-v002.patch, > HBASE-14766-v003.patch > > > Cell's getFamily() gets an array copy of the cell's family, while in the > filter function, it just needs to peek into the family and do a compare. > Replace > Bytes.toString(cell.getFamily()) > with > Bytes.toString(cell.getFamilyArray(), cell.getFamilyOffset(), > cell.getFamilyLength()) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14780) Integration Tests that run with ChaosMonkey need to specify CFs
[ https://issues.apache.org/jira/browse/HBASE-14780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-14780: --- Attachment: hbase-14780.patch > Integration Tests that run with ChaosMonkey need to specify CFs > --- > > Key: HBASE-14780 > URL: https://issues.apache.org/jira/browse/HBASE-14780 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Attachments: hbase-14780.patch, hbase-14780.patch > > > Been running some IT tests and found that some failed because getcfs was null > and didn't protecte cfs that were assumed to go unmolested. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14780) Integration Tests that run with ChaosMonkey need to specify CFs
[ https://issues.apache.org/jira/browse/HBASE-14780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14998751#comment-14998751 ] Jonathan Hsieh commented on HBASE-14780: Hm.. I don't actually touch anything related to the zombies with this patch. Aside from that it looks like a clean run. Let me try again. > Integration Tests that run with ChaosMonkey need to specify CFs > --- > > Key: HBASE-14780 > URL: https://issues.apache.org/jira/browse/HBASE-14780 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Attachments: hbase-14780.patch > > > Been running some IT tests and found that some failed because getcfs was null > and didn't protecte cfs that were assumed to go unmolested. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14498) Master stuck in infinite loop when all Zookeeper servers are unreachable.
[
https://issues.apache.org/jira/browse/HBASE-14498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14998842#comment-14998842
]
Hadoop QA commented on HBASE-14498:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12771547/HBASE-14498-V3.patch
against master branch at commit 112900d0425a8157b89041f0e353ebf5cc259c69.
ATTACHMENT ID: 12771547
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1 release
audit warnings (more than the master's current 0 warnings).
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16478//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16478//artifact/patchprocess/patchReleaseAuditWarnings.txt
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16478//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16478//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16478//console
This message is automatically generated.
> Master stuck in infinite loop when all Zookeeper servers are unreachable.
> -
>
> Key: HBASE-14498
> URL: https://issues.apache.org/jira/browse/HBASE-14498
> Project: HBase
> Issue Type: Bug
> Components: master
>Reporter: Y. SREENIVASULU REDDY
>Assignee: Pankaj Kumar
>Priority: Blocker
> Attachments: HBASE-14498-V2.patch, HBASE-14498-V3.patch,
> HBASE-14498.patch
>
>
> We met a weird scenario in our production environment.
> In a HA cluster,
> > Active Master (HM1) is not able to connect to any Zookeeper server (due to
> > N/w breakdown on master machine network with Zookeeper servers).
> {code}
> 2015-09-26 15:24:47,508 INFO
> [HM1-Host:16000.activeMasterManager-SendThread(ZK-Host:2181)]
> zookeeper.ClientCnxn: Client session timed out, have not heard from server in
> 33463ms for sessionid 0x104576b8dda0002, closing socket connection and
> attempting reconnect
> 2015-09-26 15:24:47,877 INFO
> [HM1-Host:16000.activeMasterManager-SendThread(ZK-Host1:2181)]
> client.FourLetterWordMain: connecting to ZK-Host1 2181
> 2015-09-26 15:24:48,236 INFO [main-SendThread(ZK-Host1:2181)]
> client.FourLetterWordMain: connecting to ZK-Host1 2181
> 2015-09-26 15:24:49,879 WARN
> [HM1-Host:16000.activeMasterManager-SendThread(ZK-Host1:2181)]
> zookeeper.ClientCnxn: Can not get the principle name from server ZK-Host1
> 2015-09-26 15:24:49,879 INFO
> [HM1-Host:16000.activeMasterManager-SendThread(ZK-Host1:2181)]
> zookeeper.ClientCnxn: Opening socket connection to server
> ZK-Host1/ZK-IP1:2181. Will not attempt to authenticate using SASL (unknown
> error)
> 2015-09-26 15:24:50,238 WARN [main-SendThread(ZK-Host1:2181)]
> zookeeper.ClientCnxn: Can not get the principle name from server ZK-Host1
> 2015-09-26 15:24:50,238 INFO [main-SendThread(ZK-Host1:2181)]
> zookeeper.ClientCnxn: Opening socket connection to server
> ZK-Host1/ZK-Host1:2181. Will not attempt to authenticate using SASL (unknown
> error)
> 2015-09-26 15:25:17,470 INFO [main-SendThread(ZK-Host1:2181)]
> zookeeper.ClientCnxn: Client session timed out, have not heard from server in
> 30023ms for sessionid 0x2045762cc710006, closing socket connection and
> attempting reconnect
> 2015-09-26 15:25:17,571 WARN [master/HM1-Host/HM1-IP:16000]
> zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper,
> quorum=ZK-Host:2181,ZK-Host1:2181,ZK-Host2:2181,
>
[jira] [Commented] (HBASE-14498) Master stuck in infinite loop when all Zookeeper servers are unreachable.
[
https://issues.apache.org/jira/browse/HBASE-14498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14998894#comment-14998894
]
Ted Yu commented on HBASE-14498:
Last comment:
{code}
641 Thread t = new Thread() {
{code}
The thread should be daemon thread, right ?
> Master stuck in infinite loop when all Zookeeper servers are unreachable.
> -
>
> Key: HBASE-14498
> URL: https://issues.apache.org/jira/browse/HBASE-14498
> Project: HBase
> Issue Type: Bug
> Components: master
>Reporter: Y. SREENIVASULU REDDY
>Assignee: Pankaj Kumar
>Priority: Blocker
> Attachments: HBASE-14498-V2.patch, HBASE-14498-V3.patch,
> HBASE-14498.patch
>
>
> We met a weird scenario in our production environment.
> In a HA cluster,
> > Active Master (HM1) is not able to connect to any Zookeeper server (due to
> > N/w breakdown on master machine network with Zookeeper servers).
> {code}
> 2015-09-26 15:24:47,508 INFO
> [HM1-Host:16000.activeMasterManager-SendThread(ZK-Host:2181)]
> zookeeper.ClientCnxn: Client session timed out, have not heard from server in
> 33463ms for sessionid 0x104576b8dda0002, closing socket connection and
> attempting reconnect
> 2015-09-26 15:24:47,877 INFO
> [HM1-Host:16000.activeMasterManager-SendThread(ZK-Host1:2181)]
> client.FourLetterWordMain: connecting to ZK-Host1 2181
> 2015-09-26 15:24:48,236 INFO [main-SendThread(ZK-Host1:2181)]
> client.FourLetterWordMain: connecting to ZK-Host1 2181
> 2015-09-26 15:24:49,879 WARN
> [HM1-Host:16000.activeMasterManager-SendThread(ZK-Host1:2181)]
> zookeeper.ClientCnxn: Can not get the principle name from server ZK-Host1
> 2015-09-26 15:24:49,879 INFO
> [HM1-Host:16000.activeMasterManager-SendThread(ZK-Host1:2181)]
> zookeeper.ClientCnxn: Opening socket connection to server
> ZK-Host1/ZK-IP1:2181. Will not attempt to authenticate using SASL (unknown
> error)
> 2015-09-26 15:24:50,238 WARN [main-SendThread(ZK-Host1:2181)]
> zookeeper.ClientCnxn: Can not get the principle name from server ZK-Host1
> 2015-09-26 15:24:50,238 INFO [main-SendThread(ZK-Host1:2181)]
> zookeeper.ClientCnxn: Opening socket connection to server
> ZK-Host1/ZK-Host1:2181. Will not attempt to authenticate using SASL (unknown
> error)
> 2015-09-26 15:25:17,470 INFO [main-SendThread(ZK-Host1:2181)]
> zookeeper.ClientCnxn: Client session timed out, have not heard from server in
> 30023ms for sessionid 0x2045762cc710006, closing socket connection and
> attempting reconnect
> 2015-09-26 15:25:17,571 WARN [master/HM1-Host/HM1-IP:16000]
> zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper,
> quorum=ZK-Host:2181,ZK-Host1:2181,ZK-Host2:2181,
> exception=org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase/master
> 2015-09-26 15:25:17,872 INFO [main-SendThread(ZK-Host:2181)]
> client.FourLetterWordMain: connecting to ZK-Host 2181
> 2015-09-26 15:25:19,874 WARN [main-SendThread(ZK-Host:2181)]
> zookeeper.ClientCnxn: Can not get the principle name from server ZK-Host
> 2015-09-26 15:25:19,874 INFO [main-SendThread(ZK-Host:2181)]
> zookeeper.ClientCnxn: Opening socket connection to server ZK-Host/ZK-IP:2181.
> Will not attempt to authenticate using SASL (unknown error)
> {code}
> > Since HM1 was not able to connect to any ZK, so session timeout didnt
> > happen at Zookeeper server side and HM1 didnt abort.
> > On Zookeeper session timeout standby master (HM2) registered himself as an
> > active master.
> > HM2 is keep on waiting for region server to report him as part of active
> > master intialization.
> {noformat}
> 2015-09-26 15:24:44,928 | INFO | HM2-Host:21300.activeMasterManager | Waiting
> for region servers count to settle; currently checked in 0, slept for 0 ms,
> expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval
> of 1500 ms. |
> org.apache.hadoop.hbase.master.ServerManager.waitForRegionServers(ServerManager.java:1011)
> ---
> ---
> 2015-09-26 15:32:50,841 | INFO | HM2-Host:21300.activeMasterManager | Waiting
> for region servers count to settle; currently checked in 0, slept for 483913
> ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms,
> interval of 1500 ms. |
> org.apache.hadoop.hbase.master.ServerManager.waitForRegionServers(ServerManager.java:1011)
> {noformat}
> > At other end, region servers are reporting to HM1 on 3 sec interval. Here
> > region server retrieve master location from zookeeper only when they
> > couldn't connect to Master (ServiceException).
> Region Server will not report HM2 as per current design until unless HM1
> abort,so HM2 will exit(InitializationMonitor) and again wait for region
> servers in loop.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-14791) [0.98] CopyTable is extremely slow when moving delete markers
Lars Hofhansl created HBASE-14791: - Summary: [0.98] CopyTable is extremely slow when moving delete markers Key: HBASE-14791 URL: https://issues.apache.org/jira/browse/HBASE-14791 Project: HBase Issue Type: Bug Affects Versions: 0.98.16 Reporter: Lars Hofhansl We found that some of our copy table job run for many hours, even when there isn't that much data to copy. [~vik.karma] did his magic and found that the issue with copying delete markers (we use raw mode to also move deletes across). Looking at the code in 0.98 it's immediately obvious that deletes (unlike puts) are not batched and hence sent to the other side one by one, cause a network RTT for each delete marker. Looks like in trunk it's doing the right thing (using BufferedMutators for all mutations in TableOutputFormat). So likely only a 0.98 (and 1.0, 1.1, 1.2?) issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14780) Integration Tests that run with ChaosMonkey need to specify CFs
[
https://issues.apache.org/jira/browse/HBASE-14780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999059#comment-14999059
]
Hadoop QA commented on HBASE-14780:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12771567/hbase-14780.patch
against master branch at commit 112900d0425a8157b89041f0e353ebf5cc259c69.
ATTACHMENT ID: 12771567
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 15 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1 release
audit warnings (more than the master's current 0 warnings).
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16479//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16479//artifact/patchprocess/patchReleaseAuditWarnings.txt
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16479//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16479//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16479//console
This message is automatically generated.
> Integration Tests that run with ChaosMonkey need to specify CFs
> ---
>
> Key: HBASE-14780
> URL: https://issues.apache.org/jira/browse/HBASE-14780
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Attachments: hbase-14780.patch, hbase-14780.patch
>
>
> Been running some IT tests and found that some failed because getcfs was null
> and didn't protecte cfs that were assumed to go unmolested.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12790) Support fairness across parallelized scans
[ https://issues.apache.org/jira/browse/HBASE-12790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999067#comment-14999067 ] stack commented on HBASE-12790: --- bq. Glad to see you're working on that over at Cloudera. Hopefully you're testing with Phoenix too. You did not read it. It is not "cloudera" work. It is apache hbase work. See listed JIRAs. It is a summary of the state of scheduling art in apache hbase as of a while ago. bq. I don't think having an extra optional attribute on an operation adds "a bunch of new complexity". That's fine if we disagree. Andrews' considered response 'On complexity' plainly left no mark and you can't have reviewed the attached patch and comments. Only a superficial engagement with this issue and what all is involved could result in a characterization of what is going on here as just "having an extra optional attribute" (or that the cited, pertinent blog post is 'cloudera' work). bq. Andrew Purtell made the point that if you're round robining on reads you should be consistent and do it on writes too - I think this is a fair point. Our immediate need is on the read side - I'll share our data when the analysis is complete... Our requirement is simple: the latency of point lookups and small-ish scans shouldn't be impacted by other workloads on the cluster. What ever implementation you come up with is fine by us. Your requirement changes every time you comment and you do not know what you are asking for. Let me try and write something up and situate it relative to work already done. > Support fairness across parallelized scans > -- > > Key: HBASE-12790 > URL: https://issues.apache.org/jira/browse/HBASE-12790 > Project: HBase > Issue Type: New Feature >Reporter: James Taylor >Assignee: ramkrishna.s.vasudevan > Labels: Phoenix > Attachments: AbstractRoundRobinQueue.java, HBASE-12790.patch, > HBASE-12790_1.patch, HBASE-12790_5.patch, HBASE-12790_callwrapper.patch, > HBASE-12790_trunk_1.patch, PHOENIX_4.5.3-HBase-0.98-2317-SNAPSHOT.zip > > > Some HBase clients parallelize the execution of a scan to reduce latency in > getting back results. This can lead to starvation with a loaded cluster and > interleaved scans, since the RPC queue will be ordered and processed on a > FIFO basis. For example, if there are two clients, A & B that submit largish > scans at the same time. Say each scan is broken down into 100 scans by the > client (broken down into equal depth chunks along the row key), and the 100 > scans of client A are queued first, followed immediately by the 100 scans of > client B. In this case, client B will be starved out of getting any results > back until the scans for client A complete. > One solution to this is to use the attached AbstractRoundRobinQueue instead > of the standard FIFO queue. The queue to be used could be (maybe it already > is) configurable based on a new config parameter. Using this queue would > require the client to have the same identifier for all of the 100 parallel > scans that represent a single logical scan from the clients point of view. > With this information, the round robin queue would pick off a task from the > queue in a round robin fashion (instead of a strictly FIFO manner) to prevent > starvation over interleaved parallelized scans. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-14791) [0.98] CopyTable is extremely slow when moving delete markers
[ https://issues.apache.org/jira/browse/HBASE-14791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Araujo reassigned HBASE-14791: --- Assignee: Alex Araujo [~lhofhansl] I can take this > [0.98] CopyTable is extremely slow when moving delete markers > - > > Key: HBASE-14791 > URL: https://issues.apache.org/jira/browse/HBASE-14791 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.16 >Reporter: Lars Hofhansl >Assignee: Alex Araujo > > We found that some of our copy table job run for many hours, even when there > isn't that much data to copy. > [~vik.karma] did his magic and found that the issue with copying delete > markers (we use raw mode to also move deletes across). > Looking at the code in 0.98 it's immediately obvious that deletes (unlike > puts) are not batched and hence sent to the other side one by one, cause a > network RTT for each delete marker. > Looks like in trunk it's doing the right thing (using BufferedMutators for > all mutations in TableOutputFormat). So likely only a 0.98 (and 1.0, 1.1, > 1.2?) issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-3871) Speedup LoadIncrementalHFiles by parallelizing HFile splitting
[ https://issues.apache.org/jira/browse/HBASE-3871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Francke updated HBASE-3871: Fix Version/s: 0.92.0 > Speedup LoadIncrementalHFiles by parallelizing HFile splitting > -- > > Key: HBASE-3871 > URL: https://issues.apache.org/jira/browse/HBASE-3871 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Affects Versions: 0.90.2 >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 0.92.0 > > Attachments: 3871.patch > > > From Adam w.r.t. HFile splitting: > There's actually a good number of messages of that type (HFile no longer fits > inside a single region), unfortunately I didn't take a timestamp on just when > I was running with the patched jars vs the regular ones, however from the > logs I can say that this is occurring fairly regularly on this system. The > cluster I tested this on is our backup cluster, the mapreduce jobs on our > production cluster output HFiles which are copied to the backup and then > loaded into HBase on both. Since the regions may be somewhat different on > the backup cluster I would expect it to have to split somewhat regularly. > This JIRA complements HBASE-3721 by parallelizing HFile splitting which is > done in the main thread. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HBASE-3879) Add way to control the split position of region
[ https://issues.apache.org/jira/browse/HBASE-3879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Francke reopened HBASE-3879: - > Add way to control the split position of region > --- > > Key: HBASE-3879 > URL: https://issues.apache.org/jira/browse/HBASE-3879 > Project: HBase > Issue Type: Improvement > Components: regionserver >Reporter: Ophir Cohen >Priority: Minor > > Currently the split point picked to be the mid key of the HFile. > From the code comments: > "Midkey for this file. We work with block boundaries only so > * returned midkey is an approximation only." > It could be good to have a way to override the default option and use other > method. > Possible implementation: > Create interface (similar to Hadoop's partitioner) that gets StoreFile / > StoreFile.Reader and return the splitting point. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HBASE-3933) Hmaster throw NullPointerException
[
https://issues.apache.org/jira/browse/HBASE-3933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Francke reopened HBASE-3933:
-
> Hmaster throw NullPointerException
> --
>
> Key: HBASE-3933
> URL: https://issues.apache.org/jira/browse/HBASE-3933
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 0.90.6
>Reporter: gaojinchao
>Assignee: Eugene Koontz
> Attachments: HBASE-3933.patch, HBASE-3933.patch, Hmastersetup0.90
>
>
> NullPointerException while hmaster starting.
> {code}
> java.lang.NullPointerException
> at java.util.TreeMap.getEntry(TreeMap.java:324)
> at java.util.TreeMap.get(TreeMap.java:255)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.addToServers(AssignmentManager.java:1512)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.regionOnline(AssignmentManager.java:606)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.processFailover(AssignmentManager.java:214)
> at
> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:402)
> at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:283)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14780) Integration Tests that run with ChaosMonkey need to specify CFs
[ https://issues.apache.org/jira/browse/HBASE-14780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999179#comment-14999179 ] Enis Soztutar commented on HBASE-14780: --- +1. > Integration Tests that run with ChaosMonkey need to specify CFs > --- > > Key: HBASE-14780 > URL: https://issues.apache.org/jira/browse/HBASE-14780 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Attachments: hbase-14780.patch, hbase-14780.patch > > > Been running some IT tests and found that some failed because getcfs was null > and didn't protecte cfs that were assumed to go unmolested. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14611) Backport HBASE-14473 (Compute region locality in parallel) to 0.98
[ https://issues.apache.org/jira/browse/HBASE-14611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14998920#comment-14998920 ] Hudson commented on HBASE-14611: FAILURE: Integrated in HBase-0.98-matrix #258 (See [https://builds.apache.org/job/HBase-0.98-matrix/258/]) HBASE-14611 Backport HBASE-14473 (Compute region locality in parallel) (apurtell: rev 70b9015a0ec8de5ed9f9d9302b67b8e79bc4c558) * hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/RegionLocationFinder.java > Backport HBASE-14473 (Compute region locality in parallel) to 0.98 > -- > > Key: HBASE-14611 > URL: https://issues.apache.org/jira/browse/HBASE-14611 > Project: HBase > Issue Type: Task >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 0.98.16 > > Attachments: HBASE-14611-0.98.patch > > > [~eclark] contributed a nice change on HBASE-14473 that scales calculation of > locality balance cost to larger clusters. I'd like to bring this back to 0.98 > for folks running it on larger clusters. The changes require a partial > rewrite of RegionLocationFinder hence a backport issue. > The difference between this backport and RegionLocationFinder on later > branches is we preserve the ability to change the expiration time of cache > items with the configuration parameter > "hbase.master.balancer.regionLocationCacheTime". > The difference between RegionLocationFinder in 0.98 before and after this > change is the expiration time of cache entries will be set according to when > they are written into the cache instead of from when they are last accessed, > which seems better to me. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14693) Add client-side metrics for received pushback signals
[ https://issues.apache.org/jira/browse/HBASE-14693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14998918#comment-14998918 ] Hudson commented on HBASE-14693: FAILURE: Integrated in HBase-0.98-matrix #258 (See [https://builds.apache.org/job/HBase-0.98-matrix/258/]) HBASE-14693 Add client-side metrics for received pushback signals (apurtell: rev 70964f48a7fbc03bade66d6f075fd78e6dcd8627) * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestClientPushback.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/MetricsConnection.java > Add client-side metrics for received pushback signals > - > > Key: HBASE-14693 > URL: https://issues.apache.org/jira/browse/HBASE-14693 > Project: HBase > Issue Type: Improvement >Reporter: Andrew Purtell >Assignee: Heng Chen > Fix For: 2.0.0, 1.2.0, 1.3.0, 0.98.16 > > Attachments: HBASE-14693-0.98.patch, HBASE-14693.patch, > HBASE-14693_v1.patch, HBASE-14693_v2.patch, HBASE-14693_v3.patch, > HBASE-14693_v4.patch > > > HBASE-12911 added client side metrics. HBASE-5162 added a mechanism for > sending advisory backpressure signals to clients when the server is heavily > loaded, and HBASE-12702 and subtasks backported this to all active branches. > Add client-side metrics for received pushback signal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-3779) Allow split regions to be placed on different region servers
[ https://issues.apache.org/jira/browse/HBASE-3779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Francke resolved HBASE-3779. - Resolution: Won't Fix > Allow split regions to be placed on different region servers > > > Key: HBASE-3779 > URL: https://issues.apache.org/jira/browse/HBASE-3779 > Project: HBase > Issue Type: Improvement > Components: master >Affects Versions: 0.90.2 >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 3779-v2.patch > > > Currently daughter regions are placed on the same region server where the > parent region was. > Stanislav Barton mentioned the idea that load information should be > considered when placing the daughter regions. > The rationale is that the daughter regions tend to receive more writes. So it > would be beneficial to place at least one daughter region on a different > region server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HBASE-3779) Allow split regions to be placed on different region servers
[ https://issues.apache.org/jira/browse/HBASE-3779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Francke reopened HBASE-3779: - > Allow split regions to be placed on different region servers > > > Key: HBASE-3779 > URL: https://issues.apache.org/jira/browse/HBASE-3779 > Project: HBase > Issue Type: Improvement > Components: master >Affects Versions: 0.90.2 >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 3779-v2.patch > > > Currently daughter regions are placed on the same region server where the > parent region was. > Stanislav Barton mentioned the idea that load information should be > considered when placing the daughter regions. > The rationale is that the daughter regions tend to receive more writes. So it > would be beneficial to place at least one daughter region on a different > region server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14693) Add client-side metrics for received pushback signals
[ https://issues.apache.org/jira/browse/HBASE-14693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14998943#comment-14998943 ] Hudson commented on HBASE-14693: FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #1131 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1131/]) HBASE-14693 Add client-side metrics for received pushback signals (apurtell: rev 70964f48a7fbc03bade66d6f075fd78e6dcd8627) * hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestClientPushback.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/MetricsConnection.java > Add client-side metrics for received pushback signals > - > > Key: HBASE-14693 > URL: https://issues.apache.org/jira/browse/HBASE-14693 > Project: HBase > Issue Type: Improvement >Reporter: Andrew Purtell >Assignee: Heng Chen > Fix For: 2.0.0, 1.2.0, 1.3.0, 0.98.16 > > Attachments: HBASE-14693-0.98.patch, HBASE-14693.patch, > HBASE-14693_v1.patch, HBASE-14693_v2.patch, HBASE-14693_v3.patch, > HBASE-14693_v4.patch > > > HBASE-12911 added client side metrics. HBASE-5162 added a mechanism for > sending advisory backpressure signals to clients when the server is heavily > loaded, and HBASE-12702 and subtasks backported this to all active branches. > Add client-side metrics for received pushback signal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14611) Backport HBASE-14473 (Compute region locality in parallel) to 0.98
[ https://issues.apache.org/jira/browse/HBASE-14611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14998945#comment-14998945 ] Hudson commented on HBASE-14611: FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #1131 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1131/]) HBASE-14611 Backport HBASE-14473 (Compute region locality in parallel) (apurtell: rev 70b9015a0ec8de5ed9f9d9302b67b8e79bc4c558) * hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/RegionLocationFinder.java > Backport HBASE-14473 (Compute region locality in parallel) to 0.98 > -- > > Key: HBASE-14611 > URL: https://issues.apache.org/jira/browse/HBASE-14611 > Project: HBase > Issue Type: Task >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 0.98.16 > > Attachments: HBASE-14611-0.98.patch > > > [~eclark] contributed a nice change on HBASE-14473 that scales calculation of > locality balance cost to larger clusters. I'd like to bring this back to 0.98 > for folks running it on larger clusters. The changes require a partial > rewrite of RegionLocationFinder hence a backport issue. > The difference between this backport and RegionLocationFinder on later > branches is we preserve the ability to change the expiration time of cache > items with the configuration parameter > "hbase.master.balancer.regionLocationCacheTime". > The difference between RegionLocationFinder in 0.98 before and after this > change is the expiration time of cache entries will be set according to when > they are written into the cache instead of from when they are last accessed, > which seems better to me. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14791) [0.98] CopyTable is extremely slow when moving delete markers
[ https://issues.apache.org/jira/browse/HBASE-14791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999064#comment-14999064 ] Lars Hofhansl commented on HBASE-14791: --- Cool! > [0.98] CopyTable is extremely slow when moving delete markers > - > > Key: HBASE-14791 > URL: https://issues.apache.org/jira/browse/HBASE-14791 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.16 >Reporter: Lars Hofhansl >Assignee: Alex Araujo > > We found that some of our copy table job run for many hours, even when there > isn't that much data to copy. > [~vik.karma] did his magic and found that the issue with copying delete > markers (we use raw mode to also move deletes across). > Looking at the code in 0.98 it's immediately obvious that deletes (unlike > puts) are not batched and hence sent to the other side one by one, cause a > network RTT for each delete marker. > Looks like in trunk it's doing the right thing (using BufferedMutators for > all mutations in TableOutputFormat). So likely only a 0.98 (and 1.0, 1.1, > 1.2?) issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14792) Latest docs fail to build when packaging 0.98
[
https://issues.apache.org/jira/browse/HBASE-14792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999088#comment-14999088
]
Nick Dimiduk commented on HBASE-14792:
--
bq. Do I need to make POM updates?
Must be; I had to do some fixup on branch-1 before the 1.1.0 release:
https://github.com/apache/hbase/commit/d6ac86c3b8ad603e4e5fb1e7a1dffc2dcb7fa336
> Latest docs fail to build when packaging 0.98
> -
>
> Key: HBASE-14792
> URL: https://issues.apache.org/jira/browse/HBASE-14792
> Project: HBase
> Issue Type: Bug
>Reporter: Andrew Purtell
>Priority: Minor
> Fix For: 0.98.17
>
>
> For releasing 0.98 we typically copy back the latest docs from master and
> then build.
> When I try generating the latest docs, I get:
> {noformat}
> [INFO]
>
> [INFO] Forking Apache HBase - Assembly 0.98.16-hadoop2
> [INFO]
>
> [INFO]
> [INFO] --- maven-enforcer-plugin:1.0.1:enforce (enforce) @ hbase-assembly ---
> [INFO]
> [INFO] --- buildnumber-maven-plugin:1.3:create-timestamp (default) @
> hbase-assembly ---
> [INFO]
> [INFO] <<< maven-javadoc-plugin:2.9.1:aggregate (report:aggregate) <
> generate-sources @ hbase <<<
> [INFO] configuring report plugin
> org.apache.maven.plugins:maven-checkstyle-plugin:2.13
> [INFO] Parent project loaded from repository: org.apache:apache:pom:12
> [INFO] Relativizing decoration links with respect to project URL:
> http://hbase.apache.org
> [INFO] Rendering site with org.apache.maven.skins:maven-fluido-skin:jar:1.4
> skin.
> [INFO] Skipped "About" report, file "index.html" already exists for the
> English version.
> [INFO] Skipped "Source Xref" report, file "xref/index.html" already exists
> for the English version.
> [INFO] Skipped "Test Source Xref" report, file "xref-test/index.html" already
> exists for the English version.
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-site-plugin:3.3:site (default-site) on project
> hbase: Error during page generation: Could not find the template
> 'META-INF/maven/site.vm: Encountered "source" at META-INF/maven/site.vm[line
> 1162, column 52]
> [ERROR] Was expecting one of:
> [ERROR] "," ...
> [ERROR] ")" ...
> [ERROR] ...
> {noformat}
> Do I need to make POM updates?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-3862) Race conditions in aggregate calculation
[ https://issues.apache.org/jira/browse/HBASE-3862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Francke updated HBASE-3862: Fix Version/s: 0.92.0 > Race conditions in aggregate calculation > > > Key: HBASE-3862 > URL: https://issues.apache.org/jira/browse/HBASE-3862 > Project: HBase > Issue Type: Bug > Components: Coprocessors >Affects Versions: 0.92.0 >Reporter: John Heitmann > Fix For: 0.92.0 > > Attachments: coprocessor_race.patch > > > The AggregationClient requests aggregations from multiple region servers in > parallel. The calculations in the reducer callbacks of the AggregationClient > are not thread safe, and therefore could return an incorrect result due to > simultaneous/interleaved execution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-3877) Determine Proper Defaults for Compaction ThreadPools
[ https://issues.apache.org/jira/browse/HBASE-3877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Francke updated HBASE-3877: Fix Version/s: 0.92.0 > Determine Proper Defaults for Compaction ThreadPools > > > Key: HBASE-3877 > URL: https://issues.apache.org/jira/browse/HBASE-3877 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 0.92.0 >Reporter: Nicolas Spiegelberg >Assignee: Nicolas Spiegelberg >Priority: Trivial > Labels: compaction > Fix For: 0.92.0 > > > With the introduction of HBASE-1476, we now have multithreaded compactions + > 2 different ThreadPools for large and small compactions. However, this is > disabled by default until we can determine a proper default throttle point. > Opening this JIRA to log all discussion on how to select a good default for > this case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14473) Compute region locality in parallel
[ https://issues.apache.org/jira/browse/HBASE-14473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14998919#comment-14998919 ] Hudson commented on HBASE-14473: FAILURE: Integrated in HBase-0.98-matrix #258 (See [https://builds.apache.org/job/HBase-0.98-matrix/258/]) HBASE-14611 Backport HBASE-14473 (Compute region locality in parallel) (apurtell: rev 70b9015a0ec8de5ed9f9d9302b67b8e79bc4c558) * hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/RegionLocationFinder.java > Compute region locality in parallel > --- > > Key: HBASE-14473 > URL: https://issues.apache.org/jira/browse/HBASE-14473 > Project: HBase > Issue Type: Bug > Components: Balancer >Affects Versions: 1.2.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14473-v1.patch, HBASE-14473-v2.patch, > HBASE-14473-v3.patch, HBASE-14473-v4.patch, HBASE-14473.patch > > > Right now on large clusters it's necessary to turn off the locality balance > cost as it takes too long to compute the region locality. This is because > it's computed when need in serial. > We should compute this in parallel before it's needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14792) Latest docs fail to build when packaging 0.98
Andrew Purtell created HBASE-14792:
--
Summary: Latest docs fail to build when packaging 0.98
Key: HBASE-14792
URL: https://issues.apache.org/jira/browse/HBASE-14792
Project: HBase
Issue Type: Bug
Reporter: Andrew Purtell
Priority: Minor
For releasing 0.98 we typically copy back the latest docs from master and then
build.
When I try generating the latest docs, I get:
{noformat}
[INFO]
[INFO] Forking Apache HBase - Assembly 0.98.16-hadoop2
[INFO]
[INFO]
[INFO] --- maven-enforcer-plugin:1.0.1:enforce (enforce) @ hbase-assembly ---
[INFO]
[INFO] --- buildnumber-maven-plugin:1.3:create-timestamp (default) @
hbase-assembly ---
[INFO]
[INFO] <<< maven-javadoc-plugin:2.9.1:aggregate (report:aggregate) <
generate-sources @ hbase <<<
[INFO] configuring report plugin
org.apache.maven.plugins:maven-checkstyle-plugin:2.13
[INFO] Parent project loaded from repository: org.apache:apache:pom:12
[INFO] Relativizing decoration links with respect to project URL:
http://hbase.apache.org
[INFO] Rendering site with org.apache.maven.skins:maven-fluido-skin:jar:1.4
skin.
[INFO] Skipped "About" report, file "index.html" already exists for the English
version.
[INFO] Skipped "Source Xref" report, file "xref/index.html" already exists for
the English version.
[INFO] Skipped "Test Source Xref" report, file "xref-test/index.html" already
exists for the English version.
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-site-plugin:3.3:site (default-site) on project
hbase: Error during page generation: Could not find the template
'META-INF/maven/site.vm: Encountered "source" at META-INF/maven/site.vm[line
1162, column 52]
[ERROR] Was expecting one of:
[ERROR] "," ...
[ERROR] ")" ...
[ERROR] ...
{noformat}
Do I need to make POM updates?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14792) Latest docs fail to build when packaging 0.98
[
https://issues.apache.org/jira/browse/HBASE-14792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-14792:
---
Fix Version/s: 0.98.17
> Latest docs fail to build when packaging 0.98
> -
>
> Key: HBASE-14792
> URL: https://issues.apache.org/jira/browse/HBASE-14792
> Project: HBase
> Issue Type: Bug
>Reporter: Andrew Purtell
>Priority: Minor
> Fix For: 0.98.17
>
>
> For releasing 0.98 we typically copy back the latest docs from master and
> then build.
> When I try generating the latest docs, I get:
> {noformat}
> [INFO]
>
> [INFO] Forking Apache HBase - Assembly 0.98.16-hadoop2
> [INFO]
>
> [INFO]
> [INFO] --- maven-enforcer-plugin:1.0.1:enforce (enforce) @ hbase-assembly ---
> [INFO]
> [INFO] --- buildnumber-maven-plugin:1.3:create-timestamp (default) @
> hbase-assembly ---
> [INFO]
> [INFO] <<< maven-javadoc-plugin:2.9.1:aggregate (report:aggregate) <
> generate-sources @ hbase <<<
> [INFO] configuring report plugin
> org.apache.maven.plugins:maven-checkstyle-plugin:2.13
> [INFO] Parent project loaded from repository: org.apache:apache:pom:12
> [INFO] Relativizing decoration links with respect to project URL:
> http://hbase.apache.org
> [INFO] Rendering site with org.apache.maven.skins:maven-fluido-skin:jar:1.4
> skin.
> [INFO] Skipped "About" report, file "index.html" already exists for the
> English version.
> [INFO] Skipped "Source Xref" report, file "xref/index.html" already exists
> for the English version.
> [INFO] Skipped "Test Source Xref" report, file "xref-test/index.html" already
> exists for the English version.
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-site-plugin:3.3:site (default-site) on project
> hbase: Error during page generation: Could not find the template
> 'META-INF/maven/site.vm: Encountered "source" at META-INF/maven/site.vm[line
> 1162, column 52]
> [ERROR] Was expecting one of:
> [ERROR] "," ...
> [ERROR] ")" ...
> [ERROR] ...
> {noformat}
> Do I need to make POM updates?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14791) [0.98] CopyTable is extremely slow when moving delete markers
[ https://issues.apache.org/jira/browse/HBASE-14791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-14791: -- Description: We found that some of our copy table job run for many hours, even when there isn't that much data to copy. [~vik.karma] did his magic and found that the issue is with copying delete markers (we use raw mode to also move deletes across). Looking at the code in 0.98 it's immediately obvious that deletes (unlike puts) are not batched and hence sent to the other side one by one, causing a network RTT for each delete marker. Looks like in trunk it's doing the right thing (using BufferedMutators for all mutations in TableOutputFormat). So likely only a 0.98 (and 1.0, 1.1, 1.2?) issue. was: We found that some of our copy table job run for many hours, even when there isn't that much data to copy. [~vik.karma] did his magic and found that the issue with copying delete markers (we use raw mode to also move deletes across). Looking at the code in 0.98 it's immediately obvious that deletes (unlike puts) are not batched and hence sent to the other side one by one, cause a network RTT for each delete marker. Looks like in trunk it's doing the right thing (using BufferedMutators for all mutations in TableOutputFormat). So likely only a 0.98 (and 1.0, 1.1, 1.2?) issue. > [0.98] CopyTable is extremely slow when moving delete markers > - > > Key: HBASE-14791 > URL: https://issues.apache.org/jira/browse/HBASE-14791 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.16 >Reporter: Lars Hofhansl >Assignee: Alex Araujo > > We found that some of our copy table job run for many hours, even when there > isn't that much data to copy. > [~vik.karma] did his magic and found that the issue is with copying delete > markers (we use raw mode to also move deletes across). > Looking at the code in 0.98 it's immediately obvious that deletes (unlike > puts) are not batched and hence sent to the other side one by one, causing a > network RTT for each delete marker. > Looks like in trunk it's doing the right thing (using BufferedMutators for > all mutations in TableOutputFormat). So likely only a 0.98 (and 1.0, 1.1, > 1.2?) issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14793) Allow limiting size of block into L1 block cache.
Elliott Clark created HBASE-14793: - Summary: Allow limiting size of block into L1 block cache. Key: HBASE-14793 URL: https://issues.apache.org/jira/browse/HBASE-14793 Project: HBase Issue Type: Bug Reporter: Elliott Clark Assignee: Elliott Clark G1GC does really badly with long lived large objects. Lets allow limiting the size of a block that can be kept in the block cache. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-3879) Add way to control the split position of region
[ https://issues.apache.org/jira/browse/HBASE-3879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Francke resolved HBASE-3879. - Resolution: Duplicate > Add way to control the split position of region > --- > > Key: HBASE-3879 > URL: https://issues.apache.org/jira/browse/HBASE-3879 > Project: HBase > Issue Type: Improvement > Components: regionserver >Reporter: Ophir Cohen >Priority: Minor > > Currently the split point picked to be the mid key of the HFile. > From the code comments: > "Midkey for this file. We work with block boundaries only so > * returned midkey is an approximation only." > It could be good to have a way to override the default option and use other > method. > Possible implementation: > Create interface (similar to Hadoop's partitioner) that gets StoreFile / > StoreFile.Reader and return the splitting point. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-3933) Hmaster throw NullPointerException
[
https://issues.apache.org/jira/browse/HBASE-3933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Francke resolved HBASE-3933.
-
Resolution: Won't Fix
> Hmaster throw NullPointerException
> --
>
> Key: HBASE-3933
> URL: https://issues.apache.org/jira/browse/HBASE-3933
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 0.90.6
>Reporter: gaojinchao
>Assignee: Eugene Koontz
> Attachments: HBASE-3933.patch, HBASE-3933.patch, Hmastersetup0.90
>
>
> NullPointerException while hmaster starting.
> {code}
> java.lang.NullPointerException
> at java.util.TreeMap.getEntry(TreeMap.java:324)
> at java.util.TreeMap.get(TreeMap.java:255)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.addToServers(AssignmentManager.java:1512)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.regionOnline(AssignmentManager.java:606)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.processFailover(AssignmentManager.java:214)
> at
> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:402)
> at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:283)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12790) Support fairness across parallelized scans
[ https://issues.apache.org/jira/browse/HBASE-12790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999119#comment-14999119 ] Andrew Purtell commented on HBASE-12790: bq. Hopefully there's enough information to feed into requirements from a user perspective: round robin across parallelized operations and MR jobs running on the same cluster to prevent one job from locking out others. There may be an implicit "in 0.98" here too. Let's remove that, if so, because: bq. (In 1.1 and up) Scanners can return after a certain size and/or time threshold has been crossed Step 1: Have both Phoenix scanners and those MR jobs set these parameters to constrain the amount of time each scanner.next call will run. Let's double check that we can set the defaults we want in site configuration. Step 2: On the server, as a generic and transparent improvement, have the scheduler round-robin requests between connections. With both of these in place, we get: - No one client can starve other clients. That means Phoenix work is interleaved with MR work on the server side. This is what you want. - Within your own single connection, no unit of work will exceed time X. This doesn't give you everything you want "within the connection" but you can work with this, because the server will give you ~deterministic performance per op. Now you can take the queue of local work - you own this, this is client side, HBase server side doesn't (and can't) know about internal client priorities - and make sure if you have internal notions of "this is for query A" and "that is for query B" that you interleave calls to scanner.next for A and B. It's more work than naively blasting ops at the servers and expecting the server side to handle differentiated QoS "within the connection", but this is the step too far the community doesn't want (yet). Leave this out and we might arrive at agreement. > Support fairness across parallelized scans > -- > > Key: HBASE-12790 > URL: https://issues.apache.org/jira/browse/HBASE-12790 > Project: HBase > Issue Type: New Feature >Reporter: James Taylor >Assignee: ramkrishna.s.vasudevan > Labels: Phoenix > Attachments: AbstractRoundRobinQueue.java, HBASE-12790.patch, > HBASE-12790_1.patch, HBASE-12790_5.patch, HBASE-12790_callwrapper.patch, > HBASE-12790_trunk_1.patch, PHOENIX_4.5.3-HBase-0.98-2317-SNAPSHOT.zip > > > Some HBase clients parallelize the execution of a scan to reduce latency in > getting back results. This can lead to starvation with a loaded cluster and > interleaved scans, since the RPC queue will be ordered and processed on a > FIFO basis. For example, if there are two clients, A & B that submit largish > scans at the same time. Say each scan is broken down into 100 scans by the > client (broken down into equal depth chunks along the row key), and the 100 > scans of client A are queued first, followed immediately by the 100 scans of > client B. In this case, client B will be starved out of getting any results > back until the scans for client A complete. > One solution to this is to use the attached AbstractRoundRobinQueue instead > of the standard FIFO queue. The queue to be used could be (maybe it already > is) configurable based on a new config parameter. Using this queue would > require the client to have the same identifier for all of the 100 parallel > scans that represent a single logical scan from the clients point of view. > With this information, the round robin queue would pick off a task from the > queue in a round robin fashion (instead of a strictly FIFO manner) to prevent > starvation over interleaved parallelized scans. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14766) WALEntryFilter's filter implement, cell.getFamily() needs to be replaced with the new low-cost implementation.
[ https://issues.apache.org/jira/browse/HBASE-14766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huaxiang sun updated HBASE-14766: - Attachment: HBASE-14766-v003.patch > WALEntryFilter's filter implement, cell.getFamily() needs to be replaced with > the new low-cost implementation. > -- > > Key: HBASE-14766 > URL: https://issues.apache.org/jira/browse/HBASE-14766 > Project: HBase > Issue Type: Improvement >Reporter: huaxiang sun >Assignee: huaxiang sun > Attachments: HBASE-14766-v001.patch, HBASE-14766-v002.patch, > HBASE-14766-v003.patch, HBASE-14766-v003.patch > > > Cell's getFamily() gets an array copy of the cell's family, while in the > filter function, it just needs to peek into the family and do a compare. > Replace > Bytes.toString(cell.getFamily()) > with > Bytes.toString(cell.getFamilyArray(), cell.getFamilyOffset(), > cell.getFamilyLength()) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14766) WALEntryFilter's filter implement, cell.getFamily() needs to be replaced with the new low-cost implementation.
[ https://issues.apache.org/jira/browse/HBASE-14766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14998955#comment-14998955 ] huaxiang sun commented on HBASE-14766: -- Thanks, I was wondering how to re-trigger another QA run. I will re-attach the v3 version. > WALEntryFilter's filter implement, cell.getFamily() needs to be replaced with > the new low-cost implementation. > -- > > Key: HBASE-14766 > URL: https://issues.apache.org/jira/browse/HBASE-14766 > Project: HBase > Issue Type: Improvement >Reporter: huaxiang sun >Assignee: huaxiang sun > Attachments: HBASE-14766-v001.patch, HBASE-14766-v002.patch, > HBASE-14766-v003.patch, HBASE-14766-v003.patch > > > Cell's getFamily() gets an array copy of the cell's family, while in the > filter function, it just needs to peek into the family and do a compare. > Replace > Bytes.toString(cell.getFamily()) > with > Bytes.toString(cell.getFamilyArray(), cell.getFamilyOffset(), > cell.getFamilyLength()) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14766) WALEntryFilter's filter implement, cell.getFamily() needs to be replaced with the new low-cost implementation.
[ https://issues.apache.org/jira/browse/HBASE-14766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huaxiang sun updated HBASE-14766: - Attachment: HBASE-14766-v004.patch Reload the patch to get a clean QA run. > WALEntryFilter's filter implement, cell.getFamily() needs to be replaced with > the new low-cost implementation. > -- > > Key: HBASE-14766 > URL: https://issues.apache.org/jira/browse/HBASE-14766 > Project: HBase > Issue Type: Improvement >Reporter: huaxiang sun >Assignee: huaxiang sun > Attachments: HBASE-14766-v001.patch, HBASE-14766-v002.patch, > HBASE-14766-v003.patch, HBASE-14766-v003.patch, HBASE-14766-v004.patch > > > Cell's getFamily() gets an array copy of the cell's family, while in the > filter function, it just needs to peek into the family and do a compare. > Replace > Bytes.toString(cell.getFamily()) > with > Bytes.toString(cell.getFamilyArray(), cell.getFamilyOffset(), > cell.getFamilyLength()) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14473) Compute region locality in parallel
[ https://issues.apache.org/jira/browse/HBASE-14473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14998944#comment-14998944 ] Hudson commented on HBASE-14473: FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #1131 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1131/]) HBASE-14611 Backport HBASE-14473 (Compute region locality in parallel) (apurtell: rev 70b9015a0ec8de5ed9f9d9302b67b8e79bc4c558) * hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/RegionLocationFinder.java > Compute region locality in parallel > --- > > Key: HBASE-14473 > URL: https://issues.apache.org/jira/browse/HBASE-14473 > Project: HBase > Issue Type: Bug > Components: Balancer >Affects Versions: 1.2.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14473-v1.patch, HBASE-14473-v2.patch, > HBASE-14473-v3.patch, HBASE-14473-v4.patch, HBASE-14473.patch > > > Right now on large clusters it's necessary to turn off the locality balance > cost as it takes too long to compute the region locality. This is because > it's computed when need in serial. > We should compute this in parallel before it's needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14791) [0.98] CopyTable is extremely slow when moving delete markers
[ https://issues.apache.org/jira/browse/HBASE-14791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999028#comment-14999028 ] Vikas Vishwakarma commented on HBASE-14791: --- also thanks to [~sukuna...@gmail.com] for helping with debugging the issue > [0.98] CopyTable is extremely slow when moving delete markers > - > > Key: HBASE-14791 > URL: https://issues.apache.org/jira/browse/HBASE-14791 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.16 >Reporter: Lars Hofhansl > > We found that some of our copy table job run for many hours, even when there > isn't that much data to copy. > [~vik.karma] did his magic and found that the issue with copying delete > markers (we use raw mode to also move deletes across). > Looking at the code in 0.98 it's immediately obvious that deletes (unlike > puts) are not batched and hence sent to the other side one by one, cause a > network RTT for each delete marker. > Looks like in trunk it's doing the right thing (using BufferedMutators for > all mutations in TableOutputFormat). So likely only a 0.98 (and 1.0, 1.1, > 1.2?) issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12790) Support fairness across parallelized scans
[ https://issues.apache.org/jira/browse/HBASE-12790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999155#comment-14999155 ] Andrew Purtell commented on HBASE-12790: Just throwing out ideas, considering existing interfaces: We might also consider if Phoenix's overloading of scanner.next processing on the server can help here. Scan parameters can be changed on the way in on the server side (in the preXXX hooks) according to Phoenix's view of the arriving requests. A static adjustment for predictable performance could be enough. However, there are interesting opportunities for making dynamic changes here: As workload increases, perhaps measured by arrival rate or by an estimation of query perf characteristics (like estimated cardinality), the amount of work by time performed by each scanner.next iteration can be made smaller, providing lower latency / better responsiveness when work is interleaved at the expense of throughput. As workload decreases, the quanta can be increased, optimizing for better throughput. In any case it's really up to the coprocessor application what wants to do with respect to rewriting scan parameters on the server. (And up to the client / query planner how it wants to set up scan parameters in the first place.) > Support fairness across parallelized scans > -- > > Key: HBASE-12790 > URL: https://issues.apache.org/jira/browse/HBASE-12790 > Project: HBase > Issue Type: New Feature >Reporter: James Taylor >Assignee: ramkrishna.s.vasudevan > Labels: Phoenix > Attachments: AbstractRoundRobinQueue.java, HBASE-12790.patch, > HBASE-12790_1.patch, HBASE-12790_5.patch, HBASE-12790_callwrapper.patch, > HBASE-12790_trunk_1.patch, PHOENIX_4.5.3-HBase-0.98-2317-SNAPSHOT.zip > > > Some HBase clients parallelize the execution of a scan to reduce latency in > getting back results. This can lead to starvation with a loaded cluster and > interleaved scans, since the RPC queue will be ordered and processed on a > FIFO basis. For example, if there are two clients, A & B that submit largish > scans at the same time. Say each scan is broken down into 100 scans by the > client (broken down into equal depth chunks along the row key), and the 100 > scans of client A are queued first, followed immediately by the 100 scans of > client B. In this case, client B will be starved out of getting any results > back until the scans for client A complete. > One solution to this is to use the attached AbstractRoundRobinQueue instead > of the standard FIFO queue. The queue to be used could be (maybe it already > is) configurable based on a new config parameter. Using this queue would > require the client to have the same identifier for all of the 100 parallel > scans that represent a single logical scan from the clients point of view. > With this information, the round robin queue would pick off a task from the > queue in a round robin fashion (instead of a strictly FIFO manner) to prevent > starvation over interleaved parallelized scans. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14387) Compaction improvements: Maximum off-peak compaction size
[ https://issues.apache.org/jira/browse/HBASE-14387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-14387: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 1.3.0 1.2.0 Status: Resolved (was: Patch Available) I've run the TestGenerateDelegationToken locally. Does not seem related. Pushed to 1.2+. Thanks Vladimir for the patch. > Compaction improvements: Maximum off-peak compaction size > - > > Key: HBASE-14387 > URL: https://issues.apache.org/jira/browse/HBASE-14387 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14387-v1.patch, HBASE-14387-v2.patch, > HBASE-14387-v3.patch, HBASE-14387-v3.patch > > > Make max compaction size for peak and off peak separate config options. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14785) Hamburger menu for mobile site
[ https://issues.apache.org/jira/browse/HBASE-14785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999206#comment-14999206 ] stack commented on HBASE-14785: --- bq. Only way to know for sure is to test it live. Yeah, some stuff is like this. +1 on testing live. Patch looks good (except for Ted note above). Whats this about? 35 000385458301414556862:sq1bb0xugjg 36 > Hamburger menu for mobile site > -- > > Key: HBASE-14785 > URL: https://issues.apache.org/jira/browse/HBASE-14785 > Project: HBase > Issue Type: Sub-task > Components: website >Affects Versions: 2.0.0 >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Fix For: 2.0.0 > > Attachments: HBASE-14785-addendum-v1.patch, > HBASE-14785-addendum.patch, HBASE-14785-v1.patch, HBASE-14785-v2.patch, > HBASE-14785.patch, maven-fluido-skin-1.5-HBASE.jar > > > Figure out how to do a hamburger menu on mobile. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14771) RpcServer.getRemoteAddress always returns null.
[
https://issues.apache.org/jira/browse/HBASE-14771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999218#comment-14999218
]
Appy commented on HBASE-14771:
--
Perfect!
> RpcServer.getRemoteAddress always returns null.
> ---
>
> Key: HBASE-14771
> URL: https://issues.apache.org/jira/browse/HBASE-14771
> Project: HBase
> Issue Type: Bug
> Components: IPC/RPC
>Affects Versions: 1.2.0
>Reporter: Abhishek Kumar
>Assignee: Abhishek Kumar
>Priority: Minor
> Attachments: HBASE-14771-V1.patch, HBASE-14771.patch
>
>
> RpcServer.getRemoteAddress always returns null, because Call object is
> getting initialized with null.This seems to be happening because of using
> RpcServer.getRemoteIp() in Call object constructor before RpcServer thread
> local 'CurCall' being set in CallRunner.run method:
> {noformat}
> // --- RpcServer.java ---
> protected void processRequest(byte[] buf) throws IOException,
> InterruptedException {
> .
> // Call object getting initialized here with address
> // obtained from RpcServer.getRemoteIp()
> Call call = new Call(id, this.service, md, header, param, cellScanner, this,
> responder,
> totalRequestSize, traceInfo, RpcServer.getRemoteIp());
> scheduler.dispatch(new CallRunner(RpcServer.this, call));
> }
> // getRemoteIp method gets address from threadlocal 'CurCall' which
> // gets set in CallRunner.run and calling it before this as in above case,
> will return null
> // --- CallRunner.java ---
> public void run() {
> .
> Pair resultPair = null;
> RpcServer.CurCall.set(call);
> ..
> }
> // Using 'this.addr' in place of getRemoteIp method in RpcServer.java seems
> to be fixing this issue
> Call call = new Call(id, this.service, md, header, param, cellScanner, this,
> responder,
> totalRequestSize, traceInfo, this.addr);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14784) Port conflict is not resolved in HBaseTestingUtility.randomFreePort()
[
https://issues.apache.org/jira/browse/HBASE-14784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-14784:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
Reresolving after committing test addendum to master branch (will reopen if the
test hadoopqa submission fails)
> Port conflict is not resolved in HBaseTestingUtility.randomFreePort()
> -
>
> Key: HBASE-14784
> URL: https://issues.apache.org/jira/browse/HBASE-14784
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 1.1.2
>Reporter: Youngjoon Kim
>Assignee: Youngjoon Kim
>Priority: Minor
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14784.patch, HBASE-14784_Test.patch
>
>
> If takenRandomPorts.contains(port) == true, it means port conflict, so
> randomFreePort() should rerun the loop. But continue statement leads to exit
> the loop, because port != 0.
> {code:title=hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java}
> public static int randomFreePort() {
> int port = 0;
> do {
> port = randomPort();
> if (takenRandomPorts.contains(port)) {
> continue;
> }
> takenRandomPorts.add(port);
> ...
> } while (port == 0);
> return port;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14784) Port conflict is not resolved in HBaseTestingUtility.randomFreePort()
[
https://issues.apache.org/jira/browse/HBASE-14784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999215#comment-14999215
]
stack commented on HBASE-14784:
---
Nice test. Thank you [~y0un5]. I pushed the test to master branch only. Let me
reopen this issue here so it runs hadoopqa just to be safe. Thanks.
> Port conflict is not resolved in HBaseTestingUtility.randomFreePort()
> -
>
> Key: HBASE-14784
> URL: https://issues.apache.org/jira/browse/HBASE-14784
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 1.1.2
>Reporter: Youngjoon Kim
>Assignee: Youngjoon Kim
>Priority: Minor
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14784.patch, HBASE-14784_Test.patch
>
>
> If takenRandomPorts.contains(port) == true, it means port conflict, so
> randomFreePort() should rerun the loop. But continue statement leads to exit
> the loop, because port != 0.
> {code:title=hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java}
> public static int randomFreePort() {
> int port = 0;
> do {
> port = randomPort();
> if (takenRandomPorts.contains(port)) {
> continue;
> }
> takenRandomPorts.add(port);
> ...
> } while (port == 0);
> return port;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Reopened] (HBASE-14784) Port conflict is not resolved in HBaseTestingUtility.randomFreePort()
[
https://issues.apache.org/jira/browse/HBASE-14784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack reopened HBASE-14784:
---
> Port conflict is not resolved in HBaseTestingUtility.randomFreePort()
> -
>
> Key: HBASE-14784
> URL: https://issues.apache.org/jira/browse/HBASE-14784
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 1.1.2
>Reporter: Youngjoon Kim
>Assignee: Youngjoon Kim
>Priority: Minor
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14784.patch, HBASE-14784_Test.patch
>
>
> If takenRandomPorts.contains(port) == true, it means port conflict, so
> randomFreePort() should rerun the loop. But continue statement leads to exit
> the loop, because port != 0.
> {code:title=hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java}
> public static int randomFreePort() {
> int port = 0;
> do {
> port = randomPort();
> if (takenRandomPorts.contains(port)) {
> continue;
> }
> takenRandomPorts.add(port);
> ...
> } while (port == 0);
> return port;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-14794) Cleanup TestAtomicOperation
stack created HBASE-14794: - Summary: Cleanup TestAtomicOperation Key: HBASE-14794 URL: https://issues.apache.org/jira/browse/HBASE-14794 Project: HBase Issue Type: Sub-task Components: flakey, test Reporter: stack Assignee: stack Test fails with some frequency. Complaint is that can't parse the test xml which doesn't give much clue as to what is going wrong. Looking in the test, it leaks a load of resources (It does not close regions post open) and there is a NPE parsing a long on increment. Let me clean up this test. Hopefully that will get the root failure to show if still present. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14780) Integration Tests that run with ChaosMonkey need to specify CFs
[ https://issues.apache.org/jira/browse/HBASE-14780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999189#comment-14999189 ] stack commented on HBASE-14780: --- bq. Hm.. I don't actually touch anything related to the zombies with this patch. Aside from that it looks like a clean run. Let me try again. Our zombie detection was sloppy. Long running tests from an adjacent test run would be picked up by the current hadoopqa run and considered 'zombies' since they were tests running longer than the wait period of 30 seconds. Should be better now. > Integration Tests that run with ChaosMonkey need to specify CFs > --- > > Key: HBASE-14780 > URL: https://issues.apache.org/jira/browse/HBASE-14780 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Attachments: hbase-14780.patch, hbase-14780.patch > > > Been running some IT tests and found that some failed because getcfs was null > and didn't protecte cfs that were assumed to go unmolested. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14784) Port conflict is not resolved in HBaseTestingUtility.randomFreePort()
[
https://issues.apache.org/jira/browse/HBASE-14784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-14784:
--
Status: Patch Available (was: Reopened)
> Port conflict is not resolved in HBaseTestingUtility.randomFreePort()
> -
>
> Key: HBASE-14784
> URL: https://issues.apache.org/jira/browse/HBASE-14784
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 1.1.2
>Reporter: Youngjoon Kim
>Assignee: Youngjoon Kim
>Priority: Minor
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14784.patch, HBASE-14784_Test.patch
>
>
> If takenRandomPorts.contains(port) == true, it means port conflict, so
> randomFreePort() should rerun the loop. But continue statement leads to exit
> the loop, because port != 0.
> {code:title=hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java}
> public static int randomFreePort() {
> int port = 0;
> do {
> port = randomPort();
> if (takenRandomPorts.contains(port)) {
> continue;
> }
> takenRandomPorts.add(port);
> ...
> } while (port == 0);
> return port;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14789) Provide an alternative spark-hbase connector
[ https://issues.apache.org/jira/browse/HBASE-14789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999385#comment-14999385 ] Ted Malaska commented on HBASE-14789: - Can you help me understand what components this has that don't already exist in the current HBase-Spark module and also what requirements are not met by the current Spark-Module implementation but are supported with this code? Thanks > Provide an alternative spark-hbase connector > > > Key: HBASE-14789 > URL: https://issues.apache.org/jira/browse/HBASE-14789 > Project: HBase > Issue Type: Bug >Reporter: Zhan Zhang >Assignee: Zhan Zhang > Attachments: shc.pdf > > > This JIRA is to provide user an option to choose different Spark-HBase > implementation based on requirements. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14793) Allow limiting size of block into L1 block cache.
[ https://issues.apache.org/jira/browse/HBASE-14793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-14793: -- Fix Version/s: 1.3.0 1.2.0 2.0.0 Affects Version/s: 1.2.0 1.1.2 Status: Patch Available (was: Open) > Allow limiting size of block into L1 block cache. > - > > Key: HBASE-14793 > URL: https://issues.apache.org/jira/browse/HBASE-14793 > Project: HBase > Issue Type: Bug >Affects Versions: 1.1.2, 1.2.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14793.patch > > > G1GC does really badly with long lived large objects. Lets allow limiting the > size of a block that can be kept in the block cache. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14785) Hamburger menu for mobile site
[ https://issues.apache.org/jira/browse/HBASE-14785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999350#comment-14999350 ] Misty Stanley-Jones commented on HBASE-14785: - It's the ID of the custom search I created, that searches hbase.apache.org, issues.apache.org/browse/HBASE-*, and the mailing list archives. I submitted a pull request to the Maven Fluido plugin to support this feature. > Hamburger menu for mobile site > -- > > Key: HBASE-14785 > URL: https://issues.apache.org/jira/browse/HBASE-14785 > Project: HBase > Issue Type: Sub-task > Components: website >Affects Versions: 2.0.0 >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Fix For: 2.0.0 > > Attachments: HBASE-14785-addendum-v1.patch, > HBASE-14785-addendum.patch, HBASE-14785-v1.patch, HBASE-14785-v2.patch, > HBASE-14785.patch, maven-fluido-skin-1.5-HBASE.jar > > > Figure out how to do a hamburger menu on mobile. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14786) TestProcedureAdmin hangs
[
https://issues.apache.org/jira/browse/HBASE-14786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999349#comment-14999349
]
stack commented on HBASE-14786:
---
+1
> TestProcedureAdmin hangs
>
>
> Key: HBASE-14786
> URL: https://issues.apache.org/jira/browse/HBASE-14786
> Project: HBase
> Issue Type: Sub-task
> Components: hangingTests, test
>Reporter: stack
>Assignee: Matteo Bertozzi
> Attachments: HBASE-14786-v0.patch
>
>
> This test was added by HBASE-14108 in September [~syuanjiang] Mind taking a
> look? Hope you don't mind me assigning it to you to take a look.
> This test run was incomplete:
> https://builds.apache.org/job/HBase-Trunk_matrix/jdk=latest1.8,label=Hadoop/443/
> If I do below...
> kalashnikov:hbase.git.commit stack$ python ./dev-support/findHangingTests.py
> https://builds.apache.org/job/HBase-Trunk_matrix/jdk=latest1.8,label=Hadoop/443/consoleText
> Fetching
> https://builds.apache.org/job/HBase-Trunk_matrix/jdk=latest1.8,label=Hadoop/443/consoleText
> Building remotely on H4 (Mapreduce zookeeper Hadoop Pig falcon Hdfs) in
> workspace
> /home/jenkins/jenkins-slave/workspace/HBase-Trunk_matrix/jdk/latest1.8/label/Hadoop
> Printing hanging tests
> Hanging test : org.apache.hadoop.hbase.master.procedure.TestProcedureAdmin
> Printing Failing tests
> ... it reports TestProcedureAdmin as reporting it started but there is no
> corresponding loggiing of end-of-test.
> If I look in the output for the above test run, I see all this spew:
> https://builds.apache.org/job/HBase-Trunk_matrix/jdk=latest1.8,label=Hadoop/443/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.master.procedure.TestProcedureAdmin-output.txt
> about...
> {code}
> 2015-11-07 10:19:59,052 INFO
> [B.defaultRpcServer.handler=4,queue=0,port=37253] master.HMaster(1911):
> Client=jenkins/null disable testListProcedure
> 2015-11-07 10:19:59,052 ERROR
> [B.defaultRpcServer.handler=4,queue=0,port=37253] ipc.RpcServer(2228):
> Unexpected throwable object
> java.lang.IllegalArgumentException
> at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:76)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:618)
> at
> org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1916)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.disableTable(MasterRpcServices.java:572)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:57936)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2184)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> That look right to you?
> Maybe it is something else. Would be good to clean up hanging tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14793) Allow limiting size of block into L1 block cache.
[ https://issues.apache.org/jira/browse/HBASE-14793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-14793: -- Attachment: HBASE-14793.patch > Allow limiting size of block into L1 block cache. > - > > Key: HBASE-14793 > URL: https://issues.apache.org/jira/browse/HBASE-14793 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0, 1.1.2 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14793.patch > > > G1GC does really badly with long lived large objects. Lets allow limiting the > size of a block that can be kept in the block cache. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14766) WALEntryFilter's filter implement, cell.getFamily() needs to be replaced with the new low-cost implementation.
[
https://issues.apache.org/jira/browse/HBASE-14766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999275#comment-14999275
]
Hadoop QA commented on HBASE-14766:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12771589/HBASE-14766-v003.patch
against master branch at commit 112900d0425a8157b89041f0e353ebf5cc259c69.
ATTACHMENT ID: 12771589
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1 release
audit warnings (more than the master's current 0 warnings).
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16480//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16480//artifact/patchprocess/patchReleaseAuditWarnings.txt
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16480//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16480//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16480//console
This message is automatically generated.
> WALEntryFilter's filter implement, cell.getFamily() needs to be replaced with
> the new low-cost implementation.
> --
>
> Key: HBASE-14766
> URL: https://issues.apache.org/jira/browse/HBASE-14766
> Project: HBase
> Issue Type: Improvement
>Reporter: huaxiang sun
>Assignee: huaxiang sun
> Attachments: HBASE-14766-v001.patch, HBASE-14766-v002.patch,
> HBASE-14766-v003.patch, HBASE-14766-v003.patch, HBASE-14766-v004.patch
>
>
> Cell's getFamily() gets an array copy of the cell's family, while in the
> filter function, it just needs to peek into the family and do a compare.
> Replace
> Bytes.toString(cell.getFamily())
> with
> Bytes.toString(cell.getFamilyArray(), cell.getFamilyOffset(),
> cell.getFamilyLength())
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14790) Implement a new DFSOutputStream for logging WAL only
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999330#comment-14999330
]
Sean Busbey commented on HBASE-14790:
-
excellent! I had a first pass of this done several months ago to correct for
the assumptions around write size and clean up the excessive locking done in
the existing DFSOutputStream implementation.
+1 and happy to review, please encapsulate this in a WAL Provider so that the
existing filesystem implementation can live alongside it until we have a chance
for burn in.
hopefully once we have some perf numbers we can look into getting changes
available upstream for more general use.
> Implement a new DFSOutputStream for logging WAL only
>
>
> Key: HBASE-14790
> URL: https://issues.apache.org/jira/browse/HBASE-14790
> Project: HBase
> Issue Type: Improvement
>Reporter: Duo Zhang
>
> The original {{DFSOutputStream}} is very powerful and aims to serve all
> purposes. But in fact, we do not need most of the features if we only want to
> log WAL. For example, we do not need pipeline recovery since we could just
> close the old logger and open a new one. And also, we do not need to write
> multiple blocks since we could also open a new logger if the old file is too
> large.
> And the most important thing is that, it is hard to handle all the corner
> cases to avoid data loss or data inconsistency(such as HBASE-14004) when
> using original DFSOutputStream due to its complicated logic. And the
> complicated logic also force us to use some magical tricks to increase
> performance. For example, we need to use multiple threads to call {{hflush}}
> when logging, and now we use 5 threads. But why 5 not 10 or 100?
> So here, I propose we should implement our own {{DFSOutputStream}} when
> logging WAL. For correctness, and also for performance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14793) Allow limiting size of block into L1 block cache.
[
https://issues.apache.org/jira/browse/HBASE-14793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999355#comment-14999355
]
Appy commented on HBASE-14793:
--
[~eclark], how do you plan to use the new variable 'maxBlockSize'?
Btw, there's an incomplete log message here. Also, please consider writing a
comment about why 12MB was chosen.
{code:java}
+
+if (buf.heapSize() > 12 * 1024 * 1024) {
+ LOG.warn("Trying to cache too large a ");
+}
+
{code}
> Allow limiting size of block into L1 block cache.
> -
>
> Key: HBASE-14793
> URL: https://issues.apache.org/jira/browse/HBASE-14793
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0, 1.1.2
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14793.patch
>
>
> G1GC does really badly with long lived large objects. Lets allow limiting the
> size of a block that can be kept in the block cache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14387) Compaction improvements: Maximum off-peak compaction size
[ https://issues.apache.org/jira/browse/HBASE-14387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999188#comment-14999188 ] Hudson commented on HBASE-14387: SUCCESS: Integrated in HBase-1.2-IT #274 (See [https://builds.apache.org/job/HBase-1.2-IT/274/]) HBASE-14387 Compaction improvements: Maximum off-peak compaction size (enis: rev f4c70bfc1da63ae2742272a1409615ce507aed15) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionConfiguration.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/ExploringCompactionPolicy.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerOnlineConfigChange.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/RatioBasedCompactionPolicy.java > Compaction improvements: Maximum off-peak compaction size > - > > Key: HBASE-14387 > URL: https://issues.apache.org/jira/browse/HBASE-14387 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14387-v1.patch, HBASE-14387-v2.patch, > HBASE-14387-v3.patch, HBASE-14387-v3.patch > > > Make max compaction size for peak and off peak separate config options. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13408) HBase In-Memory Memstore Compaction
[ https://issues.apache.org/jira/browse/HBASE-13408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel updated HBASE-13408: -- Attachment: HBASE-13408-trunk-v10.patch > HBase In-Memory Memstore Compaction > --- > > Key: HBASE-13408 > URL: https://issues.apache.org/jira/browse/HBASE-13408 > Project: HBase > Issue Type: New Feature >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel > Fix For: 2.0.0 > > Attachments: HBASE-13408-trunk-v01.patch, > HBASE-13408-trunk-v02.patch, HBASE-13408-trunk-v03.patch, > HBASE-13408-trunk-v04.patch, HBASE-13408-trunk-v05.patch, > HBASE-13408-trunk-v06.patch, HBASE-13408-trunk-v07.patch, > HBASE-13408-trunk-v08.patch, HBASE-13408-trunk-v09.patch, > HBASE-13408-trunk-v10.patch, > HBaseIn-MemoryMemstoreCompactionDesignDocument-ver02.pdf, > HBaseIn-MemoryMemstoreCompactionDesignDocument-ver03.pdf, > HBaseIn-MemoryMemstoreCompactionDesignDocument.pdf, > InMemoryMemstoreCompactionEvaluationResults.pdf, > InMemoryMemstoreCompactionMasterEvaluationResults.pdf, > InMemoryMemstoreCompactionScansEvaluationResults.pdf, > StoreSegmentandStoreSegmentScannerClassHierarchies.pdf > > > A store unit holds a column family in a region, where the memstore is its > in-memory component. The memstore absorbs all updates to the store; from time > to time these updates are flushed to a file on disk, where they are > compacted. Unlike disk components, the memstore is not compacted until it is > written to the filesystem and optionally to block-cache. This may result in > underutilization of the memory due to duplicate entries per row, for example, > when hot data is continuously updated. > Generally, the faster the data is accumulated in memory, more flushes are > triggered, the data sinks to disk more frequently, slowing down retrieval of > data, even if very recent. > In high-churn workloads, compacting the memstore can help maintain the data > in memory, and thereby speed up data retrieval. > We suggest a new compacted memstore with the following principles: > 1.The data is kept in memory for as long as possible > 2.Memstore data is either compacted or in process of being compacted > 3.Allow a panic mode, which may interrupt an in-progress compaction and > force a flush of part of the memstore. > We suggest applying this optimization only to in-memory column families. > A design document is attached. > This feature was previously discussed in HBASE-5311. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14791) [0.98] CopyTable is extremely slow when moving delete markers
[ https://issues.apache.org/jira/browse/HBASE-14791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999243#comment-14999243 ] Alex Araujo commented on HBASE-14791: - The patch for HBASE-12728 moved buffering of Puts from HTable into BufferedMutator for 1.0+. Since BufferedMutator is not Put specific, it also made TableOutputFormat and MultiTableOutputFormat use BufferedMutator for all Mutation types. In 0.98 there is no buffering of Deletes in HTable or elsewhere as far as I can tell. Essentially, we'd need to implement a basic BufferedMutator and use it for both OutputFormat types. The downside is that we would be duplicating some of the buffering code in HTable. > [0.98] CopyTable is extremely slow when moving delete markers > - > > Key: HBASE-14791 > URL: https://issues.apache.org/jira/browse/HBASE-14791 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.16 >Reporter: Lars Hofhansl >Assignee: Alex Araujo > > We found that some of our copy table job run for many hours, even when there > isn't that much data to copy. > [~vik.karma] did his magic and found that the issue is with copying delete > markers (we use raw mode to also move deletes across). > Looking at the code in 0.98 it's immediately obvious that deletes (unlike > puts) are not batched and hence sent to the other side one by one, causing a > network RTT for each delete marker. > Looks like in trunk it's doing the right thing (using BufferedMutators for > all mutations in TableOutputFormat). So likely only a 0.98 (and 1.0, 1.1, > 1.2?) issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14794) Cleanup TestAtomicOperation
[ https://issues.apache.org/jira/browse/HBASE-14794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-14794: -- Attachment: 14794.patch Close WAL when test is done. Run with less threads (25 vs 100). > Cleanup TestAtomicOperation > --- > > Key: HBASE-14794 > URL: https://issues.apache.org/jira/browse/HBASE-14794 > Project: HBase > Issue Type: Sub-task > Components: flakey, test >Reporter: stack >Assignee: stack > Attachments: 14794.patch > > > Test fails with some frequency. Complaint is that can't parse the test xml > which doesn't give much clue as to what is going wrong. Looking in the test, > it leaks a load of resources (It does not close regions post open) and there > is a NPE parsing a long on increment. Let me clean up this test. Hopefully > that will get the root failure to show if still present. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14793) Allow limiting size of block into L1 block cache.
[ https://issues.apache.org/jira/browse/HBASE-14793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-14793: -- Attachment: HBASE-14793-v1.patch > Allow limiting size of block into L1 block cache. > - > > Key: HBASE-14793 > URL: https://issues.apache.org/jira/browse/HBASE-14793 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0, 1.1.2 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14793-v1.patch, HBASE-14793.patch > > > G1GC does really badly with long lived large objects. Lets allow limiting the > size of a block that can be kept in the block cache. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14785) Hamburger menu for mobile site
[ https://issues.apache.org/jira/browse/HBASE-14785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-14785: Attachment: 0001-HBASE-14774-Addendum-Exclude-src-main-site-resources.patch Patch to exclude the repo from license checking for now. Already pushed to master. > Hamburger menu for mobile site > -- > > Key: HBASE-14785 > URL: https://issues.apache.org/jira/browse/HBASE-14785 > Project: HBase > Issue Type: Sub-task > Components: website >Affects Versions: 2.0.0 >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Fix For: 2.0.0 > > Attachments: > 0001-HBASE-14774-Addendum-Exclude-src-main-site-resources.patch, > HBASE-14785-addendum-v1.patch, HBASE-14785-addendum.patch, > HBASE-14785-v1.patch, HBASE-14785-v2.patch, HBASE-14785.patch, > maven-fluido-skin-1.5-HBASE.jar > > > Figure out how to do a hamburger menu on mobile. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14791) [0.98] CopyTable is extremely slow when moving delete markers
[ https://issues.apache.org/jira/browse/HBASE-14791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999342#comment-14999342 ] Andrew Purtell commented on HBASE-14791: bq. Essentially, we'd need to implement a basic BufferedMutator and use it for both OutputFormat types. The downside is that we would be duplicating some of the buffering code in HTable. I think this is fine given the trade-offs here. We don't want to make a significant change in HTable semantics in 0.98. We do want to improve performance of TableOutputFormat substantially for Deletes. A reasonable option is some 0.98-specific buffering in TableOutputFormat. It would be more palatable a change if the new behavior is configurable, defaulting to off (backwards compatible FWIW) > [0.98] CopyTable is extremely slow when moving delete markers > - > > Key: HBASE-14791 > URL: https://issues.apache.org/jira/browse/HBASE-14791 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.16 >Reporter: Lars Hofhansl >Assignee: Alex Araujo > > We found that some of our copy table job run for many hours, even when there > isn't that much data to copy. > [~vik.karma] did his magic and found that the issue is with copying delete > markers (we use raw mode to also move deletes across). > Looking at the code in 0.98 it's immediately obvious that deletes (unlike > puts) are not batched and hence sent to the other side one by one, causing a > network RTT for each delete marker. > Looks like in trunk it's doing the right thing (using BufferedMutators for > all mutations in TableOutputFormat). So likely only a 0.98 (and 1.0, 1.1, > 1.2?) issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-14791) [0.98] CopyTable is extremely slow when moving delete markers
[ https://issues.apache.org/jira/browse/HBASE-14791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999342#comment-14999342 ] Andrew Purtell edited comment on HBASE-14791 at 11/10/15 8:58 PM: -- bq. Essentially, we'd need to implement a basic BufferedMutator and use it for both OutputFormat types. The downside is that we would be duplicating some of the buffering code in HTable. I think this is fine given the trade-offs here. We don't want to make a significant change in HTable semantics in 0.98. We do want to improve performance of TableOutputFormat substantially for Deletes. A reasonable option is some 0.98-specific buffering in TableOutputFormat. It could be more palatable a change if the new behavior is configurable, defaulting to off (backwards compatible FWIW) was (Author: apurtell): bq. Essentially, we'd need to implement a basic BufferedMutator and use it for both OutputFormat types. The downside is that we would be duplicating some of the buffering code in HTable. I think this is fine given the trade-offs here. We don't want to make a significant change in HTable semantics in 0.98. We do want to improve performance of TableOutputFormat substantially for Deletes. A reasonable option is some 0.98-specific buffering in TableOutputFormat. It would be more palatable a change if the new behavior is configurable, defaulting to off (backwards compatible FWIW) > [0.98] CopyTable is extremely slow when moving delete markers > - > > Key: HBASE-14791 > URL: https://issues.apache.org/jira/browse/HBASE-14791 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.16 >Reporter: Lars Hofhansl >Assignee: Alex Araujo > > We found that some of our copy table job run for many hours, even when there > isn't that much data to copy. > [~vik.karma] did his magic and found that the issue is with copying delete > markers (we use raw mode to also move deletes across). > Looking at the code in 0.98 it's immediately obvious that deletes (unlike > puts) are not batched and hence sent to the other side one by one, causing a > network RTT for each delete marker. > Looks like in trunk it's doing the right thing (using BufferedMutators for > all mutations in TableOutputFormat). So likely only a 0.98 (and 1.0, 1.1, > 1.2?) issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14794) Cleanup TestAtomicOperation
[ https://issues.apache.org/jira/browse/HBASE-14794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-14794: -- Status: Patch Available (was: Open) > Cleanup TestAtomicOperation > --- > > Key: HBASE-14794 > URL: https://issues.apache.org/jira/browse/HBASE-14794 > Project: HBase > Issue Type: Sub-task > Components: flakey, test >Reporter: stack >Assignee: stack > Attachments: 14794.patch > > > Test fails with some frequency. Complaint is that can't parse the test xml > which doesn't give much clue as to what is going wrong. Looking in the test, > it leaks a load of resources (It does not close regions post open) and there > is a NPE parsing a long on increment. Let me clean up this test. Hopefully > that will get the root failure to show if still present. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14786) TestProcedureAdmin hangs
[
https://issues.apache.org/jira/browse/HBASE-14786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matteo Bertozzi updated HBASE-14786:
Resolution: Fixed
Fix Version/s: 1.3.0
2.0.0
Status: Resolved (was: Patch Available)
> TestProcedureAdmin hangs
>
>
> Key: HBASE-14786
> URL: https://issues.apache.org/jira/browse/HBASE-14786
> Project: HBase
> Issue Type: Sub-task
> Components: hangingTests, test
>Reporter: stack
>Assignee: Matteo Bertozzi
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-14786-v0.patch
>
>
> This test was added by HBASE-14108 in September [~syuanjiang] Mind taking a
> look? Hope you don't mind me assigning it to you to take a look.
> This test run was incomplete:
> https://builds.apache.org/job/HBase-Trunk_matrix/jdk=latest1.8,label=Hadoop/443/
> If I do below...
> kalashnikov:hbase.git.commit stack$ python ./dev-support/findHangingTests.py
> https://builds.apache.org/job/HBase-Trunk_matrix/jdk=latest1.8,label=Hadoop/443/consoleText
> Fetching
> https://builds.apache.org/job/HBase-Trunk_matrix/jdk=latest1.8,label=Hadoop/443/consoleText
> Building remotely on H4 (Mapreduce zookeeper Hadoop Pig falcon Hdfs) in
> workspace
> /home/jenkins/jenkins-slave/workspace/HBase-Trunk_matrix/jdk/latest1.8/label/Hadoop
> Printing hanging tests
> Hanging test : org.apache.hadoop.hbase.master.procedure.TestProcedureAdmin
> Printing Failing tests
> ... it reports TestProcedureAdmin as reporting it started but there is no
> corresponding loggiing of end-of-test.
> If I look in the output for the above test run, I see all this spew:
> https://builds.apache.org/job/HBase-Trunk_matrix/jdk=latest1.8,label=Hadoop/443/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.master.procedure.TestProcedureAdmin-output.txt
> about...
> {code}
> 2015-11-07 10:19:59,052 INFO
> [B.defaultRpcServer.handler=4,queue=0,port=37253] master.HMaster(1911):
> Client=jenkins/null disable testListProcedure
> 2015-11-07 10:19:59,052 ERROR
> [B.defaultRpcServer.handler=4,queue=0,port=37253] ipc.RpcServer(2228):
> Unexpected throwable object
> java.lang.IllegalArgumentException
> at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:76)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:618)
> at
> org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1916)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.disableTable(MasterRpcServices.java:572)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:57936)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2184)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> That look right to you?
> Maybe it is something else. Would be good to clean up hanging tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14793) Allow limiting size of block into L1 block cache.
[ https://issues.apache.org/jira/browse/HBASE-14793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999371#comment-14999371 ] Elliott Clark commented on HBASE-14793: --- Yeah for some reason I hadn't committed the last version. oops. > Allow limiting size of block into L1 block cache. > - > > Key: HBASE-14793 > URL: https://issues.apache.org/jira/browse/HBASE-14793 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0, 1.1.2 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14793.patch > > > G1GC does really badly with long lived large objects. Lets allow limiting the > size of a block that can be kept in the block cache. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14785) Hamburger menu for mobile site
[
https://issues.apache.org/jira/browse/HBASE-14785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999431#comment-14999431
]
Hadoop QA commented on HBASE-14785:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12771629/0001-HBASE-14774-Addendum-Exclude-src-main-site-resources.patch
against master branch at commit c817aa386cdbadbfd8aae2721862b11a7034ab14.
ATTACHMENT ID: 12771629
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+0 tests included{color}. The patch appears to be a
documentation, build,
or dev-support patch that doesn't require tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16484//console
This message is automatically generated.
> Hamburger menu for mobile site
> --
>
> Key: HBASE-14785
> URL: https://issues.apache.org/jira/browse/HBASE-14785
> Project: HBase
> Issue Type: Sub-task
> Components: website
>Affects Versions: 2.0.0
>Reporter: Misty Stanley-Jones
>Assignee: Misty Stanley-Jones
> Fix For: 2.0.0
>
> Attachments:
> 0001-HBASE-14774-Addendum-Exclude-src-main-site-resources.patch,
> HBASE-14785-addendum-v1.patch, HBASE-14785-addendum.patch,
> HBASE-14785-v1.patch, HBASE-14785-v2.patch, HBASE-14785.patch,
> maven-fluido-skin-1.5-HBASE.jar
>
>
> Figure out how to do a hamburger menu on mobile.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14793) Allow limiting size of block into L1 block cache.
[
https://issues.apache.org/jira/browse/HBASE-14793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999440#comment-14999440
]
Hadoop QA commented on HBASE-14793:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12771612/HBASE-14793.patch
against master branch at commit 1a6ec1bac65fa04a30a57891fb66dd248f3bed9c.
ATTACHMENT ID: 12771612
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1 release
audit warnings (more than the master's current 0 warnings).
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.io.TestHeapSize
org.apache.hadoop.hbase.io.hfile.TestLruBlockCache
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16483//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16483//artifact/patchprocess/patchReleaseAuditWarnings.txt
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16483//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16483//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16483//console
This message is automatically generated.
> Allow limiting size of block into L1 block cache.
> -
>
> Key: HBASE-14793
> URL: https://issues.apache.org/jira/browse/HBASE-14793
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0, 1.1.2
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14793-v1.patch, HBASE-14793.patch
>
>
> G1GC does really badly with long lived large objects. Lets allow limiting the
> size of a block that can be kept in the block cache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14387) Compaction improvements: Maximum off-peak compaction size
[ https://issues.apache.org/jira/browse/HBASE-14387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999453#comment-14999453 ] Hudson commented on HBASE-14387: SUCCESS: Integrated in HBase-1.3 #359 (See [https://builds.apache.org/job/HBase-1.3/359/]) HBASE-14387 Compaction improvements: Maximum off-peak compaction size (enis: rev 3de347f6e0c8a44bd2d04c7ffa0fbd914024968c) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/RatioBasedCompactionPolicy.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionConfiguration.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerOnlineConfigChange.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/ExploringCompactionPolicy.java > Compaction improvements: Maximum off-peak compaction size > - > > Key: HBASE-14387 > URL: https://issues.apache.org/jira/browse/HBASE-14387 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14387-v1.patch, HBASE-14387-v2.patch, > HBASE-14387-v3.patch, HBASE-14387-v3.patch > > > Make max compaction size for peak and off peak separate config options. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14766) In WALEntryFilter, cell.getFamily() needs to be replaced with the new low-cost implementation
[ https://issues.apache.org/jira/browse/HBASE-14766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-14766: --- Summary: In WALEntryFilter, cell.getFamily() needs to be replaced with the new low-cost implementation (was: WALEntryFilter's filter implement, cell.getFamily() needs to be replaced with the new low-cost implementation.) > In WALEntryFilter, cell.getFamily() needs to be replaced with the new > low-cost implementation > - > > Key: HBASE-14766 > URL: https://issues.apache.org/jira/browse/HBASE-14766 > Project: HBase > Issue Type: Improvement >Reporter: huaxiang sun >Assignee: huaxiang sun > Attachments: HBASE-14766-v001.patch, HBASE-14766-v002.patch, > HBASE-14766-v003.patch, HBASE-14766-v003.patch, HBASE-14766-v004.patch > > > Cell's getFamily() gets an array copy of the cell's family, while in the > filter function, it just needs to peek into the family and do a compare. > Replace > Bytes.toString(cell.getFamily()) > with > Bytes.toString(cell.getFamilyArray(), cell.getFamilyOffset(), > cell.getFamilyLength()) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14778) Make block cache hit percentages not integer in the metrics system
[ https://issues.apache.org/jira/browse/HBASE-14778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999560#comment-14999560 ] Hudson commented on HBASE-14778: SUCCESS: Integrated in HBase-1.3-IT #304 (See [https://builds.apache.org/job/HBase-1.3-IT/304/]) HBASE-14778 Make block cache hit percentages not integer in the metrics (eclark: rev e8e0e86b49c3a034f9c75d115c3e191d4fff07bb) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerWrapperImpl.java * hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerWrapper.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerWrapperStub.java > Make block cache hit percentages not integer in the metrics system > -- > > Key: HBASE-14778 > URL: https://issues.apache.org/jira/browse/HBASE-14778 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0, 1.1.2 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14778-v1.patch, HBASE-14778.patch, Screen Shot > 2015-11-07 at 10.03.13 AM.png, Screen Shot 2015-11-07 at 10.20.03 AM.png > > > Once you're close to the 90%+ it's hard to see a difference because getting a > full percent change is rare. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14387) Compaction improvements: Maximum off-peak compaction size
[ https://issues.apache.org/jira/browse/HBASE-14387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999558#comment-14999558 ] Hudson commented on HBASE-14387: SUCCESS: Integrated in HBase-1.3-IT #304 (See [https://builds.apache.org/job/HBase-1.3-IT/304/]) HBASE-14387 Compaction improvements: Maximum off-peak compaction size (enis: rev 3de347f6e0c8a44bd2d04c7ffa0fbd914024968c) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionConfiguration.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerOnlineConfigChange.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/RatioBasedCompactionPolicy.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/ExploringCompactionPolicy.java > Compaction improvements: Maximum off-peak compaction size > - > > Key: HBASE-14387 > URL: https://issues.apache.org/jira/browse/HBASE-14387 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14387-v1.patch, HBASE-14387-v2.patch, > HBASE-14387-v3.patch, HBASE-14387-v3.patch > > > Make max compaction size for peak and off peak separate config options. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14786) TestProcedureAdmin hangs
[
https://issues.apache.org/jira/browse/HBASE-14786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999559#comment-14999559
]
Hudson commented on HBASE-14786:
SUCCESS: Integrated in HBase-1.3-IT #304 (See
[https://builds.apache.org/job/HBase-1.3-IT/304/])
HBASE-14786 TestProcedureAdmin hangs (matteo.bertozzi: rev
1d29d8c004af64891050345bcba324ed0018ae13)
*
hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/WALProcedureStore.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/MasterProcedureTestingUtility.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestProcedureAdmin.java
> TestProcedureAdmin hangs
>
>
> Key: HBASE-14786
> URL: https://issues.apache.org/jira/browse/HBASE-14786
> Project: HBase
> Issue Type: Sub-task
> Components: hangingTests, test
>Reporter: stack
>Assignee: Matteo Bertozzi
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-14786-v0.patch
>
>
> This test was added by HBASE-14108 in September [~syuanjiang] Mind taking a
> look? Hope you don't mind me assigning it to you to take a look.
> This test run was incomplete:
> https://builds.apache.org/job/HBase-Trunk_matrix/jdk=latest1.8,label=Hadoop/443/
> If I do below...
> kalashnikov:hbase.git.commit stack$ python ./dev-support/findHangingTests.py
> https://builds.apache.org/job/HBase-Trunk_matrix/jdk=latest1.8,label=Hadoop/443/consoleText
> Fetching
> https://builds.apache.org/job/HBase-Trunk_matrix/jdk=latest1.8,label=Hadoop/443/consoleText
> Building remotely on H4 (Mapreduce zookeeper Hadoop Pig falcon Hdfs) in
> workspace
> /home/jenkins/jenkins-slave/workspace/HBase-Trunk_matrix/jdk/latest1.8/label/Hadoop
> Printing hanging tests
> Hanging test : org.apache.hadoop.hbase.master.procedure.TestProcedureAdmin
> Printing Failing tests
> ... it reports TestProcedureAdmin as reporting it started but there is no
> corresponding loggiing of end-of-test.
> If I look in the output for the above test run, I see all this spew:
> https://builds.apache.org/job/HBase-Trunk_matrix/jdk=latest1.8,label=Hadoop/443/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.master.procedure.TestProcedureAdmin-output.txt
> about...
> {code}
> 2015-11-07 10:19:59,052 INFO
> [B.defaultRpcServer.handler=4,queue=0,port=37253] master.HMaster(1911):
> Client=jenkins/null disable testListProcedure
> 2015-11-07 10:19:59,052 ERROR
> [B.defaultRpcServer.handler=4,queue=0,port=37253] ipc.RpcServer(2228):
> Unexpected throwable object
> java.lang.IllegalArgumentException
> at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:76)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:618)
> at
> org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1916)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.disableTable(MasterRpcServices.java:572)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:57936)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2184)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> That look right to you?
> Maybe it is something else. Would be good to clean up hanging tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14795) Provide an alternative spark-hbase SQL implementations for Scan
[ https://issues.apache.org/jira/browse/HBASE-14795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999621#comment-14999621 ] Ted Malaska commented on HBASE-14795: - There is no real negative to this proposed approach other then a second implementation of table scan. To bad the existing TableInputFormat can not be updated to handle this because then this would be in one local. As for implementation these is no reason this can't just be invoked straight from line 330 from DefaultSource or could be an alternate implementation in hbaseRDD that tables multi scan objects. https://github.com/apache/hbase/blob/master/hbase-spark/src/main/scala/org/apache/hadoop/hbase/spark/DefaultSource.scala#L330 > Provide an alternative spark-hbase SQL implementations for Scan > --- > > Key: HBASE-14795 > URL: https://issues.apache.org/jira/browse/HBASE-14795 > Project: HBase > Issue Type: Improvement >Reporter: Ted Malaska >Assignee: Zhan Zhang >Priority: Minor > > This is a sub-jira of HBASE-14789. This jira is to focus on the replacement > of TableInputFormat for a more custom scan implementation that will make the > following use case more effective. > Use case: > In the case you have multiple scan ranges on a single table with in a single > query. TableInputFormat will scan the the outer range of the scan start and > end range where this implementation can be more pointed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14789) Provide an alternative spark-hbase connector
[ https://issues.apache.org/jira/browse/HBASE-14789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999628#comment-14999628 ] Ted Malaska commented on HBASE-14789: - Put comments related to the bulk get implementation in jira HBASE-14796 > Provide an alternative spark-hbase connector > > > Key: HBASE-14789 > URL: https://issues.apache.org/jira/browse/HBASE-14789 > Project: HBase > Issue Type: Improvement >Reporter: Zhan Zhang >Assignee: Zhan Zhang > Attachments: shc.pdf > > > This JIRA is to provide user an option to choose different Spark-HBase > implementation based on requirements. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14796) Provide an alternative spark-hbase SQL implementations for Gets
[ https://issues.apache.org/jira/browse/HBASE-14796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999635#comment-14999635 ] Ted Malaska commented on HBASE-14796: - If implemented this code would fit great right around https://github.com/apache/hbase/blob/master/hbase-spark/src/main/scala/org/apache/hadoop/hbase/spark/DefaultSource.scala#L347 > Provide an alternative spark-hbase SQL implementations for Gets > --- > > Key: HBASE-14796 > URL: https://issues.apache.org/jira/browse/HBASE-14796 > Project: HBase > Issue Type: Improvement >Reporter: Ted Malaska >Assignee: Zhan Zhang >Priority: Minor > > Current the Spark-Module Spark SQL implementation gets records from HBase > from the driver if there is something like the following found in the SQL. > rowkey = 123 > The reason for this original was normal sql will not have many equal > operations in a single where clause. > Zhan, had brought up too points that have value. > 1. The SQL may be generated and may have many many equal statements in it so > moving the work to an executor protects the driver from load > 2. In the correct implementation the drive is connecting to HBase and > exceptions may cause trouble with the Spark application and not just with the > a single task execution -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14796) Provide an alternative spark-hbase SQL implementations for Gets
Ted Malaska created HBASE-14796: --- Summary: Provide an alternative spark-hbase SQL implementations for Gets Key: HBASE-14796 URL: https://issues.apache.org/jira/browse/HBASE-14796 Project: HBase Issue Type: Improvement Reporter: Ted Malaska Assignee: Zhan Zhang Priority: Minor Current the Spark-Module Spark SQL implementation gets records from HBase from the driver if there is something like the following found in the SQL. rowkey = 123 The reason for this original was normal sql will not have many equal operations in a single where clause. Zhan, had brought up too points that have value. 1. The SQL may be generated and may have many many equal statements in it so moving the work to an executor protects the driver from load 2. In the correct implementation the drive is connecting to HBase and exceptions may cause trouble with the Spark application and not just with the a single task execution -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14788) Splitting a region does not support the hbase.rs.evictblocksonclose config when closing source region
[ https://issues.apache.org/jira/browse/HBASE-14788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999470#comment-14999470 ] Jerry He commented on HBASE-14788: -- The patch would fix the problem. Alternatively, it would be cleaner if you can just consolidate all the checking at the bottom level StoreFile.closeReader(). We are checking the same cacheConf in all the places. > Splitting a region does not support the hbase.rs.evictblocksonclose config > when closing source region > - > > Key: HBASE-14788 > URL: https://issues.apache.org/jira/browse/HBASE-14788 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 1.0.0 >Reporter: Randy Fox >Assignee: Ted Yu > Attachments: 14788-v1.txt, 14788-v2.txt, 14788-v3.txt > > > i have a table with bucket cache turned on and hbase.rs.evictblocksonclose > set to false. I split a region and watched that the closing of the source > region did not complete until the bucketcache was flushed for that region. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13982) Add info for visibility labels/cell TTLs to ImportTsv
[ https://issues.apache.org/jira/browse/HBASE-13982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999590#comment-14999590 ] Jerry He commented on HBASE-13982: -- The patch looks good to me. Can fix the long line and format on commit. > Add info for visibility labels/cell TTLs to ImportTsv > - > > Key: HBASE-13982 > URL: https://issues.apache.org/jira/browse/HBASE-13982 > Project: HBase > Issue Type: Bug > Components: mapreduce >Affects Versions: 1.1.0.1 >Reporter: Lars George >Assignee: NIDHI GAMBHIR > Labels: beginner > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-13982.patch, HBASE-13982_2.patch, > HBASE_13982_1.patch > > > HBASE-9832 added support for two more optional, special TSV columns, but no > usage info was added. Add. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14778) Make block cache hit percentages not integer in the metrics system
[ https://issues.apache.org/jira/browse/HBASE-14778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-14778: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks for the review [~jmhsieh] > Make block cache hit percentages not integer in the metrics system > -- > > Key: HBASE-14778 > URL: https://issues.apache.org/jira/browse/HBASE-14778 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0, 1.1.2 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14778-v1.patch, HBASE-14778.patch, Screen Shot > 2015-11-07 at 10.03.13 AM.png, Screen Shot 2015-11-07 at 10.20.03 AM.png > > > Once you're close to the 90%+ it's hard to see a difference because getting a > full percent change is rare. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14789) Provide an alternative spark-hbase connector
[ https://issues.apache.org/jira/browse/HBASE-14789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999422#comment-14999422 ] Ted Malaska commented on HBASE-14789: - Cool I read the doc so there are two points. * Bulk Get - Do bulk Gets on an executor * TableInputFormat - Don't use this because or the thought that only one can run at a time * Change the table description format - Add more JSON like definition * Add write support - For SparkSQL writes to HBase #First lets talk to each point first: * Bulk Get: - As we have talked about in other jira's executing this on the executor side really doesn't add much value. It would be vary odd if people would have more then a 1000 equals in a where cause. If they did then we need to figure out at what point 1000, 1, 5 does it become faster to run the code on the executor. The normal use case is just a couple = per where cause so this is not a real concern, now if you want to do a real bulk get then use the bulk get command, that will be much better for a lot of reasons. * Not Using TableInputFormat: In the code today Spark if given the TablInputFormat in different requests so they are at different points on the DAG. So why does Spark not read from both? Also the locality is given and we are not reinventing the wheel. * Change the table description format: This is a preference thing is current version is more like the HBase shell. Ether way makes sense it makes no real difference. * Add write support: Yes we should add this. #Summery First I think any and all changes would fit into the current implementation of the HBase-Spark module with little changes. This are pretty pointed changes that effect a scoped area of the code. Second we should separate out this jira into 4 different jiras each focusing on the different points, for these different points are not dependent or related. We should open up a jira to address each features and then discuss the approach for each one and how it can be added and or if it should be added. Thanks Zhan Let me know if I missed anything > Provide an alternative spark-hbase connector > > > Key: HBASE-14789 > URL: https://issues.apache.org/jira/browse/HBASE-14789 > Project: HBase > Issue Type: Bug >Reporter: Zhan Zhang >Assignee: Zhan Zhang > Attachments: shc.pdf > > > This JIRA is to provide user an option to choose different Spark-HBase > implementation based on requirements. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14789) Provide an alternative spark-hbase connector
[ https://issues.apache.org/jira/browse/HBASE-14789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-14789: Issue Type: Improvement (was: Bug) > Provide an alternative spark-hbase connector > > > Key: HBASE-14789 > URL: https://issues.apache.org/jira/browse/HBASE-14789 > Project: HBase > Issue Type: Improvement >Reporter: Zhan Zhang >Assignee: Zhan Zhang > Attachments: shc.pdf > > > This JIRA is to provide user an option to choose different Spark-HBase > implementation based on requirements. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14788) Splitting a region does not support the hbase.rs.evictblocksonclose config when closing source region
[ https://issues.apache.org/jira/browse/HBASE-14788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999475#comment-14999475 ] Randy Fox commented on HBASE-14788: --- I tested V3 of the patch and it fixes the split issue. > Splitting a region does not support the hbase.rs.evictblocksonclose config > when closing source region > - > > Key: HBASE-14788 > URL: https://issues.apache.org/jira/browse/HBASE-14788 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 1.0.0 >Reporter: Randy Fox >Assignee: Ted Yu > Attachments: 14788-v1.txt, 14788-v2.txt, 14788-v3.txt > > > i have a table with bucket cache turned on and hbase.rs.evictblocksonclose > set to false. I split a region and watched that the closing of the source > region did not complete until the bucketcache was flushed for that region. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14708) Use copy on write Map for region location cache
[ https://issues.apache.org/jira/browse/HBASE-14708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999533#comment-14999533 ] Elliott Clark commented on HBASE-14708: --- bq.Will the COW behave well with 30K+ regions? Yeah it's pretty darn fast at that scale. By the end of the tests I had 100's of thousands in the map. > Use copy on write Map for region location cache > --- > > Key: HBASE-14708 > URL: https://issues.apache.org/jira/browse/HBASE-14708 > Project: HBase > Issue Type: Improvement > Components: Client >Affects Versions: 1.1.2 >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Critical > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14708-v10.patch, HBASE-14708-v11.patch, > HBASE-14708-v12.patch, HBASE-14708-v13.patch, HBASE-14708-v2.patch, > HBASE-14708-v3.patch, HBASE-14708-v4.patch, HBASE-14708-v5.patch, > HBASE-14708-v6.patch, HBASE-14708-v7.patch, HBASE-14708-v8.patch, > HBASE-14708-v9.patch, HBASE-14708.patch, anotherbench.zip, > location_cache_times.pdf, result.csv > > > Internally a co-worker profiled their application that was talking to HBase. > > 60% of the time was spent in locating a region. This was while the cluster > was stable and no regions were moving. > To figure out if there was a faster way to cache region location I wrote up a > benchmark here: https://github.com/elliottneilclark/benchmark-hbase-cache > This tries to simulate a heavy load on the location cache. > * 24 different threads. > * 2 Deleting location data > * 2 Adding location data > * Using floor to get the result. > To repeat my work just run ./run.sh and it should produce a result.csv > Results: > ConcurrentSkiplistMap is a good middle ground. It's got equal speed for > reading and writing. > However most operations will not need to remove or add a region location. > There will be potentially several orders of magnitude more reads for cached > locations than there will be on clearing the cache. > So I propose a copy on write tree map. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14703) not collect stats when call HTable.mutateRow
[
https://issues.apache.org/jira/browse/HBASE-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999564#comment-14999564
]
Jesse Yates commented on HBASE-14703:
-
Looking closer at the patch, I'm a bit concerned with how retries will work.
Since the AsyncProcess handles retries, we shouldn't be calling the
callable.callWithRetries, right? Then we would get two sets of retries with the
callable nested under the AsyncProcess. Can't we just use the standard AP
timeout then as well (its not something the user changes per-call)?
bq. it seems that processed flag has no relates with results
Ah, yes. For this, IIRC its all or nothing with the check so all you need to
send back is a single 'everything worked' flag, rather than a flag per
mutation, which would all be the same value. You added that with the
'processed' flag in the MultiResponse, but that's only going to be applicable
for the checkAndXXX cases, right?
I don't particularly like bolting on a processed flag onto the MulitResponse,
just to support this one case. Looking at the checkAnd methods, they all
follow the same basic logic and return a MutateResponse#processed, except for
checkAndMutate. That makes me think we can do the same thing we are doing here
for the other checkAndXXX methods, but fork logic a little bit in AsyncProcess
to support that kind of processing instead (as a follow up JIRA, not here).
In the meantime, lets just focus on supporting the mutateRow and then later we
can fix checkAndMutate along with the other checkAnd methods.
For mutateRow, maybe its better to just check to see if it has any Results,
rather than checking to see if there is a statistic tracker? It ties more into
why the ResponseConverter won't work, rather than why there are no Results in
the response and leave that information in a comment around the check for the
count of results.
Good stuff Heng Chen
> not collect stats when call HTable.mutateRow
> -
>
> Key: HBASE-14703
> URL: https://issues.apache.org/jira/browse/HBASE-14703
> Project: HBase
> Issue Type: Bug
>Reporter: Heng Chen
>Assignee: Heng Chen
> Attachments: HBASE-14703-async.patch, HBASE-14703-start.patch,
> HBASE-14703.patch, HBASE-14703_v1.patch, HBASE-14703_v2.patch
>
>
> In {{AsyncProcess.SingleServerRequestRunnable}}, it seems we update
> serverStatistics twice.
> The first one is that we wrapper {{RetryingCallable}} by
> {{StatsTrackingRpcRetryingCaller}}, and do serverStatistics update when we
> call {{callWithRetries}} and {{callWithoutRetries}}. Relates code like below:
> {code}
> @Override
> public T callWithRetries(RetryingCallable callable, int callTimeout)
> throws IOException, RuntimeException {
> T result = delegate.callWithRetries(callable, callTimeout);
> return updateStatsAndUnwrap(result, callable);
> }
> @Override
> public T callWithoutRetries(RetryingCallable callable, int callTimeout)
> throws IOException, RuntimeException {
> T result = delegate.callWithRetries(callable, callTimeout);
> return updateStatsAndUnwrap(result, callable);
> }
> {code}
> The secondary one is after we get response, in {{receiveMultiAction}}, we do
> update again.
> {code}
> // update the stats about the region, if its a user table. We don't want to
> slow down
> // updates to meta tables, especially from internal updates (master, etc).
> if (AsyncProcess.this.connection.getStatisticsTracker() != null) {
> result = ResultStatsUtil.updateStats(result,
> AsyncProcess.this.connection.getStatisticsTracker(), server, regionName);
> }
> {code}
> It seems that {{StatsTrackingRpcRetryingCaller}} is NOT necessary, remove it?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14789) Provide an alternative spark-hbase connector
[ https://issues.apache.org/jira/browse/HBASE-14789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999566#comment-14999566 ] Zhan Zhang commented on HBASE-14789: [~malaskat] Thanks for reviewing this. I agree that table write, json format, and customized sedes are all able to fit into the current implementation with some changes, and I would love to contribute on this part. Given that, there is no straightforward way to support multi-scans and bulkget with TableInputFormat with the scenario mentioned in the document. The argument regarding the BulkGet may be reasonable in most cases, but there do have chances that driver becomes bottleneck. More importantly, driver is actually processing data with HBase client library. If there is any exception happens, the whole job will crash. Regarding the multiple scan, here is my understanding. Correct me if I am wrong. Current implementation will construct a RDD for each non-overlapping scan. Then all these RDDs are union together. With the current TableInputFormat limitation, there is no easy way to walkaround this, unless TableInputFormat is changed to handle multiple scans in one shot. Now suppose we have 10 regions, but the user query may consists of 100 non-overlapping scans. Then there will be 100 RDDs constructed and union together. The Union RDD returned by buildScan will consists of at least 100 partitions, assume that each RDD only have one partitions (there is high chance that each RDD may consists of multiple partitions). 100 partitions means that Spark will has to launch 100 tasks to process the scan. Given that we only have 10 regions (10 servers to simplify the discussion). The scheduler cannot allocate 100 tasks to 10 severs (10 executors co-located with region server with 1 core each to simply the discussion). In this scenario, the executors finishes its assigned task earlier will get more tasks, which may retrieve data from other region server (hurt the data locality). In addition, the scheduler has to schedule and serialize 100 tasks, which increase the overhead. In the architecture proposed in this JIRA, the driver will construct 10 tasks (or less) with each consists of multiple scans. These 10 tasks can be scheduled to 10 executors concurrently, which achieve better data locality, sedes and scheduling overhead. In addition, in multi-tenant environment the current approach may suffer more because it has to construct much more tasks but the overall executor slot is limited. Also in this architecture, the Scan and Get are treated in a unified way, which seems to be more natural. I think in real deployment, the proposed architecture does have its advantage in many scenarios. > Provide an alternative spark-hbase connector > > > Key: HBASE-14789 > URL: https://issues.apache.org/jira/browse/HBASE-14789 > Project: HBase > Issue Type: Improvement >Reporter: Zhan Zhang >Assignee: Zhan Zhang > Attachments: shc.pdf > > > This JIRA is to provide user an option to choose different Spark-HBase > implementation based on requirements. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14795) Provide an alternative spark-hbase SQL implementations for Scan
Ted Malaska created HBASE-14795: --- Summary: Provide an alternative spark-hbase SQL implementations for Scan Key: HBASE-14795 URL: https://issues.apache.org/jira/browse/HBASE-14795 Project: HBase Issue Type: Improvement Reporter: Ted Malaska Assignee: Zhan Zhang Priority: Minor This is a sub-jira of HBASE-14789. This jira is to focus on the replacement of TableInputFormat for a more custom scan implementation that will make the following use case more effective. Use case: In the case you have multiple scan ranges on a single table with in a single query. TableInputFormat will scan the the outer range of the scan start and end range where this implementation can be more pointed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14763) Remove usages of deprecated HConnection
[ https://issues.apache.org/jira/browse/HBASE-14763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-14763: --- Attachment: hbase-14763.v4.patch v4: The javac warns were actually from scalac -- this should fix that warning now. > Remove usages of deprecated HConnection > > > Key: HBASE-14763 > URL: https://issues.apache.org/jira/browse/HBASE-14763 > Project: HBase > Issue Type: Bug > Components: hbase >Affects Versions: 2.0.0 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: 2.0.0 > > Attachments: hbase-14763.patch, hbase-14763.v2.patch, > hbase-14763.v3.patch, hbase-14763.v3.patch, hbase-14763.v4.patch > > > HConnection was deprecated in 1.0.0. There are two interfaces that are > supposed to be used instead -- Connection for client programs and > ClusterConnection for internal hbaes and special tools (LoadIncremental, > HBCK, etc). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14793) Allow limiting size of block into L1 block cache.
[ https://issues.apache.org/jira/browse/HBASE-14793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999446#comment-14999446 ] Appy commented on HBASE-14793: -- Emm, what about a metric to count the blocks which couldn't be cached because of this new threshold. Can that be useful? Maybe for tuning? what do you think? > Allow limiting size of block into L1 block cache. > - > > Key: HBASE-14793 > URL: https://issues.apache.org/jira/browse/HBASE-14793 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0, 1.1.2 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14793-v1.patch, HBASE-14793.patch > > > G1GC does really badly with long lived large objects. Lets allow limiting the > size of a block that can be kept in the block cache. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14785) Hamburger menu for mobile site
[ https://issues.apache.org/jira/browse/HBASE-14785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999527#comment-14999527 ] stack commented on HBASE-14785: --- Grand. Add comment to that effect on commit. > Hamburger menu for mobile site > -- > > Key: HBASE-14785 > URL: https://issues.apache.org/jira/browse/HBASE-14785 > Project: HBase > Issue Type: Sub-task > Components: website >Affects Versions: 2.0.0 >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Fix For: 2.0.0 > > Attachments: > 0001-HBASE-14774-Addendum-Exclude-src-main-site-resources.patch, > HBASE-14785-addendum-v1.patch, HBASE-14785-addendum.patch, > HBASE-14785-v1.patch, HBASE-14785-v2.patch, HBASE-14785.patch, > maven-fluido-skin-1.5-HBASE.jar > > > Figure out how to do a hamburger menu on mobile. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14789) Provide an alternative spark-hbase connector
[ https://issues.apache.org/jira/browse/HBASE-14789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999623#comment-14999623 ] Ted Malaska commented on HBASE-14789: - Put response to TableInputFormat design in HBASE-14795 > Provide an alternative spark-hbase connector > > > Key: HBASE-14789 > URL: https://issues.apache.org/jira/browse/HBASE-14789 > Project: HBase > Issue Type: Improvement >Reporter: Zhan Zhang >Assignee: Zhan Zhang > Attachments: shc.pdf > > > This JIRA is to provide user an option to choose different Spark-HBase > implementation based on requirements. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14789) Provide an alternative spark-hbase connector
[ https://issues.apache.org/jira/browse/HBASE-14789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999622#comment-14999622 ] Ted Malaska commented on HBASE-14789: - This is a sub jira > Provide an alternative spark-hbase connector > > > Key: HBASE-14789 > URL: https://issues.apache.org/jira/browse/HBASE-14789 > Project: HBase > Issue Type: Improvement >Reporter: Zhan Zhang >Assignee: Zhan Zhang > Attachments: shc.pdf > > > This JIRA is to provide user an option to choose different Spark-HBase > implementation based on requirements. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14766) WALEntryFilter's filter implement, cell.getFamily() needs to be replaced with the new low-cost implementation.
[
https://issues.apache.org/jira/browse/HBASE-14766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999434#comment-14999434
]
Hadoop QA commented on HBASE-14766:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12771598/HBASE-14766-v004.patch
against master branch at commit 112900d0425a8157b89041f0e353ebf5cc259c69.
ATTACHMENT ID: 12771598
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1 release
audit warnings (more than the master's current 0 warnings).
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16481//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16481//artifact/patchprocess/patchReleaseAuditWarnings.txt
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16481//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16481//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16481//console
This message is automatically generated.
> WALEntryFilter's filter implement, cell.getFamily() needs to be replaced with
> the new low-cost implementation.
> --
>
> Key: HBASE-14766
> URL: https://issues.apache.org/jira/browse/HBASE-14766
> Project: HBase
> Issue Type: Improvement
>Reporter: huaxiang sun
>Assignee: huaxiang sun
> Attachments: HBASE-14766-v001.patch, HBASE-14766-v002.patch,
> HBASE-14766-v003.patch, HBASE-14766-v003.patch, HBASE-14766-v004.patch
>
>
> Cell's getFamily() gets an array copy of the cell's family, while in the
> filter function, it just needs to peek into the family and do a compare.
> Replace
> Bytes.toString(cell.getFamily())
> with
> Bytes.toString(cell.getFamilyArray(), cell.getFamilyOffset(),
> cell.getFamilyLength())
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14766) In WALEntryFilter, cell.getFamily() needs to be replaced with the new low-cost implementation
[ https://issues.apache.org/jira/browse/HBASE-14766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-14766: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.0.0 Status: Resolved (was: Patch Available) > In WALEntryFilter, cell.getFamily() needs to be replaced with the new > low-cost implementation > - > > Key: HBASE-14766 > URL: https://issues.apache.org/jira/browse/HBASE-14766 > Project: HBase > Issue Type: Improvement >Reporter: huaxiang sun >Assignee: huaxiang sun > Fix For: 2.0.0 > > Attachments: HBASE-14766-v001.patch, HBASE-14766-v002.patch, > HBASE-14766-v003.patch, HBASE-14766-v003.patch, HBASE-14766-v004.patch > > > Cell's getFamily() gets an array copy of the cell's family, while in the > filter function, it just needs to peek into the family and do a compare. > Replace > Bytes.toString(cell.getFamily()) > with > Bytes.toString(cell.getFamilyArray(), cell.getFamilyOffset(), > cell.getFamilyLength()) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14387) Compaction improvements: Maximum off-peak compaction size
[ https://issues.apache.org/jira/browse/HBASE-14387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999486#comment-14999486 ] Hudson commented on HBASE-14387: FAILURE: Integrated in HBase-1.2 #358 (See [https://builds.apache.org/job/HBase-1.2/358/]) HBASE-14387 Compaction improvements: Maximum off-peak compaction size (enis: rev f4c70bfc1da63ae2742272a1409615ce507aed15) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/RatioBasedCompactionPolicy.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionConfiguration.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/ExploringCompactionPolicy.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerOnlineConfigChange.java > Compaction improvements: Maximum off-peak compaction size > - > > Key: HBASE-14387 > URL: https://issues.apache.org/jira/browse/HBASE-14387 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14387-v1.patch, HBASE-14387-v2.patch, > HBASE-14387-v3.patch, HBASE-14387-v3.patch > > > Make max compaction size for peak and off peak separate config options. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14789) Provide an alternative spark-hbase connector
[ https://issues.apache.org/jira/browse/HBASE-14789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999569#comment-14999569 ] Zhan Zhang commented on HBASE-14789: [~malaskat] By the way, I didn't mean "do not use TableInputFormat". I mean to make two approaches as plugins so that users can choose the one that fitting into their requirements. > Provide an alternative spark-hbase connector > > > Key: HBASE-14789 > URL: https://issues.apache.org/jira/browse/HBASE-14789 > Project: HBase > Issue Type: Improvement >Reporter: Zhan Zhang >Assignee: Zhan Zhang > Attachments: shc.pdf > > > This JIRA is to provide user an option to choose different Spark-HBase > implementation based on requirements. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14793) Allow limiting size of block into L1 block cache.
[
https://issues.apache.org/jira/browse/HBASE-14793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999575#comment-14999575
]
Hadoop QA commented on HBASE-14793:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12771630/HBASE-14793-v1.patch
against master branch at commit c817aa386cdbadbfd8aae2721862b11a7034ab14.
ATTACHMENT ID: 12771630
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.io.TestHeapSize
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16486//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16486//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16486//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16486//console
This message is automatically generated.
> Allow limiting size of block into L1 block cache.
> -
>
> Key: HBASE-14793
> URL: https://issues.apache.org/jira/browse/HBASE-14793
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0, 1.1.2
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14793-v1.patch, HBASE-14793.patch
>
>
> G1GC does really badly with long lived large objects. Lets allow limiting the
> size of a block that can be kept in the block cache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14780) Integration Tests that run with ChaosMonkey need to specify CFs
[ https://issues.apache.org/jira/browse/HBASE-14780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999576#comment-14999576 ] Jonathan Hsieh commented on HBASE-14780: The rat error has to do with fluido -- new website related. -- not from this patch. Thanks for the explanation stack. Thanks for the review enis. committed to master, branch-1, branch-1.1 and branch-1.0 (with minor import tweaks) > Integration Tests that run with ChaosMonkey need to specify CFs > --- > > Key: HBASE-14780 > URL: https://issues.apache.org/jira/browse/HBASE-14780 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: 2.0.0, 1.2.0, 1.0.3, 1.1.3 > > Attachments: hbase-14780.patch, hbase-14780.patch > > > Been running some IT tests and found that some failed because getcfs was null > and didn't protecte cfs that were assumed to go unmolested. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14780) Integration Tests that run with ChaosMonkey need to specify CFs
[ https://issues.apache.org/jira/browse/HBASE-14780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-14780: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 1.1.3 1.0.3 1.2.0 2.0.0 Status: Resolved (was: Patch Available) > Integration Tests that run with ChaosMonkey need to specify CFs > --- > > Key: HBASE-14780 > URL: https://issues.apache.org/jira/browse/HBASE-14780 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: 2.0.0, 1.2.0, 1.0.3, 1.1.3 > > Attachments: hbase-14780.patch, hbase-14780.patch > > > Been running some IT tests and found that some failed because getcfs was null > and didn't protecte cfs that were assumed to go unmolested. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
